dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Analysis Performance in the game was assessed as a d' score from signal detection theory  (Green and Swets, 1966) , which measured discrimination between targets and distractors. The d' calculation was adjusted for instances of zero false alarms  (Snodgrass and Corwin, 1988; Mill and O'Connor, 2014). JoAs and JoPs were quantified as a percentage of the VAS scale. Mean d', JoAs and JoPs were submitted to repeated measures ANOVAs, with the factors masking (unmasked vs. masked) and turbulence (no turbulence vs. turbulence). Additionally, we assessed whether any effects of visual masking on JoAs could be explained by a reduction in JoPs. For this, a hierarchical linear regression model (also known as linear mixed-effects models) was used to model singletrial level data. JoAs were modeled by the factors masking and turbulence (coded as unmasked = 0, masked = 1; no turbulence = 0, turbulence = 1), as well as their interaction.
JoPs were added as a covariate, after standardizing within participants, as we predicted that JoPs would inform JoAs across conditions. All fixed effects were also allowed vary between participants (i. e. , participant random intercepts and slopes). This analysis was conducted using the lme4 package  (Bates et al. , 2014)  in R (  R Core Team, 2015). Parameter estimates (b) and their associated t-tests (t, p), calculated using the Satterthwaite approximation for degrees of freedom  (Kuznetsova et al. , 2015) , are presented to show the magnitude of the effects, with bootstrapped 95% confidence intervals.
Data Analysis Mean d', JoPs and JoAs were submitted to repeated measures ANOVAs with the factors ambiguity (low vs. high) and turbulence (no turbulence vs. turbulence). Simple effects t-tests were used to probe interaction between ambiguity and turbulence. To characterize participants' strategies, the response bias (c) measure from signal detection theory  (Green and Swets, 1966)  was used, with a correction for zero false alarms  (Snodgrass and Corwin, 1988; Mill and O'Connor, 2014). Zero reflects an unbiased criterion, negative values reflect a liberal criterion, and positive values denote a conservative criterion. Essentially, here, a larger positive criterion would be associated with a lower number of items caught (targets or distractors), i. e. , more careful responding. This bias measure was computed for each trial, and then averaged for each participant (collapsing across conditions).
FUNDING NS was supported by a  University College London Impact Scholarship and Bogue Fellowship , the  Belgian Science Policy Office Project  " Mechanisms of conscious and unconscious learning " ( IAP P7/33 ), a  Fyssen Foundation Post-doctoral grant ,  ANR-10-LABX-0087 IEC  and  ANR-10-IDEX-0001-02 PSL. MV was supported by a  Columbia University Dean 's Fellowship, and by a  James S. McDonnell collaborative grant  ( 220020166 ), awarded to JM. PH was supported by an  ESRC   Professorial Fellowship  ( ES/J023140/1 ), an  ERC Advanced Grant HUMVOL  ( 323943 ), and  EU FP7 VERE WP1 (257695 ).
Modeling the semantic information of words with Word2vec To approximate semantic structuring in language, semantic models are typically trained on some corpus data. For the purposes of the present study, the data were extracted from the Finnish Internet Parsebank. This corpus contains approximately 3. 7 billion tokens  (Kanerva et al. 2014 ). The corpus is automatically tagged for syntactic and morphological information. The performance of the parser is estimated to have a labeled attachment score of 81. 4%. The resources are publicly available and can be found at <http://bionlp. utu. fi>. To construct the semantic vector presentation for the Finnish lexicon, we used the skip-gram version of the Word2vec algorithm  (Mikolov et al. 2013). This type of model learns to predict the context words of a given target word by changing the activation weights of the nodes in the hidden layer.
The training data contained 1,428,439 unique transitive triplets consisting of a subject, verb and object. The model both reconstructs the original subject and verb word vectors, and also produces a predicted word vector for the object. Since the model predicts an object which is based on examples on the training corpus, we can use the distance between the predicted, mapped object vector, and an arbitrary object vector to model the semantic fit of a given lexical realization of the object in a transitive construction. This semantic fit can be expressed using a cosine similarity between the predicted object vector and the semantic vector of the object. We illustrate the semantic structure learnt by the AE with the verb hoitaa 'take care of' in Table  2. The left side of the table contains the estimated six closest semantic neighbors for the object when the subject slot was filled with äiti 'mother'.
7 (SD = 3. 42). Each list had 23 participants and, in total, 10,350 completions were produced. The completions were automatically lemmatized and morphologically tagged using the software OMorFi  (Pirinen 2011). After which, the results were manually verified and cleaned by removing words containing typographical errors (less than 1% of the data). For the purposes of the present study, we only included in the final data set those instances that can be considered to function as objects in the transitive construction excluding, for example, adverbs. The final data set contains 9681 completions. For the purposes of this study, we present two analyzes of the data in which the performance of the vector-based models are compared to the productions in the cloze task. The first one concentrates on the correlation between the estimated semantic fit by the models and cloze probability discussed in Section 6.
dbscan In this work, we have chosen to identify spatial features at a given scale using local density properties and a connectivity rule to link together adjacent stars having similar stellar density neighborhood. The spatial structures thus identified are defined as connected "density cluster", following previous graph-principled work  (Hartigan 1975). To efficiently identify density clusters, we have chosen the non-parametric, onelevel dbscan (Density-Based Spatial Clustering Applications with Noise) clustering algorithm developed in the Knowledge Discovery Database field  (Ester et al. 1996, see appendix B). dbscan has several advantages over other clustering algorithms (see appendix C): the partition of stars within density components is unique, it allows the detection of clusters of arbitrary shape and size at a global scale from local requirements, and it is the only clustering algorithm that explicitly labels noise and outliers.
The second point outlines the fact that a point is never density reachable from a border point, even if this point is in the -neighborhood of the border point. So there are two basic situations that stop the ongoing growth of a cluster. One is based on the -neighborhood length threshold since the algorithm excludes as candidate members all the points that are not -indirect neighbors of the seed (first condition above). The second is based on N min parameter which is the parameter that is indirectly used in order to decide whether a cluster should be increased at a certain stage of the algorithm. This parameter describes in fact the minimum number of reachable points that a given point must have in its -neighborhood not to be a border point. A significant local drop of -neighborhood density within the dataset therefore stops the growth process (second condition quoted above).
In other words, the cluster growth stops when no more path is found from a core point, when the edges of a cluster are surrounded by less dense regions. Either way, the first conditions implies that two clusters are separated by a distance of at least to discriminate them from being one single cluster. However, this threshold value is only true when the distance between clusters is estimated from core points. This separability distance may indeed be shorter when border points are implied in this inter-distance evaluation (second condition). When this occurs, the two clusters are marginally separated due to a localized drop in density. Choosing a shorter -neighborhood length to scan the dataset would lead to identifying these border points as noise and would separate more clearly the two clusters.
Our contribution Our approach combines and leverages existing lowlevel verification, validation, and transformation means to provide the end-user with appropriate debugging information. It relies on the construction of transitive relations between the data produced by the various tools, and the user requirements for the model simulator. We illustrate this approach on the development of a model simulator for the Fiacre language  (Berthomieu et al. , 2008)  using the existing model checking toolbox TINA  (Berthomieu et al. , 2004). This paper is structured as follows. Section 2 presents the context of the study and the use-case. It gives an overview of the user requirements for the model simulator. Section 3 proposes different design method for such tools. It details their implementation and discuss about the adopted solution.
To comply with the availability and safety requirements, the mission subsystem is broken down into two channels (left/right) with two units per channel (COM/MON). A clock synchronization (CS) protocol  (Rodrigues et al. , 1998)  ensures a synchronous behavior of all units. This CS protocol model is around 700 lines of Fiacre code covering both the units to be synchronized and the communication network (CAN). Verification is performed using the TINA toolchain. Even though directly inspired from  (Rodrigues et al. , 1998) , building the model of the CS protocol required a significant design and debugging effort due to the various abstractions and simplifications that were required to obtain a tractable model  (Bourdil et al. , 2016a). In the rest of the document, we will take small samples of this model as illustrate specific issues encountered during the design of the model simulator.
The TINA verification toolbox Verifications are performed by the TINA toolbox, a set of tools used to edit and analyze Timed Transition System (TTS), an extension of Time Petri Nets (TPN) with data manipulation. The following toolbox components are considered in this paper: • TINA constructs reachability graphs and Kripke transitions systems from TTS and TPN • PLAY is a TTS and TPN animator. To be processed by TINA, a Fiacre model must be translated to TTS using the dedicated tool FRAC 3. Due to the semantic gap between the two languages, some constructs present in the Fiacre input may be hidden from the TTS output. Fortunately, FRAC can also generate transformation traceability data (using the -G option) that can be exploited to build the Fiacre simulator feedback. Now, let us consider the designers needs in terms of debugging features.
Solution 2: Use traceability data Solution 2 (in red in Figure  4 ) extends solution 1 by combining information given by the initial trace model (see Listing 2) with traceability information generated by FRAC (-G option). Historically, this information was used for debugging purposes during the development of FRAC. Later, it was also used to feed verification results from the TTS level back to Fiacre  (Zalila et al. , 2012; Zalila et al. , 2013). It is produced during the last step of the translation phase, before the generation of the TTS description. It contains TPN and data processing constructs  (guards, assignments, etc. ). Listing 3 shows a subset of a compilation trace model related to the CAN process shown in List-ing1. It contains two TTS transitions: Can 1 t0 and Can 1 t1. In order to locate these transitions in the Fiacre source model using the initial compilation trace model, it is necessary to understand how the FRAC compiler generates TTS identifiers from Fiacre-level identifiers.
Comparison of solutions In this subsection, we compare the previous solutions by estimating the cost required to recover information degraded during the compilation phase. Figure  5  illustrates this process. In this example, we consider a Fiacre source code containing a process, proc, instantiated in the compo component. The proc process has a transition containing a non-deterministic statement in which each choice contains the same source code block (x:=x+1; to s1). Let us consider the following user requirements: • REQ-a: the FS shall display the executed Fiacre statements. • REQ-b: the FS shall display the executed Fiacre statements in the source model. The problem consists in satisfying those requirements when the TTS transition proc 1 t1 is fired during animation. First, we generate the abstract syntax tree (AST) of the Fiacre model using Xtext.
RELATED WORKS Language engineering toolsets target the modeling of languages and the implementation of associated tools but the behavioral concern are usually not handled. More recent toolsets, like xCore, xMOF  (Mayerhofer et al. , 2013) , the K framework  (Rosu, 2013)  or the GEMOC studio  (Combemale et al. , 2016)  allow to handle it and ease the development of model simulators. However, they usually do not target model checkers except the K framework. But, this one do not allow to manage efficiently the combinatorial explosion of concurrent behaviors and thus does not scale to many realistic models. We could have used these toolsets to build the Fiacre simulator with the risk of semantic inconsistency between the separate implementation of the behavioral concern in the model checker and the simulator.
This notional model depicts the cognitive stages of people as they use visual analytic tools to gain understanding of their data. From Pirolli and Card's perspective, sensemaking can be categorized into two primary phases: foraging and synthesis. Foraging refers to the stages of the process where models filter and users gather collections of interesting or relevant information. This phase emphasizes the computational ability of models, as the datasets are typically much larger than what a user can handle. Then, using that foraged information, users advance through the synthesis stages of the process, where they construct and test hypotheses about how the submitted to COMPUTER GRAPHICS Forum (2/2018). foraged information may relate to the larger plot. In contrast to foraging, synthesis is more "cognitively intensive", as much of the insights stem from the user's intuition and domain expertise.
Another useful model that can be employed to describe the human-centered sense-making process is the "data-frame model" by Klein et al. [KMH06b, KMH06a]. Their model (Figure  2 ) depicts an exchange of information between the human and the data in terms of frames. People make sense of a situation by interpreting the data they are presented with in relation to what they already know to create a new understanding. A user has an internal "frame" that represents her current understanding of the world. The data connects with the frame. As she continues to explore a particular dataset, her frames of the world are mapped against the information she uncovers. If the information supports a specific frame, that frame is thought to strengthen in a process they call elaboration. As she understands the situation better, she searches for more relevant information, learning that there may be other factors to the problem than originally thought or known, therefore driving the demand for more information, and building her frame.
In the following, we review such works again under a categorization of different ML algorithm types involved. Notice that integrating user knowledge in this way in unsupervised learning contexts falls into the general semi-supervised framework, which is a submitted to COMPUTER GRAPHICS Forum (2/2018). principled way in ML for making unsupervised problems less ill-posed. Dimension Reduction Dimension reduction algorithms are suitable candidates for such approaches due to the often "unsupervised" nature of the algorithms and the possibility that errors and losses within the reduction phase are high, in particular with datasets with high numbers of dimensions. As one of the early works along these lines, Endert et al. [EHM * 11] introduce observation level interactions to assist computational analysis tools to deliver more interpretable/reliable results.
At this point, it is worthwhile to describe a visual analytics system that combines all these characteristics. VAiRoma [CDW * 16] (shown in Figure  11 ) creates a narrative that tells the whole 3,000 year history of Rome, the Empire, and the state of Italy derived from a collection of 189,000 Wikipedia articles. The articles are selected from the nearly 5M English language article collection in Wikipedia using a short list of keyword, but otherwise the initial topic modeling and named entity extraction are done automatically. The interface for VAiRoma is displayed in Figure  11. The individual topics are depicted as color-coded streams in the timeline view (A). The circular topic view in (C) provides a compact way of depicting topics, the weights of their contributions for a given time range, and topic keywords.
In this case, the user can focus from thousands of articles to a few hundred articles overall, which she can then quickly peruse. See the VAiRoma article for more details. VAiRoma shows the power of the overall model depicted in Figure  3. Though it is not complete w. r. t. this model (no current VA system is), it provides an integrated approach to data handling, interactive visualization, ML (in this case topic modeling) combined with other techniques, and exploration and knowledge building techniques. It shows the power of an integrated approach. The approach is general and is now being applied to large, heterogeneous collections of climate change documents. In addition, full text journal article collections are being analyzed using extensions of the topic modeling and entity extraction methods.
People in charge of the safety and reliability of large networks analyze large amounts of streaming data and alerts throughout their day, thus the temporal component of making a decision from the analysis is emphasized. For example, Fisher et al. presented Event Browser, a visual analytic system for analyzing and monitoring network events  [FMK12]. Their work emphasizes how different tasks of the analyst have to happen at different time scales. That is, some tasks are "real-time", while others can be taken "offline" and performed for a longer duration of time. The persistent updating of new data into the offline tasks presents challenges. Social media data can also be analyzed using visual analytic systems. For example, Storylines [ZC07] and Even-tRiver [LYK * 12] are two examples of how visual analytic applications can help people understand the evolution of events, topics, and themes from news sources and social media feeds.
In these systems, similar machine learning techniques are used as for text. However, the temporality of the data is more directly emphasized and taken into account. Lu et al. [LKT * 14] showed how appropriate social media analysis could have predictive power, in their case predicting movie box office grosses from early word of mouth discussion on Twitter, YouTube, and IMDB. A dictionary-based sentiment analysis was used along with analytics from the R statistical computing environment and the Weka machine learning workbench. This permitted a choice of modeling in terms of multivariate regression, support vector machines, and neural networks. The paper promoted an integrated visual analytics approach where the interactive visualizations, based on D3, permitted users to investigate comments and sentiment, classify similar movies, and follow trends and identify features.
The user could then improve a base line regression model based on trends and features identified in the visaulizations. Results of the use cases were positive with several of the non-expert participants being able to outperform experts in predicting opening weekend grosses for 4 films, according to the criteria set up by the authors. The paper has the usual limitation of supervised learning approaches in that a training dataset must first be collected and analyzed as a preliminary step, but it does successfully allow for improvement of the analytic model within the VA environment. Also, like many papers dealing with more complex analysis, it defines a process for best use of the system; this appears to be an important and effective approach for VA + ML systems. Yeon et al. [YKJ16]  covered similar ground in their identification and analysis of interesting past abnormal events as a precursor for predicting future events.
A similar idea on the stochastic neighbor embedding algorithm (SNE) was also proposed in [DCP * 14]. • The user might also have insight on the similarities between items. In [BLBC12], a system called dis-function was developed, featuring DR visualization that allows the user to modify the distance matrix Dij = aiaj Ω between items i, j, by moving points in the visualization based on his/her understanding of their similarity, and see new results after a recomputation of the projections with the new metrics. • Also, prior knowledge on class information can be inserted by the user, suggesting techniques to increase the similarity of items belonging to the same class. In [PZS * 15] a method is proposed to allow the user to include prior class knowledge in the DR projections by extending the original dataset with transformations of the original feature space based on his existing class knowledge.
Designing ad hoc ML algorithms with this approach in mind that pave the way for new and useful kinds of interaction mechanisms opens new and exciting research paths. There has been some prior work on this topic. For example, Stolper et al. developed a system for progressive visual analytics, where intermediate results of a sequence-mining algorithm running on medical treatment events can be shown to clinicians  [SPG14]. Their work gave analysts the ability to see broader results sooner to help decide if the entire computation needed to be executed. Similarly, systems to show partial query results of large datasets  [FPDs12]  and partial dimension reduction and clustering results  [TKBH17]  have been recently developed. These works raise important questions about the tradeoff between accuracy and execution time of these algorithms, and also about how to incorporate user feedback into computation during runtime.
This is the base dataset that can be used for traceability, responsibility and provenance: providing an argued case for others (such as collaborators or managers) to critique and use to make decisions. However, beyond this use, the database is also a resource to mine in order to clarify the decisions that are made in the course of visual analytics, leading to the potential to develop adaptable interfaces and a greater depth of understanding of users' mental models, which can then be used to guide other, perhaps less skilled or experienced, users. It would not be feasible (nor practically useful) to track every single change in a visualisation. It is essential that the process involves minimal interruption to cognitive flow (so as to avoid damaging the very process we are trying to understand).
As a complement to this database of successful analytic practice, what many users need is a way of avoiding bad practice (or errors). A catalogue of 'typical'errors that is searchable (using case-based reasoning tools) could be crowdsourced from teachers (and their students!) or training courses. How can machine learning aid the understanding of user processes? At the simplest level, user interactions are a linear sequence of actions: discovering the underlying sequence and the transitions between items is relatively straight-forward, since a Markov (or hidden Markov) model can easily be trained to uncover this structure. However, an unstructured and unannotated sequential list does not contain enough structure to infer the analytical process. Firstly, we need to understand the reasons why a user has made choices (which requires annotations).