dataesr/scientific-paragraphs-categorization huggingface dataset:
One of their views utilizes bar charts in order to rank the most valuable features that affect the model's performance. Ji et al. [JSR *  19] propose visual exploration of a neural document embedding with the goal to gain insights into the underlying embedding space and encourage this utilization in standard infrared (IR) spectroscopy applications. In their paper, they use a scatterplot visualization, i. e. , a projection. LSAView [CDS09] is a system for interactive, latent semantic analysis (LSA) models. Multiple views, linked matrix-graph views, and data views in the form of lists are used to choose parameters and see the effects of them. Other papers apply different visual representations, some rare cases are waterfall charts, violin charts, Voronoi diagrams, and bipartite graphs [Aup07, HSD19, KKZE20, LA11, LGG * 18, WGSY19, WGYS18, ZTR16].
Evaluation/User Expectation (TL5) The Agreement of colleagues is related to provenance and the possibility to enable users to collaborate with each other. Wongsuphasawat et al. [WSW * 18] present a design study of the TensorFlow Graph Visualizer, which is a module of the shareable TensorFlow platform. This tool improves users' understanding of complicated ML architectures by visualizing data-flow graphs. These flows can be investigated, and at each point in time, provenance can be considered as a way to return back to a previous situation. Visualization evaluation, as mentioned earlier in Sect. 6. 6, is activated when the visualization techniques and tools are evaluated or if any type of feedback is provided. Showing metrics validation/results is the most common way of enhancing trust until now.
The top eight terms for each topic are displayed in Table  4  together with the papers belonging to a topic (see S6 for further details). From the terms that occurred, we removed any terms related to the structure of the analyzed texts and not to the actual content, for instance, "figure" and "fig". Our implementation is based on Python with NLTK  [Bir06]  for the pre-processing Table  4 : For each of the ten topics, we present the top eight terms that we extracted from the results of the latent Dirichlet allocation (LDA) that has been applied to all papers. The suggested topic titles are shown in italics. Each topic is encoded by one specific color. In each topic, we cite the papers that mostly belong to them. of stop words and Gensim  [ ŘS10]  for the topic modeling part. The names of the topics were manually assigned by us after several discussion cycles considering both the top terms and the contents of the papers in each topic.
Ensemble learning along with DL are positively correlated, as the former includes the latter in many cases (a fact already mentioned before). Mapped instances lead to instance-based visualizations, in general. With regard to domains, continuous target variables and business are positively correlated (potentially due to trend predictions); as well as computer vision and multi-class data in an even stronger fashion. The latter correlation is also supported by our data set analysis (see Table  5 ). Finally, visualization interactions are positively correlated among each other, with the exception of verbalization which is negatively correlated with the remaining categories in this group. This possibly means that verbalization is not frequently used by the VA system developers. For more details, we refer the reader to Figure  6 (b) and supplementary material S7.
Research Opportunities The impact of bias. By looking at our categorization, we can infer that some level of bias might be represented in all our defined trust levels in different forms: (a) data bias (equality), (b) previous familiarity with algorithms, (c) model bias, and (d) user bias. Also, it is known that visualization techniques ordinarily do not scale very well when analyzing massive volumes of data. Furthermore, some of the ML approaches have inherent challenges to face, for example, the curse of dimensionality  [Bel03]  in case of DR. Thus, considerable levels of selection bias might be unintentionally ignored by the user, for instance, when users have to choose from a selection while not seeing the entire picture and/or the alternatives  [GSC16, LA11]. Hence, the research question here is: "what novel solutions can help users to minimize the impact of bias with regard to the data?" A potential answer would be to consider various interaction logs with the VA system.
[MXLM20] , explaining how visualization can assist in avoiding vulnerabilities of adversarial attacks in ML. Specifically, their focus is on how to avoid data poisoning attacks from the models, data instances, features, and local structures perspectives with the use of their VA approach. Nowadays, visualization systems are deployed online for users to access them easily. Such internet-accessibility leads to further problems concerning security vulnerabilities. This is one of the advantages of TensorFlow. js, which utilizes the WebGL-accelerated implementation of JavaScript in web browsers to implement and use ML models on local computers. Fairness of the decisions. Going beyond interpretability towards more explainability is another open challenge. However, general proposals of frameworks in the visualization community combining ML and visualizations have been already described in recent research papers  [MXQR19, SSSEA20].
Conclusion In this survey, we study the state of the art in enhancing trust in machine learning (ML) models with the use of visualizations. We introduced the background necessary for defining trustworthiness of ML models and explained the methodology used to select relevant papers in the literature. Based on the selected 200 peer-reviewed publications that introduce a large variety of visualization techniques to increase trust in ML models and their results, we proposed a fine-grained categorization comprising 8 high-level aspects partitioned into 18 category groups that on their part contain 119 categories in total. In addition, we performed a topic analysis to be able to discover connections and emerging topics among the 200 papers. Further analyses of the categorized data involved category correlations, temporal trends, and data sets used in the respective publications.
In order to make our categorization and the assignment of papers into categories accessible for the public, an interactive survey browser-called TrustMLVis Browser-was implemented and made available online. It supports the readers of this STAR in the exploration of the rich information provided in this work, thus facilitating future research in enhancing trustworthiness of ML models with the help of interactive visualizations. Our findings indicate the growing interest for developing visualizations in ML to improve trustworthiness in the context of various data domains, tasks, and multidisciplinary applications. As future work, we intend to continue extending and refining the survey data set, categorization, and corresponding analyses, as well as maintaining the online survey browser.
For rivers again, Hening Huang proposed in 2018  [19]  a theoretical and a semiempirical model calibrated on transect datasets, to estimate the uncertainty of streamflow measurements made by an acoustic Doppler current profiler (ADCP) mounted on a moving platform. This method could be applied to oceanography using vessel-mounted ADCPs, though it is not adapted to standalone Teledyne RD Instruments ADCPs or equivalent instruments of Nortek group called AQPs for 'AQuadopp Profilers'. The idea came to perfect a test using an acoustic transducer attached to the profiler transducers and linked to a frequency generator. This allows echoes received by the profiler to be simulated. The exploitation of the Doppler effect formula and of the speeds sensed by the instrument has allowed a test method of the device's measurement channels to be perfected.
For Doppler current meters, the low particle concentration in these facilities is a problem, and by taking into account the profiling range of profilers used in oceanography, this method cannot be applied. It was still possible to carry out intercomparisons at sea as carried out in rivers by Boldt and Oberg in 2015  [18] , but these intercomparisons are expensive, difficult to organize, and allow only one part of the velocity range of instruments to be tested. For rivers again, Hening Huang proposed in 2018  [19]  a theoretical and a semiempirical model calibrated on transect datasets, to estimate the uncertainty of streamflow measurements made by an acoustic Doppler current profiler (ADCP) mounted on a moving platform. This method could be applied to oceanography using vessel-mounted ADCPs, though it is not adapted to standalone Teledyne RD Instruments ADCPs or equivalent instruments of Nortek group called AQPs for 'AQuadopp Profilers'.
The difference between the impedance of the piezo-electrical transducer (which is very high) and that of the air (which is very low), however, results in nearly 99% of pulse energy being reflected to the interface. The other part, along with pulses transmitted by the profiler's other transducers, can increase the detection noise according to the wavelength of the instrument. All this equipment is remote controlled by a program developed using LabVIEW© software. It automates testing by decoding the messages from different profiler models, to determine and change their configuration, to extract speed values and to drive the frequency generator. Using a frequency generator allows the adaptation of the power of the signal heard by the profiler. It can be adjusted to optimize the noise of the measured speeds, taking into account the saturation of the measurement channel.
The value 1525 m/s was chosen to obtain profiler responses in the range of ±6 m/s (except for the Nortek versions called DeepWater where f r is different and leads phase wrapping). This value is programmed in the instrument and in the LabVIEW program to calculate a reference velocity V ref. It is important that the device under test (DUT) and the LabVIEW program contains the same value of c to see the same speed variations for given frequency variations. Calibration consists in calculating a speed deviation δv from the Doppler effect formula, such that: δv = V re f -V i = c 2 δ f re f -δ f i f 0 ( 10 ) with δf ref = kδf s. δf i is defined by relationship (8), i being the index of the instrument's transducer.
A numerical tool has been also deployed to identify the interface model parameters. As the reliability of the numerical model depends strongly on the extent of accuracy of its parameters, great attention has been given to this phase of study. The analytical ply model has been implemented in Abaqus by the finite element method via the user material (UMAT) subroutine. The simulation of the interface behaviour has been performed directly on the CAE interface of the software. In parallel with the development of the whole model, a set of experimental tensile tests has been carried out on representative specimens to validate the reliability of the numerical model.
Finite element model The theoretical model presented and identified above has been implemented in the Abaqus finite element software. In fact, the tensile behaviour of ±55 • longitudinal representative specimens has been simulated and observed with the aim of evaluating the performance of the model. These tests have been accompanied by strain measurements using classical methods that provide information, allowing a comparison with numerical analysis. The choice of semi-structures (specimens cut from tubes) for the validation of the numerical model has been based on the fact that carrying out experimental tests on a whole tube is difficult, laborious and expensive. Thus, it appeared appropriate to use representative specimens able to capture the phenomena appearing in the tubes during their loading.
Therefore, the same experimental set-up in the model identification has been used in its validation. The numerical model developed to simulate inter-laminar and intra-laminar damages appearing in the representative specimen loaded is a 3D meso-scale finite element model. The whole laminate has been divided into 10 sections (plies) of 0. 5 mm thickness each, separated by cohesive layers of zero thickness. The plies have been discretized using an eight-node integration element (SC8R) called the "continuum shell". Regarding the cohesive layers, they have been modelled using an eight-node cohesive element called COH3D8  [26]. Two different types of materials have been included according to the two constituents of the structure. The material of the cohesive layers has been defined directly on the CAE Abaqus interface thanks to the material definition module already integrated in its library.
However, the definition of the ply material has required the "user definition" module. The computational algorithm describing the ply behaviour has been written and implemented in the Abaqus code through the UMAT subroutine. An assembly of 10 plies and 9 cohesive layers across the thickness defines the complete model. This assembly has been constructed in such a way so as to maintain continuity between plies before the damage occurs. In fact, a connection has been made between the cohesive layers and the plies using the boundary condition "tie". To evaluate the performance of the model, tensile behaviours of ±55 • longitudinal specimens have been simulated (Figure  13 ) and observed. It should be noted that in the simulation of the longitudinal specimens, the curvature has been neglected and only the area between the jaws has been modelled.
We present PANENE, a progressive algorithm for approximate nearest neighbor indexing and querying. Although the use of k-nearest neighbor (KNN) libraries is common in many data analysis methods, most KNN algorithms can only be queried when the whole dataset has been indexed, i. e. , they are not online. Even the few online implementations are not progressive in the sense that the time to index incoming data is not bounded and cannot satisfy the latency requirements of progressive systems. This long latency has significantly limited the use of many machine learning methods, such as t-SNE, in interactive visual analytics. PANENE is a novel algorithm for Progressive Approximate k-NEarest NEighbors, enabling fast KNN queries while continuously indexing new batches of data. Following the progressive computation paradigm, PANENE operations can be bounded in time, allowing analysts to access running results within an interactive latency.
condition: ∀i ∈ KNN k (q) ∀j ∈ [1, N ] -KNN k (q), q, p i ≤ q, p j , where q, p is the distance between q and p. The KNN problem is a building block of many data mining and visualization methods, such as clustering  [4] , classification  [5] , embedding  [6] , and non-parametric density estimation  [7]. Thus, designing an efficient progressive algorithm for the KNN problem is an important step towards extending the applicability of progressive computation. One straightforward approach is to calculate the distances from a query point q to every point in the dataset and take the k closest. However, this method is inefficient, since it has to iterate over all points and thus has time complexity of θ(N ). A more efficient approach is to use a search data structure or an indexing method such as a k-d tree, which usually reduces the time complexity to a logarithmic scale.
Progressive Systems The latency of interactive systems has become a primary concern for visualization researchers and practitioners. As the size of datasets increases and analytic methods become more sophisticated, delivering results within a time limit becomes a new challenge in designing visual exploration systems. Reducing the latency is an essential problem in visual analytics, because long latency not only delays analysis but also outpaces humans' ability to focus their attention, eventually degrading the quality of the analysis  [27]. Nielsen  [11]  distinguishes three time limits for users' perception and attention: 0. 1 second for feeling that the system is continuous (e. g. , for animations), 1. 0 second for maintaining the user's flow of thought, and 10 seconds for keeping the user's attention.
Algorithm 1 A sequential algorithm for recursively building a randomized k-d tree of the given l points in L Algorithm 1 has three strong limitations: First, it requires all the points in L to be already loaded in main memory before it can build the randomized k-d trees. Such a constraint forces analysts to wait until all data is read from disk or database before performing any analysis. Second, once the data structures are built, the algorithm does not allow any modification, such as inserting new points or removing points. Finally, the running time of the algorithm solely depends on the size of input (i. e. , l), and thus its latency cannot be controlled.
An Online Algorithm In contrast to a sequential algorithm, an online algorithm allows adding new points to the trees even after they are built. This benefits interactive systems in that analysts do not have to wait until all data is loaded. Rather, the data is split into batches, loaded onto the system incrementally, and can be used for further online algorithms. Analysts can access the running result between the batches, obtaining improved approximations of the final results. However, we only found one implementation of online AKNN: the FLANN library  [10]. It can build an initial k-d tree of the points in the first batch using Algorithm 1, and other points can be added into the tree thereafter. The insertion procedure of the FLANN library is very akin to that of a binary tree: Starting from the root node, each point moves to either the left or the right child of an internal node by comparing its value at the dimension chosen to split the values at that node-called the cutdim-against the median value computed initially for that node-the cutvaluntil it reaches a leaf node.
This implies the need to rebalance the tree when it is too unbalanced. In real cases, the imbalance is never that extreme but can vary substantially if the data added has a different distribution than the initial tree. The imbalance leads to a slower query time with little degradation of the result quality. On the other side, when updating a tree for a large dataset, assuming the data is stationary, the distribution of incoming data will at some point converge to the distribution of the whole dataset, and the tree will remain balanced even after new points are inserted. FLANN's implementation of k-d trees uses a simple strategy for rebalancing the trees: It reconstructs all trees each time the dataset doubles in size from the initial dataset (i. e. , the first batch). Therefore, the k-d trees can become unbalanced as new data is loaded but eventually will be reconstructed.
When loading a large dataset progressively, even if the incoming distribution matches the current k-d tree structure, FLANN will always reconstruct its k-d trees when the dataset doubles in size. To sum up, the FLANN implementation suffers from three problems: 1) the k-d tree may become unbalanced when data is added, leading to longer KNN searches; 2) the k-d tree is always reconstructed when the dataset doubles in size, leading to long interruptions in the KNN search at unpredictable moments; and 3) the k-d tree is always re-created when the dataset doubles in size, even when it remains balanced.
Filtered AKNN Search Visual analytics should allow users to explore data by applying filters dynamically. To avoid rebuilding whole k-d trees when the user filters points, our progressive search algorithm can restrict its search to a selection of points. This selection is implemented through a very fast bit vector library  [37] : The list of filtered points is converted into a compressed bitmap and passed to the search function that gathers the k neighbors, making the search function ignore the points in the bitmap. This filtered search is slightly slower than a non-filtered search, depending on the number of points filtered, but always faster than rebuilding whole k-d trees before performing the query. The original FLANN algorithm has a provision for removing points from the k-d trees, marking them as deleted.
BENCHMARK Online algorithms are usually evaluated using competitive analysis  [39] : their performance are compared against an equivalent offline algorithm and the ratio is reported. This ratio only makes sense when the algorithm needs to complete to its end, but one premise of progressive data analysis is that some decision can be made before the algorithm ends. Competitive analysis does not account for these important early termination cases. On the other extreme, Eichmann et al. [40]  have proposed benchmarks to compare the effectiveness of databases to fulfill the requirements of data analysis sessions with or without progressive support. This benchmark requires a full system used on a realistic task, which is not our purpose. In our work, we remain at the algorithm level and compare our implementation with the only online implementation available, provided by FLANN.
Online and Progressive k-d Trees In this benchmark, we compare the performance of online and progressive k-d trees. We used one real dataset (the GloVe  [41]  dataset) and one synthetic but more structured dataset (a Blob dataset). Both datasets had 1 million of 100-dimensional points. The GloVe dataset had embedding vectors of English words, while the synthetic Blob dataset had points sampled from 100 isotropic Gaussian blobs, 10,000 points for each. For reproducibility, the Blob dataset was generated through scikit-learn's blob generator  [5]. The points in the Blob dataset appear in an increasing order of clusters; for example, the first 10,000 points were from the Algorithm 5 An algorithm for building a progressive KNN lookup table end while 39: end procedure first Gaussian blob, the next 10,000 points were from the second Gaussian blob, and so on.
As query points, we used another 1,000 embedding vectors for the GloVe dataset and random 100-dimensional vectors for the Blob dataset. To see the effect of the inserting order of points, we used two conditions: the order was 1) kept as in the original dataset (original) and 2) shuffled (shuffled), potentially producing balanced k-d trees earlier. For each iteration, we gave 5,000 operations to both the online and the progressive k-d tree (i. e. , ops = 5, 000). The online version used up all operations to add new points (i. e. , 5,000 points were inserted to k-d trees during one iteration). Fig. 1 : Benchmark results of online and progressive k-d trees according to datasets (GloVe  [41]  and Blob) and ordering conditions (original and shuffled). FLANN's online k-d tree (the red line) rebuilt the k-d trees each time the data size doubles, producing insertion times longer than 10 seconds (e.
g. , leftmost charts at the 62nd and 126th iterations). In contrast, progressive k-d trees (the green, blue, and purple lines) alleviated the spikes in insertion time by spreading the reconstruction load. Our progressive k-d trees yield a small loss in queries per second (middle column), yet with faster convergence to a mean distance error of 7% for the GloVe dataset and 3% for the Blob dataset (rightmost column). For the progressive k-d tree, we used three different values for τ : 0. 2, 0. 35, and 0. 5. A progressive tree with a higher value of τ prioritizes insertion tasks, assigning fewer operations to maintaining the balance of the tree. Note that the value of τ only affects the algorithms when reconstruction occurs. For example, if the accumulated loss due to imbalance has not reached a certain threshold, all operations will be assigned to insertion tasks for the progressive k-d trees.