rare), and completeness relates to more patterns as well as more interesting patterns.

There are a number of early studies of progressive systems that gather multiple values. Stolper et al. [293] propose a system, “Progressive Insights,” that is designed to search for common patterns in event sequence data. The system progressively returns a growing set of matches. Their evaluation is qualitative: they report use cases and observations of a doctor using the system and commenting (positively) on its capabilities. Ravenau et al. [248] also report on sequential pattern mining, describing six tasks they want their system to support. They evaluate their system using competitive analysis [30], showing that their progressive version is slightly slower but more flexible than its eager equivalent. They also performed a controlled study with 14 participants to compare their interface with and without steering. They show that participants with steering were both faster for their tasks and were also more accurate with their answers. They also report on feedback after the controlled study using a questionnaire and show that users preferred a progressive system and its steering capabilities. Gogolou et al. [102] report about a method to perform data series progressive similarity search. They evaluate their method using synthetic and real datasets and evaluate the quality of results as well as the time savings of the early stopping strategy. For the quality of results, they measure how much of the total search space has been covered over time and evaluate the confidence intervals used in finding the best match. Another aspect of the quality of results they evaluate is the time guarantees for the exact answers. The time savings are measured by the time saved using early stopping relative to the exactness of answers. Their evaluation does not take into account the benefit of progressive intermediate results returned, only the quality of the final “best” result, which remains the recognized criteria in the database literature.

All the articles on search/gather techniques mention useful properties, such as early partial results, but they do not evaluate their importance specifically. **Research Agenda:** There are substantial opportunities to refine methods and methodologies to evaluate search/gather progressive results. A promising theoretical framework for this is *optimal stopping* [52], which consists of choosing a time to take a particular action in order to maximize an expected reward or minimize an expected cost. However, choosing a good reward/cost function might not be obvious for exploratory tasks.

<figure><img src="image_97.png" /></figure>

### 9.2.3 Visualization and Interaction

Evaluation of progressive visualization and interactions is concerned with enabling user interaction during progression (R12), and it seeks to provide effective visual
feedback on computation progress (R5, R11) to support users in generating informed insights and making early observations/decisions. In some cases, the eventual completion time of the progressive process, i.e., the time that is waited for the final results (R10), is also an important factor. To support users in decision-making, other properties of the progressive visualization can also play a role, including how fast the progressive computation converges and how users visually assess the quality of the visualization at a given step (R7).

We can learn about the evaluation of progressive visualization by looking at analogous systems in computer graphics more generally. Progressive rendering has been investigated for use in tools for editing 3D realistic graphics, such as the shapes of computer-aided design (CAD) models or lightning and materials of a movie sequence for computer-animated films. Interactive previews are essential for editing tasks that require making careful enhancements while being mindful of images that take hours to render. For example, progressive rendering is increasingly becoming a standard in relighting engines [267], systems that allow users to interactively configure and refine lights in a scene. The primary challenge is to provide adequate fidelity in the initial approximation to enable an initial placement and configuration of lights. Although many rendering algorithms enable straightforward, progressive updates, methods differ in how predictable they are and how fast they converge for complex scenes [267]. These tools are evaluated following an established user methodology where users run both open- and non-open-ended tasks in the interfaces being compared. Measures of success are both objective (time to completion, image error) and subjective (user ratings for the effectiveness of specific tasks and globally) [267]. Some work has also focused on improving the refinement strategy of rendering algorithms to match the vision model [81], suggesting perceptible image error as an alternative success criterion. **Research Agenda:** Similar user methodologies could be developed for comparing progressive visualization methods in the future.

<figure><img src="image_98.png" /></figure>

Progressive visualization introduces dedicated visualizations and interactions designed to help users make sense of the progression, assess intermediate results, and manipulate intermediate views. Existing systems often convey feedback on progress and quality using separate visual views, such as progress bars for progress and line plots for the quality history (e.g., [16]). Evaluations focus on the utility and effectiveness of a new visualization, in some cases in comparison with an alternative eager interface. Measures of success include usability scores, insights [342], and interaction metrics, time and accuracy of user answers, user confidence, and criteria for visual stability [224]. Patil et al. [224] (also discussed in Section 6.3) perform a controlled user study for progressive bar charts using multiple conditions and also
compare human results with those of sequential statistical tests (humans are good) that remain heuristics for data with unknown distributions.

**Usability** Progression and quality feedback (R10, R11) are essential to users in assessing the trustworthiness of the visualization, and support them in making decisions with confidence and at an appropriate time. In particular, any change in the context of application and tasks, e.g., toward more sensemaking or more time-pressure [161], will influence how a user assesses the time-quality trade-off and the time point at which they determine that they can or need to decide. Subjective feedback collected during user studies showed that simple progress bars were found to be useful to users overall [16]. Progress feedback can take different forms, and thus different visual representations, depending on the type and complexity of the upstream progression process (relative or absolute, estimated or not). Therefore, checking to make sure that users can make sense of the progression is an important aspect of the evaluation.

There can be multiple progress indicators calling for new representations, for instance, when users have the ability to start multiple processes in parallel. In ProReveal [141], users can compose multiple progressive charts, although only one can run at a time. A side-view list summarizes the progress of every ongoing visualization by displaying progress indicators, and allows alternate computation between visualizations. The major focus of the interface is interactive guards, called PVA-Guards, which are operators and visual reminders whose purpose is to help conduct speculative analysis of the data. Users can formulate their current hypotheses and register them to the system such that their validity is tested continuously as intermediate results improve. The guards are visually displayed on the side and provide a notification if the hypothesis proves incorrect. As a proof-of-concept, PVA-Guards have been evaluated in a user study using usability as a measure of success: testing whether users were able to correctly interact and gain insights, while collecting subjective usability ratings. Finally, usability studies also report user comments, observations, and descriptions of their preference [16, 304, 342].

**Research Agenda:** With regard to usability, there is also a need to evaluate how well system or algorithm guarantees, such as probabilistic bounds on the error in intermediate results, are visualized, understood, and leveraged by users in their analysis strategies.

**Time, Accuracy, and Confidence** Together with indicators of progression, indicators of uncertainty are essential components in making informed decisions (R7). As presented in Section 6.5, the existing body of work and guidelines on uncertainty visualization can be utilized in progressive visualization, such as traditional

<figure><img src="image_99.png" /></figure>
static error bars or transparency coloring. Patil et al. [224] compare four uncertainty representations for progressive bar charts, showing that the representations using error bars and near-history are significantly superior to the baseline bar chart and the combination of both error bars and near-history, for simple tasks. **Research Agenda:** More work is needed to evaluate how well these visual indicators are understood in the progressive context, since they are critical for making informed decisions.

Generally, there is a tension between visual complexity and the display of local or fine measures of uncertainty (e.g., single quality measure vs. per-element measure). More advanced techniques use interaction to provide answers to specific queries with corresponding uncertainty. The interactive annotations by Ferreira et al. [85], for instance, support various comparative tasks on bar charts. To compare each bar to a range of values, for example, the system estimates the probability that any given bar is inside the user-selected range and displays it using a color scale. These interactive annotations have been evaluated for answer accuracy and user confidence, and have been shown to improve user confidence compared to confidence intervals.

In a set of controlled experiments, Procopio et al. measured the effect of properties of progressive results (e.g., false patterns or high variations) and user steering ability on accuracy, confidence, and completion time for progressive gradient plots [240]. Their experiments examine the four types of cognitive biases associated with progressive visualization (see Section 7.5) and are an example of a precise experiment design.

In evaluations using time, accuracy, and user confidence as performance measures, we find a resemblance with the measures commonly used for user experiments of non-progressive visualizations or interfaces. However, time and accuracy have, to some extent, a different meaning in PVA: when completing a task in a progressive system, time is not only used to complete the task but also to wait for the result quality to improve. To evaluate how well users are able to balance waiting time with accuracy, user performance should be compared to those of optimal or near-optimal strategies, such as a sequential statistical test appropriate for the given task [321].

In addition to user confidence, it is also important to evaluate the level of confidence inspired in users with respect to the quality of results, i.e., build and calibrate trust [109]. Evaluating this relationship provides information on how efficiently progress and quality are being conveyed, but it also greatly depends on the literacy of users in the statistical tools on which uncertainty indicators are built. Evaluations of uncertainty visualization, surveyed by Hullman et al. [131], follow two main approaches: measuring user accuracy and collecting user feedback on how helpful the visualization was in making decisions. Authors of the survey provide recommen-

<figure><img src="image_100.png" /></figure>
dations drawn from other fields that should also be considered when designing human
experiments for progressive systems, such as using decision frameworks to create a
realistic decision context for participants. Some examples of user measures from the
decision-making literature, beyond time and accuracy, are decision consistency and
utility maximization [131].

**Research Agenda:** More study is needed to determine the trade-offs between different measures of user performance in the particular context of user experiments for progressive visualization as compared to current evaluation practices.

<figure><img src="image_101.png" /></figure>

**Visual Stability** For progressive visualization that computes a layout, visual insta-
bility can pose problems for usability (R2, R13, see also Section 5.2.1). Preserving
the general structure of the representation between updates is important so as not to
interfere with users’ cognitive flow (R3) and to help them spot changes. However,
there is generally a tension between accuracy and stability. To support visual stability,
new results can be introduced through animated transitions and changes with color or
other cues for emphasis.

Visual stability is often part of the design goals (e.g., constraining the spatial area of change [233]), but it can also be needed indirectly to assess the convergence of the visualization [16, 230]. For layout and clustering algorithms, a quantitative measure related to visual stability is usually computed (e.g., looking at the MDS criterion value over iterations [51]). Giving users control over updates, pause and rewind can compensate for a lack of visual stability [16]. Angelini and Santucci [10] propose multiple data and visual metrics to characterize stability. They propose a preliminary framework comparing visual stability and visual consistency, where consistency represents the secondary goal of preserving the coherence of the visual and data changes between updates.

Currently, progressive visualization evaluations are, for the most part, conducted at
the system level, evaluating progressive visualizations and interfaces in the context of
a particular processing system. **Research Agenda:** In order to evaluate visualizations
in controlled studies, a characterization of the outputs of progressive algorithms
and progressive database queries is needed so that the efficiency of progressive
visualizations and interactions can be compared more precisely and generalized to a
variety of contexts.

<figure><img src="image_102.png" /></figure>
### 9.2.4 Human Factors: Perception and Cognition

Ultimately, progressive analytics systems are meant to be used by people. As such, an evaluation of human factors as described in Chapter 7 will play an essential role. Humans bring perceptual and cognitive skills that can be leveraged throughout the analysis process, but they also have their limitations and biases that need to be taken into account (see Section 7.5). Compared to traditional systems, the incremental changes in a progressive visualization may give rise to additional errors related to perception and cognitive factors. At the same time, perceptual studies will help us to provide guidelines for setting the many open design parameters in a progressive visualization system.

Methodologically, these evaluations will need to rely on quantitative and qualitative human-subject studies, as commonly used in the area of human-computer interaction (HCI) [169] and cognitive psychology [159]. While the primary focus of studies in HCI is on usability, studies that more specifically target perceptual and cognitive models will likely also need to leverage research methods from these areas. Elliot et al.'s design space of vision science methods for visualization research [78], for instance, provides a good starting point for the use of vision science methods in visualization and visual analytics research.

**Changes over Time** A core characteristic of progressive systems is that (intermediate) results are gradually changing over time. In streaming data scenarios, we can simply show new data as it comes in. But in progressive scenarios, it is already clear that this endeavor poses many challenges. From a human factors perspective, for instance, we need to ensure that users are aware of these updates (or are deliberately not being made aware). Furthermore, intermediate results not only introduce changes in results, but also in their associated error bounds or uncertainty measures, and they do not always follow the expected trend.

*Change blindness* [286] refers to a well-known phenomenon of human perception: under certain circumstances, people will fail to spot a change in a stimulus despite it being obvious as soon as it is pointed out to them. But even aside from change blindness, changes might simply not be visible to a user [27]. For instance, the magnitude of the change in the data might not be commensurately reflected in the visualization [158], and as such, a change might be missed. Likewise, a slow progression of small changes may simply go unnoticed. Studies on Just Noticeable Differences and Weber's law, such as those of Harrison et al. [110], are relevant in this context. **Research Agenda:** Human subject studies are needed to investigate these

<figure><img src="image_103.png" /></figure>
effects in progressive visualization and to systematically consider different ways to counteract them.

These considerations also call for explicit investigations into the question of how frequently changes should be shown to the user. This frequency might be too slow or too fast [38]. Too slow will lead to an interrupted workflow [41] or missed changes. Too fast might overwhelm the user or, counter-intuitively, even slow down decisions [38]. While in the election example of Section 1.1, we would likely show new data as soon as it comes in (possibly too slow), in other scenarios we might have new results coming in on a millisecond basis. In this case, we would need to evaluate different strategies and frequencies in deciding how frequently to update users in progressive systems. These studies will not only need to take into account user performance in analytical tasks but also health and fatigue-related issues [141, 278]. Outcomes where users are either exposed to many hours of rapidly-flicking updates, or stare at a static screen showing no changes at all, are equally to be avoided. Even in conventional eager contexts, the effects of latency are not yet well understood [180], with some results showing that, while affecting user strategies, they are not always detrimental to user performance in higher complexity tasks [20]. In the progressive context, systems might offer updates automatically, or, following “anytime” strategies (Section 2.3), could allow users to request changes on demand, enabling even more complex user strategies. However, it would be ensured that the user is not missing important updates, which calls for studies on peripheral vision in the context of progressive systems.

**Mental Models and Sensemaking** A central aspect of traditional visualization and visual analytics systems is to support sensemaking [234], that is, the process of lending meaning to the data that users investigate (see also Section 7.2.2). Sensemaking will also be a key task for progressive analyses. However, since new results can come in progressively, the classical sensemaking process might undergo substantial changes in such setups [342]. Instead of operating on a static problem, users might engage in speculative exploration, more rapidly coming up with new hypotheses about the data, but also needing to correct or “up-believe” them [100, 101]. In the process of sensemaking, mental models (see also Section 7.4) play a key role, that is, the users’ internal explanation of how something works. Rapid changes could be problematic in the slower process of coming up with coherent mental models about the data. The faithful perception of uncertainty will also be key in this process. Further studies will be needed to understand these exciting new facets of the sensemaking process. **Research Agenda:** While studying visual sensemaking is already a

<figure><img src="image_104.png" /></figure>
non-trivial endeavor for traditional visual analytics systems [151], progressivity will present new challenges.

**Risks and Biases** As described in Section 7.5, properly understanding and potentially mitigating cognitive biases will play an important role in progressive analytics. There have been many studies on such biases in general [149], but also some specifically addressing visualization and visual analytics [68, 310]. There is a need for more studies that specifically consider the characteristics and specificities of progressive systems, following the experiment on cognitive biases of Procopio et al. [240]. **Research Agenda:** How exactly do these biases manifest in progressive setups, how severe are they, and how effective might different countermeasures be? Are the initial results anchoring or priming the user [309]? Is the user subject to illusion biases in the presence of false patterns? These questions necessitate a careful and rigorous empirical investigation. While initial hypotheses can be derived from existing theory as described in Section 7.5, empirical studies will need to confirm, refine, or refute them.

<figure><img src="image_105.png" /></figure>

**Decision-making** Evaluations of progressive systems also share similarities with decision-making environments. The decision-making literature uses consistency of decisions as a success measure [32], which is also relevant to progressive systems. Measures of user performance from progressive system experiments are most likely strongly dependent on decision strategies followed by participants and what they consider to be good enough responses. As a consequence, evaluations need to report on strategies deployed by users to mitigate any waiting time between updates, as mentioned previously, but should also clarify elements that affect how participants calibrate their goal in terms of the trade-off between accuracy and response time. This concerns any scenario or domain requirements that create specific time pressure or implicitly define acceptable thresholds, but also the reward mechanisms to financially incentivize participants. An example of a decision-making strategy is “satisficing”, where the user first defines acceptable thresholds in advance and then chooses, or stops at, the first alternative that satisfies the thresholds [67]. Decision strategies applying to progressive tasks differ from those described in non-progressive contexts, since progressive tasks can include strategic waiting time to increase accuracy. **Research Agenda:** Methodologies are needed to describe decision-making requirements and strategies, and assess the optimality of user behaviors. Part of a system’s evaluation would also be to determine how well it performs under different requirements (e.g., fast decision or minimum accuracy).

<figure><img src="image_106.png" /></figure>
### 9.2.5 Machine Learning

As discussed in Chapter 8, progressive machine learning can be used by Architects, Trainers, and Model Users. The evaluation criteria for the three groups are different. For the Model User using a pre-trained machine learning algorithm, evaluation is very similar to progressive algorithm evaluation. For the Architect and Trainer, few progressive machine learning algorithms are reported in the literature, although most of the iterative algorithms could be made progressive, and online machine learning is also a first step toward progressive machine learning. Machine learning algorithms are diverse; a typical machine learning pipeline is made up of several stages, e.g., data preprocessing, feature preprocessing, and an estimator [87]. Data processing stages belong to standard algorithms, whereas we consider feature processing and estimators as machine learning algorithms.

Progressive machine learning has been evaluated at the algorithm and system levels. With the advent of explainable AI, **Research Agenda:** they should also be evaluated regarding their capability of explaining or adding transparency to the learning process. Several machine learning algorithms have been evaluated at the algorithm level. Many of them are dimensionality reduction methods that are feature processing methods for machine learning, but can be used for visualization in an exploratory context. Several articles on progressive algorithms highlight their capability to return a progressive result at an acceptable pace [94, 233, 334].

PCA is one of the most popular dimensionality reduction methods. Fujiwara et al. [94] describe a streaming PCA algorithm that can be considered progressive-in-chunks and tries to “maintain the user’s mental map”. Stability is critical with the Incremental PCA algorithm [255] because the sign of the eigenvectors can flip at each iteration, resulting in a strong visual disruption if not controlled properly. Fujiwara et al.’s technique is useful for computing uncertainty and stabilizing the visualization. Their algorithm is evaluated using the run-time performance of the algorithm and two use cases. Streaming PCA is not progressive by iteration natively since it does not guarantee a quantum, i.e., a time limit for its execution, but by tuning the size of its chunks its latency can be controlled.

Pezzotti et al. Approximate *t*-SNE (At-SNE) [233] is evaluated based on precision over time, the number of points handled over time, and a specific quality measure of *t*-SNE over the number of iterations. This latter measure facilitates reproducibility and replicability of the article's evaluations, as running times are not reproducible. However, At-SNE is only progressive-in-iterations after the approximate *k*-Nearest Neighbors (kNN) of the dataset has been fully computed. In contrast, Jo et al. [142]

<figure><img src="image_107.png" /></figure>
describe a *progressive-in-chunks* kNN algorithm and implement a *progressive-in-chunks* and *progressive-in-iterations* t-SNE algorithm called Responsive *t*-SNE as a use case of their progressive kNN. As intended, this implementation is fully progressive and allows for monitoring the computation of a multidimensional projection when data is loaded progressively (*progressive-in-chunks*). The evaluation reports absolute loss over iteration instead of relative loss over time. Each of the two versions of progressive *t*-SNE implements specific features, such as computational steering for At-SNE and interactive exaggeration for Responsive *t*-SNE. Steering allows users to devote more resources to faithfully compute the projection around a point of interest. Interactive exaggeration applies a global optimization phase at the user's initiative, since it could disrupt the visual stability of the progressive projection if triggered automatically (requirements R2 and R13). These features are described in the articles but not evaluated in any way. Finally, Responsive *t*-SNE is designed to recover from non-stationary input. Its progressive kNN algorithm adapts to the non-stationarity and influences Responsive *t*-SNE to fix its computations when needed. This feature is not formally evaluated, but images of the progression are shown as a visual assessment of the process (see Figure 4.2). Showing images of the results is an accepted assessment in computer graphics and visualization, although newer articles tend to prefer computational evaluation approaches. However, judging the interpretability of a set of progressive results is currently ill-defined and can only be done with human judgment.

Several progressive estimators are mentioned in the literature. Jo et al. [142] show examples of kNN regression and density estimation and can easily be used to implement a kNN classifier, but none of them are evaluated. Fekete and Primet [83] mention a *k*-means clustering module in their ProgressiVis toolkit but also do not evaluate it.

### 9.2.6 System Level

Due to the lack of characterization in the requirements for the data processing, machine learning, and visualization parts of the pipeline, many PDA approaches are evaluated at a system level. Since conducting controlled studies for entire systems is difficult, a common evaluation approach is to conduct case [283] or design studies [277] with experts. These evaluations are typically conducted in an open-ended manner and report on observations made during interviews with domain experts, while or after they use the system. For example, for the sampleAction system [89], experts reconstructed a question they had recently addressed, or could have addressed,
and were asked to give their interpretation of intermediate results at several points of
the progression.

A specific form of this type of evaluation is Multi-dimensional In-depth Long-term Case studies (MILCs), proposed by Shneiderman and Plaisant [283]. For instance, Stolper et al. [293] conducted a two-month case study with a domain expert following the MILC protocol of Perer and Shneiderman [227]. This protocol has five phases: an initial interview, training, a first usage phase with possible modification of the software, a second usage phase, and an exit interview in which the expert can report on their level of achievement for each goal originally set. During the usage phases, observations are collected. This approach provides a longitudinal and in-depth understanding of how experts use the tool and provides guidance on the redesign of the tool. For DimXplorer [304], observations were collected for four analysis sessions over two months. During interviews with experts, authors seek to interpret progressive results and generate preliminary insights.

Design and evaluation aspects are commonly combined for these types of projects,
resulting in design study contributions [277]. Systems are iteratively evaluated along
the entire design process in close collaboration with domain experts. The qualitative
reflection on this process allows then to improve, extend, or challenge existing
design guidelines. Design studies are very common in the visualization literature,
and examples range from reports on systems designed for biologists [195], system
managers [194], and automotive engineers [276].

Other systems are evaluated based on controlled user studies of non-domain expert
users working on both open- and non-open-ended tasks, for instance, using Saraiya et
al.'s insight-based methodology [265]. For InsightsFeed [16], the experiments collect
quantitative measures, subjective ratings, and qualitative observations collected from
user recordings. Measures of success include the ability to make early observations,
to answer with confidence, serendipitous observations, and usability rate for different
UI parts. Zgraggen et al. use a hybrid experiment design with both log-based and
speech-based metrics to measure insight and interaction rates, insight originality, and
visualization coverage [342]. They also report surprising discoveries, such as the use
of the system in ways other than the intended purpose.

**Research Agenda:** While a plethora of such system-level evaluations exist in the area of visual analytics and visualization, very few of those explicitly incorporate aspects of progressivity. Rich and qualitative descriptions from such case and design studies [44] will help to uncover previously unexplored facets in the realm of PDA systems. They will help us to better characterize and formalize metrics of success, but also to learn more about the utility and usability characteristics that PDA solutions

<figure><img src="image_108.png" /></figure>
will need to offer if they are to be fully integrated into daily working practices.

## 9.3 Summary

PDA has introduced two new dimensions for evaluation: trading accuracy/complete-ness for time and balancing stability and quality. These criteria will need to be well-defined to be integrated into the existing evaluation frameworks of progressive systems. To improve the evaluation of progressive systems, we also need a better separation of concerns among the multiple parts of the progressive pipeline: data management, algorithm or machine learning (if relevant), and visualization and interaction. Right now, these parts are difficult to separate because of the integrated nature of progressive systems. This makes it difficult to design experiments targeting just one part rather than the whole system. To separate them, we would need a better understanding of the requirements and guarantees from one part to the next. In particular, algorithms should be able to let users know they can trade better quality for less stability, but interactive interventions in algorithms are not currently considered in algorithm evaluation, and visualization evaluation does not consider algorithm interventions in their evaluation. **Research Agenda:** More research will be needed to enable communication between algorithms and visualizations for control of the stability/quality trade-off, especially with a view to discovering meaningful evaluation criteria. For example, evaluating a progressive visualization would need a characterization of its possible requirements in terms of progressive results convergence and pace for the parts coming upstream in the pipeline. Progressive visualization techniques could have constraints on how progress feedback and uncertainty should be available to match its encoding. In turn, evaluating data management and machine learning techniques would require a characterization of steerability capabilities or interface requirements, describing how a downstream interface might have control over loading speed parameters or filtering queries progressively.

**Research Agenda:** Additionally, there is a need for more empirical studies comparing progressive systems and traditional systems in different tasks and domain applications with a view to identifying potential perceptual and cognitive bias and developing design guidelines to help progressive systems mitigate them. As progressive systems become more widely available for different applications, it will also become more important to evaluate the adoption of PDA systems. Finally, it will be critical to orchestrate the various forms of evaluation into a larger framework for PDA evaluation. For the areas of visualization and visual analytics, researchers have

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>
provided such frameworks, for instance, as nested models [196, 211] and as guiding
scenarios [137, 168]. These frameworks help researchers pick adequate evaluation
methods and methodologies throughout the entire visualization design and research
process. PDA will necessitate reflecting on, adapting, and extending these models.
# 10 Challenges and Research Agenda

This book has outlined the state of the art in Progressive Data Analysis research. Throughout, it has highlighted challenges—open questions that have yet to be resolved, and opportunities where new algorithms, techniques, or technologies could open the way to greater adoption of Progressive Data Analysis.

This chapter consolidates some of the cross-cutting themes and brings together broader questions. The goal is to provide a research agenda for Progressive Data Analysis that could lead to the broad adoption of these techniques. Ultimately, of course, it would be desirable for off-the-shelf PDA systems to be available as industrial-strength systems to bring these benefits to end users.

We begin with a discussion of the overarching challenges in the field—questions that cut across multiple topics and require new thinking about how to build infrastructures. We follow by describing a selection of high-risk / high-value challenges that, while difficult, have the potential to dramatically reshape the field. We then conclude with a selection of other research challenges from the previous chapters.

## 10.1 Overarching Challenges to Broader Adoption of PDA

An analytics system can be thought of as a pipeline: data starts from a data source and, through a series of computation algorithms, flows through the system, coming out on the other end as a user experience and visualization, and ultimately, as a final result and report. While this pipeline metaphor is common in data analysis, it is true in a more literal way in the case of Progressive Data Analysis: a system has to take into account the continuous (or repeated) flow of data through the pipeline stages. This means that optimally, the database, the algorithms, and the visualization should be designed together: each must be aware that the other components may be part of a progressive system and comply with its requirements (Table 2.1).

Conversely, in a steerable system, the user input must flow backward through the system to drive different computations.

This gives rise to a number of challenges:

* Building an infrastructure to manage this pipeline, while providing consistent mechanisms throughout
* Ensuring that databases, algorithms, and user experiences are all aware of the concepts of progressivity and uncertainty

* Making sure that algorithms and user experiences are aware of the concept of stability, and the trade-offs between stability and quality (Section 5.2.1)

* Designing system infrastructure that allows for reproducible computations, so that the same run will produce the same results in the same order when needed (Section 9.2.2)

* Evaluation of progressive systems, algorithms, and visualizations should take into account at least three different deadlines: the first meaningful progressive result, the earliest result accurate for decision-making, and the total computation time if needed (Section 9.1.2)

* Steering should also be considered for evaluation, even though there is currently no standard methodology for taking that into account

* Most scalable eager exploratory systems assume that data has been loaded locally and, in some cases, indexed in a database before it can be used for exploration (e.g., Falcon [206], Mosaic [114], Cartolabe [39], Nanocubes [177]). Progressive systems have the potential to cold-start an exploration. The assumptions and the preprocessing time (i.e., overall time from intent to first meaningful results as mentioned in Section 9.1.2), should be clarified to allow fair comparisons between exploratory systems and supporting algorithms [249].

## 10.2 Data Management

* Databases often prefer data stored in an efficient, ordered format. By contrast, progressive algorithms prefer shuffled data to ensure fair sampling. Both arbitrary access to the database and random selection of rows are expensive and are slow operations that can act as a bottleneck to efficient Progressive Data Analysis systems. It would be helpful to find new ways to ensure that data can be efficiently shuffled for real applications (Section 3.5.1).

* More research is needed to identify and categorize sampling strategies that can be used by progressive algorithms, as well as to characterize their efficiency, especially for large-scale distributed storage.

* If data is accessed through a database, it would be helpful to provide a mechanism to read it by chunk in random order (shuffled) to improve the quality of progressive algorithms down the pipeline. OLA-RAW [50] (see Section 3.5.1) shuffles the data incrementally using sophisticated mechanisms to avoid global
pre-processing. While most databases can sort the data (at high cost), very few are able to shuffle data. Although the problem has been explored in research [237], few solutions are available with existing systems, and those are not readily applicable to PDA.

* Modern data management systems rely on well-specified SQL standard(s) and novel JSON-based formats and APIs. Unfortunately, these standards are not applicable to PDA, as they assume eager, one-shot function calls. New conventions and standards should be designed to allow progressive data systems to interoperate in a standard way with the rest of the PDA pipeline.

* More experience is needed to understand the level of support required between the progressive database and the analytical operations down the PDA pipeline to avoid redundant work, while releasing the analytical front-end from performing operations, which are more cost-effective to do in the database.

* Query steering is important for exploration in PDA, but is far from standard in data management systems. More research is needed on adapting existing systems or designing new systems for query steering, as well as on standardizing the information to communicate through an API to process the results of this steering downstream.

* More progressive services are needed to use PDA over the network for exploring large databases. These services should support important operations, including progressive (min, max) for a specified query, approximate attribute distributions, approximate distinct-values counts, approximate quantiles, and possibly different levels of shuffling to trade speed for quality.

## 10.3 Data Structures and Algorithms

As explained in Chapter 4, orchestrating a general PDA application calls for consistent mechanisms and requirements at the system level and at each level of libraries for scientific computing. This consistency is currently difficult to achieve because a PDA pipeline encompasses components designed by different scientific and technical domains, whereas consistency is required between all the domains.

The challenges for data structures and algorithms are:

* More research is needed to find general strategies, when possible, to adapt iterative algorithms to be applicable to data arriving in chunks.

* Ideally, all progressive algorithms should be able to handle data growing by chunks, but this can come at a high cost. Therefore, it also makes sense for
progressive libraries to provide multiple variations of the same algorithm that are better suited to data arriving in chunks or all at once.

* More generally, solutions can be explored, with different trade-offs, to support the progressive implementation of important algorithms.

* The same is true for steering: most fundamental algorithms are not designed to deal with data changing dynamically.

* A specific compiler is needed to transform a general set of linear algebra operations written in conventional notation into an efficient progressive implementation, performing matrix and vector operations in an appropriate order (e.g., row-wise or column-wise) to allow continuously generating meaningful partial results.

* Generalizing progressive data analysis to networks is still an open problem. Many graph/network algorithms are based on visitors that are iterative but do not always deliver useful partial results. For example, computing the shortest path in a graph uses an iterative algorithm (e.g., Dijkstra's) and can be interrupted at any time, but the partial results are not always solutions, whereas progressive algorithms are expected to deliver intermediate solutions. Could such graph algorithms be adapted to provide useful or meaningful partial results?

* More research will be needed to adapt existing data sketching methods to handle general progressive operations efficiently. They need two adaptations: bounding their computation time (latency) and, for steerable progressivity, dealing with deletions or value updates in a smarter manner than complete recomputation.

* More work is needed to characterize existing algorithms regarding their ability to become progressive, to help in finding generic transformation methods or strategies to make more of them progressive.

## 10.4 Visualization

Progressive Data Analysis offers new opportunities for data visualization. Chapter 5 outlines many of the specialized needs for visualization in Progressive Data Analysis systems. Some other interesting research challenges include:

* We lack good representations for scalability in visualizations. While some visualizations are inherently based on aggregations, and thus are scalable “for free”, we are also accustomed to seeing visualizations on an individual data
level. We should articulate scalable versions of classic visualizations — like a scalable line chart with many data points or millions of lines or area charts with multiple dimensions. We should carefully articulate which dimensions can take extreme scalability (e.g., number of points represented), and which need to be constrained to reasonable numbers (e.g., number of series).

* We need to learn more about how to manage stable visualizations as progressive information arrives. Some algorithms are intrinsically stable for certain types of changes and datasets. For other visualization algorithms, we can find stabilized variants. For others, we might be able to trade off stability and other measures of quality. (For example, a stable treemap might get increasingly bad aspect ratios on its rectangles). In those algorithms, we may want to allow the user to interactively trigger “repairing” the visualization, explicitly breaking the stability to achieve better quality, as mentioned by Jo et al. [142]. This strategy has implications for the whole progressive system since algorithms should inform the user that repair actions are available. Changing the visualization layout certainly breaks the user’s mental map and would require mechanisms to help users recover it.

Designing interfaces for Progressive Visual Analytics will also drive a need to create ways to communicate with users—a set of guidelines, conventions, and controls that we refer to here as a “design language.” We will need to learn how to communicate uncertainty, progress, and partial results to users. We will need to design steerable interfaces that can help users understand what changes to expect.

We see opportunities for new design languages in each of the following:

* Visualizing uncertainty on graphs, especially as it changes with additional progress (but see also Section 10.5).

* Selecting the best visual encoding for a progressive visualization and which features it has to provide.

* Using the user's interactions to direct visualization and selecting controls to best help steer visualization and computation in a progressive system.

* Choosing controls that appropriately cue the user for proper responsiveness and computation expectations (see Section 7.1).

Overall, more research should be conducted on the advantages, disadvantages, and design patterns to choose where progression and quality indicators should be shown in the UI and visualizations. A systematic approach is needed to identify the design space of choices in controlling and interacting with PVA. (See also Section 5.5).
## 10.5 Quality and Uncertainty

Fundamental to the concept of allowing users to explore the results of progressive analytics is giving them the ability to interpret their partial and progressive results. In Chapter 6, we outlined contemporary issues in visualizing uncertainty and progressive uncertainty. Some of the major issues in uncertainty include:

* Modeling uncertainty, quality, and progression in a consistent way across the whole PDA pipeline to help combine and visualize them.

* When uncertainty is too complex to be combined across pipeline stages, a consistent way of using the Monte Carlo method is designed to collect ensemble uncertainty from full computation pipelines.

* Improve methods for computing the uncertainty levels algorithmically, including finding uncertainty guarantees and good approximations.

* Choosing good representations for uncertainty that allow users to interpret both the uncertainty and the quality of the estimates.

* Carry out perceptual and psychological research to gain a better understanding of how users make decisions under progressive uncertainty.

* Uncertainty comes in several parts: a computation might have measures of both local (per-element) and global uncertainty; both of these may be proxies for quality measures. We need to research design languages to help better communicate the different sorts of uncertainty.

* It is unclear how to compute or communicate uncertainty for high-dimensional data analysis and complex estimators. Only a small set of partial measures have been found—for example, in a k-means model, the amount of movement in the centroids can serve as a proxy for quality.

* Some quality measures take longer to compute than the progressive result. When designing systems, we could encounter a situation where the quality measure could either slow down the main visualization or appear with a delay. How should we help users interpret such a situation?

* Distribution-based probability methods, such as Monte Carlo estimation, may create different models of uncertainty. There are opportunities to learn new ways to visualize them.

* More work is needed to study uncertainty in the context of machine learning progressive pipelines, where the uncertainty at the end of the pipeline depends on choices made all through the pipeline but only shows up late in the computation after the model has learned for long enough.
## 10.6 Human Aspects

We turn to users themselves to examine the human implications of progressive visualization. The taxonomy of task and user types described in Chapter 7) presents new research challenges.

* We need to build a design language that helps clarify to users what the capabilities each of their roles holds.

* We need to understand how users move between these roles, and under what circumstances, during the analysis process.

* We need to learn how analytical tasks in Progressive Visual Analytics relate to their more traditional counterparts, to find analogues to guidelines like Shneiderman's Visual Information-Seeking Mantra [282], and to expand traditional task typologies and taxonomies to progressive workflows.

* Explainable approaches are needed to shed light on the progressive usage and training of black-box models.

* We need to find ways for PVA systems to mitigate various forms of bias Section 7.5.

* We need to learn the appropriate requirements for interactivity, accuracy, and timing. When does it make sense to produce a more refined output compared to producing responses to users? What sorts of users will tolerate a PDA system that takes longer than a baseline system to reach a final response, but can provide partial results much more quickly?

* The study of collaborative PDA solutions is another research challenge that largely remains open.

## 10.7 Machine Learning

We now turn to the challenges for progressive methods in Machine Learning (ML), as discussed in Chapter 8. While in an ideal world, ML model building and usage would be fully automatic, both processes are in reality tied to human experts/users interacting with them. Given that datasets tend to be large in ML, progressive methods offer a viable opportunity to make this interaction more efficient and effective by letting users steer the respective algorithms on the fly. However, more research is needed to overcome the inherent challenges of making ML a progressive process.

* Many fundamental ML algorithms (beyond kNN search) may be reformulated as progressive versions so that they can be used in a PDA pipeline. These
algorithms need to be able to work incrementally with partial data, be robust in terms of data stationarity, and ensure proper convergence rates. In addition, issues of quality and uncertainty are key in establishing a good interplay between users and the algorithms. Work in online learning could provide interesting starting points for such algorithms.

* Training very large deep neural networks is slow and error-prone. Progressive methods could help to support monitoring and steering for this process. However, integrating PDA into deep-learning frameworks, such as Torch or Keras, is an open problem.

* While methods for reinforcement learning through human feedback (RLHF) are progressive by design, investigating how to steer and verify their feedback process is an open research question.

* To support progressive model building, we need informative and consistent progress and performance indicators that address the inherent uncertainties of progressive techniques.

* We need more advanced methods of “active learning” that can elicit complex tacit knowledge from people to facilitate effective steering of models in progressive learning settings.

* With regard to designing effective progressive ML techniques, it is also imperative to gain a better understanding of human factors (see also Section 10.6), e.g., by conducting “in the wild” studies (see also Section 10.8).

## 10.8 Evaluation

As discussed in Chapter 9, the new PDA paradigm will also substantially impact how we evaluate these analytical systems. The core difference between PDA and classical eager systems is the inherent trade-off between time and quality/confidence. This central trade-off needs to be reflected in the evaluation of PDA approaches along the entire pipeline. Not only are new evaluation methods and methodologies needed that can foster this inherent trade-off, but more studies will also be needed to better understand PDA and how it compares to traditional approaches.

In terms of new and adapted methods, methodologies, and metrics, we are facing the following challenges:

* The overarching challenge of adopting existing evaluation methods and methodologies from the visualization and database communities, and extending them so that they account for the new requirements of PDA systems. A new method-
ology is needed that combines different methods (quantitative and qualitative, objective and subjective) along the entire PDA pipeline.

* There already exist a plethora of objective measures available for the quantitative evaluation of analytical systems. An important challenge will be to characterize when to use which measures, for what purpose, under what condition/scenario, and how to weigh the trade-off between them. Many of the old tasks and measures will also remain valid for PDA. However, at the same time, new representative tasks and measures will need to be devised that account for the time/quality trade-off that is inherent to PDA. Measures to address this trade-off need to account for aspects such as early-stopping [52], steering of the progressive process, and sometimes the stability of the intermediary results.

* Another important characteristic of evaluating PDA systems is the intrinsic interdependence of human and computational factors. For instance, how can we quantitatively evaluate and compare approaches while involving human interventions? New evaluation methods are needed that tightly integrate and bridge these two worlds, given the specific requirements of PDA.

* Assessing whether a progressive system meets the requirements listed in Table 2.1 will require that their authors, and ultimately the PDA community, clarify these concepts, in addition to stating that their system implements them fully, partially, or given specific definitions. Properly articulating assumptions is critical for a faithful evaluation of PDA approaches, and will be invaluable in devising a higher-level framework for PDA evaluation.

* Methodologies are needed to describe decision-making requirements, sense-making strategies, and assess the optimality of user behaviors. Part of a system's evaluation would also be to determine how well it performs under different requirements (e.g., fast decision, or minimum accuracy).

* New, better, standardized benchmarks are needed with both representative data distributions and query tasks.

So far, only a few empirical evaluations of approaches along the PDA pipeline exist. More work will be needed that helps to empirically characterize, assess, and compare the benefits and drawbacks of different approaches:

* More studies are needed that characterize the differences between traditional and PDA approaches, and provide guidance on when to use which.

* PDA interfaces will bring a new understanding of how data is visualized, understood, and leveraged by users in their analysis strategies. One important challenge, for instance, regards the update rate: How often should changes be shown in order not to either overwhelm the user with too frequent changes or
fail to maintain their attention with too infrequent changes? PDA interfaces pose an inherent trade-off between stability (only small visual changes) vs. accuracy (accurate representation at each time), and human-subject studies are needed to devise different strategies for dealing with this and other trade-offs in the user interface.

* Careful and rigorous perceptual and cognitive studies will be needed for various aspects of the PDA pipeline. These include aspects such as change blindness, latency perception, and cognitive biases. How exactly do such perceptual and cognitive aspects appear in progressive setups, do they cause new problems beyond traditional systems, and how well do countermeasures work?

* Progressive machine learning has been evaluated at the algorithm level and the system level. With the advent of explainable AI, they should also be evaluated regarding their capability of explaining or adding transparency to the learning process.

* Longitudinal case, design, and field studies "in the wild" will help to qualitatively understand the implications of PDA on real-world analytical practices and how end users will adopt them.

* PDA is a highly faceted research area. Studies will need to focus on specific aspects, and will usually not be able to cover the full evaluation breadth along the entire pipeline. A proper description of the focus and its limitations will help build up larger-scale models based on different individual studies.

* Last but not least, many studies will be needed to understand the implications of new and refined evaluation methods and methodologies as discussed above. For instance, the trade-offs between different measures of user performance are dependent on the particular context of user experiments for progressive visualization and will differ from each other as well as from current evaluation practices. Reflecting on PDA evaluation experiences can help others in similar endeavors.

## 10.9 Threats and Opportunities

To understand the state of the evolution of PDA, we can refer to the work of Brian Gaines [95] and his BRETAM model. Gaines argues that new knowledge or products in science and technology evolve in stages, which begins with a *breakthrough*, when the knowledge or artifact is presented to a community. This then requires a *replication* stage during which the community tries to replicate them with variations, followed
by an *empiricism* stage when success and failures improve the knowledge/product.
This leads to the development of *theories, automation* and, eventually, to *maturity*, at
which point the knowledge/product is assimilated and used without question. Without
a doubt, at the time this book is published, PDA is at the early *replication* stage,
although with a few specific issues it is sometimes in the *empiricism* stage. In the past,
some technologies have taken many years to get past the replication stage. Greenberg
mentions as food for thought and inspiration the groupware toolkits in [104]:

The big catch is that because groupware is hard to build, we have
essentially throttled its development at the replication stage. This has
minimized product invention and innovation as well as hands-on expe-
rience to all but the CSCW [Computer Supported Collaborative Work]
research community and a few well-resourced developers. Thus, the
necessary creativity leading to product evolution was stifled.

Nowadays, groupware tools are assimilated and used without question, but it
took fifteen years to go beyond the replication stage. The seminal article on Online
Aggregation [117] was published in 1997. More than twenty five years have passed
and it has still not been adopted. So, why has Online Aggregation still not been able
to break through? Is it because we still need to build a successful complete system,
or are there deeper challenges to adoption? What would it take to show that PDA is
actually a Terrible Idea? The authors of this book believe, on the contrary, that it is a
promising idea that should be pursued—until it grows beyond the replication stage
and eventually reaches maturity.

Another important point that is worth mentioning, although so far only based on
anecdotal evidence, is the ability of progressive visualizations to reveal the inner
workings of some algorithms and phenomena. For example, Li *et al*. report that “By
visualizing the dynamics of the t-SNE computation, we identified smooth phenotypic
transitions from cells within the [...] cluster to both the [...], revealing potential
differentiation trajectories.” [174]. Following the unfolding of a complex algorithm
progressively might convey the way it operates and add transparency, as well as some
level of explainability to data analysis. It may also reveal data characteristics only
visible through the animation of a progressive algorithm. More work is needed to
understand where, when, and why these properties of PDA are effective.
# 11 Conclusion

The starting point of this book is the glaring fact that current data exploration and visualization systems are severely limited in scalability because they require low latency, and this latency is currently tied to computation time. This time grows unbounded with data size and the complexity of data analysis algorithms; modern data sizes and algorithm complexity involve compute times that easily exceed human limits. Without a solution, the existing data exploration and visualization systems might become unusable for data analysts who routinely need to analyze terabytes of data using sophisticated analytical algorithms.

In this book, we explained how Progressive Data Analysis addresses this scalability problem by decoupling response time (latency) from computation time, and why it, therefore, could become an important paradigm to support data exploration at scale in the near future. However, while addressing the latency issue, this new paradigm introduces many new and complex challenges that need to be addressed before PDA can become usable and mainstream.

We have explained what PDA is and how it works at different levels of the data analysis and exploration pipeline, from data management to the human analysts who need to use it. Along the way, we listed several exciting challenges and opportunities to develop PDA and turn it into a mainstream paradigm. We also discussed the applicability of PDA for real applications.

We believe that, in Gaines's framework [95] (see Section 10.9), PDA remains stuck in the replication stage, and the purpose of this book is to unlock the paradigm and provide a clear roadmap to replication, empiricism, and the stages beyond. Passing the replication stage will require joining efforts between separate disciplines instead of working in silos, as academic research tends to encourage. We hope many students will be inspired by the multiple cross-disciplinary challenges raised in this book to contribute to the development of PDA.

Although the vision of this book is to achieve fully progressive systems, partially progressive systems are also possible and certainly suitable for important purposes. More work will be needed to understand how progressive and non-progressive systems can live together and support data exploration at scale in a wide variety of contexts and setups. Whether fully progressive or partially progressive, they can be
transformational for data exploration, as well as for helping machine learning and complex simulations with humans in the loop.

Over the last decade, data-driven approaches have fundamentally changed the world around us, and opened door to tremendous advances in science, technology, and society. Many of these advances are now subsumed under the umbrella of Artificial Intelligence (AI). A core pillar of their success can be largely attributed to the substantially increased amount and size of data underlying these models. Similarly important, however, is the fact that human experts, scientists, and engineers built and tested these models. To this end, it has been and will continue to be extremely important that humans are able to properly visualize, interact with, and understand the data at hand. Or as Michael Jordan, distinguished professor for statistics at Berkeley, puts it: *“We will need well-thought-out interactions of humans and computers to solve our most pressing problems”* [148]. To this end, we believe that methods for progressive data analysis will become ever more essential as a way of dealing with the continuous exponential growth of data. While challenges remain, PDA will eventually allow us to interactively work with, analyze, and understand data at scales that are not yet feasible today.
Bibliography

[1] S. Afzal, R. Maciejewski, and D. S. Ebert. Visual analytics decision support environ-
ment for epidemic modeling and response evaluation. In Proceedings of IEEE VAST,
pages 191–200. IEEE, 2011. doi:10.1109/VAST.2011.6102457. 113

[2] S. Agarwal, B. Mozafari, A. Panda, H. Milner, S. Madden, and I. Stoica. BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data. In *Proc. of the 8th ACM European Conference on Computer Systems*, EuroSys '13, pages 29–42. ACM, 2013. doi:10.1145/2465351.2465355. 34

[3] I. Alagiannis, R. Borovica, M. Branco, S. Idreos, and A. Ailamaki. NoDB: Effi-
cient Query Execution on Raw Data Files. In *International Conference on Man-
agement of Data*, SIGMOD '12, page 241–252, New York, NY, USA, 2012. ACM.
doi:10.1145/2213836.2213864. 34

[4] F. Alam, F. Ofli, and M. Imran. Processing Social Media Images by Combining Human and Machine Computing during Crises. *International Journal of Human-Computer Interaction*, 34(4):311–327, 2018. doi:10.1080/10447318.2018.1427831. 125

[5] F. Alam, F. Ofli, M. Imran, and M. Aupetit. A Twitter Tale of Three Hurricanes: Harvey, Irma, and Maria. In *International Conference on Information Systems for Crisis Response and Management (ISCRAM)*, 2018. 124

[6] S. Albers. Online algorithms: a survey. *Mathematical Programming*, 97(1-2):3–26, 2003. doi:10.1007/s10107-003-0436-0. 24

[7] G. Alicioglu and B. Sun. A survey of visual analytics for Explainable Artificial Intelligence methods. *Computers & Graphics*, 102:502–520, 2022. doi:10.1016/j.cag.2021.09.002. 119

[8] M. Angelini, T. May, G. Santucci, and H.-J. Schulz. On Quality Indicators for Progressive Visual Analytics. In *Proc. of the 10th International EuroVis Workshop on Visual Analytics (EuroVA'19)*, pages 25–29. Eurographics Association, 2019. doi:10.2312/eurova.20191120. 92, 93, 100, 112, 115

[9] M. Angelini and G. Santucci. Modeling Incremental Visualizations. In *EuroVis Workshop on Visual Analytics*. The Eurographics Association, 2013. doi:10.2312/PE.EuroVAST.EuroVA13.013-017. 22, 73, 80

[10] M. Angelini and G. Santucci. On Visual Stability and Visual Consistency for Progressive Visual Analytics. In *Proceedings of IVAPP*, pages 335–341. SciTePress, 2017. doi:10.5220/0006269703350341. 112, 115, 162

[11] M. Angelini, G. Santucci, H. Schumann, and H.-J. Schulz. A Review and Characterization of Progressive Visual Analytics. *Informatics*, 5(3), 2018.
doi:10.3390/informatics5030031. 10, 22, 80, 119, 121, 123

[12] Apache Arrow. https://arrow.apache.org/. Accessed: 2024-01-06. 62

[13] M. Aumüller, E. Bernhardsson, and A. Faithfull. ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. *Information Systems*, 87:101374, 2020. doi:10.1016/j.is.2019.02.006. 60

[14] M. Aupetit and M. Imran. Interactive Monitoring of Critical Situational Information on Social Media. In *International Conference on Information Systems for Crisis Response And Management (ISCRAM)*, pages 673–683, 2017. 124, 125, 127

[15] H. Avron, M. Kapralov, C. Musco, C. Musco, A. Velingker, and A. Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In *International Conference on Machine Learning*, pages 253–262. PMLR, 2017. 142

[16] S. K. Badam, N. Elmqvist, and J.-D. Fekete. Steering the Craft: UI Elements and Visualizations for Supporting Progressive Visual Analytics. *Computer Graphics Forum*, 36(3):491–502, 2017. doi:10.1111/cgf.13205. 19, 22, 78, 79, 80, 82, 85, 86, 90, 95, 97, 104, 105, 106, 112, 119, 121, 123, 159, 160, 162, 168, 210

[17] J. Baron. *Thinking and deciding*. Cambridge University Press, 2000. doi:10.1017/CBO9780511840265. 123

[18] J. Baron, J. Beattie, and J. C. Hershey. Heuristics and biases in diagnostic reasoning: II. Congruence, information, and certainty. *Org. Behavior and Human Decision Processes*, 42(1):88–110, 1988. doi:10.1016/0749-5978(88)90021-0. 123

[19] L. Battle, M. Angelini, C. Binnig, T. Catarci, P. Eichmann, J. Fekete, G. Santucci, M. Sedlmair, and W. Willett. Evaluating Visual Data Analysis Systems: A Discussion Report. In *Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA@SIGMOD 2018, Houston, TX, USA, June 10, 2018*, pages 4:1–4:6, 2018. 154

[20] L. Battle, R. J. Crouser, A. Nakeshimana, A. Montoly, R. Chang, and M. Stonebraker. The role of latency and task complexity in predicting visual search behavior. *IEEE Trans. Vis. Comput. Graph.*, 26(1):1246–1255, 2020. doi:10.1109/TVCG.2019.2934556. 164

[21] L. Battle, P. Eichmann, M. Angelini, T. Catarci, G. Santucci, Y. Zheng, C. Binnig, J.-D. Fekete, and D. Moritz. Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data. In *International Conference on Management of Data*, volume 17 of SIGMOD '20, Portland, OR, United States, June 2020. ACM. doi:10.1145/3318464.3389732. 34, 154

[22] Y. Bengio. *Learning deep architectures for AI*. Now Publishers Inc, 2009. 138, 143

[23] L. Berg, T. Ziegler, C. Binnig, and U. Röhm. ProgressiveDB: Progressive Data Analytics as a Middleware. *Proc. VLDB Endow.*, 12(12):1814–1817, Aug. 2019. doi:10.14778/3352063.3352073. 42, 45, 150

[24] J. Bernard, M. Zeppelzauer, M. Sedlmair, and W. Aigner. VIAL: a unified pro- cess for visual interactive labeling. *The Visual Computer*, 34(9):1189–1207, 2018.
doi:10.1007/s00371-018-1500-3, 146

[25] E. Bertini and G. Santucci. Give Chance a Chance: Modeling Density to Enhance Scatter Plot Quality through Random Data Sampling. *Information Visualization*, 5(2):95-110, 2006. doi:10.1057/palgrave.ivs.9500122.70, 72

[26] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A Fresh Approach to Numerical Computing. *SIAM Review*, 59(1):65-98, Jan. 2017. doi:10.1137/141000671.49

[27] A. Bezerianos, P. Dragicevic, and R. Balakrishnan. Mnemonic Rendering: An Image-Based Approach for Exposing Hidden Changes in Dynamic Displays. In *Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology*, UIST '06, page 159-168, New York, NY, USA, 2006. ACM. doi:10.1145/1166253.1166279.163

[28] P. Billingsley. *Convergence of probability measures*. Wiley Series in Probability and Statistics: Probability and Statistics. John Wiley & Sons Inc., New York, second edition, 1999. A Wiley-Interscience Publication. 21

[29] P. A. Boncz, M. Zukowski, and N. Nes. MonetDB/X100: Hyper-Pipelining Query Execution. In *Conference on Innovative Data Systems Research, CIDR*, volume 5, pages 225-237, 2005. 55

[30] A. Borodin and R. El-Yaniv. *Online Computation and Competitive Analysis*. Cambridge University Press, 1998. 156, 158

[31] L. Bottou. Online Algorithms and Stochastic Approximations. In *Online Learning and Neural Networks*. Cambridge University Press, 1998. URL http://leon.bottou.org/papers/bottou-98x. revised, oct 2012. 136

[32] E. H. Bowman. Consistency and optimality in managerial decision making. *Management Science*, 9(2):310-321, 1963. 165

[33] S. Boyd and L. Vandenberghe. *Convex Optimization*. Cambridge University Press, March 2004. URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike-20&path=ASIN/0521833787. 140

[34] U. Brandes. Force-Directed Graph Drawing. In *Encyclopedia of Algorithms*, pages 768-773. Springer, New York, NY, 2016. doi:10.1007/978-1-4939-2864-4_648. 57

[35] M. Brehmer and T. Munzner. A Multi-Level Typology of Abstract Visualization Tasks. *IEEE Trans. Vis. Comput. Graph.*, 19(12):2376-2385, 2013. doi:10.1109/TVCG.2013.124. 114, 128

[36] L. Breiman. Random forests. *Machine learning*, 45(1):5-32, 2001. doi:10.1023/A:1010933404324. 137

[37] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Nee-lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners, 2020,
2005.14165. 134

[38] J. S. Bruner and M. C. Potter. Interference in Visual Recognition. *Science*, 144(3617):424-425, 1964. doi:10.1126/science.144.3617.424. 164

[39] P. Caillou, J. Renault, J.-D. Fekete, A.-C. Letournel, and M. Sebag. Cartolabe: A Web-Based Scalable Visualization of Large Document Collections. *IEEE Comput. Graph. Appl. Mag*, 41(2):76-88, 2021. doi:10.1109/MCG.2020.3033401. 9, 172

[40] M. A. Cameron, R. Power, B. Robinson, and J. Yin. Emergency Situation Awareness from Twitter for Crisis Management. In *International Conference on World Wide Web*, WWW’12, pages 695–698. ACM, 2012. doi:10.1145/2187980.2188183. 124

[41] S. K. Card, J. D. Mackinlay, and B. Shneiderman, editors. *Readings in Information Visualization: Using Vision to Think*. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1999. 13, 164, 210

[42] S. K. Card, G. G. Robertson, and J. D. Mackinlay. The Information Visualizer, an Information Workspace. In *Proceedings of CHI*, CHI '91, page 181–186, New York, NY, USA, 1991. ACM. doi:10.1145/108844.108874. 108, 110

[43] H. Cardot and D. Degras. Online principal component analysis in high dimension: Which algorithm to choose? *International Statistical Review*, 86(1):29-50, 2018. doi:10.1111/insr.12220. 60

[44] S. Carpendale. Evaluating information visualizations. In *Information visualization*, pages 19–45. Springer, 2008. 168

[45] C. Carpineto, S. Osiundefinedski, G. Romano, and D. Weiss. A Survey of Web Clustering Engines. *ACM Comput. Surv.*, 41(3), July 2009. doi:10.1145/1541880.1541884. 117

[46] CarrotSearch Clustering and Visualization of search results. https://carrotsearch.com/. Accessed: 2020-03-08. 117

[47] S. Chambi, D. Lemire, O. Kaser, and R. Godin. Better Bitmap Performance with Roar- ing Bitmaps. *Softw. Pract. Exper.*, 46(5):709-719, May 2016. doi:10.1002/spe.2325. 67

[48] X. Chen, J. Zhang, C.-W. Fu, J.-D. Fekete, and Y. Wang. Pyramid-based Scatterplots Sampling for Progressive and Streaming Data Visualization. *IEEE Trans. Vis. Comput. Graph.*, 28(1):593-603, 2022. doi:10.1109/TVCG.2021.3114880. 72, 156, 157

[49] Y. Cheng and F. Rusu. Parallel In-Situ Data Processing with Speculative Loading. In *International Conference on Management of Data*, SIGMOD '14, page 1287-1298, New York, NY, USA, 2014. ACM. doi:10.1145/2588555.2593673. 34

[50] Y. Cheng, W. Zhao, and F. Rusu. Bi-Level Online Aggregation on Raw Data. In *International Conference on Scientific and Statistical Database Management*, SSDBM '17. ACM, 2017. doi:10.1145/3085504.3085514. 42, 62, 155, 172

[51] J. Choo, C. Lee, H. Kim, H. Lee, C. Reddy, B. Drake, and H. Park. PIVE: Per-Iteration visualization environment for supporting real-time interactions with computational methods. In *Visual Analytics Science and Technology (VAST)*, 2014 IEEE Conference
on, pages 241–242, Oct 2014. doi:10.1109/VAST.2014.7042510. 162

[52] Y. S. Chow. *Great expectations: The theory of optimal stopping*. Houghton Mifflin, Jan. 1971. 1, 94, 101, 152, 158, 179

[53] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. *Advances in Neural Information Processing Systems*, 30:4299–4307, 2017. 144

[54] J. D. Cohen. Drawing Graphs to Convey Proximity: An Incremental Arrangement Method. *ACM Trans. Comput.-Hum. Interact.*, 4(3):197–229, sep 1997. doi:10.1145/264645.264657. 73

[55] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. *Journal of artificial intelligence research*, 4:129–145, 1996. doi:10.1613/jair.295. 146

[56] A. Collette. *Python and HDF5*. O'Reilly, 2013. 55

[57] G. Cormode. Data Sketching. Commun. ACM, 60(9):48–55, Aug. 2017. doi:10.1145/3080008. 52, 97

[58] G. Cormode, M. Garofalakis, P. J. Haas, and C. Jermaine. Synopses for Massive Data: Samples, Histograms, Wavelets, Sketches. *Found. Trends Databases*, 4(1–3):1–294, Jan. 2012. doi:10.1561/1900000004. 44, 52, 61

[59] M. Correll and M. Gleicher. Error bars considered harmful: Exploring alternate encodings for mean and error. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2142–2151, 2014. doi:10.1109/TVCG.2014.2346298. 88

[60] M. Correll, D. Moritz, and J. Heer. Value-Suppressing Uncertainty Palettes. In *Proceedings of CHI*, CHI ’18, pages 1–11. ACM, 2018. doi:10.1145/3173574.3174216. 77, 82, 88

[61] C. Cortes and V. Vapnik. Support-vector networks. *Machine Learning*, 20(3):273–297, Sept. 1995. doi:10.1007/bf00994018. 138, 140, 141

[62] N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy. Optimal transport for domain adaptation. *IEEE Trans. Pattern Anal. Mach. Intell.*, 39(9):1853–1865, 2017. doi:10.1109/TPAMI.2016.2615921. 145

[63] A. Crisan and B. Fiore-Gartland. Fits and Starts: Enterprise Use of AutoML and The Role of humans in the loop. In *Proceedings of CHI*, CHI ’21, pages 601:1–601:15, 2021. doi:10.1145/3411764.3445775. 137

[64] Crisis Computing, Dec 2019. URL https://crisiscomputing.qcri.org/. 124

[65] R. J. Crutcher and A. F. Healy. Cognitive operations and the generation effect. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 15(4):669–675, 1989. doi:10.1037/0278-7393.15.4.669. 131

[66] Z. Cui, J. Kancherla, H. C. Bravo, and N. Elmqvist. Sherpa: Leveraging User Attention for Computational Steering in Visual Analytics. In *Proceedings of the Visualization in Data Science (VDS) Symposium*, pages 48–57. IEEE, 2019. doi:10.1109/VDS48975.2019.8973384. 113, 116, 118

[67] E. Dimara, G. Bailly, A. Bezerianos, and S. Franconeri. Mitigating the Attraction
Effect with Visualizations. *IEEE Trans. Vis. Comput. Graph.*, 25(1):850–860, Jan. 2019. doi:10.1109/TVCG.2018.2865233. 165

[68] E. Dimara, S. Franconeri, C. Plaisant, A. Bezerianos, and P. Dragicevic. A Task-Based Taxonomy of Cognitive Biases for Information Visualization. *IEEE Trans. Vis. Comput. Graph.*, 26(2):1413–1432, 2020. doi:10.1109/TVCG.2018.2872577. 165

[69] E. Dimara and C. Perin. What is Interaction for Data Visualization? *IEEE Trans. Vis. Comput. Graph.*, 26(1):119–129, Jan 2020. doi:10.1109/TVCG.2019.2934283. 108

[70] C. Domeniconi and D. Gunopulos. Incremental support vector machine construction. In *International Conference on Data Mining (ICDM)*, pages 589–592, 2001. doi:10.1109/ICDM.2001.989572. 140, 141

[71] J. Dongarra, M. Gates, A. Haidar, J. Kurzak, P. Luszczek, P. Wu, I. Yamazaki, A. Yarkhan, M. Abalenkovs, N. Bagherpour, and et al. PLASMA: Parallel Linear Algebra Software for Multicore Using OpenMP. *ACM Trans. Math. Softw.*, 45(2), May 2019. doi:10.1145/3264491. 55, 59

[72] E. P. dos Santos Amorim, E. V. Brazil, J. Daniels, P. Joia, L. G. Nonato, and M. C. Sousa. iLAMP: Exploring high-dimensional spacing through backward multidimensional projection. In *2012 IEEE Conference on Visual Analytics Science and Technology (VAST)*, pages 53–62, Oct 2012. doi:10.1109/VAST.2012.6400489. 117

[73] D. H. Douglas and T. K. Peucker. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. *Cartographica*, 10(2):112–122, 1973. doi:10.3138/FM57-6770-U75U-7727. 103

[74] P. Eades. A heuristic for graph drawing. *Congressus Numerantium*, 42:149–160, 1984. URL https://ci.nii.ac.jp/naid/10000023432/en/. 10, 57, 73

[75] K. Echihabi, T. Tsandilas, A. Gogolou, A. Bezerianos, and T. Palpanas. ProS: Data Series Progressive k-NN Similarity Search and Classification with Probabilistic Quality Guarantees. *The VLDB Journal*, 32(4):763–789, nov 2022. doi:10.1007/s00778-022-00771-z. 60

[76] B. Efron and R. J. Tibshirani. *An Introduction to the Bootstrap*. Chapman & Hall, 1993. 102, 153

[77] P. Eichmann, E. Zgraggen, Z. Zhao, C. Binnig, and T. Kraska. Towards a Benchmark for Interactive Data Exploration. *IEEE Data Eng. Bull.*, 39(4):50–61, 2016. 155

[78] M. A. Elliott, C. Nothelfer, C. Xiong, and D. A. Szafir. A Design Space of Vision Science Methods for Visualization Research. *IEEE Trans. Vis. Comput. Graph.*, 27(2):1117–1127, 2021. doi:10.1109/TVCG.2020.3029413. 163

[79] J. S. B. Evans, J. L. Barston, and P. Pollard. On the conflict between logic and belief in syllogistic reasoning. *Memory & Cognition*, 11(3):295–306, 1983. doi:10.3758/BF03196976. 120

[80] J. Farman. *Delayed Response: The Art of Waiting from the Ancient to the Instant World*. Yale University Press, 2019. 149

[81] J.-P. Farrugia and B. Péroche. A progressive rendering algorithm using an adaptive
perceptually based image metric. *Computer Graphics Forum*, 23(3):605–614, 2004.
doi:10.1111/j.1467-8659.2004.00792.x. 159

[82] J.-D. Fekete, D. Fisher, A. Nandi, and M. Sedlmair. Progressive Data Analysis and Visualization (Dagstuhl Seminar 18411). *Dagstuhl Reports*, 8(10):1–40, 2019.
doi:10.4230/DagRep.8.10.1. 11

[83] J.-D. Fekete and R. Primet. *Progressive Analytics: A Computation Paradigm for Exploratory Data Analysis*. *arXiv preprint arXiv:1607.05162*, abs/1607.05162, 2016.
doi:10.48550/arXiv.1607.05162. 64, 65, 82, 105, 106, 167

[84] M. Fernandes, L. Walls, S. Munson, J. Hullman, and M. Kay. Uncertainty displays using quantile dotplots or cdfs improve transit decision-making. In *Proceedings of CHI*, CHI '18, pages 1–12. ACM, 2018. doi:10.1145/3173574.3173718. 106

[85] N. Ferreira, D. Fisher, and A. C. König. Sample-oriented task-driven visualizations: allowing users to make better, more confident decisions. In *Proceedings of CHI*, CHI '14, pages 571–580. ACM, 2014. doi:10.1145/2556288.2557131. 107, 114, 161

[86] F. Ferstl, K. Bürger, and R. Westermann. Streamline Variability Plots for Character-
izing the Uncertainty in Vector Field Ensembles. *IEEE Trans. Vis. Comput. Graph.*,
22(1):767–776, 2016. doi:10.1109/TVCG.2015.2467204. 102

[87] M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-Sklearn 2.0: The Next Generation. *arXiv preprint arXiv:2007.04074*, abs/2007.04074, 2020. doi:10.48550/arXiv.2007.04074. 166

[88] D. Fisher, S. M. Drucker, and A. C. Koenig. Exploratory Visualization Involving Incremental, Approximate Database Queries and Uncertainty. *IEEE Comput. Graph. Appl.*, 32(4):55–62, July 2012. doi:10.1109/MCG.2012.48. 52

[89] D. Fisher, I. O. Popov, S. M. Drucker, and m c schraefel. Trust Me, I'm Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster. In *Proceedings of CHI*, CHI '12, pages 1673–1682. ACM, 2012. doi:10.1145/2207676.2208294. 10, 73, 74, 82, 97, 116, 121, 167

[90] E. Fix. *Discriminatory analysis: nonparametric discrimination, consistency properties*. USAF School of Aviation Medicine, 1951. 136, 138

[91] S. Frederick, G. Loewenstein, and T. O'Donoghue. Time Discounting and Time Preference: A Critical Review. *Journal of Economic Literature*, 40(2):351–401, 2002.
doi:10.1257/002205102320161311. 122

[92] S. Frey, F. Sadlo, K.-L. Ma, and T. Ertl. Interactive progressive visualization with space-time error control. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2397–2406, 2014.
doi:10.1109/TVCG.2014.2346319. 10

[93] J. H. Friedman, J. L. Bentley, and R. A. Finkel. An algorithm for finding best matches in logarithmic expected time. *ACM Transactions on Mathematical Software (TOMS)*, 3(3):209–226, 1977. doi:10.1145/355744.355745. 139

[94] T. Fujiwara, J. Chou, Shilpika, P. Xu, L. Ren, and K. Ma. An Incremental Dimensional-
ity Reduction Method for Visualizing Streaming Multidimensional Data. *IEEE Trans.*
Vis. Comput. Graph., 26(1):418-428, 1 2020. doi:10.1109/TVCG.2019.2934433. 61, 166

[95] B. R. Gaines. Modeling and forecasting the information sciences. *Inf. Sci.*, 57-58:3-22, 1991. doi:10.1016/0020-0255(91)90066-4. 180, 182

[96] E. Gamma, R. Helm, R. Johnson, and J. Vlissides. *Design Patterns: Elements of Reusable Object-Oriented Software*. Addison-Wesley Longman Publishing Co., Inc., USA, 1995. 66

[97] M. Garofalakis, J. Gehrke, and R. Rastogi. *Data Stream Management: Processing High-Speed Data Streams (Data-Centric Systems and Applications)*. Springer-Verlag, Berlin, Heidelberg, 2007. doi:10.1007/978-3-540-28608-0. 52, 61

[98] K. Ghiasi-Shirazi, R. Safabakhsh, and M. Shamsi. Learning Translation Invariant Kernels for Classification. *Journal of Machine Learning Research*, 11(4), 2010. doi:10.5555/1756006.1859896. 143

[99] A. Gijsberts, R. Bohra, D. Sierra González, A. Werner, M. Nowak, B. Caputo, M. A. Roa, and C. Castellini. Stable myoelectric control of a hand prosthesis using non-linear incremental learning. *Frontiers in Neurorobotics*, 8:8, 2014. doi:10.3389/fnbot.2014.00008. 142

[100] D. T. Gilbert. How mental systems believe. *American Psychologist*, 46(2):107–119, 1991. doi:10.1037/0003-066x.46.2.107. 164

[101] D. T. Gilbert, D. S. Krull, and P. S. Malone. Unbelieving the unbelievable: Some problems in the rejection of false information. *Journal of Personality and Social Psychology*, 59(4):601–613, 1990. doi:10.1037/0022-3514.59.4.601. 164

[102] A. Gogolou, T. Tsandilas, K. Echihabi, A. Bezerianos, and T. Palpanas. Data Series Progressive Similarity Search with Probabilistic Quality Guarantees. In *International Conference on Management of Data*, SIGMOD '20, pages 1857–1873. ACM, 2020. doi:10.1145/3318464.3389751. 150, 158

[103] J. Grass and S. Zilberstein. Anytime algorithm development tools. *ACM SIGART Bulletin*, 7(2):20–27, 1996. doi:10.1145/242587.242592. 27

[104] S. Greenberg. Toolkits and interface creativity. *Multim. Tools Appl.*, 32(2):139–159, 2007. doi:10.1007/S11042-006-0062-Y. 181

[105] H. Griethe and H. Schumann. The Visualization of Uncertain Data: Methods and Problems. In *Proceedings of SimVis '06*, pages 143–156. SCS Publishing House, 2006. 94, 96

[106] P. J. Haas and J. M. Hellerstein. Ripple Joins for Online Aggregation. *SIGMOD Rec.*, 28(2):287–298, jun 1999. doi:10.1145/304181.304208. 39

[107] P. J. Haas and C. König. A Bi-Level Bernoulli Scheme for Database Sampling. In *International Conference on Management of Data*, SIGMOD '04, page 275–286, New York, NY, USA, 2004. ACM. doi:10.1145/1007568.1007601. 43

[108] N. Halko, P. G. Martinsson, and J. A. Tropp. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. *SIAM
Review, 53(2):217-288, 2011. doi:10.1137/090771806. 60

[109] W. Han and H.-J. Schulz. Beyond Trust Building — Calibrating Trust in Visual Analytics. 2020 IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX), pages 9–15, 2020. 161

[110] L. Harrison, F. Yang, S. Franconeri, and R. Chang. Ranking visualizations of correlation using Weber's law. *IEEE Trans. Vis. Comput. Graph.*, 20(12):1943–1952, 2014. doi:10.1109/TVCG.2014.2346979. 163

[111] L. Hasher, D. Goldstein, and T. Toppino. Frequency and the conference of referential validity. *Journal of Verbal Learning and Verbal Behavior*, 16(1):107–112, 1977. doi:10.1016/S0022-5371(77)80012-1. 122

[112] X. He, K. Zhao, and X. Chu. AutoML: A Survey of the State-of-the-Art. *Knowledge-Based Systems*, 212:106622, 2019. doi:10.1016/j.knosys.2020.106622. 5, 137

[113] S. M. Hedetniemi, S. T. Hedetniemi, and A. L. Liestman. A survey of gossiping and broadcasting in communication networks. *Networks*, 18(4):319–349, 1988. doi:10.1002/net.3230180406. 58

[114] J. Heer and D. Moritz. Mosaic: An Architecture for Scalable & Interoperable Data Views. *IEEE Trans. Vis. Comput. Graph.*, 30(1):436–446, 2024. doi:10.1109/TVCG.2023.3327189. 172

[115] J. Heinrich and D. Weiskopf. State of the Art of Parallel Coordinates. In *Eurographics 2013 - State of the Art Reports*. The Eurographics Association, 2013. doi:10.2312/conf/EG2013/stars/095-116. 71

[116] J. M. Hellerstein, R. Avnur, A. Chou, C. Hidber, C. Olston, V. Raman, T. Roth, and P. J. Haas. Interactive Data Analysis: The Control Project. *Computer*, 32(8):51–59, Aug. 1999. doi:10.1109/2.781635. 11, 24, 121

[117] J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online Aggregation. In *International Conference on Management of Data*, SIGMOD '97, pages 171–182, New York, NY, USA, 1997. ACM. doi:10.1145/253260.253291. 10, 11, 24, 34, 52, 56, 73, 181

[118] R. Hema, M. Omid, and J. Rosie. Active Learning with Feedback on Both Features and Instances. *Journal of Machine Learning Research*, 7:1655–1686, 2006. 146

[119] E. G. Hetzler, V. L. Crow, D. A. Payne, and A. E. Turner. Turning the Bucket of Text into a Pipe. In *Proceedings of the IEEE Symposium on Information Visualization*, pages 89–94. IEEE, 2005. doi:0.1109/INFOVIS.2005.34. 19, 22

[120] R. J. Heuer Jr. *Analysis of competing hypotheses*, pages 95–110. US Government Printing Office, 1999. 122

[121] R. Hipp. SQLite. https://www.sqlite.org/, 2000. Accessed: 2019-07-20. 56

[122] T. K. Ho. Random decision forests. In *International Conference on Document Analysis and Recognition*, volume 1, pages 278–282. IEEE, 1995. doi:10.1109/ICDAR.1995.598994. 137

[123] M. Hogräfer, M. Angelini, G. Santucci, and H.-J. Schulz. Steering-by-example for Progressive Visual Analytics. *ACM Trans. Intell. Syst. Technol.*, 13(6):96:1–96:26,
2022. doi:10.1145/3531229. 78, 116

[124] M. Hogräfer, J. Burkhardt, and H.-J. Schulz. A Pipeline for Tailored Sampling for Progressive Visual Analytics. In *Proceedings of the 13th International EuroVis Workshop on Visual Analytics (EuroVA'22)*, pages 49–53. Eurographics Association, 2022. doi:10.2312/eurova.20221079. 31

[125] M. Hogräfer, D. Moritz, A. Perer, and H.-J. Schulz. Combining Degree of Interest Functions and Progressive Visualization. In *Short Paper Proceedings of the IEEE Visualization Conference (VIS'23)*, pages 251–255. IEEE, 2023. doi:10.1109/VIS45172.2023.00059. 118

[126] M. Hogräfer and H.-J. Schulz. Tailorable Sampling for Progressive Visual Analytics. *IEEE Trans. Vis. Comput. Graph.*, pages 1–13, 2023. doi:10.1109/TVCG.2023.3278084. 31, 97

[127] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau. Visual analytics in deep learning: An interrogative survey for the next frontiers. *IEEE Trans. Vis. Comput. Graph.*, 25(8):2674–2693, 2018. doi:10.1109/TVCG.2018.2843369. 146, 147

[128] T. Höllt, N. Pezzotti, V. van Unen, F. Koning, E. Eisemann, B. Lelieveldt, and A. Vitanova. Cytosplore: Interactive Immune Cell Phenotyping for Large Single-Cell Datasets. *Computer Graphics Forum*, 35(3):171–180, 2016. doi:10.1111/cgf.12893. 11

[129] A. Hughes and L. Palen. Twitter adoption and use in mass convergence and emergency events. *International Journal of Emergency Management*, 6(3):248–260, 2009. doi:10.1504/IJEM.2009.031564. 124

[130] J. Hullman, M. Kay, Y.-S. Kim, and S. Shrestha. Imagining replications: Graphical prediction & discrete visualizations improve recall & estimation of effect uncertainty. *IEEE Trans. Vis. Comput. Graph.*, 24(1):446–456, 2017. doi:10.1109/TVCG.2017.2743898. 106

[131] J. R. Hullman, X. Qiao, M. Correll, A. Kale, and M. Kay. In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation. *IEEE Trans. Vis. Comput. Graph.*, 25:903–913, 2019. doi:10.1109/TVCG.2018.2864889. 96, 105, 161, 162

[132] S. Huron, R. Vuillemot, and J.-D. Fekete. Visual Sedimentation. *IEEE Trans. Vis. Comput. Graph.*, 19(12):2446–2455, 2013. doi:10.1109/TVCG.2013.227. 118, 121

[133] S. Idreos, O. Papaemmanouil, and S. Chaudhuri. Overview of Data Exploration Techniques. In *International Conference on Management of Data*, SIGMOD '15, pages 277–281, New York, NY, USA, 2015. ACM. doi:10.1145/2723372.2731084. 9

[134] M. Imran, C. Castillo, F. Diaz, and S. Vieweg. Processing Social Media Messages in Mass Emergency: A Survey. *ACM Computing Surveys*, 47(4):67:1–67:38, 2015. doi:10.1145/2771588. 124

[135] M. Imran, C. Castillo, J. Lucas, P. Meier, and S. Vieweg. AIDR: artificial intelligence for disaster response. In *International Conference on World Wide Web*, WWW'14, pages 159–162, 2014. doi:10.1145/2567948.2577034. 124, 125
[136] M. Imran, P. Mitra, and C. Castillo. Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages. In *International Conference on Language Resources and Evaluation (LREC 2016)*, pages 1638–1643, 2016. 125

[137] T. Isenberg, P. Isenberg, J. Chen, M. Sedlmair, and T. Möller. A Systematic Review on the Practice of Evaluating Visualization. *IEEE Trans. Vis. Comput. Graph.*, 19(12):2818–2827, 2013. doi:10.1109/TVCG.2013.126. 170

[138] ISO 5721-1. Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions, 1994. URL https://www.iso.org/obp/ui/#iso:std:11833:en. 20

[139] A. C. Janes, D. A. Pizzagalli, S. Richardt, B. de B Frederick, A. J. Holmes, J. Sousa, M. Fava, A. E. Evins, and M. J. Kaufman. Neural substrates of attentional bias for smoking-related cues: An FMRI study. *Neuropsychopharmacology*, 35(12):2339–2345, 2010. doi:10.1038/npp.2010.103. 122

[140] A. Jena, U. Engelke, T. Dwyer, V. Raiamanickam, and C. Paris. Uncertainty visualisation: an interactive visual survey. In *2020 IEEE Pacific Visualization Symposium (PacificVis)*, IEEE Pacific Visualization Symposium, pages 201–205. IEEE, 2020. doi:10.1109/PacificVis48177.2020.1014. 94, 96, 105

[141] J. Jo, S. L'Yi, B. Lee, and J. Seo. ProReveal: Progressive Visual Analytics with Safeguards. *IEEE Trans. Vis. Comput. Graph.*, 27(7):3109–3122, 2021. doi:10.1109/TVCG.2019.2962404. 87, 88, 97, 105, 107, 113, 122, 160, 164

[142] J. Jo, J. Seo, and J.-D. Fekete. PANENE: A Progressive Algorithm for Indexing and Querying Approximate k-Nearest Neighbors. *IEEE Trans. Vis. Comput. Graph.*, 26(2):1347–1360, 2020. doi:10.1109/TVCG.2018.2869149. 31, 60, 61, 62, 63, 72, 77, 79, 98, 104, 140, 156, 157, 166, 167, 175

[143] J. Jo, F. Vernier, P. Dragicevic, and J.-D. Fekete. A Declarative Rendering Model for Multiclass Density Maps. *IEEE Trans. Vis. Comput. Graph.*, 25(1):470–480, 2019. doi:10.1109/TVCG.2018.2865141. 72

[144] S. Jo and I. Trummer. BitGourmet: Deterministic Approximation via Optimized Bit Selections. In *CIDR 2020, Online Proceedings*, pages 1–7, 2020. 38

[145] T. Joachims. *Learning to Classify Text Using Support Vector Machines – Methods, Theory, and Algorithms*. Kluwer/Springer, 2002. doi:10.1007/978-1-4615-0907-3. 141

[146] J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. *arXiv preprint arXiv:1702.08734*, 2017. doi:10.48550/arXiv.1702.08734. 140

[147] E. Jones, T. Oliphant, P. Peterson, et al. SciPy: Open source scientific tools for Python, 2001–. URL http://www.scipy.org/. [Online; accessed 2016-03-11]. 50

[148] M. I. Jordan. Artificial intelligence—the revolution hasn’t happened yet. *Harvard Data Science Review*, 1(1):1–9, 2019. doi:10.1162/99608f92.f06c6e61. 183

[149] D. Kahneman. *Thinking, fast and slow*. Macmillan, 2011. 165

[150] S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer. Wrangler: Interactive Visual Specification of Data Transformation Scripts. In *Proceedings of CHI*, CHI ’11, pages
3363-3372, New York, NY, USA, 2011. ACM. doi:10.1145/1978942.1979444. 5

[151] Y.-A. Kang and J. Stasko. Examining the use of a visual analytics system for sense-making tasks: Case studies with domain experts. *IEEE Trans. Vis. Comput. Graph.*, 18(12):2869-2878, 2012. doi:10.1109/TVCG.2012.224. 165

[152] Z. S. Karnin, K. J. Lang, and E. Liberty. Almost Optimal Streaming Quan-tiles Algorithms. *arXiv preprint arXiv:1603.05346*, abs/1603.05346, 2016. doi:10.48550/arXiv.1603.05346. 52, 53

[153] R. M. Karp. On-Line Algorithms Versus Off-Line Algorithms: How Much is it Worth to Know the Future? In *IFIP congress (1)*, volume 12, pages 416-429, 1992. 24

[154] D. Keim, G. Andrienko, J.-D. Fekete, C. Görg, J. Kohlhammer, and G. Mélancon. Visual Analytics: Definition, Process and Challenges. In *Information Visualization - Human-Centered Issues and Perspectives*, number 4950 in LNCS, pages 154-175. Springer, Aug. 2008. URL http://hal-lirmm.ccsd.cnrs.fr/lirmm-00272779. State-of-the-Art Survey. 13, 14, 73, 210

[155] A. Kerren and J. T. Stasko. Algorithm Animation. In *Software Visualization*, pages 1-15. Springer, 2002. doi:10.1007/3-540-45875-1_1. 18, 119, 121

[156] M. Khan, L. Xu, A. Nandi, and J. M. Hellerstein. Data tweening: incremen-tal visualization of data transforms. *Proc. VLDB Endow.*, 10(6):661-672, 2017. doi:10.14778/3055330.3055333. 121

[157] H. Kim, J. Choo, C. Lee, H. Lee, C. Reddy, and H. Park. PIVE: per-iteration visualiza-tion environment for real-time interactions with dimension reduction and clustering. In *Proceedings of the AAAI Conference on Artificial Intelligence*, AAAI'17, page 1001-1009. AAAI Press, 2017. doi:10.1609/AAAI.V31I1.10628. 10

[158] G. Kindlmann and C. Scheidegger. An algebraic process for visualiza-tion design. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2181-2190, 2014. doi:10.1109/TVCG.2014.2346325. 163

[159] F. A. A. Kingdom and N. Prins. *Psychophysics: A Practical Introduction*. Elsevier Science, 2016. 163

[160] H.-K. Ko, J. Jo, and J. Seo. Progressive Uniform Manifold Approximation and Projection. In *EuroVis 2020 - Short Papers*. The Eurographics Association, 2020. doi:10.2312/evs.20201061. 72, 98

[161] M. Korporaal, I. T. Ruginski, and S. I. Fabrikant. Effects of Uncertainty Visualization on Map-Based Decision Making Under Time Pressure. *Frontiers in Computer Science*, 2, 2020. doi:10.3389/fcomp.2020.00032. 150, 160

[162] N. Kourtellis, G. De Francisci Morales, and F. Bonchi. Scalable online betweenness centrality in evolving graphs. In *International Conference on Data Engineering (ICDE)*, pages 1580-1581, May 2016. doi:10.1109/ICDE.2016.7498421. 57

[163] C. Kozyrkov. Your dataset is a giant inkblot test, 2019. URL https://towardsdatascience.com/your-dataset-is-a-giant-inkblot-test-b9bf4c53ee5. retrieved 04-
MAR-2020. 122

[164] G. E. Krasner and S. T. Pope. A Cookbook for Using the Model-View Controller User Interface Paradigm in Smalltalk-80. *J. Object Oriented Program.*, 1(3):26–49, Aug. 1988. 66

[165] J. F. Kruiger, A. Hassoumi, H.-J. Schulz, A. C. Telea, and C. Hurter. Multidimensional Data Exploration by Explicitly Controlled Animation. *Informatics*, 4(3):26:1–26:21, 2017. doi:10.3390/informatics4030026. 117

[166] S. Kullback and R. A. Leibler. On Information and Sufficiency. *The Annals of Mathematical Statistics*, 22(1):79–86, Mar. 1951. doi:10.1214/aoms/1177729694. 95

[167] B. C. Kwon, M.-J. Choi, J. T. Kim, E. Choi, Y. B. Kim, S. Kwon, J. Sun, and J. Choo. Retainvis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. *IEEE Trans. Vis. Comput. Graph.*, 25(1):299–309, 2018. doi:10.1109/TVCG.2018.2865027. 147

[168] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Empirical studies in information visualization: Seven scenarios. *IEEE Trans. Vis. Comput. Graph.*, 18(9):1520–1536, 2011. doi:10.1109/TVCG.2011.279. 170

[169] J. Lazar, J. H. Feng, and H. Hochheiser. *Research methods in human-computer interaction*. Morgan Kaufmann, 2017. 163

[170] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. *nature*, 521(7553):436–444, 2015. doi:10.1038/nature14539. 136

[171] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. *Neural Computation*, 1(4):541–551, 1989. doi:10.1162/neco.1989.1.4.541. 143

[172] F. Li, B. Wu, K. Yi, and Z. Zhao. XDB: approximate DB. https://initialdlab.github.io/XDB/. Accessed on May, 19th, 2020. 37, 56

[173] M. Li, T. Zhang, Y. Chen, and A. J. Smola. Efficient Mini-Batch Training for Stochastic Optimization. In *International Conference on Knowledge Discovery and Data Mining*, KDD '14, page 661–670, New York, NY, USA, 2014. ACM. doi:10.1145/2623330.2623612. 31, 104

[174] N. Li, V. van Unen, T. Höllt, A. Thompson, J. van Bergen, N. Pezzotti, E. Eise-
mann, A. Vilanova, S. M. Chuva de Sousa Lopes, B. P. Lelieveldt, and F. Kon-
ing. Mass cytometry reveals innate lymphoid cell differentiation pathways in the
human fetal intestine. *Journal of Experimental Medicine*, 215(5):1383–1396, 03 2018.
doi:10.1084/jem.20171934. 181

[175] E. Liberty. Simple and Deterministic Matrix Sketching. In *International Conference on Knowledge Discovery and Data Mining*, KDD '13, page 581–588, New York, NY, USA, 2013. ACM. doi:10.1145/2487575.2487623. 61

[176] E. Liberty, F. Woolfe, P.-G. Martinsson, V. Rokhlin, and M. Tygert. Randomized algorithms for the low-rank approximation of matrices. *Proceedings of the National Academy of Sciences*, 104(51):20167–20172, 2007. doi:10.1073/pnas.0709640104. 60
[177] L. Lins, J. T. Klosowski, and C. Scheidegger. Nanocubes for Real-Time Exploration of Spatiotemporal Datasets. *IEEE Trans. Vis. Comput. Graph.*, 19(12):2456–2465, Dec 2013. doi:10.1109/TVCG.2013.179. 9, 172

[178] D. Liu, W. Cui, K. Jin, Y. Guo, and H. Qu. Deeptracker: Visualizing the training process of convolutional neural networks. *ACM Trans. Intell. Syst. Technol.*, 10(1):1–25, 2018. doi:10.1145/3200489. 147

[179] Z. Liu and J. Heer. The Effects of Interactive Latency on Exploratory Vi-
sual Analysis. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2122–2131, Dec 2014.
doi:10.1109/TVCG.2014.2346452. 2, 5, 20, 33

[180] Z. Liu and J. Heer. The Effects of Interactive Latency on Exploratory Vi-
sual Analysis. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2122–2131, 2014.
doi:10.1109/TVCG.2014.2346452. 108, 110, 164

[181] S. B. Loeschke, M. Hogräfer, and H.-J. Schulz. Progressive Parameter Space Visualization for Task-Driven SAX Configuration. In *Proceedings of the International EuroVis Workshop on Visual Analytics (EuroVA)*, pages 43–47. Eurographics, 2020. doi:10.2312/eurova.20201085. 116

[182] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. *arXiv preprint arXiv:1706.08840*, 2017. doi:10.48550/arXiv.1706.08840. 136

[183] V. Losing, B. Hammer, and H. Wersing. kNN classifier with self adjusting memory for heterogeneous concept drift. In *International Conference on Data Mining (ICDM)*, pages 291–300. IEEE, 2016. doi:10.1109/ICDM.2016.0040. 145

[184] V. Losing, B. Hammer, and H. Wersing. Incremental on-line learning: A review and comparison of state of the art algorithms. *Neurocomputing*, 275:1261–1274, 2018. doi:10.1016/j.neucom.2017.06.084. 145

[185] K.-L. Ma. In Situ Visualization at Extreme Scale: Challenges and Opportunities. *IEEE Comput. Graph. Appl. Mag*, 29(6):14–19, Nov 2009. doi:10.1109/MCG.2009.120. 18

[186] J. MacQueen. Some methods for classification and analysis of multivariate observations. In *Proc. of the Fifth Berkeley Symposium on Mathematical Statistics and Probability*, volume 1: Statistics, pages 281–297. University of California Press, Berkeley, Calif., 1967. 85, 136

[187] A. Macy. Visualizing Sorting Algorithms. https://bl.ocks.org/alexmacy/770f14e11594623320db1270361331dc. Accessed: 2020-03-08. 115

[188] M. J. Mahoney. Publication prejudices: An experimental study of confirmatory bias in the peer review system. *Cognitive Therapy and Research*, 1(2):161–175, 1977. doi:10.1007/BF01173636. 120, 122

[189] MATLAB. version 7.10.0 (R2010a). The MathWorks Inc., Natick, Massachusetts, 2010. 49

[190] A. Mayorga and M. Gleicher. Splatterplots: Overcoming Overdraw in Scatter Plots. *IEEE Trans. Vis. Comput. Graph.*, 19(9):1526–1538, Sept 2013. doi:10.1109/TVCG.2013.65. 70
[191] A. McGregor. Graph Stream Algorithms: A Survey. *SIGMOD Rec.*, 43(1):9–20, May 2014. doi:10.1145/2627692.2627694. 61

[192] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. *arXiv preprint arXiv:1802.03426*, 2018. doi:10.48550/arXiv.1802.03426. 72, 136

[193] W. McKinney. pandas: a Foundational Python Library for Data Analysis and Statistics. *Python for High Performance and Scientific Computing*, pages 1–9, 2011. 50, 57

[194] P. McLachlan, T. Munzner, E. Koutsofios, and S. North. LiveRAC: Interactive Visual Exploration of System Management Time-Series Data. In *Proceedings of CHI*, CHI '08, pages 1483–1492. 2008. doi:10.1145/1357054.1357286. 168

[195] M. Meyer, T. Munzner, and H. Pfister. MizBee: A Multiscale Synteny Browser. *IEEE Trans. Vis. Comput. Graph.*, 15(6):897–904, 2009. doi:10.1109/TVCG.2009.167. 168

[196] M. Meyer, M. Sedlmair, P. S. Quinan, and T. Munzner. The nested blocks and guidelines model. *Information Visualization*, 14(3):234–249, 2015. doi:10.1177/1473871613510429. 170

[197] L. Micallef, H.-J. Schulz, M. Angelini, M. Aupetit, R. Chang, J. Kohlhammer, A. Perer, and G. Santucci. The Human User in Progressive Visual Analytics. In *Short Paper Proceedings of EuroVis'19*, pages 19–23. Eurographics Association, 2019. doi:10.2312/evs.20191164. 109, 122, 152

[198] P. M. Miller and N. S. Fagley. The Effects of Framing, Problem Variations, and Providing Rationale on Choice. *Personality and Social Psychology Bulletin*, 17(5):517–522, 1991. doi:10.1177/0146167291175006. 124

[199] R. B. Miller. Response Time in Man-computer Conversational Transactions. In *Proc. of the Fall Joint Computer Conference, Part I*, pages 267–277. ACM, 1968. doi:10.1145/1476589.1476628. 2

[200] T. Miller. Explanation in artificial intelligence: Insights from the social sciences. *Artificial Intelligence*, 267:1–38, 2019. doi:10.1016/j.artint.2018.07.007. 146

[201] M. Monroe, R. Lan, H. Lee, C. Plaisant, and B. Shneiderman. Temporal Event Sequence Simplification. *IEEE Trans. Vis. Comput. Graph.*, 19(12):2227–2236, 2013. doi:10.1109/TVCG.2013.200. 7

[202] J. Moody, D. McFarland, and S. Bender-deMoll. Dynamic Network Visualization. *American Journal of Sociology*, 110(4):1206–1241, 2005. doi:10.1086/421509. 75

[203] D. A. Moore and P. J. Healy. The trouble with overconfidence. *Psychological Review*, 115(2):502–517, 2008. doi:10.1037/0033-295X.115.2.502. 122

[204] D. A. Moore and D. Schatz. The three faces of overconfidence. *Social and Personality Psychology Compass*, 11(8):e12331, 2017. doi:10.1111/spc3.12331. 122

[205] D. Moritz, D. Fisher, B. Ding, and C. Wang. Trust, but Verify: Optimistic Visualizations of Approximate Queries for Exploring Big Data. In *Proceedings of CHI*, CHI '17, pages 2904–2915, New York, NY, USA, 2017. ACM. doi:10.1145/3025453.3025456. 77, 97, 105, 106
[206] D. Moritz, B. Howe, and J. Heer. Falcon: Balancing Interactive Latency and Resolution Sensitivity for Scalable Linked Visualizations. In Proceedings of CHI, CHI '19, New York, NY, USA, 2019. ACM. doi:10.1145/3290605.3300924. 9, 36, 41, 78, 156, 172

[207] T. Mühlbacher, H. Piringer, S. Gratzl, M. Sedlmair, and M. Streit. Opening the Black Box: Strategies for Increased User Involvement in Existing Algorithm Implementations. IEEE Trans. Vis. Comput. Graph., 20(12):1643–1652, Dec 2014. doi:10.1109/TVCG.2014.2346578. 19, 22, 80, 98, 111, 117, 119, 135

[208] M. Muja and D. G. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Applications – Vol.1: VISAPP, volume 2, pages 331–340, 2009. doi:10.5220/0001787803310340. 139

[209] M. Muja and D. G. Lowe. Scalable Nearest Neighbor Algorithms for High Dimensional Data. IEEE Trans. Pattern Anal. Mach. Intell., 36, 2014. doi:10.1109/TPAMI.2014.2321376. 61

[210] J. D. Mulder, J. J. van Wijk, and R. van Liere. A Survey of Computational Steering Environments. Future Gener. Comput. Syst., 15(1):119–129, Feb. 1999. doi:10.1016/S0167-739X(98)00047-8. 28, 116

[211] T. Munzner. A Nested Process Model for Visualization Design and Validation. IEEE Trans. Vis. Comput. Graph., 15(6):921–928, 2009. doi:10.1109/TVCG.2009.111. 170

[212] S. Muthukrishnan. Data Streams: Algorithms and Applications. Found. Trends Theor. Comput. Sci., 1(2):117–236, Aug. 2005. doi:10.1561/0400000002. 26, 157

[213] L. Nachmanson, R. Prutkin, B. Lee, N. H. Riche, A. E. Holroyd, and X. Chen. GraphMaps: Browsing Large Graphs as Interactive Maps. In Proceedings of Graph Drawing, pages 3–15. Springer, 2015. doi:10.1007/978-3-319-27261-0_1. 112

[214] New York Taxis & Limousine Commission. TLC Trip Record Data. https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page, 2024. 62, 71, 75, 122, 210

[215] D. T. Nguyen, K. Al-Mannai, S. R. Joty, H. Sajjad, M. Imran, and P. Mitra. Robust Classification of Crisis-Related Data on Social Networks Using Convolutional Neural Networks. In International Conference on Web and Social Media, ICWSM, pages 632–635, 2017. 124, 125

[216] S. C. North. Incremental layout in DynaDAG. In Graph Drawing, pages 409–418, Berlin, Heidelberg, 1995. Springer Berlin Heidelberg. doi:10.1007/BFB0021824. 73

[217] M. I. Norton, D. Mochon, and D. Ariely. The IKEA effect: When labor leads to love. Journal of Consumer Psychology, 22(3):453–460, 2012. doi:10.1016/j.jcps.2011.08.002. 131

[218] P. C. O'Brien and T. R. Fleming. A Multiple Testing Procedure for Clinical Trials. Biometrics, 35(3):549, Sept. 1979. doi:10.2307/2530245. 101

[219] T. O'Donoghue and M. Rabin. Present Bias: Lessons Learned and to Be Learned. American Economic Review, 105(5):273–279, 2015. doi:10.1257/aer.p20151085. 122
[220] F. Ofli, P. Meier, M. Imran, C. Castillo, D. Tuia, N. Rey, J. Briant, P. M. Rudd, F. Reinhard, M. Parkan, and S. Joost. Combining Human Computing and Machine Learning to Make Sense of Big (Aerial) Data for Disaster Response. Big Data, 4(1):47-59, 2016. doi:10.1089/big.2014.0064. 125

[221] L. Padilla, M. Kay, and J. Hullman. Uncertainty Visualization, pages 1-18. John Wiley & Sons, Ltd, 2021. doi:https://doi.org/10.1002/9781118445112.stat08296. 96, 105

[222] M. Panju. Iterative methods for computing eigenvalues and eigenvectors. arXiv preprint arXiv:1105.1185, 2011. doi:10.48550/arXiv.1104.1185. 60

[223] Y. Park, B. Mozafari, J. Sorenson, and J. Wang. VerdictDB: Universalizing Approximate Query Processing. In International Conference on Management of Data, SIGMOD '18, pages 1461-1476, New York, NY, USA, 2018. ACM. doi:10.1145/3183713.3196905. 34, 155

[224] A. Patil, G. Richer, C. Jermaine, D. Moritz, and J.-D. Fekete. Studying Early Decision Making with Progressive Bar Charts. IEEE Trans. Vis. Comput. Graph., 29(1):407-417, 2023. doi:10.1109/TVCG.2022.3209426. 52, 54, 77, 95, 97, 102, 105, 107, 159, 161

[225] P. P. Pebay. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments. Technical Report SAND2008-6212, Sandia National Laboratories, 9 2008. doi:10.2172/1028931. 60

[226] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. doi:10.5555/1953048.2078195. 57, 60

[227] A. Perer and B. Shneiderman. Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis. In Proceedings of CHI, CHI '08, pages 265-274, 2008. doi:10.1145/1357054.1357101. 168

[228] D. Petersohn, S. Macke, D. Xin, W. Ma, D. Lee, X. Mo, J. E. Gonzalez, J. M. Hellerstein, A. D. Joseph, and A. Parameswaran. Towards Scalable Dataframe Systems. Proc. VLDB Endow., 13(12):2033-2046, July 2020. doi:10.14778/3407790.3407807. 55, 56, 59

[229] E. Petraki, S. Idreos, and S. Manegold. Holistic Indexing in Main-memory Column-stores. In International Conference on Management of Data, SIGMOD '15, pages 1153-1166, New York, NY, USA, 2015. ACM. doi:10.1145/2723372.2723719. 56

[230] N. Pezzotti, J.-D. Fekete, T. Höllt, B. P. Lelieveldt, E. Eisemann, and A. Vilanova. Multiscale Visualization and Exploration of Large Bipartite Graphs. Computer Graphics Forum, 37(3):549-560, 2018. doi:10.1111/cgf.13441. 57, 162

[231] N. Pezzotti, T. Höllt, J. Van Gemert, B. P. Lelieveldt, E. Eisemann, and A. Vilanova. Deepeyes: Progressive visual analytics for designing deep neural networks. IEEE Trans. Vis. Comput. Graph., 24(1):98-108, 2017. doi:10.1109/TVCG.2017.2744358. 119, 147
[232] N. Pezzotti, T. Höllt, B. Lelieveldt, E. Eisemann, and A. Vilanova. Hierarchical Stochastic Neighbor Embedding. *Computer Graphics Forum*, 35(3):21–30, 2016. doi:10.1111/cgf.12878. 72, 103, 136

[233] N. Pezzotti, B. P. Lelieveldt, L. van der Maaten, T. Höllt, E. Eisemann, and A. Vilanova. Approximated and user steerable tSNE for progressive visual analytics. *IEEE Trans. Vis. Comput. Graph.*, 23(7):1739–1752, 2016. doi:10.1109/TVCG.2016.2570755. 31, 72, 98, 104, 136, 140, 146, 162, 166

[234] P. Pirolli and S. Card. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In *International Conference on Intelligence Analysis*, volume 5, pages 2–4, 2005. 164

[235] C. Plaisant, G. Grinstein, and J. Scholtz. Visual-Analytics Evaluation. *IEEE Trans. Vis. Comput. Graph.*, 29(3):16–17, 2009. doi:10.1109/MCG.2009.56. 149

[236] S. J. Pocock. When (Not) to Stop a Clinical Trial for Benefit. *JAMA*, 294(17):2228–2230, 11 2005. doi:10.1001/jama.294.17.2228. 101

[237] A. Pol, C. M. Jermaine, and S. Arumugam. Maintaining very large random samples using the geometric file. *The VLDB Journal*, 17(5):997–1018, 2008. doi:10.1007/s00778-007-0048-z. 62, 173

[238] R. Polikar. Ensemble based systems in decision making. *IEEE Circuits and Systems Magazine*, 6(3):21–45, 2006. doi:10.1109/MCAS.2006.1688199. 113

[239] N. Potti and J. M. Pate. DAQ: A New Paradigm for Approximate Query Processing. *Proc. VLDB Endow.*, 8(9):898–909, 2015. 38

[240] M. Procopio, A. Mosca, C. E. Scheidegger, E. Wu, and R. Chang. Impact of Cognitive Biases on Progressive Visualization. *IEEE Trans. Vis. Comput. Graph.*, 28(9):3093–3112, 2022. doi:10.1109/TVCG.2021.3051013. 161, 165

[241] M. Procopio, C. Scheidegger, E. Wu, and R. Chang. Load-n-Go: Fast Approximate Join Visualizations That Improve Over Time. In *Proceedings of DSIA*, 2017. 39, 56, 118

[242] M. Procopio, C. Scheidegger, E. Wu, and R. Chang. Selective Wander Join: Fast progressive visualizations for data joins. *Informatics*, 6(1), 3 2019. doi:10.3390/informatics6010014. 31, 37, 39, 56, 155, 156

[243] Y. Qiu, Y. Wang, K. Yi, F. Li, B. Wu, and C. Zhan. Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries. In *International Conference on Management of Data*, SIGMOD '21, pages 1465–1477. ACM, 2021. doi:10.1145/3448016.3452821. 155

[244] R Core Team. *R: A Language and Environment for Statistical Computing*. R Foundation for Statistical Computing, Vienna, Austria, 2014. URL http://www.R-project.org/. 49

[245] M. Raasveldt and H. Mühleisen. DuckDB: An Embeddable Analytical Database. In *International Conference on Management of Data*, SIGMOD '19, pages 1981–1984, New York, NY, USA, 2019. ACM. doi:10.1145/3299869.3320212. 56
[246] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. *OpenAI blog*, 1(8):9, 2019. 144

[247] G. Ramalingam and T. Reps. A categorized bibliography on incremental computation. In *Proceedings of the 20th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages*, pages 502–510, 1993. doi:10.1145/158511.158710. 25

[248] V. Raveneau, J. Blanchard, and Y. Prié. Progressive sequential pattern mining: steerable visual exploration of patterns with PPMT. In *Visualization in Data Science (VDS at IEEE VIS 2018)*, Berlin, Germany, 2018. IEEE. 158

[249] G. Richer, A. Pister, M. Abdelaal, J. Fekete, M. Sedlmair, and D. Weiskopf. Scalability in Visualization. *arXiv preprint arXiv:2210.06562*, abs/2210.06562, 2022. doi:10.48550/arXiv.2210.06562. 70, 172

[250] M. Rocklin. Dask: Parallel Computation with Blocked algorithms and Task Scheduling. In *Proceedings of the 14th Python in Science Conference*, pages 130 – 136, 2015. doi:10.25080/MAJORA-7B98E3ED-013. 55, 59

[251] C. Rohrdantz, D. Oelke, M. Krstajić, and F. Fischer. Real-Time Visualization of Streaming Text Data: Tasks and Challenges. In *Proceedings of TextVis*, 2011. 115

[252] R. Rosenbaum and B. Hamann. Progressive Presentation of Large Hierarchies Using Treemaps. In *Advances in Visual Computing*, pages 71–80. Springer, 2009. doi:10.1007/978-3-642-10520-3_7. 77, 111

[253] R. Rosenbaum and H. Schumann. Progressive refinement: more than a means to overcome limited bandwidth. In *Proceedings of VDA*, page 72430I. SPIE, 2009. doi:10.1117/12.810501. 115, 118, 121

[254] R. Rosenbaum, J. Zhi, and B. Hamann. Progressive parallel coordinates. In *2012 IEEE Pacific Visualization Symposium*, pages 25–32. IEEE, 2012. doi:10.1109/PacificVis.2012.6183570. 77, 115

[255] D. A. Ross, J. Lim, R.-S. Lin, and M.-H. Yang. Incremental Learning for Robust Visual Tracking. *Int. J. Comput. Vision*, 77(1-3):125–141, May 2008. doi:10.1007/s11263-007-0075-7. 60, 83, 166

[256] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. ORB: An efficient alternative to SIFT or SURF. In *International Conference on Computer Vision*, pages 2564–2571. IEEE, 2011. doi:10.1109/ICCV.2011.6126544. 140

[257] K. Rudra, S. Banerjee, N. Ganguly, P. Goyal, M. Imran, and P. Mitra. Summarizing Situational Tweets in Crisis Scenario. In *ACM Conference on Hypertext and Social Media*, pages 137–147. ACM, 2016. doi:10.1145/2914586.2914600. 124

[258] S. Ruping. Incremental learning with support vector machines. In *International Conference on Data Mining (ICDM)*, pages 641–642, 2001. doi:10.1109/ICDM.2001.989589. 140, 141

[259] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive Neural Networks, 2016, 1606.04671. 145

[260] F. Rusu, C. Qin, and M. Torres. Scalable Analytics Model Calibration with Online
Aggregation. *IEEE Data Eng. Bull.*, 38(3):30–43, 2015. 41

[261] R. B. Rusu and S. Cousins. 3D is here: Point cloud library (PCL). In *International Conference on Robotics and Automation*, pages 1–4. IEEE, 2011. doi:10.1109/ICRA.2011.5980567. 140

[262] Y. Saad. *Iterative methods for sparse linear systems*. SIAM, 2003. doi:10.1137/1.9780898718003. 25

[263] D. Sacha, A. Stoffel, F. Stoffel, B. C. Kwon, G. Ellis, and D. A. Keim. Knowledge Generation Model for Visual Analytics. *IEEE Trans. Vis. Comput. Graph.*, 20(12):1604–1613, 2014. doi:10.1109/TVCG.2014.2346481. 114

[264] W. Saeed and C. Omlin. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. *Knowledge-Based Systems*, 263:110273, 2023. doi:https://doi.org/10.1016/j.knosys.2023.110273. 119

[265] P. Saraiya, C. North, and K. Duca. An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. *IEEE Trans. Vis. Comput. Graph.*, 11(4):443–456, 2005. doi:10.1109/TVCG.2005.53. 168

[266] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer. Vega-Lite: A Grammar of Interactive Graphics. *IEEE Trans. Vis. Comput. Graph.*, 23(1):341–350, 2017. doi:10.1109/TVCG.2016.2599030. 89

[267] T.-W. Schmidt, F. Pellacini, D. Nowrouzezahrai, W. Jarosz, and C. Dachsbacher. State of the Art in Artistic Editing of Appearance, Lighting and Material. *Computer Graphics Forum*, 35(1):216–233, 2016. doi:10.1111/cgf.12721. 159

[268] B. Schneider, D. Jäckle, F. Stoffel, A. Diehl, J. Fuchs, and D. Keim. Integrating Data and Model Space in Ensemble Learning by Visual Analytics. *IEEE Transactions on Big Data*, 7(3):483–496, 2021. doi:10.1109/TBDATA.2018.2877350. 117

[269] T. Schreck, J. Bernard, T. von Landesberger, and J. Kohlhammer. Visual cluster analysis of trajectory data with interactive Kohonen maps. *Information Visualization*, 8(1):14–29, 2009. doi:10.1109/VAST.2008.4677350. 117, 120, 121

[270] H.-J. Schulz, M. Angelini, G. Santucci, and H. Schumann. An enhanced visualization process model for incremental visualization. *IEEE Trans. Vis. Comput. Graph.*, 22(7):1830–1842, July 2016. doi:10.1109/TVCG.2015.2462356. 53, 82

[271] H. Schwerdtfeger. *Introduction to linear algebra and the theory of matrices*. Noordhoff, 1950. 141

[272] D. Sculley. Web-scale K-means Clustering. In *International Conference on World Wide Web*, WWW ’10, pages 1177–1178. ACM, 2010. doi:10.1145/1772690.1772862. 31, 60, 83

[273] R. Sedgewick and K. Wayne. *Algorithms*. Addison-Wesley, 4th edition, 2011. 139

[274] M. Sedlmair and M. Aupetit. Data-Driven Evaluation of Visual Quality Measures. *Computer Graphics Forum*, 34(3):201–210, June 2015. doi:10.1111/CGF.12632. 151

[275] M. Sedlmair, C. Heinzl, S. Bruckner, H. Piringer, and T. Möller. Visual Parameter Space Analysis: A Conceptual Framework. *IEEE Trans. Vis. Comput. Graph.*, 20(12):2161–
2170, 2014. doi:10.1109/TVCG.2014.2346321. 116

[276] M. Sedlmair, P. Isenberg, D. Baur, M. Mauerer, C. Pigorsch, and A. Butz. Cardiogram: visual analytics for automotive engineers. In *Proceedings of CHI*, CHI '11, pages 1727-1736, 2011. doi:10.1145/1978942.1979194. 168

[277] M. Sedlmair, M. Meyer, and T. Munzner. Design study methodology: Reflections from the trenches and the stacks. *IEEE Trans. Vis. Comput. Graph.*, 18(12):2431–2440, 2012. doi:10.1109/TVCG.2012.213. 167, 168

[278] S. C. Seow. *Designing and Engineering Time: The Psychology of Time Perception in Software*. Addison-Wesley Professional, 2008. 149, 164

[279] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. *arXiv preprint arXiv:1705.08690*, 2017. doi:10.48550/arXiv.1705.08690. 136

[280] K. Shin and P. Ramanathan. Real-time computing: a new discipline of computer science and engineering. *Proceedings of the IEEE*, 82(1):6–24, 1994. doi:10.1109/5.259423. 20

[281] B. Shneiderman. Response Time and Display Rate in Human Performance with Computers. *ACM Comput. Surv.*, 16(3):265–285, Sept. 1984. doi:10.1145/2514.2517. 2, 36, 110

[282] B. Shneiderman. The eyes have it: a task by data type taxonomy for information visualizations. In *Proceedings 1996 IEEE Symposium on Visual Languages*, pages 336–343. IEEE, 1996. doi:10.1109/VL.1996.545307. 115, 177

[283] B. Shneiderman and C. Plaisant. Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In *Proceedings of the AVI workshop on BEyond time and errors: novel evaluation methods for information visualization*, pages 1–7, 2006. doi:10.1145/1168149.1168158. 167, 168

[284] S. Si, S. Kumar, and Y. Li. Nonlinear Online Learning with Adaptive Nyström Approximation. *arXiv preprint arXiv:1802.07887*, 2018. doi:10.48550/arXiv.1802.07887. 142

[285] F. Siddiqui, T. Höllt, and A. Vilanova. A Progressive Approach for Uncertainty Visualization in Diffusion Tensor Imaging. *Computer Graphics Forum*, 40(3):411–422, 2021. doi:10.1111/CGF.14317. 102

[286] D. J. Simons and R. A. Rensink. Change blindness: Past, present, and future. *Trends in cognitive sciences*, 9(1):16–20, 2005. doi:10.1016/j.tics.2004.11.006. 163

[287] M. M. Skeels, B. Lee, G. Smith, and G. G. Robertson. Revealing uncertainty for information visualization. *Inf. Vis.*, 9(1):70–81, 2010. doi:10.1057/ivs.2009.1. 96

[288] D. Smilkov, S. Carter, D. Sculley, F. B. Viégas, and M. Wattenberg. Direct-manipulation visualization of deep networks. *arXiv preprint arXiv:1708.03788*, 2017. doi:10.48550/arXiv.1708.03788. 119, 121

[289] M. Sondag, B. Speckmann, and K. Verbeek. Stable Treemaps via Local Moves. *IEEE Trans. Vis. Comput. Graph.*, 24(1):729–738, 2018. doi:10.1109/TVCG.2017.2745140. 75
[290] D. Song and E. Golin. Fine-grain Visualization Algorithms in Dataflow Environments. In Proc. of the 4th Conference on Visualization '93, pages 126–133. IEEE, 1993. doi:10.1109/VISUAL.1993.398860. 73

[291] K. Starbird, L. Palen, A. L. Hughes, and S. Vieweg. Chatter on the Red: What Hazards Threat Reveals About the Social Life of Microblogged Information. In ACM Conference on Computer Supported Cooperative Work, pages 241–250. ACM, 2010. doi:10.1145/1718918.1718965. 124

[292] I. Steinwart. Support Vector Machines are Universally Consistent. Journal of Complexity, 18(3):768–791, 2002. doi:https://doi.org/10.1006/jcom.2002.0642. 140

[293] C. D. Stolper, A. Perer, and D. Gotz. Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE Trans. Vis. Comput. Graph., 20(12):1653–1662, 2014. doi:10.1109/TVCG.2014.2346574. 11, 19, 21, 22, 73, 82, 105, 119, 121, 151, 157, 158, 168

[294] M. Stonebraker, P. Brown, J. Becla, and D. Zhang. SciDB: A Database Management System for Applications with Complex Analytics. Computing in Science and Engg., 15(3):54–62, May 2013. doi:10.1109/MCSE.2013.19. 55

[295] H. Strobelt, B. Hoover, A. Satyanaryan, and S. Gehrmann. LMdiff: A Visual Diff Tool to Compare Language Models. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 96–105, Nov. 2021. doi:10.18653/v1/2021.emnlp-demo.12. 147

[296] M. Sugiyama and M. Kawanabe. Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT Press, 2012. doi:10.7551/mitpress/9780262017091.001.0001. 145

[297] Tableau Software. https://www.tableau.com/, 2003. Accessed: 2019-07-20. 36

[298] Tensorflow Playground. Github, 2016. URL https://playground.tensorflow.org/. 106

[299] A. K. Thomas and P. R. Millar. Reducing the Framing Effect in Older and Younger Adults by Encouraging Analytic Processing. The Journals of Gerontology: Series B, 67B(2):139–149, 2011. doi:10.1093/geronb/gbr076. 124

[300] TPC-H. http://www.tpc.org/tpch/, 2016. Accessed: 2021-11-02. 155

[301] C.-H. Tsai, C.-Y. Lin, and C.-J. Lin. Incremental and Decremental Training for Linear Classification. In International Conference on Knowledge Discovery and Data Mining, KDD '14, page 343–352, New York, NY, USA, 2014. ACM. doi:10.1145/2623330.2623661. 142

[302] M. Tukan, C. Baykal, D. Feldman, and D. Rus. On coresets for support vector machines. In International Conference on Theory and Applications of Models of Computation, pages 287–299. Springer, 2020. doi:10.1007/978-3-030-59267-7_25. 142

[303] J. W. Tukey. Exploratory data analysis. Addison-Wesley series in behavioral science : quantitative methods. Addison-Wesley, Reading (Mass.), 1977. URL http://opac.inria.fr/record=b1080310. 4
[304] C. Turkay, E. Kaya, S. Balcisoy, and H. Hauser. Designing Progressive and Interactive Analytics Processes for High-Dimensional Data Analysis. *IEEE Trans. Vis. Comput. Graph.*, 23(1):131–140, 2017. doi:10.1109/TVCG.2016.2598470. 60, 78, 80, 82, 83, 84, 116, 146, 156, 160, 168

[305] C. Turkay, N. Pezzotti, C. Binnig, H. Strobelt, B. Hammer, D. A. Keim, J.-D. Fekete, T. Palpanas, Y. Wang, and F. Rusu. Progressive data science: Potential and challenges. *arXiv preprint arXiv:1812.08032*, 2018. doi:10.48550/arXiv.1812.08032. 132

[306] A. Tversky and D. Kahneman. The framing of decisions and the psychology of choice. *Science*, 211(4481):453–458, 1981. doi:10.1126/science.7455683. 123

[307] A. Ulmer, M. Angelini, J.-D. Fekete, J. Kohlhammer, and T. May. A Survey on Progressive Visualization. *IEEE Trans. Vis. Comput. Graph.*, 30(9):6447–6467, 2024. doi:10.1109/TVCG.2023.3346641. 81

[308] S. D. Urban, T. B. Abdellatif, S. W. Dietrich, and A. Sundermier. Delta abstractions: A technique for managing database states in runtime debugging of active database rules. *IEEE Trans. Knowl. Data Eng*, 15(3):597–612, 2003. doi:10.1109/TKDE.2003.1198393. 119, 121

[309] A. C. Valdez, M. Ziefle, and M. Sedlmair. Priming and anchoring effects in visualization. *IEEE Trans. Vis. Comput. Graph.*, 24(1):584–594, 2017. doi:10.1109/TVCG.2017.2744138. 122, 165

[310] A. C. Valdez, M. Ziefle, and M. Sedlmair. Studying biases in visualization research: Framework and methods. In *Cognitive Biases in Visualizations*, pages 13–27. Springer, 2018. doi:10.1007/978-3-319-95831-6_2. 165

[311] L. van der Maaten and G. Hinton. Visualizing Data using t-SNE. *Journal of Machine Learning Research*, 9(86):2579–2605, 2008. URL http://jmlr.org/papers/v9/vandermaaten08a.html. 72, 85, 95, 104

[312] S. van der Walt, S. C. Colbert, and G. Varoquaux. The NumPy Array: A Structure for Efficient Numerical Computation. *Computing in Science Engineering*, 13(2):22–30, March 2011. doi:10.1109/MCSE.2011.37. 50

[313] G. van Rossum. Python tutorial. Technical Report CS-R9526, Centrum voor Wiskunde en Informatica (CWI), Amsterdam, May 1995. 49

[314] J. Vermeulen, K. Luyten, K. Coninx, and N. Marquardt. The Design of Slow-motion Feedback. In *Proceedings of DIS*, pages 267–270. ACM, 2014. doi:10.1145/2598510.2598604. 79, 112

[315] E. Vernier, M. Sondag, J. Comba, B. Speckmann, A. Telea, and K. Verbeek. Quantitative Comparison of Time-Dependent Treemaps. *Computer Graphics Forum*, 39(3):393–404, 2020. doi:https://doi.org/10.1111/cgf.13989. 75

[316] J. Vidal, P. Guillou, and J Tierny. A Progressive Approach to Scalar Field Topology. *IEEE Trans. Vis. Comput. Graph.*, 27(06):2833–2850, June 2021. doi:10.1109/TVCG.2021.3060500. 151

[317] J. S. Vitter. Random Sampling with a Reservoir. *ACM Trans. Math. Softw.*, 11(1):37–57,
Mar. 1985. doi:10.1145/3147.3165. 44, 60, 72

[318] D. Vohra. Apache Parquet. In *Practical Hadoop Ecosystem*, pages 325–335. Apress, 2016. doi:10.1007/978-1-4842-2199-0-8. 55, 62

[319] W. A. Wagenaar and G. B. Keren. Calibration of probability assessments by profes-
sional blackjack dealers, statistical experts, and lay people. *Org. Behavior and Human
Dec. Proc.*, 36(3):406–416, 1985. doi:10.1016/0749-5978(85)90008-1. 120

[320] K. Wagstaff, C. Cardie, S. Rogers, and S. Schrödl. Constrained K-means Clustering with Background Knowledge. In *International Conference on Machine Learning* (ICML), pages 577–584. Morgan Kaufmann, 2001. 117

[321] A. Wald. Sequential Analysis. John Wiley and Son, Inc, New York., 1947. 94, 101, 161

[322] E. Wall, L. M. Blaha, L. Franklin, and A. Endert. Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Bias in Interactive Visual Analytics. In *2017 IEEE Conference on Visual Analytics Science and Technology (VAST)*, pages 104–115. IEEE, 2017. doi:10.1109/VAST.2017.8585669. 122

[323] Z. Wan, W. Taha, and P. Hudak. Event-Driven FRP. In *Practical Aspects of Declarative Languages*, pages 155–172, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. doi:10.1007/3-540-45587-6_11. 66

[324] D. Wang, J. D. Weisz, M. Muller, P. Ram, W. Geyer, C. Dugan, Y. Tausczik, H. Samu-
lowitz, and A. Gray. Human-AI Collaboration in Data Science: Exploring Data Sci-
entists’ Perceptions of Automated AI. *Proceedings of the ACM on Human-Computer
Interaction*, 3(CSCW):1–24, 2019. doi:10.1145/3359313. 132

[325] J. Waser, R. Fuchs, H. Ribicic, B. Schindler, G. Blöschl, and E. Gröller. World lines. *IEEE Trans. Vis. Comput. Graph.*, 16(6):1458–1467, 2010. doi:10.1109/TVCG.2010.223. 117, 120, 121

[326] G. I. Webb, R. Hyde, H. Cao, H. L. Nguyen, and F. Petitjean. Characteriz-
ing concept drift. *Data Mining and Knowledge Discovery*, 30(4):964–994, 2016.
doi:10.1007/s10618-015-0448-4. 145

[327] B. P. Welford. Note on a Method for Calculating Corrected Sums of Squares and Products. *Technometrics*, 4(3):419–420, 1962. doi:10.1080/00401706.1962.10490022. 30, 51, 60

[328] A. Weller. Transparency: motivations and challenges. In *Explainable AI: Inter-
preting, Explaining and Visualizing Deep Learning*, pages 23–40. Springer, 2019.
doi:10.1007/978-3-030-28954-6_2. 146

[329] H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2009. URL http://ggplot2.org. 89

[330] H. Wickham. Bin-summarise-smooth: a framework for visualising large data. Technical report, had.co.nz, 2013. URL https://vita.had.co.nz/papers/bigvis.pdf. 52, 68, 70, 97

[331] H. Wickham, D. Cook, H. Hofmann, and A. Buja. Graphical inference for infovis. *IEEE*
*Trans. Vis. Comput. Graph.*, 16(6):973–979, Nov 2010. doi:10.1109/TVCG.2010.161.122

[332] L. Wilkinson. *The Grammar of Graphics*. Springer-Verlag New York, Inc., 2005. 89

[333] L. Wilkinson, A. Anand, and R. Grossman. Graph-Theoretic Scagnostics. In *Proceedings of the Proceedings of the 2005 IEEE Symposium on Information Visualization*, INFOVIS '05, page 21, USA, 2005. IEEE. doi:10.1109/INFOVIS.2005.14.72

[334] M. Williams and T. Munzner. Steerable, Progressive Multidimensional Scaling. In *Proceedings of the IEEE Symposium on Information Visualization*, pages 57–64. IEEE, IEEE, 2004. doi:10.1109/INFVIS.2004.60.11.73.166

[335] W. Wright, D. Schroh, P. Proulx, A. Skaburskis, and B. Cort. The Sandbox for analysis: concepts and methods. In *Proceedings of CHI*, CHI '06, pages 801–810. ACM, 2006. doi:10.1145/1124772.1124890.122

[336] S. Xu, X. Zhang, and S. Liao. A Linear Incremental Nyström Method for Online Kernel Learning. *International Conference on Pattern Recognition (ICPR)*, pages 2256–2261, 2018. doi:10.1109/ICPR.2018.8545404.142

[337] T. B. Yacoubian, D. A. S. A. Thani, and M. Aupetit. The Role of a Facilitator in Co-Design Applications for Exploratory Analysis in Domains of High Complexity: The Case of MAHiCGO. *IEEE Access*, 9:38296–38317, 2021. doi:10.1109/ACCESS.2021.3063468.110.130

[338] M. A. Yalçin, N. Elmqvist, and B. B. Bederson. Cognitive Stages in Visual Data Exploration. In *Proceedings of BELIV*, pages 86–95. ACM, 2016. doi:10.1145/2993901.2993902.117

[339] M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica. Discretized Streams: Fault-tolerant Streaming Computation at Scale. In *Proc. of the Twenty-Fourth ACM Symposium on Operating Systems Principles*, SOSP '13, pages 423–438. ACM, 2013. doi:10.1145/2517349.2522737.52

[340] K. Zeng, S. Gao, B. Mozafari, and C. Zaniolo. The Analytical Bootstrap: A New Method for Fast Error Estimation in Approximate Query Processing. In *International Conference on Management of Data*, SIGMOD '14, page 277–288, New York, NY, USA, 2014. ACM. doi:10.1145/2588555.2588579.102

[341] W. Zeng, C. Lin, J. Lin, J. Jiang, J. Xia, C. Turkay, and W. Chen. Revisiting the Modifiable Areal Unit Problem in Deep Traffic Prediction with Visual Analytics. *IEEE Trans. Vis. Comput. Graph.*, 2020. doi:10.1109/TVCG.2020.3030410.147

[342] E. Zgraggen, A. Galakatos, A. Crotty, J.-D. Fekete, and T. Kraska. How Progressive Visualizations Affect Exploratory Analysis. *IEEE Trans. Vis. Comput. Graph.*, 23(8):1977–1987, Aug 2017. doi:10.1109/TVCG.2016.2607714.2, 33, 97, 105, 159, 160, 164, 168

[343] J. Zhang. Modern Monte Carlo methods for efficient uncertainty quantification and propagation: A survey. *WIREs Computational Statistics*, 13(5):e1539, 2021. doi:https://doi.org/10.1002/wics.1539.104
[344] K. Zhang, L. Lan, Z. Wang, and F. Moerchen. Scaling up Kernel SVM on Limited Resources: A Low-rank Linearization Approach. In *International Conference on Artificial Intelligence and Statistics*, volume 22 of *Proceedings of Machine Learning Research*, pages 1425–1434, La Palma, Canary Islands, 21–23 Apr 2012. PMLR. URL https://proceedings.mlr.press/v22/zhang12d.html. 142

[345] F. Zhao, S. Maiyya, R. Wiener, D. Agrawal, and A. E. Abbadi. KLL$^{±}$ approximate quantile sketches over dynamic datasets. *Proc. VLDB Endow.*, 14(7):1215–1227, Mar. 2021. doi:10.14778/3450980.3450990. 53

[346] J.-J. Zhu and J. Bento. Generative adversarial active learning. *arXiv preprint* arXiv:1702.07956, 2017. doi:10.48550/arXiv.1702.07956. 146

[347] S. Zilberstein. Using anytime algorithms in intelligent systems. *AI magazine*, 17(3):73, 1996. doi:10.1609/aimag.v17i3.1232. 27
List of Figures

<table><tr><td>1.1</td><td>Icicle tree visualization of 10 million patients events aggregated</td><td>8</td></tr><tr><td>1.2</td><td>The Information Visualization Reference Model, adapted from [41]</td><td>13</td></tr><tr><td>1.3</td><td>The Visual Analytics Reference Model, adapted from [154]</td><td>14</td></tr><tr><td>1.4</td><td>Book chapters related to the data analysis pipeline.</td><td>14</td></tr><tr><td>3.1</td><td>OLA-RAW architecture</td><td>42</td></tr><tr><td>3.2</td><td>ProgressiveDB architecture</td><td>44</td></tr><tr><td>4.1</td><td>Progressive bar charts without and with uncertainty.</td><td>54</td></tr><tr><td>4.2</td><td>Responsive t-SNE steps of the MNIST dataset loaded in a skewed order.</td><td>63</td></tr><tr><td>5.1</td><td>Standard visual representations do not scale well.</td><td>71</td></tr><tr><td>5.2</td><td>A screenshot from the SampleAction system.</td><td>74</td></tr><tr><td>5.3</td><td>In the NYC Taxi dataset [214], the density map of taxis loaded progressively (left) becomes the top left point (right) after loading a data row representing a taxi that traveled to Florida with its meter on, creating a scale problem.</td><td>75</td></tr><tr><td>5.4</td><td>Trade-off between stability and quality in progressive visualizations.</td><td>76</td></tr><tr><td>5.5</td><td>The DimXplorer System.</td><td>83</td></tr><tr><td>5.6</td><td>Visual components in DimXplorer.</td><td>84</td></tr><tr><td>5.7</td><td>The user interface of InsightsFeed [16]</td><td>86</td></tr><tr><td>5.8</td><td>UI Elements in InsightsFeed.</td><td>87</td></tr><tr><td>5.9</td><td>The ProReveal System.</td><td>88</td></tr><tr><td>6.1</td><td>The three phases of PDA uncertainty.</td><td>93</td></tr><tr><td>6.2</td><td>The user interface of InsightsFeed [16]</td><td>95</td></tr><tr><td>7.1</td><td>User roles, Tasks, Focus, and Usage in PVA.</td><td>121</td></tr><tr><td>7.2</td><td>The AIDR dashboard prototype.</td><td>127</td></tr></table>
8.1 Different users and expectations for progressive machine learning . 134
List of Tables

<table><tr><td>2.1</td><td>Design requirements for creating PVA systems</td><td>19</td></tr><tr><td>2.2</td><td>Summary of the different methods of computation relatable to PDA.</td><td>31</td></tr><tr><td>2.3</td><td>Summary of the determining characteristics of each method of computation related to PDA.</td><td>32</td></tr><tr><td>7.1</td><td>Comparison of the current AIDR system and its progressive improvement.</td><td>129</td></tr></table>
List of Definitions

<table><tr><td>1</td><td>Mathematical and Computer function</td><td>23</td></tr><tr><td>2</td><td>Online function</td><td>24</td></tr><tr><td>3</td><td>Iterative function</td><td>25</td></tr><tr><td>4</td><td>Incremental Function</td><td>26</td></tr><tr><td>5</td><td>Streaming function</td><td>27</td></tr><tr><td>6</td><td>Real-time function</td><td>27</td></tr><tr><td>7</td><td>Anytime function</td><td>28</td></tr><tr><td>8</td><td>Progressive function</td><td>29</td></tr><tr><td>9</td><td>Approximate Query Processing Function</td><td>30</td></tr><tr><td>10</td><td>PDA uncertainty</td><td>93</td></tr><tr><td>11</td><td>The PDA quality</td><td>93</td></tr></table>
## Listings

<table><tr><td>3.1</td><td>Example: SQL Progressive Query</td><td>37</td></tr><tr><td>3.2</td><td>SQL ALTER QUERY</td><td>39</td></tr><tr><td>3.3</td><td>SQL PROGRESSIVE Query</td><td>46</td></tr><tr><td>3.4</td><td>SQL PROGRESSIVE VIEW and FUTURE</td><td>47</td></tr><tr><td>4.1</td><td>Example: Progressive Mean</td><td>51</td></tr><tr><td>4.2</td><td>Example: Progressive Median</td><td>52</td></tr><tr><td>4.3</td><td>Example: Steerable Progressive Median with Filtering</td><td>53</td></tr><tr><td>4.4</td><td>Example: Filling a Tensor</td><td>55</td></tr></table>
Alphabetical Index

Accuracy, 23, 27, 58

Accuracy and Precision, 156

Aggregation, 70
decomposable, 46

Algorithm animation, 18

Algorithmic
stability, 20

ALTER QUERY, 39

Alternatives, 113

Anchor, 112

Animated transition, 77

Anytime, 14, 17, 23, 27

Apophenia, 122

Approximation, 26

AQP, 23, 29

Arrays, 49

Asynchronous, 42, 43

Attention management, 90

AutoML, 137

Bias, 152

Cognitive, 69, 78, 94, 107, 109,
124, 130, 131, 163, 165,
169

Attention, 122

Belief, 120

Confirmation, 120, 122

Framing, 123

IKEA, 131

Illusion, 165

Illusory truth, 122

Information, 123

Overconfidence, 122

Present, 122

Self-generation, 131

cognitive

belief, 120

confirmation, 120

exaggerated expectation, 120

Data, 143

Distribution, 62

Statistical, 102

System, 12

Bootstrap, 38

Bound

CPU, 44

I/O, 44

Bounded time, 27

Broadcasting operations, 58

Chunks, 24, 27, 30, 31, 35, 42–46, 53,
173

CLT-bounds, 38

Competitive analysis, 156–158

Confidence interval, 52, 77, 93, 100,
101

Control

Slow down, 112

Converge, 20

Convergence, 20
CPU-bound, 44

CSV, 43

Data distribution, 43

Data Frames, 49

Data Sketch, see Synopse

Data streaming, 52

Database
  state, 38

Decision-making, 51, 88, 165

Delta, 27, 37, 66, 68

Design requirement

R01, 18–21, 32, 80, 151

R02, 19, 22, 32, 80, 151, 157, 162, 167

R03, 18–22, 32, 87, 111, 151, 162

R04, 19, 22, 32, 58, 80, 112, 151

R05, 19, 22, 112, 151, 159

R06, 18, 19, 21, 22

R07, 18–22, 88, 112, 153, 159, 160

R08, 19, 20, 22, 32, 111

R09, 18–21, 32, 100, 111, 153

R10, 18–21, 32, 81, 100, 111, 159, 160

R11, 18–21, 32, 81, 100, 159, 160

R12, 19, 20, 32, 78, 79, 82, 158

R13, 19, 20, 22, 78, 81, 82, 157, 162, 167

R14, 19, 20, 22, 78, 82, 85

R15, 19, 20, 22, 78, 82, 85, 87

R16, 19, 20, 22, 32, 78, 79, 82, 87

Dimensionality reduction, 72

DimXplorer, 83

Disadvantages, 9

Disclosure, 77

Drill-down, 39, 45, 46

Eager, 15, 16

Election, 2

Exploratory Data Analysis, 1

Feature

Engineering, 137

Selection, 137

Filtering, 39, 70

FITS, 43

FORK QUERY, 40

Function

anytime, 27

approximate, 29

incremental, 25

iterative, 24

mathematical, 23

online, 24

progressive, 28

streaming, 26

Grammar of Graphics, 89

Graph algorithms, 57

Graph drawing, 57

Graphs, 49

Green computing, 4

Group-by, 39

History navigation, 77

I/O-bound, 44

Iid, 30, 101, 103

In-Situ, 18, 155

In-Situ Databases, 34

Incremental, 23, 25
InsightsFeed, 85

Interaction, 89

Interactivity, 110

Iterative, 23, 24

Join, 39, 155

Progressive, 155

Wander, 155, 156

JSON, 43

Julia, 49

k-means, 25, 60, 67, 79, 83, 85, 97, 98, 104, 105, 136, 167, 176

k-Nearest Neighbors, 60, 61, 98, 99, 104, 136, 138, 140, 166

Latency, 16, 45, 46, 110

Lineage, 36

Linear algebra, 49

Long-lasting computation, 89

Loss, 106

Machine learning
  interactivity, 136
  steerability, 137

MATLAB, 49

Mental map, 77, 79, 112, 166, 175

Middleware, 37, 38, 45

Mini-batch, 31

Monitoring, 51, 109, 111

Networks, 49, 56

OLA, see Online aggregation

Online, 23, 24

Online aggregation, 10, 17, 34, 181

Optimal stopping, 1, 94, 101, 152, 158, 179

Outlier, 107

Overplotting, 70

Overview, 70

Parallel, 40, 41, 50, 113, 117, 160

Partition, 45

PDA Quality, 92

Phase

Chaotic, 93

Converged, 93

Converging, 93

Pipeline, 5, 14, 28, 49

Pivoting, 39

Posterior, 93

Precision and Recall, 21, 23, 156

Progress, 69

Progressive, 17, 28

computation, 28

concurrent, 90

middleware, 37

quantum, 28, 30, 31, 84

table, 45

view, 45

Progressive-in-chunks, 21, 30, 53, 59–61, 68, 70, 96, 99, 107, 156, 166, 167

Progressive-in-iterations, 30, 31, 53, 57, 60, 68, 96, 97, 99, 107, 156, 166, 167

ProReveal, 87, 107, 160

PVA-Guards, 87

Python, 49

Quality, 28, 54, 58, 61, 86, 92–96, 98–101, 103–105, 107, loss, 106

Quantum, 27, 28, 33, 37, 84
Queries

Progressive, 46

Query

alter, 39

derived, 36

fork, 40

Join, 35

latency, 45

lineage, 36

steering, 39, 45, 46, 155

R, 49

Random samples

with replacement, 31

without replacement, 30

Real-time, 23, 27, 27, 68

Recall, 156

Reproducibility, 124, 157, 166

Roles, 109

explorer, 111, 113

observer, 111

searcher, 111, 112

Roll-up, 39

Sampling, 30, 35, 70, 97

bi-level, 43, 44

bias, 107

block-level, 43

mini-batch, 31

reservoir, 44

with replacement, 31

without replacement, 30, 38

Satisficing, 165

Scalability, 70, 80

number of categories, 70

number of dimensions, 70

number of items, 70

Search/gather, 17, 21, 157

Sensemaking, 164

Sequential analysis, 101

Server log, 5

Short-term memory, 106

Shuffling, 30, 35, 43, 62, 151, 152, 172

Significance testing, 100

Speculative

analysis, 90, 152, 160

exploration, 41, 87, 90, 164

query, 41

Stability, 12, 20, 58, 64, 80

State, 38

Static data, 17

Stationary, 21, 30

Stationary process, 62, 145, 151, 152, 156, 157, 167

Steering, 2, 28, 39, 45, 46, 52, 69, 109, 111, 113, 151, 152, 179

hyperparameter, 79

query, 79

Strategy, 50, 64

Streaming, 14, 17, 23, 26

Summarization, 72

Summary statistics

algebraic, 68

distributive, 68, 97

holistic, 68, 97

Synopse, 52, 60, 68

quantile, 52

t-SNE, 47, 62, 63, 72, 85, 86, 90, 95, 96, 98, 99, 104, 166, 167

Table, 49

Join, 155
Telemetry, 5

Tensors, 49

Time engineering, 108, 110

Time window, 26, 28, 52

Time-quality trade-off, 1, 48, 102,
130, 142, 149, 151, 157,
160, 173, 178

Uncertainty, 23, 28, 52, 69, 73, 77,
85, 87–89, 92–100, 103,
105–107, 156

Stability, 100

User

focus, 109

roles, 109, 111

tasks, 109

Vectors, 49

Visual

quality, 75, 77

stability, 73

Visualization

Aggregation, 70

overplotting, 70

overview, 70

scalability, 70

summarization, 72

What-if, 113
Progressive Data Analysis

We live in an era where data is abundant and growing rapidly; databases storing big data sprawl past memory, reach computation limits, and become increasingly distributed. Engineers are designing new hardware and software systems, with new storage management and predictive computation, to sustain this growth. Yet, while good for data at scale, these infrastructures do not support exploratory data analysis (EDA) effectively. EDA allows analysts to make sense of data with little or no known model. It is essential in many application domains, from network security and fraud detection to epidemiology and preventive medicine. Data exploration is done through an iterative loop where analysts interact with data through computations that return results, usually shown with visualizations, which in turn are interacted with by the analyst again. EDA calls for highly responsive systems: at 500 ms, users change their querying behavior; past five or ten seconds, users abandon tasks or lose attention. To address this problem, a new computation paradigm has emerged in the last decade: *Progressive Data Analysis*.

This book is an introduction to this new paradigm. It explains the main scientific and technical benefits of performing complex data analysis progressively on large data. It also introduces the challenges that it raises to become fully usable. These important issues concern research fields that are traditionally separate in computer science: databases, scientific computing, machine learning, visualization, statistics, and human-computer interaction. The book ends with a research agenda to help the scientific community converge on key research questions.

ISBN: 978-3-03868-270-7

<figure><img src="image_111.png" /></figure>

9 783038 682707

<figure><img src="image_112.png" /></figure>