# Benefits and Trade-Offs of Different Model Representations in Decision Support Systems for Non-expert Users

Francisco Gutiérrez, Xavier Ochoa, Karsten Seipp, Tom Broos, Katrien Verbert

<figure><img src="image_1.png" /></figure>

## To cite this version:

Francisco Gutiérrez, Xavier Ochoa, Karsten Seipp, Tom Broos, Katrien Verbert. Benefits and Trade-Offs of Different Model Representations in Decision Support Systems for Non-expert Users. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.576-597, 10.1007/978-3-030-29384-0_35. hal-02544619

HAL Id: hal-02544619

https://inria.hal.science/hal-02544619

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Benefits and Trade-offs of Different Model Representations in Decision Support Systems for Non-expert Users

Francisco Gutiérrez¹, Xavier Ochoa², Karsten Seipp¹, Tom Broos¹, and Katrien Verbert¹

¹ KU Leuven, Department of Computer Science
Celestijnenlaan 200A, BE-3001 Leuven, Belgium
{francisco.gutierrez, katrien.verbert, tom.broos}@kuleuven.be,
karsten.seipp@gmail.com

² NYU Steinhardt, New York, New York, United States
xavier.ochoa@nyu.edu

**Abstract.** Researchers have reported a lack of experience and low graph literacy as significant problems when making visual analytics applications available to a general audience. Therefore, it is fundamental to understand the strengths and weaknesses of different visualizations in the decision-making process. This paper explores the benefits and challenges of an intuitive, a compact, and a detailed visualization for supporting non-expert users in two different decision-making contexts. Using objective and subjective means proposed by earlier work, we determine the benefits and trade-offs of these visualizations for different task complexity levels. We found that while an intuitive visualization can be a good choice for easy level and medium level tasks, hard level tasks are best supported with a richer, yet visually more demanding visualization.

**Keywords:** Visual Analytics · Laymen · Graph literacy · Decision-making

# 1 Introduction

In our daily lives, we may consider multiple risks when planning a financial investment, a project strategy, or when finding a clean and safe place to live. Visual analytics offers a promising approach to facilitate such decision-making tasks. Thomas and Cook [60] define visual analytics (VA) as the science of analytical reasoning facilitated by interactive visual interfaces. In its core, VA supports this analytic reasoning with automated visual data analysis [32]. VA tools can provide control over a variety of settings, interactive discovery, exploration, and understanding of real-world complex systems [5]. Moreover, VA gradually supports users in their sense-making process, allowing them to gauge the effects of different parameters and to gain more insight with each step of the human-computer dialogue [66].

While the interactive steering of algorithms allows insight into massive data sets, such as social data streams [18] and geospatial data [4], there is a need for
research concerning an adequate presentation of such analysis with regards to users inexperienced with visual analysis [14] or unfamiliar with the given visual representation [37]. In the remainder of this paper, we refer to this group of users as *non-expert users*.

Despite the growing needs of non-expert users to control complex analytic processes, little work has been done to support them. Examples include researchers in the humanities who want to apply analysis techniques to large text corpora [26] and non-expert users making decisions about storm forecasts [45]. Different visual representations might have different levels of “visual literacy” that impact the cognitive activities of users [9][38]. For instance, simple and intuitive visualizations may be easier to understand, but general ones might offer more detail and insight into the underlying data and processes at the cost of “representational compatibility” [55]. Although much effort has been geared towards finding the most suitable representation of different types of data [64], researchers report challenges that non-expert users face when using these interactively in VA [23]. To address this issue, we investigate the strengths and weaknesses of different representations to support non-expert users in decision-making. Our research questions are as follows:

1. Regarding objective factors such as accuracy and efficiency, *how do the intuitive, detailed, and compact visualizations relate to the task difficulty?*

2. Regarding subjective factors such as usability and uncertainty, *what are the benefits and challenges of intuitive, detailed, and compact visualizations in an interactive decision-making process?*

To answer these questions, we designed and evaluated an intuitive, a detailed and a compact visualization to support non-expert users with their decision-making under uncertainty. Using subjective and objective means, we provided a set of tasks to the participants and evaluated the three visualizations in two contexts in two separated studies. The study focuses on stock prices investment, using market indicators to support non-expert users.

# 2 Background

## 2.1 Decision Support and Visualization

Decision support systems literature has modelled decision-making as an iterative process of problem recognition, perspective development, perspective synthesis, actions, and results [42]. Visualization plays a crucial role to support the development of these multiple perspectives.

More specifically, visualization facilitates human decision-making by providing structured views about information pertaining to the decision-making problem space [51]. Visualization is a well-established method to support decision-making in a wide variety of domains. Verbert et al. [62] showed that accuracy of recommender systems that suggest relevant items to a user increases with a set-based cluster map representation compared to a traditional ranked list of
recommendations. Speier and Morris [57] showed that particularly when the so-
lution set is large, visual query interfaces help users with greater effectiveness
than text-based interfaces. Recent studies show how visualization affects and
supports user decision-making in the context of financial services [6].

Decision-making usually relies on perception-based information [3]. However, adequate interpretations for perceptions may not always be easy to find under uncertain situations, and thus defining models of this outstanding capability is a difficult, yet a highly promising research area [3]. More specifically, visualization facilitates human decision-making by providing structured views about models related to the decision-making problem space [51]. When making decisions in the real world, information visualization not only helps decision-making, but also offers a means of knowledge creation, as well as an appropriate communication channel [2]. Several application areas have been using visual techniques, including health-care [1], supply chain management [43] and financial services [6].

However, there is a lack of knowledge about which visualization and inter-
action techniques work best for particular settings and particular users [20].
Also, research has shown that the choice of representation of data influences
decision-making outcomes [56]. For instance, Gettinger et al. [13] found that
different visual representations have an impact on the decision-making process
when giving simple and complex tasks to the participants. We are interested in
researching further the factors that influence decision-making, such as the effects
of the visualization context, uncertainty representation and trust [27].

## 2.2 Visual Analytics

In recent years, the importance of Visual Analytics to support decision-making
has grown [32]. Visual analytics extends interaction of traditional information
visualization techniques with facilities for updating, steering and improving the
analytic processes. The key objective is to incorporate feedback from end-users
to improve an automatic analysis process.

Examples of VA tools in decision making include the work of Senaratne et al. [49] that explores the use of a VA approach to tackle the inherent uncertainty in urban mobility patterns. Data collected from citizens' mobile communication is used to help to determine the urban dynamics of a city. Afterwards, city planners can use these tools to improve decision-making processes. Moreover, Höferlin et al. [22] introduced a VA approach to analyze video content supported by the communication of uncertainty generated from computer vision feature extraction to end-users. Goda and Song [15] presented a framework for visualizing Tsunami risks using VA techniques to support decision-making in emergency response capability. Their work emphasizes the importance of uncertainty modeling and visualization. The framework was developed through a case study for the 2011 Tohoku tsunami. Other examples include non-expert interpretations of hurracaine prediction [45], climate change data [19], and uncertainty in climate prediction [59]. Visual analytics is also a useful tool to improve understanding of data in predictions [34]. For instance, Ming et al. [41] introduced an interac-
tive visualization technique to support non-expert users in machine-learning to
explore and understand predictive models.

Several researchers have compared different visualizations of analysis out-
comes and uncertainty associated to these outcomes to support decision-making
for non-experts. Ibrekk and Morgan [25] compared nine different visualizations,
including error bars, pie chart and density functions, to communicate uncer-
tainty to users with no experience in statistical analysis. The authors suggest to
use a combination of cumulative and probability distribution functions (PDF) to
avoid misinterpretation. More recently, Greis et al. [16] compared different repre-
sentations of uncertainty for non-expert users. Results indicate that even though
a PDF plot is the best way to communicate uncertainty information, qualitative
factors are also important to consider in representations for decision-making. Dy-
namic visualizations have been researched as well. Hullman et al. [24] compared
the effects of an animated visualization against two static visualizations, and
measured levels of difficulty by considering one and two variables in the model.
Even though accuracy was similar in all representations for single-variable dis-
tributions, participants tended to be more accurate with their judgments when
using the animated visualization with two variables. Tak et al. [58] compared
seven visual representations of uncertainty for non-expert users, including solid
and dashed borders, colour-band, gradient, thinning and random lines, and error
bars. Results indicate that the visual representation of uncertainty affects the
perceived certainty of participants. However, further exploration is required to-
wards the use of other visual variables such as the use of colour and uncertainty
range.

In this paper, we build on this earlier research and focus specifically on the
design and evaluation of different types of visualizations (*intuitive, compact,*
* *detailed*) for non-expert users. We extend the work of [58] by including the
uncertainty range in different representations. As in work of [13], we consider
different levels of problem complexity. We evaluate different aspects of the utility
and effectiveness of different designs, including ease of use and visual appeal as
identified by [16]. We discuss these evaluation metrics in the next section.

## 2.3 Evaluation Metrics

Metrics to evaluate VA applications comprise a variety of criteria. However,
definitive, generalizable guidelines do not appear to exist. Therefore, we ana-
lyzed previous work to define a set of objective and subjective metrics that may
be considered useful in measuring an application’s adequacy for supporting the
decision-making process. Yang et al. [65] evaluated information graphics based
on user preference and their capability to provide different levels of insight. While
they could establish a link between information type and presentation, similar
to Wongsuphasawat et al. [64], they did not investigate how well the represen-
tations would perform as part of an interactive analysis in a VA application.
Mckenzie et al. [40] considered objective measures such as *time spent* and *accu-*
*racy*, whereas Bertini et al. [47] measured the *actions* in an uncertainty-aware
interactive interface.
<table><caption>**Table 1.** List of objective and subjective measures considered.</caption><thead><tr><th>Metric</th><th>Question</th><th>Ref.</th></tr></thead><tbody><tr><td colspan="3"><strong>Objective measures</strong></td></tr><tr><td>Accuracy</td><td>How does precision compare across the visualizations?</td><td>[40]</td></tr><tr><td>Time spent</td><td>Is the use of the visualization resulting in longer or shorter decision times?</td><td>[40]</td></tr><tr><td>Actions</td><td>How many interactions are required to solve the tasks?</td><td>[47]</td></tr><tr><td colspan="3"><strong>Subjective measures, usability-related</strong></td></tr><tr><td>Visual appeal</td><td>The visualization was visually appealing.</td><td>[16]</td></tr><tr><td>Ease of use</td><td>The visualization was easy to use.</td><td>[16]</td></tr><tr><td>Suitability</td><td>The visualization was suitable for solving the tasks.</td><td>[50]</td></tr><tr><td>Likability</td><td>The preference towards the uncertainty visualization method.</td><td>[50]</td></tr><tr><td>Task difficulty</td><td>Solving the tasks using the visualization was easy.</td><td>[65]</td></tr><tr><td colspan="3"><strong>Subjective measures, uncertainty-related</strong></td></tr><tr><td>Trust</td><td>The visualization was trustworthy.</td><td>[30]</td></tr><tr><td>Confidence</td><td>I felt confident using the visualization.</td><td>[11]</td></tr><tr><td>Credibility</td><td>The visualization was credible.</td><td>[39]</td></tr><tr><td>Understanding</td><td>The visualization was easy to understand.</td><td>[16]</td></tr><tr><td>Accuracy</td><td>I find the accuracy of this visualization acceptable.</td><td>[16]</td></tr></tbody></table>

When comparing visual representations, Senaratne et al. [50] ranked visual representations towards their *suitability* according to the user domain. They explored metrics such as *performance* and *likability*, defined as the preference towards the uncertainty visualization method. Using a set of tasks, Lee et al. [36] used objective measures such as *time spent* and, as the work of Greis et al. [16], they used subjective measures such as *ease of use*, *visual appeal* and *easiness to understand*. To evaluate uncertainty factors, Kay et al. [31] proposed a questionnaire to evaluate *acceptance of accuracy* in visual analysis, considering *ease of use* and *intent to use*. [46] defined guidelines towards uncertainty propagation in VA systems, indicating that *trustworthiness* is a critical factor towards uncertainty awareness. Deitrick et al. [11] evaluated different uncertainty representations and their influence on the *confidence* of participants decisions. To conceptualize a framework for uncertainty, MacEachren et al. [39] defined *credibility* as an important subjective measure that has an impact on the user's decision-making and analysis. We compiled a list of objective and subjective measures in Table 1.

## 3 Study Design

In this study, we collected financial data and implemented intuitive, compact and detailed model representations to support financial decisions for non-expert users<fnref n="3" />.

<footnote n="3">³ https://github.com/FranciscoGutierrez/stockprices</footnote>
## 3.1 Design Motivation

As indicated above, there is a lack of knowledge about which visualization and interaction techniques work best for particular settings and particular users [20]. Research has also shown that the choice of representation of data influences decision-making outcomes [56]. We are interested in researching further the factors that influence decision-making, such as the effect of the complexity of the visualization, uncertainty representation and trust. Overall, our decision motivations follow Shneiderman's visual information-seeking mantra [52]:

- With a **detailed representation**, we aim to provide a general context for understanding the predicted data. A detailed visualization addresses the representation of uncertainty considering the data used in the model.

- A **compact representation** provides a close-up, allowing to examine the distribution of data in limited screen spaces and high data complexity situations, as viewing every detail might be impractical. This visualization provides focused insight into a particular risk, clearly showing its spread and uncertainty. Yet, providing detail for a single factor comes at the price of depicting relationships between multiple factors.

- An **intuitive representation** is a visualization that is commonly used and familiar by the end-user to resolve tasks. These representations might include a minimal depiction of uncertainty and might not explain a model in detail.

A typical scenario for decision-making is the investment in stocks. Such a task is subject to uncertainty due to many variables that are involved. We used data related to stock prices to compare different model representations that support users in their decision-making while investing in stocks.

**What and When: intuitive representation (time series).** Investing capital in stocks requires to see the behavior of the company stock price in the past. Typically, stock prices fluctuate a lot over time, partly depending on different external factors, such as worldwide news, oil price, and world economy. For the intuitive representation, we asked a financial expert about typical time series visualization. Based on this expert feedback, we designed a time series visualization showing the last year historic fluctuation of the stock value and a next year forecast. The time-series visualization is suited for answering simple questions concerning aspects of what and when.

**Why: detailed representation (scatter plot).** A detailed visualization of the prediction model allows users to investigate why and how a prediction is made. By inspecting the variables in the model, users can identify possible sources of uncertainty. Differences between the distributions of observations can be depicted and users can see with high precision how data is being correlated [28]. As the approach requires a certain degree in graph literacy, it is important to investigate when and how such a representation is appropriate to support decision-making for non-expert users.

**How: compact representation (dot plots).** The compact representation used probability distribution functions (PDF), as also used by [25] and [29]. The visualization focuses on showing a clear representation of future events
to happen showing discrete outcomes. The dot plot is less detailed than the scatter plot representation, as it does not represent the observations and model. However, it does show a clear representation of probability outcomes. The most likely predicted value was shown above the mean of the distribution, while the predicted value was displayed below the visualization.

The visualization of uncertainty is important to correctly interpret an algorithm's prediction [53]. In our earlier work [48], we investigated quantitative and qualitative performance of different visual variables, such as opacity, blur, contour lines, grid size, and texture, for representing uncertainty. We found that overall opacity performed best, followed by texture. Therefore, we chose a change in opacity in the three visualizations for depicting uncertainty.

## 3.2 Application Design

We created a set of visualizations to represent the predicted stock price for different companies. A different visualization was used for each of the three study applications resulting in three identical applications, one with a set of six intuitive visualizations, a second with a set of six detailed visualizations and a third with a set of six compact visualizations.

## 3.3 Data

A typical scenario for decision-making is the investment in stocks. Such a task is subject to uncertainty due to many of the variables that are involved, (e.g., consumer trust, worldwide news, economic growth, Dow Jones index). We used stock prices data to compare different model representations that support non-expert users in their decision-making process while investing in stocks.

To simulate a real scenario we asked for advice from the financial expert. Based on experience, the financial expert selected six companies from the Dow Jones share that had a similar behaviour in their stock prices: Disney (NYSE: DIS), Home Depot (NYSE: HD), Apple (NASDAQ: AAPL), Chevron (NYSE: CVX), IBM (NYSE: IBM), and McDonalds (NYSE: MCD). We used the Quandl API [44] to get the current and one-year historical records of stocks from the selected companies. With the advice of the financial expert, we collected tweets mentioning the selected companies for topics such as worldwide news, and consumer trust that might serve as an indicator of the stock market [7]. We aimed for tweets containing the following hashtags and keywords: "news", "\#news", "trust", "confident", "confidence". Then, we used the sentiment analysis API [54], to assign a score from negative (0) to positive (1).

## 3.4 Model

To estimate the stock price, we built a simple regression model:

$$ \textit{StockPrice} = CI_i\beta_0 + \beta_1 \qquad (1) $$
where *CI* is a compound indicator, *β*₀ is the slope of the line and *β*₁ is the *Stock Price* intercept. *CI* is based on the average normalized values of the market indicators: consumer trust, worldwide news, economic growth, and the Dow Jones index. This allowed users to adjust the market indicators using sliders in the user interface, updating the predictions in the visualization section. Note that we present this simplified forecasting model solely for the purpose of demonstration.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Layout used in the applications. a) The settings section shows sliders for adjusting the variables. b) Visualization section where predictions are shown. c) Questions section that users have to answer together with the visuals.</figcaption></figure>

## 3.5 Interaction

We divided the application into three main sections: settings, visuals and questions, see Figure 1. The settings section used the following market indicators: consumer trust, worldwide news, economic growth, and Dow Jones index. The application presented the market indicators as checkboxes with sliders, see Figure 2 (left). Selecting a checkbox determines whether the model considers the market indicator as a variable in the prediction. Each slider enables the users to set the level of importance of the variable in the model, according to their beliefs.

When the user selected variables or changed their importance with sliders, the visualization outcome was updated instantly, see Figure 1b. Next, the questions section, see Figure 1c, shows a set of easy-to-hard questions reviewed by the financial expert. The users had to interact with a combination of variable settings to respond to all the questions.

## 3.6 Visualization

The intuitive, compact and detailed visualizations were separated in three different applications, see Figure 2. Each of the visualizations represented stock price predictions of the same six companies.
<figure><img src="image_4.png" /><figcaption>Fig. 2. The three different applications used in the financial decisions study: a) intuitive, b) detailed, and c) compact.</figcaption></figure>
**Figure 2a: intuitive visualization (time-series).** In this visualization, the stock prices predictions were shown using time-series. We implemented a line chart to show the variation of the stock price divided in two sections: historical and forecast. The historical section depicted the stock prices from the past 12 months as historical data. The forecast section showed the expected predicted value for the next 12 months. We also showed the value of the last stock price for the historical data, the value of the forecasted stock price and the percentage difference between these two values, coloured by red (negative) or green (positive). In addition, we showed the prediction interval for such prediction. We used an opacity scale that indicated the likelihood for the predicted value from unlikely (light grey) to likely (dark grey). In other words, the opacity scale highlighted the quantiles of the predicted probability distribution and showed the range of accuracy for the value. The visualization was updated immediately after when the user adjusted the importance of different variables in the settings section.

**Figure 2b: detailed visualization (scatterplots).** The visualization consisted of the plot of the regression model together with the observations of trust, worldwide news, economic growth and the Dow Jones index. The spread of stock prices of a variable was mapped to the X-axis, the corresponding distribution of the variables describing the economic factors to the Y-axis and then plotted into the coordinate system. Different colours were used to distinguish between the variables. The predicted value for the stock price was written next to the visualization at the Y-axis coordinate of the corresponding point of the regression line.

**Figure 2c: compact visualization (dot plots).** We used opacity in the dots to represent the accuracy in the forecast. The most likely predicted stock value was shown above the graph. The percent differences were displayed below. Colour was used to depict increase (green) or decrease (red) of the stock price in all visualizations.

## 3.7 Study Formulation

The financial expert analyzed the data, the model and the visualizations to create a set of 20 candidate questions for the main study. We conducted a preliminary study in Amazon Mechanical Turk (AMT), where we evaluated the 20 questions with ten participants (10 males) with no expertise with data analysis or visualization. Based on the number of actions required by the participants to answer the questions, we ranked them and selected the top 12 questions. After, we organized the questions into three different categories (easy, medium and hard).

## 3.8 Main study

We recruited 360 participants on AMT, 120 per application. To qualify for the study, users had to indicate that they had no professional experience in data analysis and that they were reasonably proficient in the English language. Also,
<table><caption>**Table 2.** Final groups of participants from the user study</caption><thead><tr><th>group</th><th># participants</th><th>gender</th><th>mean age</th><th>SD</th></tr></thead><tbody><tr><td>Intuitive</td><td>98</td><td>36F, 61M, 1 did not say</td><td>32.3</td><td>8.9</td></tr><tr><td>Detailed</td><td>84</td><td>29F, 55M</td><td>34.8</td><td>10.5</td></tr><tr><td>Compact</td><td>83</td><td>28F, 54M, 1 did not say</td><td>32.5</td><td>9.47</td></tr></tbody></table>

participants were required to have a minimum approval rate of 95%. To catch “gaming” users that simply clicked through the questions, we included five “gold standard” questions randomly into our task list [63]. These were simple questions, such as “What is 1 + 2?”, which used similar controls as the “normal” tasks. In addition to these, we added three training questions to familiarize users with the application. We deemed this necessary as a pilot study showed low performance at the beginning with a learning effect. Each user could only participate once and received a minimum payment of $0.50. To motivate users to try harder, we offered bonuses. For answering at least 66% of questions correctly, users received $2.50, for more than 80%, they received $3.50.

After a short tutorial, the application presented the training and study questions. The average time of the study for each participant was 21 minutes. At the start of each task, the visualization and controls were hidden, with only the question and a “start” button being visible in the middle of the screen. Upon clicking “start”, the visualization, controls, and questions became visible. Recording started when users clicked “start” and ended when they clicked “submit”. The following properties were recorded: time spent, number of actions, and the answer given. After users had completed all tasks, they were invited to answer questions regarding usability: *visual appeal, ease of use, suitability, likability, ease of tasks* and questions related to uncertainty: *trust, confidence, credibility, understanding, accuracy representation*, presented in Table 1. Answers were given on a five-point Likert scale, from strongly disagree (1) to strongly agree (5). Users who had failed more than one of the gold questions, had less than three correct answers or did not finish the evaluation were removed, leaving the groups presented in Table 2.

# 4 Results

We ranked the answers of the participants by their accuracy and grouped them in three levels of difficulty, see Table 3. Based on this ranking, we assigned easy, medium and hard levels of difficulty to each question. Also, to further understand the usage of the interface and answer our research questions, we calculated the participants efficiency. Frøkjær et al. [12] defined efficiency as the relation between the accuracy with which users achieve certain goals and the resources expended in achieving them.

The evaluation of VA applications is a big challenge in the visual analytics field [61]. So far, evaluation guidelines are not well established. However, accuracy and efficiency are amongst the most accepted criteria [46]. Moreover, VA
<table><caption>**Table 3.** Questions by level of difficulty. We show the mean accuracy from all participants.</caption><thead><tr><th>Diff.</th><th>Question</th><th>Acc.</th></tr></thead><tbody><tr><td rowspan="4">easy</td><td>If you consider all four factors to be moderate (50%), which company has the least certain stock price value?</td><td>1.00</td></tr><tr><td>If you consider all four factors to be good (75%), which stocks are expected to grow more than 5%?</td><td>0.88</td></tr><tr><td>Consumer trust is moderate (50%) and economic growth is very good (100%): which stock loses more than any other?</td><td>0.87</td></tr><tr><td>If you consider all four factors to be very bad (0%), which is the most profitable stock to invest?</td><td>0.83</td></tr><tr><td rowspan="3">medium</td><td>You expect a very good consumer trust (100%) and a very bad Dow Jones index (0%). McDonald's loses more than (...)</td><td>0.80</td></tr><tr><td>You expect IBM's stock to be exactly $127.91 and you only consider one factor. Which factor needs to be very good (100%) to confirm your expectation?</td><td>0.79</td></tr><tr><td>Economic growth and consumer trust are moderate (both 50%). What is the value for the Dow Jones index when the chevron stock is $110.11?</td><td>0.78</td></tr><tr><td rowspan="5">hard</td><td>You have very precise expectations, consumer trust at 39%, worldwide news at 68%, economic growth at 38% and the Dow Jones index at 62%. The (...) stock is expected to gain more than (...) but less than (...).</td><td>0.74</td></tr><tr><td>You expect the economic growth to be moderate (50%). How does the Dow Jones index have to be, so that the growth of apple is exactly the opposite of the growth of IBM?</td><td>0.68</td></tr><tr><td>Consumer trust is moderately bad (25%), economic growth and the Dow Jones index are good (75%). What is the value for the worldwide news when the home depot's stock is $129.99?</td><td>0.58</td></tr><tr><td>Worldwide news are very good (100%). Which two companies have the highest predicted stock value?</td><td>0.46</td></tr><tr><td>If you discard all other indicators, the difference between very bad (0%) and very good (100%) worldwide news has the strongest effect on (...).</td><td>0.37</td></tr></tbody></table>

studies have used efficiency in the past to measure human-computer interaction techniques in the decision-making process. As such, efficiency is a relevant measurement in our study, as it can tell us how the visual representation is used to efficiently communicate the outcome to the end-user [33].

We define efficiency as follows:

$$ efficiency = \frac{\sum_{j=1}^{R} \sum_{i=1}^{N} a_{i,j}}{\sum_{j=1}^{R} \sum_{i=1}^{N} r_{i,j}} \quad (2) $$

where N is the total number of tasks; R is the number of users; $a_{i,j}$ is the accuracy of task i by user j; $r_{i,j}$ is the amount of resources expended by the user
$j$ to solve task $i$. Resource is defined as the actions per minute performed by the user.

Figure 3 shows the mean values for accuracy and efficiency, using 95% CI, and sorted by level of difficulty. Factorial ANOVA tests were conducted to compare the accuracy of participants, and their efficiency with the visualizations and levels of difficulty. As a follow up to the findings, a post-hoc Tukey HSD test was used. Also, we report the results in Table 4.

**Accuracy:** participant's accuracy was significantly different for all levels of difficulty ($p < 0.001$, $\eta_p^2 = 0.23$). In general, participants tended to be significantly ($p < .01$, $\eta_p^2 = 0.018$) less accurate while using the *compact* visualization. No other significant differences were found.

- *Easy tasks:* no significant differences were found. Participants tended to be more accurate with the intuitive (*mean* = 92%) and the detailed (*mean* = 91%) visualizations, followed by the *compact* (*mean* = 85%) visualization.

- *Medium tasks:* participants were significantly more accurate ($p < .05$, $\eta_p^2 = 0.005$) when using the intuitive visualization (*mean* = 82%). They were less accurate with the detailed (*mean* = 78%) and *compact* (*mean* = 69%) visualizations.

- *Hard tasks:* participants were significantly more accurate ($p < .05$, $\eta_p^2 = 0.005$) while using the detailed visualization (*mean* = 60%). They were less accurate with the intuitive (*mean* = 52%) and *compact* (*mean* = 47%) visualizations.

**Efficiency:** efficiency was significantly different for all levels of difficulty ($p < 0.01$, $\eta_p^2 = 0.035$) and visualizations, ($p < 0.001$, $\eta_p^2 = 0.24$). In general, participants were significantly more efficient ($p < 0.001$, $\eta_p^2 = 0.031$) while using the *detailed* and *intuitive* representations. No other significant differences were found.

- *Easy tasks:* no significant differences were found. Participants tended to be more efficient with the intuitive visualization (*mean* = 34%), followed by the *detailed* (*mean* = 29%) and the *compact* (*mean* = 11%) visualizations.

- *Medium tasks:* participants were significantly more efficient ($p < .001$, $\eta_p^2 = 0.005$) using the intuitive visualization (*mean* = 34%). They were less efficient with the detailed (*mean* = 18%) and *compact* (*mean* = 8%) visualizations.

- *Hard tasks:* participants were significantly more efficient ($p < .001$, $\eta_p^2 = 0.005$) using the detailed visualization (*mean* = 60%). They were less efficient with the intuitive (*mean* = 20%) and *compact* (*mean* = 5%) visualizations.

A Kruskal Wallis U non-parametric test was used to find a difference in the responses related to usability and uncertainty. Mann-Whitney tests were used as a follow-up in the findings.
<table><caption>**Table 4.** Results of the study. Objective measures are presented with the mean, subjective measures are presented with their mean and median. Best-scoring values are highlighted in **bold**.</caption><thead><tr><th>Metric</th><th>Difficulty</th><th>Intuitive</th><th>Detailed</th><th>Compact</th></tr></thead><tbody><tr><td rowspan="3">Accuracy</td><td>Easy</td><td><strong>92%</strong></td><td>91%</td><td>85%</td></tr><tr><td>Medium</td><td><strong>82%*</strong></td><td>78%</td><td>69%</td></tr><tr><td>Hard</td><td>52%</td><td><strong>60%*</strong></td><td>47%</td></tr><tr><td rowspan="3">Efficiency</td><td>Easy</td><td><strong>34%</strong></td><td>29%</td><td>11%</td></tr><tr><td>Medium</td><td><strong>34%***</strong></td><td>18%</td><td>8%</td></tr><tr><td>Hard</td><td>20%</td><td><strong>23%***</strong></td><td>5%</td></tr><tr><td><strong>Usability</strong></td><td></td><td><strong>Mdn</strong></td><td><strong>Mdn</strong></td><td><strong>M</strong></td><td><strong>Mdn</strong></td></tr><tr><td>Visual appeal</td><td></td><td><strong>5</strong></td><td>4.46</td><td>4</td><td><strong>4.47</strong> 4</td><td>4.46</td></tr><tr><td>Ease of use</td><td></td><td><strong>5*</strong></td><td><strong>4.43</strong></td><td>5</td><td>4.38</td><td>4</td><td>4.37</td></tr><tr><td>Suitability</td><td></td><td><strong>5***</strong></td><td>4.55</td><td><strong>5**</strong></td><td><strong>4.58</strong></td><td>4</td><td>4.35</td></tr><tr><td>Likability</td><td></td><td><strong>5</strong></td><td><strong>4.5</strong></td><td>4</td><td>4.46</td><td><strong>5</strong></td><td>4.46</td></tr><tr><td>Tasks were easy</td><td></td><td><strong>5***</strong></td><td><strong>4.28</strong></td><td>4**</td><td>4.16</td><td>3</td><td>4.18</td></tr><tr><td><strong>Uncertainty</strong></td><td></td><td></td><td><strong>Mdn</strong></td><td><strong>Mdn</strong></td><td><strong>M</strong></td><td><strong>Mdn</strong></td></tr><tr><td>Trust</td><td></td><td><strong>4***</strong></td><td>4.15</td><td>4</td><td>4.1</td><td>3</td><td><strong>4.21</strong></td></tr><tr><td>Confidence</td><td></td><td><strong>5***</strong></td><td>4.41</td><td>4.5***</td><td><strong>4.42</strong></td><td>3</td><td>4.35</td></tr><tr><td>Credibility</td><td></td><td>4</td><td>4.23</td><td>4</td><td>4.15</td><td>4</td><td><strong>4.31</strong></td></tr><tr><td>Understanding</td><td></td><td><strong>5***</strong></td><td><strong>4.41</strong></td><td><strong>5</strong></td><td>4.26</td><td>4</td><td>4.3</td></tr><tr><td>Accuracy</td><td></td><td><strong>5***</strong></td><td><strong>4.49</strong></td><td>4</td><td>4.23</td><td>4</td><td>4.3</td></tr></tbody></table>

* significant at *p* < .05; ** significant at *p* < .01;

*** significant at *p* < .001.

- **Visual appeal:** no significant differences were found. Participants preferred the intuitive (*Mdn* = 5) visualization, followed by the compact (*Mdn* = 4) and the detailed (*Mdn* = 4) visualizations.

- **Ease of use:** participants had a significant preference (*p* < .05) towards the intuitive (*Mdn* = 5) visualization, followed by the detailed (*Mdn* = 5) and compact (*Mdn* = 4) visualizations.

- **Suitability:** participants had a significant preference (*p* < .001) towards the intuitive (*Mdn* = 5) and detailed (*Mdn* = 5) visualizations (*p* < .01), followed by the compact (*Mdn* = 4) visualization.

- **Likability:** no significant differences were found. Participants preferred the intuitive (*Mdn* = 5) and compact (*Mdn* = 5) visualizations, followed by the detailed (*Mdn* = 4) visualization.

- **Tasks were easy:** participants had a significant preference (*p* < .001) towards the intuitive (*Mdn* = 5) and the detailed (*Mdn* = 4) visualizations (*p* < .01), followed by the compact (*Mdn* = 3) visualization.

- **Trust:** participants had a significant preference (*p* < .001) towards the intuitive (*Mdn* = 4) visualization, followed by the compact (*Mdn* = 4) and the detailed (*Mdn* = 3) visualizations.

- **Confidence:** participants had a significant preference (*p* < .001) towards the intuitive (*Mdn* = 5) and the detailed (*Mdn* = 4.5) visualizations (*p* < .001), followed by the compact (*Mdn* = 3) visualization.
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Results regarding actions, time spent, actions per minute, accuracy and efficiency, grouped by easy, medium, and hard levels of difficulty.</figcaption></figure>

- **Credibility**: no significant differences were found. Participants showed a similar preference towards the intuitive, detailed and compact ($Mdn = 4$) visualizations.

- **Understanding**: participants had a significant preference ($p < .001$) towards the intuitive ($Mdn = 5$) visualization, followed by the compact ($Mdn = 4$) and the detailed ($Mdn = 4$) visualizations.

- **Uncertainty**: participants had a significant preference ($p < .001$) towards the intuitive ($Mdn = 5$), followed by the detailed ($Mdn = 4$) and compact ($Mdn = 4$) representations.

# 5 Answering the research questions

RQ1: Regarding objective factors such as accuracy and efficiency, how do the intuitive, detailed, and compact visualizations relate to the task difficulty?
The selection of a suitable visualization for the task at hand has a signifi-
cant impact on evaluation results. Results indicate that an *intuitive* visualization
works well for easy and medium-level tasks where the participants were the most
accurate and efficient. In both studies, participants that used the *detailed visual-
ization* tended to be more accurate as the level of difficulty increased, particularly
for hard tasks. Participants tended to be more accurate and more efficient while
using this representation, particularly for hard tasks. The *compact* visualization
was in general the least accurate and efficient.

RQ2: Regarding subjective factors such as usability and uncertainty, what are the benefits and challenges of intuitive, detailed, and compact visualizations in an interactive decision-making process?

The *intuitive visualization* scored best on most usability and uncertainty factors. These results indicate that the careful selection of an *easy to use* and *suitable* visualization has a strong impact towards uncertainty factors such as *trust*, *understanding* and *confidence*. Participants also indicated that the tasks seemed easy to solve and the accuracy representation was good. Participants that used the *detailed visualization* indicated the visualization was suitable for solving the tasks and tended to show a positive attitude towards the uncertainty representation. Although the *compact visualization* was rated good for factors such as understanding, accuracy representation and likability, further exploration is required on factors such as trust, confidence, and task difficulty.

# 6 Recommendations for Implementation

Visualization for non-expert users requires careful measurement of qualitative and quantitative analysis of behaviour to solve a problem. Modern approaches such as immerse analytics demand that the choice of the visualization should be progressively adapted to the complexity of the task [8]. Communication of uncertainty is essential in the decision-making and should be an integral part of visualization. Based on our findings, we give the following recommendations.

- **Detailed visualization:** we recommend the use of this visualization as the level of difficulty increases, mainly when looking for efficiency and accuracy on hard level tasks. However, one of the potential drawbacks is the responsiveness of the visualization on small screens. A detailed visualization would shine in larger spaces, where users can manipulate the variables and inspect the details as the information is updated.

- **Intuitive visualization:** the visualization scored best on most usability and uncertainty aspects and was highly accurate for easy and medium level tasks. Participants tended to be more accurate and efficient with the intuitive representation of easy and medium level tasks.

- **Compact visualization:** we recommend being cautious using a compact visualization when accuracy is a critical factor for medium and hard tasks.
A compact visualization has potential because it can show a deep insight towards explaining uncertainty. According to participant's feedback, the visualization was credible and understandable, but they were not sure when asked about their trust or confidence. Further research is required about the effects of a compact visualization in the decision-making for VA applications.

# 7 Conclusion

Exploring the challenges of VA for non-experts is a growing research field [35]. While advice on mapping types of data to an ideal representation does exist [64], information about different visualizations' qualitative and quantitative performance in an interactive decision-making process for non-experts does not. To address this research gap, we have examined the suitability of intuitive, compact, and detailed visualizations in the risk-based decision-making process for non-experts in an application facilitating the exploration of variables involved in a prediction model. By allowing the users to interactively change the importance of different variables, users could steer an algorithm predicting these aspects from a dataset.

Using objective and subjective means proposed by earlier work, we determined the benefits and trade-offs of different uncertainty representations using tasks with different difficulty levels. We found that intuitive visualizations can be a good choice for easy and medium level tasks and that a more demanding visualization may be required for hard level tasks, potentially at the cost of usability aspects. Our research extends previous work on the challenges of supporting non-experts with VA: Kwon et al. [35] reported a set of “roadblocks” inhibiting non-experts from successfully using popular VA applications. As these are commonly built for experts, their utility can be “unreachable” to non-experts due to their limited domain experience, low graph literacy, and inaccurate mental model.

Some applications of our work include the education domain where visualization of prediction outcomes plays a critical role to support the decision-making process of academic advisers [17]. Moreover, in the employment domain, interactive VA tools can be useful to visualize prediction and recommendation output and support a dialogue between job mediator and job seeker [10]. Here VA tools increase *explainability*, providing mediators with control over the information that is presented to job seekers.

Therefore, if we want to extend the user-base of VA to non-experts, we may have to take a step back from these specialized and robust applications. Instead, this endeavour may call for a new generation of closely tied applications to a user's abilities, preferences, and the difficulty of the problem to be solved. Huang et al. [23] explored the challenges of PVA and suggested that an application that is to be successful in this field needs to consider the user's context and provide “appropriate baselines to support reasoning about data”. Our work presents a step towards providing such a baseline: defining user context as the difficulty of the problem to be solved or level of insight to be gained, we have evalu-
ated the adequacy of a certain visualization for supporting the VA process of non-experts. Following Shneiderman's visual information seeking mantra [52] of increasing information density as required, we suggest increasing the complexity and capability of a model's representation as the task difficulty and the required level of insight increases.

Regarding the relationship between accuracy and uncertainty, we observed an increment of accuracy with the intuitive visualization for medium and easy tasks where participants also indicated an increase of trust and confidence. This was followed by the detailed representation where participants where significantly more accurate for hard tasks and they reported to have confidence with the visualization. However, with the compact representation, participants where less accurate and gave the lowest scores regarding uncertainty factors.

Our results and those of Kwon et al. [35] have indicated that confronting non-experts with a powerful, but unnecessary complex visualization is detrimental to solving simple problems. Instead, our results indicate that by adapting the visualization to the difficulty of the task, a favourable balance may be struck between various objective and subjective aspects and the decision-making process optimally supported. By defining strengths and weaknesses of three visualization types for problems of varying difficulty, we provided a step towards the definition of a set of visualizations that may be chosen to support non-expert users.

## 8 Limitations

We used a crowd-sourcing platform to gather data. Previous work has shown the suitability of AMT for conducting studies in the visualization domain [21,65]. Even though our interface was simple, guiding the users with questions to facilitate evaluation, and only four control variables were used to simplify interaction, results should be interpreted with caution, as studies were unsupervised. Moreover, we have detailed the accuracy results per difficulty level, however, the subjective data related to perceived uncertainty is not captured individually for the different difficulty levels. Also, while we were able to identify the benefits and challenges of the visualizations, the effect sizes are relatively small. Future work will investigate further the perceived uncertainty and examine performance in a purely exploratory environment. We will also investigate whether the findings can be extended to other domains.

## Acknowledgements

Part of this research has been supported by the KU Leuven Research Council (grant agreement no. C24/16/017 and RUN/16/003), the Research Foundation Flanders (FWO, grant agreement no. G0C9515N), and the European Union (Horizon 2020 research and innovation programme, grant agreement no. 780751).
References

1. Abidi, S.S.R.: Knowledge management in healthcare: Towards knowledge-driven decision-support services. International Journal of Medical Informatics **63**(1), 5–18 (2001)

2. Al-Kassab, J., Ouertani, Z.M., Schiuma, G., Neely, A.: Information visualization to support management decisions. International Journal of Information Technology & Decision Making **13**(02), 407–428 (2014)

3. Aliev, R.A., Huseynov, O.H.: Fuzzy Geometry-Based Decision Making with Unprecisiated Visual Information. International Journal of Information Technology & Decision Making **13**(5), 1051–1073 (2014)

4. Andrienko, G., Andrienko, N., Jankowski, P., Keim, D., Kraak, M.J., MacEachren, A., Wrobel, S.: Geovisual analytics for spatial decision support: Setting the research agenda. International Journal of Geographical Information Science **21**(8), 839–857 (2007)

5. Basole, R.C., Qamar, A., Park, H., Paredis, C.J., McGinnis, L.F.: Visual analytics for early-phase complex engineered system design support. IEEE computer graphics and applications **35**(2), 41–51 (2015)

6. Ben-Assuli, O.: Assessing the perception of information components in financial decision support systems. Decision Support Systems **54**(1), 795–802 (2012)

7. Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock market. Journal of computational science **2**(1), 1–8 (2011)

8. Bonada, S., Veras, R., Collins, C.: Personalized views for immersive analytics. In: Proceedings of the 2016 ACM Companion on Interactive Surfaces and Spaces. pp. 83–89. ISS Companion ’16, ACM, New York, NY, USA (2016)

9. Boy, J., Rensink, R.A., Bertini, E., Fekete, J.D.: A principled way of assessing visualization literacy. IEEE transactions on visualization and computer graphics **20**(12), 1963–1972 (2014)

10. Charleer, S., Gutiérrez Hernández, F., Verbert, K.: Supporting job mediator and job seeker through an actionable dashboard. In: Proceedings of the 24th IUI conference on Intelligent User Interfaces. ACM (2018)

11. Deitrick, S., Edsall, R.: The influence of uncertainty visualization on decision making: An empirical evaluation. In: Progress in spatial data handling, pp. 719–738. Springer (2006)

12. Frøkjær, E., Hertzum, M., Hornbæk, K.: Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. pp. 345–352. ACM (2000)

13. Gettinger, J., Kiesling, E., Stummer, C., Vetschera, R.: A comparison of representations for discrete multi-criteria decision problems. Decision support systems **54**(2), 976–985 (2013)

14. Gettinger, J., Koeszegi, S.T., Schoop, M.: Shall we dance? – the effect of information presentations on negotiation processes and outcomes. Decision support systems **53**(1), 161–174 (2012)

15. Goda, K., Song, J.: Uncertainty modeling and visualization for tsunami hazard and risk mapping: a case study for the 2011 tohoku earthquake. Stochastic environmental research and risk assessment **30**(8), 2271–2285 (2016)

16. Greis, M., Ohler, T., Henze, N., Schmidt, A.: Investigating representation alternatives for communicating uncertainty to non-experts. In: Human-Computer Interaction. pp. 256–263. Springer (2015)
17. Gutiérrez, F., Seipp, K., Ochoa, X., Chiluiza, K., De Laet, T., Verbert, K.: Lada: A learning analytics dashboard for academic advising. Computers in Human Behavior (2018)

18. Hao, M., Rohrdantz, C., Janetzko, H., Dayal, U., Keim, D.A., Haug, L.E., Hsu, M.C.: Visual sentiment analysis on twitter data streams. In: Visual Analytics Science and Technology (VAST), 2011 IEEE Conference on. pp. 277–278. IEEE (2011)

19. Harold, J., Lorenzoni, I., Shipley, T.F., Coventry, K.R.: Cognitive and psychological science insights to improve climate change data visualization. Nature Climate Change 6(12), 1080 (2016)

20. He, C., Parra, D., Verbert, K.: Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications 56, 9–27 (2016)

21. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp. 203–212. ACM (2010)

22. Höferlin, M., Höferlin, B., Weiskopf, D., Heidemann, G.: Uncertainty-aware video visual analytics of tracked moving objects. Journal of Spatial Information Science 2011(2), 87–117 (2011)

23. Huang, D., Tory, M., Aseniero, B.A., Bartram, L., Bateman, S., Carpendale, S., Tang, A., Woodbury, R.: Personal visualization and personal visual analytics. Visualization and Computer Graphics, IEEE Trans. on 21(3), 420–433 (2015)

24. Hullman, J., Resnick, P., Adar, E.: Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PLoS ONE 10(11), e0142444 (2015)

25. Ibrekk, H., Morgan, M.G.: Graphical communication of uncertain quantities to nontechnical people. Risk analysis 7(4), 519–529 (1987)

26. John, M., Koch, S., Heimerl, F., Müller, A., Ertl, T., Kuhn, J.: Interactive visual analysis of german poetics. In: Digital Humanities 2015 Book of Abstracts (2015)

27. Johnson, C.R., Sanderson, A.R.: A next step: Visualizing errors and uncertainty. IEEE Computer Graphics and Applications 23(5), 6–10 (2003)

28. Kay, M., Heer, J.: Beyond Weber’s Law: A Second Look at Ranking Visualizations of Correlation. IEEE Transactions on Visualization and Computer Graphics 22(1), 469–478 (2016)

29. Kay, M., Kola, T., Hullman, J.R., Munson, S.A.: When (ish) is My Bus? User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16 pp. 5092–5103 (2016)

30. Kay, M., Morris, D., Kientz, J.A., et al.: There’s no such thing as gaining a pound: Reconsidering the bathroom scale user interface. In: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. pp. 401–410. ACM (2013)

31. Kay, M., Patel, S.N., Kientz, J.A.: How good is 85%?: A survey tool to connect classifier evaluation to acceptability of accuracy. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. pp. 347–356. ACM (2015)

32. Keim, D.A., Mansmann, F., Schneidewind, J., Thomas, J., Ziegler, H.: Visual analytics: Scope and challenges (2008)

33. Keim, D.A., Mansmann, F., Thomas, J.: Visual analytics: How much visualization and how much analytics? SIGKDD Explor. Newsl. 11(2), 5–8 (May 2010)
34. Kim, Y.S., Reinecke, K., Hullman, J.: Explaining the gap: Visualizing one's predictions improves recall and comprehension of data. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. pp. 1375-1386. CHI '17, ACM, New York, NY, USA (2017). https://doi.org/10.1145/3025453.3025592, bluehttp://doi.acm.org/10.1145/3025453.3025592

35. Kwon, B.C., Fisher, B., Yi, J.S.: Visual analytic roadblocks for novice investigators. In: VAST 2011 - IEEE Conference on Visual Analytics Science and Technology 2011, Proceedings. pp. 3-11. IEEE (2011)

36. Lee, B., Robertson, G.G., Czerwinski, M., Parr, C.S.: Candidtree: visualizing structural uncertainty in similar hierarchies. Information Visualization **6**(3), 233-246 (2007)

37. Lee, S., Kim, S.H., Hung, Y.H., Lam, H., Kang, Y.a., Yi, J.S.: How do people make sense of unfamiliar visualizations?: a grounded model of novice's information visualization sensemaking. IEEE transactions on visualization and computer graphics **22**(1), 499-508 (2016)

38. Lee, S., Kim, S.H., Kwon, B.C.: Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics **23**(1), 551-560 (2017)

39. MacEachren, A.M.A., Robinson, A., Hopper, S., Gardner, S., Murray, R., Gahegan, M., Hetzler, E.: Visualizing geospatial information uncertainty: What we know and what we need to know. Cartography and Geographic Information Science **32**(3), 139-160 (2005)

40. McKenzie, G., Hegarty, M., Barrett, T., Goodchild, M.: Assessing the effectiveness of different visualizations for judgments of positional uncertainty. International Journal of Geographical Information Science **30**(2), 221-239 (2016)

41. Ming, Y., Qu, H., Bertini, E.: Rulematrix: Visualizing and understanding classifiers with rules. IEEE transactions on visualization and computer graphics **25**(1), 342-352 (2019)

42. Park, H., Basole, R.C.: Bicentric diagrams: Design and applications of a graph-based relational set visualization technique. Decision Support Systems **84**, 64-77 (2016)

43. Park, H., Bellamy, M.A., Basole, R.C.: Visual analytics for supply network management: System design and evaluation. Decision Support Systems **91**, 89-102 (2016)

44. Quandl: Quandl api (2017), bluehttps://www.quandl.com/docs/api

45. Ruginski, I.T., Boone, A.P., Padilla, L.M., Liu, L., Heydari, N., Kramer, H.S., Hegarty, M., Thompson, W.B., House, D.H., Creem-Regehr, S.H.: Non-expert interpretations of hurricane forecast uncertainty visualizations. Spatial Cognition & Computation **16**(2), 154-172 (2016)

46. Sacha, D., Senaratne, H., Kwon, B.C., Ellis, G., Keim, D.A.: The Role of Uncertainty, Awareness, and Trust in Visual Analytics. IEEE Transactions on Visualization and Computer Graphics **22**(1), 240-249 (2016)

47. Sarkar, A., Blackwell, A.F., Jamnik, M., Spott, M.: Interaction with Uncertainty in Visualisations. In: Bertini, E., Kennedy, J., Puppo, E. (eds.) Eurographics Conference on Visualization (EuroVis) - Short Papers. The Eurographics Association (2015)

48. Seipp, K., Gutiérrez, F., Ochoa, X., Verbert, K.: Towards a visual guide for communicating uncertainty in visual analytics. Journal of Visual Languages and Computing **50**, 1-18 (2019)
49. Senaratne, H., Mueller, M., Behrisch, M., Lalanne, F., Bustos-Jiménez, J., Schneidewind, J., Keim, D., Schreck, T.: Urban mobility analysis with mobile network data: A visual analytics approach. IEEE Transactions on Intelligent Transportation Systems **19**(5), 1537–1546 (May 2018)

50. Senaratne, H., Reusser, D., Schreck, T.: Usability of uncertainty visualisation methods: A comparison between different user groups (2013)

51. Shim, J.P., Warkentin, M., Courtney, J.F., Power, D.J., Sharda, R., Carlsson, C.: Past, present, and future of decision support technology. Decision Support Systems **33**(2), 111–126 (2002)

52. Shneiderman, B.: The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings 1996 IEEE Symposium on Visual Languages. pp. 336–343. VL ’96, IEEE Computer Society (1996)

53. Skeels, M., Lee, B., Smith, G., Robertson, G.G.: Revealing uncertainty for information visualization. Information Visualization **9**(1), 70–81 (2010)

54. Smedt, T.D., Daelemans, W.: Pattern for python. Journal of Machine Learning Research **13**(Jun), 2063–2067 (2012)

55. Sparrow, J.A.: Graphical displays in information systems: some data properties influencing the effectiveness of alternative forms. Behaviour and Information Technology **8**(1), 43–56 (1989)

56. Speier, C.: The influence of information presentation formats on complex task decision-making performance. Human-Computer Studies **64**(11), 115–1131 (2006)

57. Speier, C., Morris, M.G.: MIS Quarterly (3)

58. Tak, S., Toet, A., van Erp, J.: The perception of visual uncertainty representation by non-experts. IEEE transactions on visualization and computer graphics **20**(6), 935–943 (2014)

59. Taylor, A.L., Dessai, S., de Bruin, W.B.: Communicating uncertainty in seasonal and interannual climate forecasts in europe. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences **373**(2055), 20140454 (2015)

60. Thomas, J.J., Cook, K.A.: Illuminating the path: The r&d agenda for visual analytics national visualization and analytics center. In: National Visualization and Analytics Center-US. Department of Homeland Security (2005)

61. Thomas, J., Kielman, J.: Challenges for visual analytics. Information Visualization **8**(4), 309–314 (2009)

62. Verbert, K., Parra, D., Brusilovsky, P.: Agents vs. users: Visual recommendation of research talks with multiple dimension of relevance. ACM Trans. on Interactive Intelligent Systems pp. 1–46 (2016)

63. Wang, J., Ipeirotis, P.G., Provost, F.: A framework for quality assurance in crowdsourcing p. 38 (2013)

64. Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. IEEE Transactions on Visualization and Computer Graphics **22**(1), 649–658 (2016)

65. Yang, H., Li, Y., Zhou, M.X.: Understand users’ comprehension and preferences for composing information visualizations. ACM Transactions on Computer-Human Interaction **21**(1), 1–30 (2014)

66. Yi, J.S., ah Kang, Y., Stasko, J.T., Jacko, J.A.: Toward a deeper understanding of the role of interaction in information visualization. Visualization and Computer Graphics, IEEE Trans. on **13**(6), 1224–1231 (2007)