# We Should Change How We Measure User Experience in Visual Analytics Systems

Eliane Zambon Victorelli, Anne-Flore Cabouat, Emanuele Santos, Florent Cabric, Petra Isenberg

► To cite this version:

Eliane Zambon Victorelli, Anne-Flore Cabouat, Emanuele Santos, Florent Cabric, Petra Isenberg. We Should Change How We Measure User Experience in Visual Analytics Systems. EuroVA 2025 - 16th EuroVis Workshop on Visual Analytics, Jun 2025, Luxembourg, Luxembourg. 10.2312/eurova.20251095. hal-05245376

HAL Id: hal-05245376

https://inria.hal.science/hal-05245376v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# We Should Change How We Measure User Experience in Visual Analytics Systems

Eliane Z. Victorelli<fnref n="1" />,² ID, Anne-Flore Cabouat<fnref n="1" />,³ ID, Emanuele Santos<fnref n="1" />,<fnref n="4" /> ID, Florent Cabric<fnref n="1" /> ID and Petra Isenberg<fnref n="1" /> ID

**Abstract**

The evaluation of user experience in visual analytics systems is a complex and multifaceted challenge that demands a specialized approach. Traditional user experience measurement tools, including standardized questionnaires, often fail to capture the unique interactions and cognitive demands of visual analytics environments. In this position paper, we argue that it is necessary to develop a new measurement tool specifically tailored to user experience in visual analytics systems. We explore the relationship between user experience and usability and review current methods for collecting, measuring, and analyzing user experience data. We discuss the limitations of existing methods for user experience evaluations of visual analytics systems with a focus on the problematic use of current user experience questionnaires. Our research agenda outlines the key challenges for developing a dedicated evaluation instrument and proposes strategies for exploring the dimensions of user experience relevant to the visual analytics systems context. In particular, future research on a framework for developing robust and context-appropriate measurement tools that target the many important dimensions of user experience with visual analytics systems is needed.

**Keywords:** User experience, measurement scale, evaluation, visual analytics

**CCS Concepts**

* Human-centered computing → Visualization design and evaluation methods; Visual analytics;

## 1. Introduction

When we build a visual analytics system, we pay close attention to the functionalities it needs to support. Yet, to make a system successful and adopted, we also need to pay close attention to how users experience their interactions with the system [Has13]. A good experience can lead to the adoption of visual analytics systems, better analysis results, and provide support for a range of users who may not otherwise be able to analyze their data. These benefits are supported by additional lower-level advantages such as increased engagement with the artifact, tool, or system, a greater focus on tasks, or improved effectiveness [ALC17].

In the visual analytics domain, dedicated work on studying user experience (UX) is still rare, perhaps because it is a complex and multifaceted challenge that demands a specialized approach. Measuring user's experience with computing systems is unfortunately challenging because it is subjective, context-dependent, product-dependent, and evolves over time [GB12]. In the use of visual analytics systems, humans and algorithms complement each other in handling complex problems and decision-making scenarios centered around data [SSS*14]. Standard HCI UX measurement instruments, however, do not evaluate how humans experience this interplay between themselves, the underlying algorithms, data, and visualizations, nor their subjective experience of reading the visualizations

and of gaining insights or drawing conclusions from them [SSS*14].

We, therefore, argue that researchers should be cautious about using measurement instruments developed in adjacent domains as these instruments only allow a limited view of UX with visual analytics systems. We propose a research agenda to develop a UX evaluation instrument tailored to visual analytics systems. With such an instrument, we will be able to measure UX in visual analytics systems in a standardized and reliable way while considering the unique cognitive and interactive demands of advanced data analysis tools. An added benefit of engaging in this development challenge is that the work required can provide a deeper understanding of what is important to users in data analysis activities. In the following sections, we first provide a brief overview of the concept of user experience and motivate our focus on questionnaires as an evaluation method. We then describe current questionnaires used by visual analytics researchers and their limitations. Finally, we outline the anticipated challenges in creating a measuring instrument tailored to visual analytics systems, and a research agenda to address them.

## 2. Background: User Experience

This section briefly discusses user experience as a concept and motivates our focus on using questionnaires as a study methodology.

<footnote n="1">¹Université Paris-Saclay, CNRS, Inria, LISN, France ²Universidade de São Paulo, Brazil ³Telecom Paris, i3, Institut Polytechnique de Paris, France</footnote>

<footnote n="4">⁴Department of Computing, Center of Sciences, Federal University of Ceará, Brazil</footnote>
<table><caption>**Table 1:** Standard HCI UX questionnaires mentioned in visual-ization and visual analytics papers, their study focus, context of development, reference in our bibliography, and the number of times (#) it appeared in our literature search.</caption><thead><tr><th>acronym</th><th>focus</th><th>context</th><th>ref.</th><th>#</th></tr></thead><tbody><tr><td>NASA-TLX</td><td>Subjective workload</td><td>Aviation & Piloting and other tasks</td><td>[HS88]</td><td>20</td></tr><tr><td>SUS</td><td>Usability</td><td>Industrial systems</td><td>[Bro96]</td><td>19</td></tr><tr><td>UEQ</td><td>User experience</td><td>Software products</td><td>[LHS08]</td><td>4</td></tr><tr><td>IPQ</td><td>Presence</td><td>Virtual environment</td><td>[SFR01]</td><td>3</td></tr><tr><td>Attrak-Diff</td><td>Pragmatic & hedonic quality, attractiveness, behaviours, emotions</td><td>Interactive products</td><td>[HBK03]</td><td>3</td></tr><tr><td>USE</td><td>Usefulness, satisfaction, and ease of use</td><td>Software, hardware, services, and user support materials</td><td>[Lun01]</td><td>2</td></tr><tr><td>QUIS</td><td>Satisfaction</td><td>HCI</td><td>[CDN88]</td><td>1</td></tr><tr><td>TAM</td><td>User acceptance</td><td>Information systems</td><td>[Dav86]</td><td>1</td></tr><tr><td>meCue</td><td>User experience</td><td>Technical device</td><td>[MTWK17]</td><td>1</td></tr><tr><td>PXI</td><td>Functional & psychosocial consequences</td><td>Gameplay</td><td>[ASN*20]</td><td>1</td></tr><tr><td>SAM</td><td>Affective reactions</td><td>Pictures</td><td>[BL94]</td><td>1</td></tr><tr><td>TRUST</td><td>Trust</td><td>Automated systems</td><td>[JBD00]</td><td>1</td></tr></tbody></table>

## 2.1. What is user experience?

UX is a holistic concept involving many aspects, and there is no consensus on its definition. ISO defines user experience as “*a person’s perceptions and responses resulting from the use and/or anticipated use of a product, system, or service*” [ISO18]. Despite the ISO definition, UX remains a concept open to various interpretations [DCB19]. Most importantly, user experience is often discussed in conjunction with the concept of usability, and sometimes the terms are even used interchangeably [HT06]. The complexity of the concept of UX and its many ramifications, such as trust, loyalty, identity, and engagement, raises the danger that UX is not clearly approached in people and technology studies [DCB19]. A theoretical framework that could serve as a reference for constructing a comprehensive, validated measuring tool has not been identified yet. Therefore, different interpretations of UX may have led to different scopes of measure [Bev09]. Before discussing UX evaluation details, we should clarify what it encompasses in our opinion. Inspired by the work of Tullis and Albert [TA13], we propose that a holistic view of UX, which includes users’ feelings and reactions to their interactions with the system, and the ISO-defined usability goals: effectiveness, efficiency, and satisfaction is the most appropriate to consider.

## 2.2. UX evaluation using questionnaires

UX is measured using a wide range of evaluation methodologies, often expanding on traditional methods for usability evaluation: ranging from laboratory usability testing to low-cost methods such as heuristic evaluation or cognitive walk-throughs. All UX evaluation methods have advantages and can be applied at specific stages in the development of a visual analytics system and with different types of sample sizes. Hogan et al. [HHH16] cited five methods often used in information visualization research to evaluate insights and subjective experiences: observations, diaries, questionnaires,

think-aloud, and interviews. We acknowledge that qualitative methods are a good complement to a quantitative instrument when it comes to understanding the details and contextual nuances of visual analytics systems. However, in this paper, we focus on questionnaires because they provide a practical, scalable, and structured way to start exploring UX with visual analytics systems. Darin et al. [DCB19] suggest that questionnaires can cover a wider range of UX qualities than other instruments. In addition, questionnaires can be established with psychometric properties that facilitate the comparison and dissemination of results.

In case a rigorously collected set of items has been validated and shown to target a single phenomenon (called a “construct”), it is often referred to as a psychological scale. When measuring complex phenomena like UX, a questionnaire can incorporate multiple psychological scales, each targeting a single construct. For example, the AttrakDiff questionnaire measures separately the constructs of hedonic and pragmatic qualities of UX, which together contribute to UX as a higher-level target construct [HBK03]. For the remainder of this paper, we call questionnaires that rely on one or multiple psychological scales standardized questionnaires. In contrast, we refer to ad-hoc questionnaires as a series of not-validated (therefore, not standardized) open or close-ended questions asked by evaluators to collect feedback regarding UX.

## 3. Current UX questionnaires in visual analytics

Studies in visual analytics have used both ad-hoc questionnaires and standardized HCI UX/usability questionnaires. It is common for researchers to create their own ad-hoc questionnaires for a single study, for example, when researchers want to evaluate specific aspects of how users experience their data interactions. Often, little to no information is provided about how a questionnaire was built, and articles rarely mention a validation process to ensure that the questions effectively capture the information that allows researchers to draw conclusions about the studied concept. It is even more common that questionnaires are not included in the paper, although with an increasing focus on reproducibility, it questionnaires start to appear in supplementary material. There has been an effort in the broader visualization community to propose questionnaires for assessing specific concepts of UX, such as the aesthetic pleasure of visual representations [HIDI22], engagement [HP17], trust [ESB*24], readability [CHII25], enjoyment [SES16] and value of visualization [WAM*19]. Victorelli et al. [VDR23] extended a standard HCI UX questionnaire with items concerned with human-data interaction and types of interaction with visualizations. However, from the above papers we cited, only two followed the methodological process of scale development and validation [CHII25, HIDI22].

Although standardized HCI questionnaires such as the System Usability Scale (SUS, [Bro96]) or the NASA Task Load index (NASA-TLX, [HS88]) were not specifically tailored for VA systems, researchers have employed them in their user studies (e.g., [PSS23]). To identify what are common questionnaires in the community, we searched IEEE VIS papers (1991–2023) and TVCG journal papers presented at IEEE VIS (2011–2021) using the names and acronyms of common UX questionnaires. We searched visualization papers broadly (rather than focusing on only papers mentioning VA) in order to base our analysis on a larger number of papers. We compiled
our list of UX questionnaires based on Perrig et al.'s systematic literature review [PAS*24] and chapters on UX questionnaires in HCI books [LG18, SL16]. Table 1 presents the number of studies that utilized each questionnaire. The SUS and NASA-TLX questionnaires were the most frequently mentioned. The SUS (system usability scale) is one of the best-known and most widely used standardized questionnaires for assessment of perceived usability for any type of product. It includes 10 easy-to-understand items that can be rated on a 5-point scale. Research into its psychometric properties (reliability, validity, and sensitivity) has been positive [BKM08, Lew18]. With a different focus, NASA-TLX specifically targets the construct of perceived workload, i.e. the mental and physical effort people exert in physical and/or digital experiences while conducting their tasks. As such, the NASA-TLX targets just one possible component of UX related to people's interactions with visual analytics systems—but it seems to be one of high interest to visualization and visual analytics researchers. Beyond these two questionnaires, the others often used in HCI were very infrequently reported in visualization and visual analytics papers; four times or fewer in total.

# 4. What is problematic about the current use of UX questionnaires for visual analytics research?

Here, we discuss why it is difficult currently to choose an effective existing instrument for evaluating UX with visual analytics systems.

## 4.1. Misalignment of common UX questionnaires

Even when a study is meticulous and uses a validated and well-constructed questionnaire, the terms and questions may be inappropriate for the specific application context [LK17]. VA systems frequently possess unique characteristics and application contexts that involve specific user interactions which standard UX questionnaires may not fully encompass. The SUS [Bro96], for example, asks broad questions on intent to use ("I think that I would like to use this system frequently"), perceived learnability ("I needed to learn a lot of things before I could get going with this system."), or ease of use ("I thought the system was easy to use"). However, these questions are all considered to be related to a single concept, the system's usability. They are certainly relevant to the visual analytics context but might not provide the right level of information to study complex visual analytics systems. It is possible, for example, that participants find a visualization easy to read but do not understand how to interact with an underlying model or data. In such a scenario, SUS questions would fail to capture UX for the individual parts of the system, thus limiting the questionnaire's usefulness.

Moreover, we argue that the mental effort required to interpret and understand visualizations is crucial for UX in visual analytics systems, where users must rapidly and accurately process complex data and models. Some general UX standardized questionnaires, such as the SUS, do not explicitly consider mental workload. Other, more targeted questionnaires like the NASA-TLX [HS88] assess mental workload but do not cover other essential aspects of UX, such as ease of use. VA systems require users to interpret complex data representations, and the nuances of the mental load required for data interpretation may not be captured by a generalized questionnaire, or even by the NASA-TLX that includes a single question about

this ("How mentally demanding was the task?"). Other NASA-TLX measures such as those on physical or temporal demand may not be relevant or do not make sense in the VA context ("How physically demanding was the task?" and "How hurried or rushed was the pace of the task") which may be one of the reasons why researchers developed their own version of this questionnaire [CWH*23] or used it in conjunction with other questionnaires like the SUS [FSN20]. Yet, both the NASA-TLX and the SUS scale are used frequently in visualization and visual analytics. Why? Studies show that researchers frequently choose questionnaires not based on their suitability but on their ease of use [DCB19]. Standard questionnaires are generally easy to use. Kosch et al. [KKZ*23], for example, hypothesize that NASA-TLX's simplicity of use and analysis may be the reason for the widespread use of the questionnaire in HCI. The SUS scale is similarly easy to administer and can be quickly filled out by participants. Answers are combined to generate a single score. The ease of analysis and interpretation of questionnaire results may also more generally be a factor in their adoption. There are studies about how to interpret, for example, the average score of the SUS questionnaire and an argument that the score results across studies are comparable [Lew18]. However, we did not find studies about average scores for visual analytics or visualization systems, and we hypothesize that a score from a visual analytics system may differ significantly from other types of applications.

## 4.2. Lack of validity and/or reliability evidence.

More generally, caution is advised when developing and using UX questionnaires and scales. These tools can be either well-designed, rigorously tested, and reliable, providing highly valid data, or poorly constructed, resulting in questionable validity and/or reliability. The SUS questionnaire is considered a reliable and valid tool for measuring system usability across a wide range of applications and users [BKM08]. However, even questionnaires such as the NASA-TLX, largely used in the HCI field, may have been questioned regarding their reliability and replicability; see Kosch et al. [KKZ*23] who criticize NASA-TLX's use in HCI and suggest alternatives. We would like to echo Kosch et al.'s call for further reflection on the scale's use; in our case, whether the scale is indeed the most appropriate to evaluate workload for visual analytics systems (or usability and UX) [KKZ*23]. Even well-designed and rigorously tested questionnaires may have to be validated again if the context of use demands modifications, such as after translations [LDOQRB23] or when applied outside of the original context.

In summary, we found existing UX questionnaires too broad or too specific to adequately capture UX with visual analytics systems. What is more, some lack evidence of validity or reliability of standardized tools. Although we did not find an existing measurement scale targeted at UX with visual analytics systems, we found a study highlighting the gap we aim to address [FC12]. Forsell and Cooper discussed the need to have standardized questionnaires specifically designed to measure usability in information visualization studies. While we argue that usability is only a part of UX, even the availability of such a questionnaire would significantly enhance the evidence base that the community needs to build high-quality visual analytics systems, facilitate user adoption, promote successful commercialization, and guide future research [FC12]. We reinforce their argument
that creating a validated questionnaire to measure users' perceived
experience in evaluation studies of VA systems would provide the
community with a valuable additional research tool.

5. Research Agenda and Challenges

Developing a validated UX questionnaire for visual analytics has
several significant challenges. First, UX is a broad concept encom-
passing various dimensions, and there is no consensus on its defini-
tion [Bev09] in the visual analytics context. In addition, we need to
strike a balance between the standardization of a measuring tool and
the ability of participants to make sense of the questions in all visual
analytics usage contexts. Finally, constructing and validating a scale
requires a large number of participants for statistical significance,
but visual analytics tools are typically tailored for users with specific
profiles and often have a limited user base. Therefore, it is crucial to
identify contexts with enough users to engage meaningfully with the
visual analytics system to validate a UX scale. In this section, we
elaborate on each of these identified obstacles, propose a strategy
that we plan to use to advance this research agenda ourselves, and
form a broader collection of challenges for future work on measuring
UX using alternative methods, such as interviews protocols.

**5.1. What are relevant UX dimensions for visual analytics?**

When envisioning the creation of a new measuring tool such as
a scale, or a group of scales, a first, essential step is to delineate
the boundaries of the domain [DT21]. This involves defining what
the desired instrument aims to measure—in our case, UX in visual
analytics systems. In scale development [BNF*18], a clearly defined
domain is a requirement for a thorough comprehension of the phe-
nomenon being studied, sets boundaries for the subject matter, and
streamlines the processes of item generation and content validation.

Our initial exploration of the UX and usability domains confirmed
that UX is a complex and broad construct, and there is no consensus
on its definition. It is therefore important to set conceptual bound-
aries if one wants to measure UX. In our work, we aim to measure
the experience after the user performed a data analysis interacting
with a visual analytics system, including the data, the visual represen-
tation and the models as well as the complex set of actions to handle
these elements. It must be possible to measure the user experience
including the users' thoughts, feelings, and perceptions that result
from that interaction and the potential knowledge gained [Van05]. It
will be necessary, but very difficult, to delve deeper into theoretical
frameworks that could serve as suitable references for developing
scales targeted at the different dimensions of UX in visual analytics.

To identify dimensions of UX in visual analytics, we can start
by examining all dimensions in existing standardized UX ques-
tionnaires. As such, some UX dimensions as efficiency, useful-
ness, visual aesthetics, ease of use, and satisfaction from NASA-
TLX [HS88], SUMI [KC93], PSUQ [Lew92], meCUE [MTWK17],
UEQ [LHS08], AttrakDiff [HBK03], or SUPR-Q [Sau15] can bear
a similar meaning in both HCI and visual analytics contexts. In fact,
there already exists a validated scale to measure visual aesthetics
in visualization contexts: the BeauVis scale [HIDI22]. On the other
hand, we should also start to exclude dimensions that are not or
rarely relevant for visual analytics systems, such as NASA-TLX's

physical demand scale, which is likely not relevant for assessing
UX in most visual analytics digital systems, or the spatial presence
scale in the Igroup Presence Questionnaire [SFR01], which is only
relevant for virtual environments. Some other dimensions might
require in-depth study to establish their significance for visual an-
alytics and to possibly identify their subcomponents: for example,
trust is regarded as a single dimension in SUPR-Q [Sau15] and scale
of Trust between People and Automation [JBD00]. However, when
applied to visual analytics, trust relates to both the visualization
and the underlying data [ESB*24], and measuring each construct
separately might be necessary for researchers who want to analyze
their users' behavior and experience in visual analytics studies. Fi-
nally, existing standardized UX questionnaires may lack some UX
dimensions specific to visual analytics, requiring the development
of dedicated items. We must also consider the core components of
visual analytics systems to develop a comprehensive instrument to
assess UX: analytics, visualizations, and interactions. However, it is
not clear how different constructs of UX relate to these individual
components. There are multiple ways to approach this problem. One
is to define dimensions a priori, for example by targeting UX items
related to interactivity, visualization, and analytics components sep-
arately. We are currently pursuing another by generating a global
set of items, followed by statistical analyses to extract factors that
emerge from how participants responded to these items.

**5.2. How can we adapt existing scales for visual analytics?**

It may not be practical to develop a single tool that can effectively
evaluate UX for all types of visual analytics systems and contexts.
For example, it will likely be necessary to adapt existing standard-
ized questionnaires, for example depending on the system's inter-
action or display modalities. Adapting a scale includes changing
the number of items (shortening, adding, or combining items from
multiple scales into a single scale), rewriting items whether for lan-
guage translation or to change the context or the referent within
the original language (e. g., SUS revised version replaced the word
"system" by the word "product" and the original term "cumbersome"
by the word "awkward" [BKM08]), modifying the number of rating
options or their text anchors, and using only a subset of scales from
a standardized questionnaire targeting a construct (e.g., using only
the "Extraversion" scale from the Big Five personality Inventory).
Heggestad et al. [HSB*19] point out that, while adaptation is not
necessarily a bad practice, it always raises questions about the valid-
ity and reliability of the resulting scores. Unfortunately, there is not
much guidance available on how to safely proceed with adapting
scales. As a result, we suggest that an important part of the work on
UX evaluation should involve developing a methodological frame-
work to help researchers adapt visual analytics UX questionnaires
into usable tools tailored to their specific contexts. Establishing
such a framework will also be essential to guide the transformation
of existing and relevant HCI UX questionnaires and scales into
appropriate, standardized tools for visual analytics research.

**5.3. How do we target multiple types of audiences?**

Scale and questionnaire development processes involve pre-testing
of questions among a specific target population. For this reason, it
is essential to determine which user groups will be the target of our
proposed UX in visual analytics measurement tool. Defining the
users of a visual analytics system is indeed complex. Three distinct
groups of users can be identified: casual users, domain experts, and
visualization experts [WME18]. Domain experts are a commonly
encountered group of users of visual analytics tools who aim to
apply visualization and analytics to investigate their domain but
may not have extensive literacy, training, or intrinsic motivation in
these fields [WME18]. This group may share some similarities with
casual users who may have similar levels of visualization literacy
but likely low data analysis training. We posit that understanding
the experience of end users who seek to leverage the benefits of
visualization is important regardless of whether they are domain
experts or occasional users. A UX questionnaire should therefore
not assume visualization expertise nor a high level of data analysis
training. The users targeted by the scale should, however, be moti-
vated to use visual analytics tools, have a basic understanding of the
data, and basic visualization interpretation and manipulation skills.

**5.4. How do we validate a scale in the visual analytics context?**

Newly developed measurement scales and questionnaires must un-
dergo thorough validation before they can be used as measuring
tools in studies [BNF*18,DT21]. Such a validation phase can also
prove useful when adapting a scale. Validation phases require the
participation of a large number of people to ensure statistical signifi-
cance. The challenge, therefore, lies in the fact that visual analytics
tools are typically designed for users with specific profiles and often
have a limited user base. Moreover, evaluating the user experience
with a system necessitates a substantial amount of interaction time,
which quick tasks typical of crowd-sourced studies may not ade-
quately capture. Therefore, finding a system with a large user base
for validation experiments can be challenging.

One potential alternative is to conduct multiple experiments with smaller participant groups and consolidate their results. However, this would require conducting a multi-group confirmatory factor analysis, which will present a statistical challenge. Due to the number of potential dimensions in the visual analytics UX construct, it might also be impractical to develop and validate each individual scale of our future questionnaire in a single study. Indeed, for each scale, construct validation will require at least two additional measures to establish convergent and discriminant validity. A comprehensive study tackling all dimensions of visual analytics UX at once might prove too long for a participant.

Additionally, due to the comprehensive nature of user experience,
once we have established and validated a scale for each visual analyt-
ics UX dimension, we also need to validate the overall questionnaire
by conducting a dimensionality analysis on data collected using all
scales' items together. Such a study would serve as a validation for
the entire standardized questionnaire, but can only be conducted
once we have 1) defined the UX concept thoroughly and established
its dimensions, and 2) validated a scale for each of these dimensions.

6. Final Remarks

In summary, our investigations into the topic of UX show that mov-
ing away from evaluating visual analytics systems using instruments

and practices developed for traditional HCI systems is imperative.
Given the challenges exposed above, it is not surprising that Forsell
and Cooper's [FC12] call for standardized questionnaires to evaluate
UX in visualization has not yet been answered. Establishing a valid
visual analytics UX questionnaire can only be approached as an am-
bitious, long-term research goal, which will require multiple steps
from theory-building to empirical studies encompassing the develop-
ment and validation of possibly multiple measuring scales. We have
started on this endeavor but are still in the process of finding out
how to address the multiple challenges we identified. The project
will require substantial effort but will be worth it. A future UX
standardized questionnaire for visual analytics systems can provide
better and more detailed information about the quality of the sys-
tems from the users' perspective. It will enable the development of
systems that optimize user experience, making data more accessible,
understandable, and insightful for a broader range of users.

Acknowledgements

This work was financed in part by the São Paulo Research Foun-
dation (FAPESP) (Grants #2022/08651-0 & #2023//17681-2). The
opinions expressed in this work do not necessarily reflect those of
the funding agency.

References

[ALC17] ADAGHA O., LEVY R. M., CARPENDALE S.: Towards a prod-
uct design assessment of visual analytics in decision support applica-
tions: a systematic review. *J Intell Manuf* 28, 7 (2017), 1623–1633.
doi:10/gbzdzd.

[ASN*20] ABEELE V. V., SPIEL K., NACKE L., JOHNSON D., GER-
LING K.: Development and validation of the player experience inven-
tory: A scale to measure player experiences at the level of functional
and psychosocial consequences. *Int J Hum-Comput Stud* 135 (2020),
102370:1–12. doi:10/ggjqff.

[Bev09] BEVAN N.: What is the difference between the purpose of usabil-
ity and user experience evaluation methods. In *Proc. UXEM* (2009).

[BKM08] BANGOR A., KORTUM P. T., MILLER J. T.: An empirical evaluation of the system usability scale. *Int J Hum-Comput Interact* 24, 6 (2008), 574–594. doi:10/b344sc.

[BL94] BRADLEY M. M., LANG P. J.: Measuring emotion: The self-
assessment manikin and the semantic differential. *J Behav Ther Exp
Psychiatry* 25, 1 (1994), 49–59. doi:10/bx59sf.

[BNF*18] BOATENG G. O., NEILANDS T. B., FRONGILLO E. A., MELGAR-QUIÑONEZ H. R., YOUNG S. L.: Best practices for developing and validating scales for health, social, and behavioral research: A primer. *Front Public Health* 6 (2018), 149:1–18. doi:10/gfsqzs.

[Bro96] BROOKE J.: SUS—A quick and dirty usability scale. In *Usabil-
ity Evaluation in Industry*, Jordan P. W., Thomas B., Weerdmeester B.,
McClelland I. L., (Eds.). Taylor & Francis, 1996, pp. 189–194.

[CDN88] CHIN J. P., DIEHL V. A., NORMAN K. L.: Development of an instrument measuring user satisfaction of the human-computer interface. In *Proc. CHI* (1988), ACM, pp. 213–218. doi:10/ftqjdf.

[CHI25] CABOUAT A.-F., HE T., ISENBERG P., ISENBERG T.: PRE-
Vis: Perceived readability evaluation for visualizations. *IEEE Trans Vis
Comput Graph* 31, 1 (2025), 1083–1093. doi:10/njnz.

[CWH*23] CHEN R., WENG D., HUANG Y., SHU X., ZHOU J., SUN G., WU Y.: Rigel: Transforming tabular data by declarative mapping. *IEEE Trans Vis Comput Graph* 29 (2023), 128–138. doi:10/g68rbb.

[Dav86] DAVIS F. D.: A technology acceptance model for empirically testing new end-user information systems: theory and results. PhD thesis, Indiana University, Cambridge, MA, 1986.
[DCB19] DARIN T., COELHO B., BORGES B.: Which instrument should I use? Supporting decision-making about the evaluation of user experience. In *Design, User Experience, and Usability. Practice and Case Studies* (2019), Springer, pp. 49–67. doi:10/gg9g7pz.

[DT21] DEVELLIS R. F., THORPE C. T.: *Scale Development: Theory and Applications*, 5th ed. SAGE, 2021.

[ESB*24] ELHAMDADI H., STEFKOVICS A., BEYER J., MOERTH E., PFISTER H., BEARFIELD C. X., NOBRE C.: Vistrust: A multidimensional framework and empirical study of trust in data visualizations. *IEEE Trans Vis Comput Graph* 30, 1 (2024), 348–358. doi:10/gtjwqg.

[FC12] FORSELL C., COOPER M.: Questionnaires for evaluation in information visualization. In *Proc. BELIV* (2012), ACM, pp. 16:1–3. doi:10/f24hhx.

[FSN20] FILHO J. A. W., STUERZLINGER W., NEDEL L.: Evaluating an immersive space-time cube geovisualization for intuitive trajectory data exploration. *IEEE Trans Vis Comput Graph* 26, 1 (2020), 514–524. doi:10/gtdktj.

[GB12] GROSS A., BONGARTZ S.: Why do I like it? Investigating the product-specificity of user experience. In *Proc. NordiCHI* (2012), ACM, pp. 322–330. doi:10/gg9g7p4.

[Has13] HASSENZAHL M.: *User Experience and Experience Design*, 2nd ed. The Interaction Design Foundation, 2013, ch. 3, pp. 1–14.

[HBK03] HASSENZAHL M., BURMESTER M., KOLLER F.: *AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität*. Vieweg+Teubner, 2003, pp. 187–196. doi:10/fzpnxd.

[HHH16] HOGAN T., HINRICHS U., HORNECKER E.: The elicitation interview technique: Capturing people’s experiences of data representations. *IEEE Trans Vis Comput Graph* 22, 12 (2016), 2579–2593. doi:10/f9bd2f.

[HIDI22] HE T., ISENBERG P., DACHSELT R., ISENBERG T.: Beauf Vis: A validated scale for measuring the aesthetic pleasure of visual representations. *IEEE Trans Vis Comput Graph* 29, 1 (2022), 363–373. doi:10/kt3n.

[HP17] HUNG Y. H., PARSONS P.: Assessing user engagement in information visualization. In *CHI Extended Abstracts* (2017), ACM, pp. 1708–1717. doi:10/gg9g7rs.

[HS88] HART S. G., STAVELAND L. E.: Development of NASA-TLX (task load index): Results of empirical and theoretical research. In *Human Mental Workload*, vol. 52. North-Holland, 1988, pp. 139–183. doi: 10/fbms3r.

[HSB*19] HEGGESTAD E. D., SCHEAF D. J., BANKS G. C., MOROE HAUSFELD M., TONIDANDEL S., WILLIAMS E. B.: Scale adaptation in organizational science research: A review and best-practice recommendations. *J Manage* 45, 6 (2019), 2596–2627. doi:10/gf2rfd.

[HT06] HASSENZAHL M., TRACTINSKY N.: User experience - A research agenda. *Behav Inf Technol* 25, 2 (2006), 91–97. doi:10/djx9nq.

[ISO18] ISO CENTRAL SECRETARY: *Part 11: Usability: Definitions and concepts*. Standard ISO 9241-11:2018, International Organization for Standardization, Geneva, CH, 2018.

[JBD00] JIAN J.-Y., BISANTZ A. M., DRURY C. G.: Foundations for an empirically determined scale of trust in automated systems. *Int J Cognit Ergon* 4, 1 (2000), 53–71. doi:10/fdt84r.

[KC93] KIRAKOWSKI J., CORBETT M.: SUMI: the software usability measurement inventory. *Brit J Educational Tech* 24, 3 (1993), 210–212. doi:10/fmh928.

[KKZ*23] KOSCH T., KAROLUS J., ZAGERMANN J., REITERER H., SCHMIDT A., WOŹNIAK P. W.: A survey on measuring cognitive workload in human-computer interaction. *ACM Comput Surv* 55 (2023), 283:1–283:39. doi:10/gsp9h8.

[LDOQRB23] LÓPEZ G., DÍAZ-OREIRO I., QUESADA L., RAMÍREZ-BENAVIDES K.: Translation and validation of the AttrakDiff user experience questionnaire to spanish. In *Proc. UCAMI* (2023), Springer, pp. 243–254. doi:10/gg9g7qf.

[Lew92] LEWIS J. R.: Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. *Proc Hum Factors Soc Annu Meet* 36, 16 (1992), 1259–1260. doi:10/ggp3xr.

[Lew18] LEWIS J. R.: The system usability scale: Past, present, and future. *Int J Hum-Comput Interact* 34, 7 (2018), 577–590. doi:10/ggxp24.

[LG18] LALLEMAND C., GRONIER G.: *Méthodes de design UX: 30 méthodes fondamentales pour concevoir des expériences optimales*. Eyrolles, 2018.

[LHS08] LAUGWITZ B., HELD T., SCHREPP M.: Construction and evaluation of a user experience questionnaire. In *Proc. USAB* (2008), Springer, pp. 63–76. doi:10/fgsrh6.

[LK17] LALLEMAND C., KOENIG V.: How could an intranet be like a friend to me? Why standardized UX scales don’t always fit. In *Proc. ECCE* (2017), ACM, pp. 9–16. doi:10/ggjcv9.

[Lun01] LUND A. M.: Measuring usability with the USE questionnaire. *The Usability SIG Newsletter: Usability Interface* 8, 2 (Oct. 2001), 3–6.

[MTWK17] MINGE M., THÜRING M., WAGNER I., KUHR C. V.: The meCUE questionnaire: A modular tool for measuring user experience. In *Proc. Advances in Ergonomics Modeling, Usability & Special Populations* (2017), Springer, pp. 115–128. doi:10/gh4jsr.

[PAS*24] PERRIG S. A., AESCHBACH L. F., SCHAROWSKI N., VON FELTEN N., OPWIS K., BRÜHLMANN F.: Measurement practices in user experience (UX) research: A systematic quantitative literature review. *Front Comput Sci* 6 (2024), 1368860:1–17. doi:10/m2ms.

[PSS23] PANDEY A., SRINIVASAN A., SETLUR V.: MEDLEY: Intent-based recommendations to support dashboard composition. *IEEE Trans Vis Comput Graph* 29, 1 (2023), 1135–1145. doi:10/gg9g7qk.

[Sau15] SAURO J.: SUPR-q: A comprehensive measure of the quality of the website user experience. *J User Exper* 10, 2 (2015), 68–86. URL: https://uxpajournal.org/supr-q-a-comprehensive-measure-of-the-quality-of-the-website-user-experience/.

[SES16] SAKET B., ENDERT A., STASKO J.: Beyond usability and performance: A review of user experience-focused evaluations in visualization. In *Proc. BELIV* (2016), ACM, pp. 133–142. doi:10/gfw4vt.

[SFR01] SCHUBERT T., FRIEDMANN F., REGENBRECHT H.: The experience of presence: Factor analytic insights. *Presence (Camb)* 10, 3 (2001), 266–281. doi:10/cvt2nz.

[SL16] SAURO J., LEWIS J. R.: *Quantifying the User Experience: Practical Statistics for User Research*. Morgan Kaufmann, 2016.

[SSS*14] SACHA D., STOFFEL A., STOFFEL F., KWON B. C., ELLIS G., KEIM D. A.: Knowledge generation model for visual analytics. *IEEE Trans Vis Comput Graph* 20, 12 (2014), 1604–1613. doi:10/f6qj6x.

[TA13] TULLIS T., ALBERT B.: *Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics*, 2nd ed. Elsevier, 2013. doi:10/gg9g7qz.

[Van05] VAN WIJK J. J.: The value of visualization. In *Proc. VIS* (2005), IEEE, pp. 79–86. doi:10/d3kg43.

[VDR23] VICTORELLI E. Z., DOS REIS J. C.: Evaluating user experience in information visualization systems: UXIV an evaluation questionnaire. In *Proc. HCII* (2023), Springer, pp. 320–338. doi:10/gg9g7pr.

[WAM*19] WALL E., AGNIHOTRI M., MATZEN L., DIVIS K., HAASS M., ENDERT A., STASKO J.: A heuristic approach to value-driven evaluation of visualizations. *IEEE Trans Vis Comput Graph* 25, 1 (2019), 491–500. doi:10/gf5b9g.

[WME18] WONG Y. L., MADHAVAN K., ELMQVIST N.: Towards characterizing domain experts as a user group. In *Proc. BELIV* (2018), IEEE, pp. 1–10. doi:10/gtx89h.