# Pondering the reading of visual representations

Anne-Flore Cabouat, Tingying He, Petra Isenberg, Tobias Isenberg

► To cite this version:

Anne-Flore Cabouat, Tingying He, Petra Isenberg, Tobias Isenberg. Pondering the reading of visual representations. 2023. hal-04240900

HAL Id: hal-04240900

https://inria.hal.science/hal-04240900v1

Preprint submitted on 13 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Pondering the reading of visual representations

Anne-Flore Cabouat, Tingying He, Petra Isenberg, Tobias Isenberg

**English Abstract**—We follow a theoretical approach to define the concept of reading visualizations. In the past, researchers often assessed readability based on the cognitive processes at work during an individual's engagement with a visual representation. The commonly used term “reading” in these studies, however, often lacks consistency: sometimes it refers solely to the extraction of textual information, while in other instances it is limited to the interpretation of visual signals such as patterns, color gradients, or object sizes. We argue that there exists a gap in the literature for a comprehensive, unifying definition of reading that would potentially broaden the horizons of design spaces and analytical frameworks in our field. To address this issue, we discuss models of reading text and how they can potentially relate to visualization reading.

## 1 INTRODUCTION

Consuming information is an important goal for viewers of visualizations [4]. In order to consume information from a visualization, the viewer has to actively *read* it [5, 8, 11, 13, 19, 27]. How effective this information consumption is, is often assessed through the concept of *readability* of a visualization.

We started the process of developing a validated scale to formally assess perceived readability of visual representations. As a first step, we conducted a systematic review of the questionnaire items previously used by researchers to assess perceived readability of visual designs. The examination of these items showed a wide variety of terms related to concepts such as *clutter*, *assessability*, *complexity*, *effectiveness*, *visibility*, *speed*, *recognizability*, *confidence*, or *overview*. Participants of prior studies also commented on readability using further related terms such as *visual organization*, *comprehension*, *meaningfulness*, *distraction*, *legibility*, or *obviousness*.

Unfortunately, the concept of “reading a visualization” has not been formally defined in the literature, making it difficult to determine which terms may actually measure readability rather than another related concept. The one model perhaps most related to our work is van Wijk’s [30] model of visualization (Fig. 1). The author summarized the key elements of a visualization and its usage. Reading a visualization is a core activity linking the I, P and K components in this model.

To the best of our knowledge no cognitive model of reading visualizations exists but such models are crucial for research [16, 23]. By summarizing cognitive processing theories, they help broaden our understanding of the solution space and can serve as effective scaffolding for design. Here we discuss our ongoing efforts to better understand the concept of reading visualizations.

## 2 RELATED WORK

Cognitive psychologists categorize reading as one of the most complex tasks that humans undertake [9]. As such, it has been a long term topic of interest for researchers. In this section, we describe common definitions and cognitive models of reading.

### 2.1 Definitions of reading text

Reading has been defined as the process of extracting information from printed words [14, 17]. Rayner et al. [28]

<figure><img src="image_2.png" /><figcaption>Fig. 1. A simple model of visualization from van Wijk [30].</figcaption></figure>

provide a more detailed definition: “*reading is the ability to extract visual information from the page and comprehend the meaning of the text.*” Neuroscientists, such as Fischer-Baum et al. [10] further refined the description: “*reading allows our brains to transform patterns of retinal stimulation into abstract representations of words (i.e., meanings and pronunciations)*” [10].

These definitions use terms specific to written language (e. g., text or words), which make them difficult to apply directly to visualizations. While it is easy to infer how data visualizations can produce “patterns of retinal stimulation” for the viewer, the idea of “pronouncing” visual data is not as straightforward.

### 2.2 Models of reading text

Cognitive psychologists identified two components in reading [15, 21]: **word decoding** and **linguistic comprehension**. Before discussing how to transfer the concept of reading to visualizations, we examine models of text reading to better grasp the cognitive processes at work for each component.

#### 2.2.1 Word decoding

Fig. 2 gives an overview of the Dual-Route Cascaded (DRC) model, which focuses on word decoding in read-aloud tasks [7]. While visualizations are rarely read out loud, the model can help us identify the cognitive processes at work to recover bits of meaning from visually encoded data (A to E).

The first steps involve pre-attentive processing to extract graphemes (e. g., letters, or other graphical units for non alphabetical languages such as Chinese). Then, a specific type of component called *lexicon* allows readers to recognize shapes, sounds and meanings of words. The lexicon is a part of human long-term memory; it is theorized to store words in the form of a dictionary with multiple entry indexes [18]: words can be accessed from sounds F, letters O, or meanings B.

We can observe that there are two-way connections between each step of the process from grapheme detection C to

* Anne-Flore Cabouat, Tingying He (同汀莲), Petra Isenberg, Tobias Isenberg: Inria, LISN, Université Paris-Saclay, CNRS. E-mail: first-name.lastname@inria.fr.
<figure><img src="image_3.png" /><figcaption>Fig. 2. The dual-route cascaded model of visual word recognition and reading aloud from Coltheart et al. [7].</figcaption></figure>

semantics **E**. For example, the orthographic lexicon **D** influences the recognition of graphemes **C** and vice-versa. This model also shows how meaning is reached through parallel processing of shapes (**D** → **E**) and sounds (**D** → **F** → **E**).

We have no such model for visualization and we do not yet know which components of the DRC model may transfer. For example, it would be interesting to find out if sounds for words play a role at all for reading visualizations.

### 2.2.2 Text comprehension

The second component of reading is linguistic comprehension, which proceeds from word decoding and heavily relies on the interaction of top-down (knowledge-driven) and bottom-up (word-based) reading processes. Perfetti and Satura [24] proposed the Reading Systems Framework (RSF) as a global framework for text comprehension. Fig. 3 illustrates how this model emphasizes the role of the lexicon as a hub between word recognition and comprehension processes, and brings to light recurring inputs from knowledge systems (e. g., linguistic and orthographic systems, general and contextual knowledge).

## 3 DISCUSSION

Cognitive models of reading, based on an extended corpus of research in psychology, offer valuable insights to our research question. However, these models often specifically rely on text elements, and thus would not transfer easily to a visualization context. An added difficulty is the need to consider at least two other components for a cognitive model of visualization: *spatial processing* and *numerical processing*. Existing studies suggest that spatial processing can support numerical representations [20] and visual interpretation [22, 29], and that viewers' numeracy (i. e., their ability to understand and work with numbers) and ability to read graphs have

<figure><img src="image_4.png" /><figcaption>Fig. 3. The Reading Systems Framework [24].</figcaption></figure>

shared cognitive skills dependencies [12]. Further work should seek to examine the cognitive effects of spatial and numerical processes to find out if, and how, they interact with a viewer's ability to understand a visualization.

Another obstacle in applying text reading models to a visualization context is that visual representations of data do not, in principle, translate to sounds the same way as words do—or even at all. Some visualizations present a large amount of text (e. g., word clouds, some networks, ...) and activate the phonological processes of the Dual-Route Cascade (DRC) model, but some other visualizations may not even display a single word. To the best of our knowledge, the possible activation and role of phonological cognitive processes have not been investigated for silent, wordless data visualizations. Since there is some evidence of phonological processes at work in mental manipulation of numbers [25], we suggest that further research should focus on this particular question.

Conversely, some components of the discussed models (e. g., *visual input*, *lexicon* and *knowledge systems*) could be directly transferred to a model of reading visualizations. Fig. 4 shows our first attempt to collect the possible cognitive processes at work in retrieving information from a visualization.

First, we can associate the *image* (I) component of van Wijk's model of visualization ([30], Fig. 1) with the *visual input* of the Reading Systems Framework (RSF), or *print* in the DRC model. Then, based on the central role of the mental lexicon in both models and on existing research on visualization literacy [3, 6, 26], we can also postulate the existence of a "visual lexicon" with semantics dedicated to graphical elements (e. g., orthogonal lines representing data axis in a plot).

Furthermore, both models show two-way connections between their components. Such loops are observed at low levels (e. g., word decoding sub-processes from DRC), and higher levels (e. g., comprehension processes from RSF). These interactions allow readers to integrate sensory input into meaningful representations of the content, which in return informs ongoing decoding processes. We propose that the same may be true for reading visualizations. For example, the recognition of a word could have an influence on the identification of visual features (e. g., identifying a dot on a map
<figure><img src="image_5.png" /><figcaption>Fig. 4. Draft: Simple cognitive model of reading visualizations.</figcaption></figure>

as the location of a city) and vice-versa (e. g., a quantitative y-axis labeled "capital" is understood to mean "financial assets" rather than the "administrative centre of a country").

Finally, the recurring interactions between cognitive processes and knowledge systems, a feature of the RSF model, are also highlighted in number processing [2] and visualization processing [6]. Consequently, we propose that a cognitive model of reading visualizations should show interactions between cognitive processes and the following knowledge systems: language, numbers, visualization, world and context.

Considering the aforementioned elements, we propose to define reading visualizations as *the cognitive process of converting visual encodings into information*. In this context, we use Ackoff's definition of information [1]: *data which is usable for a task*.

## 4 CONCLUSION

We propose a definition of reading in the field of visualization. We show a first attempt at collecting bottom-up and top-down cognitives processes at work in reading visualizations, and suggest that the concurrent semantic decoding of all types of visual content is deeply interlinked. Further research should aim to extend this theoretical grounding by reviewing more related work and studies of visualization perception, spatial reasoning and number processing.

## REFERENCES

[1] L. R. Ackoff. From data to wisdom. *J Applied Systems Analysis*, 16:3-9, 1989.

[2] M. H. Ashcraft. Cognitive arithmetic: A review of data and theory. *Cognition*, 44(1):75-106, 1992. doi: 10.1016/0010-0277(92)90051-1

[3] J. Boy, R. A. Rensink, E. Bertini, and J.-D. Fekete. A principled way of assessing visualization literacy. *IEEE Trans Vis Comput Graph*, 20(12):1963-1972, 2014. doi: 10.1109/TVCG.2014.2346984

[4] M. Brehmer and T. Munzner. A multi-level typology of abstract visualization tasks. *IEEE Trans Vis Comput Graph*, 19(12):2376-2385, 2013. doi: 10.1109/TVCG.2013.124

[5] A. Burns, C. Xiong, S. Franconeri, A. Cairo, and N. Mahyar. How to evaluate data visualizations across different levels of understanding. In *Proc. BELIV*, pp. 19-28. IEEE Comp. Soc., Los Alamitos, 2020. doi: 10.1109/BELIV51497.2020.00010

[6] K. Börner, A. Maltese, R. N. Balliet, and J. Heimlich. Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. *Inf Vis*, 15(3):198-213, 2016. doi: 10.1177/1473871615594652

[7] M. Coltheart, K. Rastle, C. Perry, R. Langdon, and J. Ziegler. DRC: A dual route cascaded model of visual word recognition and reading aloud. *Psychol Rev*, 108(1):204-256, 2001. doi: 10.1037/0033-295X.108.1.204

[8] F. R. Curcio. Comprehension of mathematical relationships expressed in graphs. *J Res Math Educ*, 18(5):382-393, 1987. doi: 10.2307/749086

[9] A. M. Elleman and E. L. Oslund. Reading comprehension research: Implications for practice and policy. *Policy Insights Behav Brain Sci*, 6(1):3-11, 2019. doi: 10.1177/2372732218816339

[10] S. Fischer-Baum, D. Bruggemann, I. F. Gallego, D. S. Li, and E. R. Tamez. Decoding levels of representation in reading: A representational similarity approach. *Cortex*, 90:88-102, 2017. doi: 10.1016/j.cortex.2017.02.017

[11] S. N. Friel, F. R. Curcio, and G. W. Bright. Making sense of graphs: Critical factors influencing comprehension and instructional implications. *J Res Math Educ*, 32(2):124-158, 2001. doi: 10.2307/749671

[12] M. Galesic and R. Garcia-Retamero. Graph Literacy: A Cross-Cultural Comparison. *Med Decis Making*, 31(3):444-457, 2011. doi: 10.1177/0272989X10373805

[13] M. Ghoniem, J.-D. Fekete, and P. Castagliola. On the readability of graphs using node-link and matrix-based representations: A controlled experiment and statistical analysis. *Inf Vis*, 4(2):114-135, 2005. doi: 10.1057/palgrave.ivs.9500092

[14] E. J. Gibson and H. Levin. *The Psychology of Reading*. MIT Press, Cambridge, USA, 1975.

[15] P. B. Gough and W. E. Tunmer. Decoding, reading, and reading disability. *Remedial Spec Educ*, 7(1):6-10, 1986. doi: 10.1177/074193528600700104

[16] C. Heine. Towards modeling visualization processes as dynamic Bayesian networks. *IEEE Trans Vis Comput Graph*, 27(2):1000-1010, 2021. doi: 10.1109/TVCG.2020.3030395

[17] E. Huey. *The Psychology and Pedagogy of Reading*. Macmillan, Oxford, UK, 1908.

[18] R. Jackendoff. Précis of Foundations of Language: Brain, Meaning, Grammar, Evolution. *Behav Brain Sci*, 26(6):651-665, 2003. doi: 10.1017/S0140525X03000153

[19] S. Lee, S.-H. Kim, and B. C. Kwon. VLAT: Development of a visualization literacy assessment test. *IEEE Trans Vis Comput Graph*, 23(1):551-560, 2017. doi: 10.1109/TVCG.2016.2598920

[20] K. S. Mix, S. C. Levine, Y.-L. Cheng, C. Young, D. Z. Hambrick, R. Ping, and S. Konstantopoulos. Separate but correlated: The latent structure of space and mathematics across development. *JEP: General*, 145(9):1206-1227, 2016. doi: 10.1037/xge0000182

[21] J. Oakhill, K. Cain, and P. Bryant. The dissociation of word reading and text comprehension: Evidence from component skills. *Lang Cogn*, 18(4):443-468, 2003. doi: 10.1080/0169063044000008

[22] A. Ottley, E. M. Peck, L. T. Harrison, D. Afergan, C. Ziemkiewicz, H. A. Taylor, P. K. J. Han, and R. Chang. Improving Bayesian reasoning: The effects of phrasing, visualization, and spatial ability. *IEEE Trans Vis Comput Graph*, 22(1):529-538, 2016. doi: 10.1109/TVCG.2015.2467758

[23] L. M. Padilla. A case for cognitive models in visualization research: Position paper. In *Proc. BELIV*, pp. 69-77. IEEE Comp. Soc., Los Alamitos, 2018. doi: 10.1109/BELIV.2018.8634267

[24] C. Perfetti and J. Stafura. Word knowledge in a theory of reading comprehension. *Sci Stud of Read*, 18(1):22-37, 2014. doi: 10.1080/10888438.2013.827687

[25] C. Pollack and N. C. Ashby. Where arithmetic and phonology meet: The meta-analytic convergence of arithmetic and phonological processing in the brain. *Dev Cogn Neurosci*, 30:251-264, 2018. doi: 10.1016/j.dcn.2017.05.003

[26] Y. Postigo and J. I. Pozo. On the road to graphicity: The learning of graphical representation systems. *Educ Psychol*, 24(5):623-644, 2004. doi: 10.1080/0144341042000262944

[27] M. Raschke, T. Blascheck, M. Richter, T. Agapkin, and T. Ertl. Visual analysis of perceptual and cognitive processes. In *Proc. IVAPP*, pp. 284-291. SciTePress, Setúbal, 2014. doi: 10.5220/0004687802840291

[28] K. Rayner, A. Pollatsek, J. Ashby, and C. Clifton Jr. *Psychology of Reading*. Psychology Press, New York, 2nd ed., 2012.

[29] S. B. Trickett and J. G. Trafton. Toward a comprehensive model of graph comprehension: Making the case for spatial cognition. In *Proc. Diagrams*, pp. 286-300. Springer, Berlin, 2006. doi: 10.1007/11783183_38

[30] J. van Wijk. The value of visualization. In *VIS 05. IEEE Visualization*, 2005., pp. 79-86, 2005. doi: 10.1109/VISUAL.2005.1532781