<figure><img src="image_1.png" /></figure>

HAL
open science

# Bilingual Reading Experiences: What They Could Be and How to Design for Them

Clément Pillias, Pierre Cubaud

<figure><img src="image_2.png" /></figure>

## To cite this version:

Clément Pillias, Pierre Cubaud. Bilingual Reading Experiences: What They Could Be and How to Design for Them. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.531-549, 10.1007/978-3-319-22701-6_39. hal-01599650

HAL Id: hal-01599650

https://inria.hal.science/hal-01599650

Submitted on 2 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Bilingual Reading Experiences:
What They Could Be and How To Design for Them

Clément Pillias¹ and Pierre Cubaud¹

¹Conservatoire National des Arts et Métiers, CÉDRIC
292 rue Saint Martin, Paris, France
{clement.pillias, cubaud}@cnam.fr

**Abstract.** We introduce the idea of *bilingual reading*, where a document comes in two languages and the reader can choose at will on which language to focus during the reading. Between the complete ignorance of a language (where translation is the only option) and bilingualism (where translation is useless), there exists a variety of contexts of partial bilingualism where bilingual reading interfaces would prove highly useful. We first study through interviews and reviews how the bilingual reading experience is understood today. We provide an analysis framework and highlight design challenges for the design of bilingual reading appliances. We then describe a taxonomy of the different approaches available to address these challenges, analyze them in the light of our framework and show how they can be derived to sketch future bilingual reading interfaces.

**Keywords:** Bilingual reading; mechanisms of reading; nexus of attention; e-book; e-reader; parallel text; text morphing; text animation.

# 1 Introduction

"Cognitive, social, personal, and economic benefits accrue to the individual who has an opportunity to develop a high degree of bilingual proficiency when compared with a monolingual counterpart." This is how Tucker presented in 1999 the definitive conclusions of 30 years of research on multilingualism [39]. This observation calls for the promotion of language education, but also calls for more exposure to foreign-language medias. Novels, for instance, can meet various individual interests: a reader's desire to better appreciate the author's work and culture; the need to practice a foreign language; social recognition, etc. Reading a text written in a foreign language is however difficult for people who are not fluent with the language. To alleviate this difficulty, one can bind a foreign-language text with a high-quality translation of it in the reader's mother tongue. The binding uses an *alignment structure* that links each part of the original text to the corresponding part in the translated text and reciprocally, resulting in a *bitext* (see Fig. 1). One expected benefit of this solution is to ease the transition between the two versions of the text. This paper discusses *bilingual reading experiences* based on bitexts and studies interaction designs for such experiences.
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** *Gulliver's Travels* by Jonathan Swift is available in many languages for free on the Gutenberg Project (http://www.gutenberg.org). In this picture, we show extracts of the original English version (left) and of a French translation (right). Note the differences in the layout of paragraphs and construction of sentences. Natural Language Processing techniques can now automatically find the corresponding parts of each version to create an *alignment structure*. Here, this structure is represented at the level of paragraphs with arrows and color-coding.</figcaption></figure>

People are already bilingual (at various levels of proficiency) in most parts of the world, and have accepted e-books as viable alternatives to paper books [31]. Publishing industry is familiar with translation processes and already distributes many resources in multiple languages: successful books are translated to reach an international market and old books and their translations have been massively digitalized. Natural language processing techniques are being developed to ease the creation of alignment structures at a massive scale [38] and individuals can already create such structures at reasonable costs.

With such an economical and societal importance and so little technical barriers, it is striking how little HCI works address bilingual reading. Reading has been studied many times in HCI, highlighting the variety of reading purposes and reading styles [27] and the importance of supporting activities [32,37]. More general views have also been suggested about what digital reading could be [33]. All of these works are relevant for bilingual reading, but none of them mention bilingual reading as an opportunity.

The central question of the research agenda on bilingual reading is: *How could the reader benefit the most from both the original text and its translation?* It is therefore essential to understand what it actually means for readers to benefit from each version of the text. Each version can satisfy different goals, e.g., the translated text is easier to read, but reading the original version allows one to practice the language. But how does the reader deal with conflicting goals such as “learning new foreign words and expressions” vs. “staying in the flow of the story”? More questions arise if one can switch at will and seamlessly between the two versions: What is the cost of language-switching decisions? What strategies inform them? What are the resulting switching patterns?

Answering these questions raises two methodological issues: First, existing materials support bilingual reading very poorly, so observing people using them might only provide limited knowledge about bilingual reading. Second, it is difficult to collect readers’ memories of past bilingual reading experiences or to rely on their ability to imagine themselves having such experiences. Indeed, most potential users of bilingual reading appliances are *experts* in reading in their mother tongue: they have spent many years learning how to do it and they practice their reading skills everyday (without even noticing it), up to the point that whole areas of their brain have become dedicated to reading [11]. Reading is for them an activity that relies mostly on unconscious processes, and
this holds true even for reading in a foreign language once they have mastered its writ-
ing system. These readers tend therefore to focus on remarkable events in their previous
foreign-language reading experiences, which are mainly frustrating situations such as
having to stop reading because of an unknown word. They have therefore a strongly
biased understanding of what a bilingual reading experience could be and researchers
need to find methods that let potential users reflect on their own or fictive experiences in
an unbiased way.

This understanding of bilingual reading is necessary to discover what would be a
*right thing to design*, but the complementary problem of *getting the design right* also
brings its own challenges. Indeed, designers are also experts in reading and are therefore
subject to the same biases than potential users. Moreover, text display technologies
seem to have reached a stable and mature form after centuries of evolution, it is thus
difficult to think out of the box and discover innovative ways of displaying text and
interacting with it. In this context, proposing and evaluating a simple proof-of-concept
prototype appliance might impede the discovery of alternative techniques, by focusing
designers and researchers minds on the design of this prototype or some variant of it.
Opening the design space to bring some room for unconventional techniques is there-
fore one of the main concerns that guided this work.

We describe in this paper our approach for the study of bilingual reading and discuss
some early results. We focus on addressing two issues: avoiding biases in users’ exami-
nations of their experiences, and providing support for designers while keeping the de-
sign space open. We narrow these research questions by focusing first on reading for
pleasure [9] and especially on reading novels. We also focus on a pair of languages that
share the same writing system while having both common roots and strong differences:
French and English.

After setting the technical and scientific background for bilingual reading, we intro-
duce in section 3 the method that we used to investigate what a bilingual reading experi-
ence could be, and some preliminary observations. We provide in section 4 a framework
to describe how a bitext is displayed and read. In section 5, we report on a set of design
challenges that were highlighted during our interactions with potential users and analyze
these challenges in the light of the previous framework. We also consider solutions
proposed in other domains that address some of these challenges, and show that they
follow three different approaches. Finally, we give hints about how to adapt these ap-
proaches for bilingual reading.

2 Background

## 2.1 Current Support for Bilingual Reading

**Printed Materials.** Bilingual printed books often assign a different language to the left and right side of the book, so that facing pages display corresponding parts of the text. These so-called *parallel texts* use a coarse granularity for the alignment: when the reader encounters an unknown word, finding its translation can be difficult as it might require to scan a big part of the facing page. Establishing a visual match at a finer granu-
larity (e.g. paragraphs or lines) could raise the publishing costs and would still require
the aligned text elements to follow the same ordering in both languages. This can be a
problem with non-literal translations, where parts of text can be swapped or removed, or
where sentences can be grouped differently to form paragraphs. Printed parallel texts
therefore rely often on dedicated or adapted translations.

**Digitalized Materials.** Reading a digitalized parallel text with a standard e-reader is cumbersome, as parallel texts have specific requirements for navigation between pages. Bederson et al. designed a smart-phone application to address related navigation issues in digitalized children books (including multi-lingual ones) [3]. While their interaction techniques can be used to access and read text boxes corresponding to different languages, it does not allow the reader to quickly switch its focus of attention from one language to the other. Another solution could be to use dual-display readers such as *Codex* [19], dedicating a display to each language.

**Interactive Digital Materials.** Web-based machine translation services like *Google Translate* are the best known and most used interfaces to read a web page originally written in a foreign language. When these tools are used to translate a full web page, hovering a translated sentence with the mouse triggers the display of a tool-tip showing the original version of this sentence. *Doppel Text*<fnref n="1" /> provides a similar interface for reading classical novels as e-books, but exchanges the roles of the two languages (tool-tips show the translation). Its use of professional translations certainly allows for a better reading experience but reduces the number of available books (only 32 at the time of writing). Much more content could be made available to readers with the use of natural language processing tools that compute a (sentence-level) alignment structure. These interfaces only use very simple interactions and have ergonomic issues, such as the tool-tips being hard to distinguish from the original document. More advanced interaction techniques allow disclosing supporting materials [7] but have not been tested for bilingual reading. Users' reception of these techniques being highly varied [42], they require a fine-tuning of the design and there is room to explore more design options.

## 2.2 Related Work

**Monolingual Digital Reading.** Reading with digital tools has been studied for a long time: many studies have investigated the effect of display and text parameters on reading performance [36], or the benefits of pagination overscroll layout (which is supposed to involve lower levels of mental workload [40] and to make a better use of spatial memory [28,29]), or of other visualization techniques [20]. Other studies have investigated the role of paper and the physicality of printed books, to find that they better support navigation [24,35] and recall of the text [25] than e-books.

**Enhanced Foreign-Language Reading.** Some techniques have been proposed to facilitate the reading of a text in a foreign language without relying on a bi-text. For example, *Jenga* splits and indents sentences to make their grammar easier to grab [41]. *iDict* uses

<footnote n="1">¹ http://www.doppeltext.com/</footnote>
eye-tracking techniques and algorithms to identify the foreign language words that cause difficulties to the reader, and automatically displays in a side pane the corresponding entry in a bilingual dictionary [21]. Han et al. discuss the benefits of using pictures instead of bilingual dictionaries entries to avoid reading in multiple languages [18]. In the context of written conversations, people better understand machine-translated messages when the sender highlights the key words in the message [16]. These helping techniques and other similar ones could be used for bilingual reading, however we will argue that bilingual reading cannot be reduced to the availability of such help tools.

**Generalization to Multi-Representational Reading.** Studies of interest can be found in the context of multiple representations. For example, a document written in a markup language such as HTML has two natural representations: the textual one with markup code, and the graphical one displayed in the browser. The rendering process binds these representations together, defining links between elements of each representation in a similar way than the alignment structure does for languages. This kind of links has been generalized to other relations between documents such as: an original document and its annotations; successive or concurrent versions in the document edit history (the link is then obtained by a kind of *diff* operation); etc. [26]. Animation has been used in such settings to help in understanding the differences between successive edits of wiki pages [8] or to help in understanding how a source file is compiled into a graphical representation [12]. The later also allow graphical picking of source code, which eases the document edition and avoid disruptive searches in the code. Understanding how some text has been translated and searching the translation of some part of text are similar tasks in bilingual reading.

# 3 Understanding Bilingual Reading Experiences

**Methodology.** To understand what a bilingual reading experience could be, we have conducted informal interviews with people who showed interest in this question: novel reading amateurs, foreign language teachers, professional translators and their students, linguists, and researchers in natural language processing. We interviewed around 20 people individually or in groups of less than 10, in sessions of one to four hours, with some people interviewed multiple times. The goal of these interviews was as much to get insights about bilingual reading experiences than to test methods that could be used to get such insights.

These interviews made obvious the tendency of interviewees to focus on the negative aspects of their past experiences of reading in a foreign language. They also immediately suggested solutions to their problems, and all following discussions revolved around these solutions. Almost all interviewees mentioned, for instance, the following idea: "if I encounter a word that I don't know, I could press a button and get its translation." None of the interviewees realized that parallel texts provided a different solution to this problem, although most of them had previous experience with this kind of material. So we had to tell them: such interventions seemed necessary to get more useful insights, and we were interested in finding what kind of intervention would be the most useful and if it could become part of a more formal collaborative design method.
The most useful interventions were the ones that allowed the interviewees to re-examine their concerns in the light of a different, familiar past experience. We used first the metaphor of subtitles in movies, which could be introduced like this: "As non-native English speakers, we like to watch Hollywood movies subtitled in our native language. This way, the actors' voices seem more natural and we can better appreciate their acting and the director's work. Our English is good enough to understand most of the diálogs, but we are sometime lazy or the characters use slang or have a strong accent that we cannot understand. We can then easily have a look at the subtitles to understand what is going on. Moreover, it clearly makes us practice our English speaking skills, adding the pleasure of improving ourselves to the enjoyment of being immersed in a good movie. We want to create for novels what subtitles are to movies." This metaphor was useful to communicate a vague idea of what could be a bilingual reading experience, as well as in triggering memories about particular situations where the interviewee would switch its attention from the subtitles to the audio or conversely. However, the metaphor was not strong enough that the interviewees would spontaneously refer to it during the whole interview – movies probably seemed too far away from reading.

We also asked interviewees to analyze their concerns in the light of existing materials such as parallel texts or *Doppel Text*. It helped in highlighting the benefits and drawbacks of these materials and to find other ways to use them than simply looking for the translation of unknown words. We finally introduced and tested *reading traces*, a more formalized approach where a group is asked at the beginning of the session to read a short novel printed in the parallel text format. The reading takes less than 30 minutes and the sheets of the document can be detached to let the readers adopt the configuration of printed materials that they judge the most satisfying. Participants are asked to annotate the document with a pen, using their own marks, to create *"a trace of what and how they read, that they could use in the following discussion, and that we could use in a later analysis"*. They also write at the end of the document a description of the mark system that they used and other comments. A round table follows the reading session with each participant commenting on her experience. A group discussion ends the session. We found that asking participants to write traces of their reading activities incited them to reflect on their actions both during the reading and during the discussion.

**What we Have Learned.** The first lesson is that bilingual reading should not be confused with an enhanced form of foreign-language reading, but encompasses it. For example, another important aspect of bilingual reading is to satisfy the reader's curiosity, e.g. when she asks herself "how can one translate that pun?" or "did they translate the double entendre?" These examples show other punctual and opportunistic needs for the translation, but bilingual reading experiences can rely on a much intensive use of the two texts, in a way that redefines the reader's engagement with the novel. We experienced this while reading parallel texts: at some point the reading turned into a game where we tried to predict how each sentence had been translated.

In our experimentation with reading traces, we observed a great variety of reading behaviors: reading the whole original text first and then the translation (or the opposite); reading each sentence or paragraph in both languages before reading the following one; switching language at each new paragraph; having a quick overview of one text to pick
interesting words or expressions that the reader were interested in knowing the translation; being so immersed in the story as to forget to switch back to the first language; trying to improve the translation; marking corresponding paragraphs to ease language switching; etc. This variety of strategies reflects the diversity of readers' skills and interests. It also highlights the role of the medium: the participants generally judged parallel text as providing a poor support for bilingual reading. It however supports reading strategies that would be hard to sustain with other techniques such as tool-tips.

# 4 Describing Reader's and System's Behaviors

While the bilingual reading experience still has to be invented, it will necessarily follow a few principles imposed by the mechanisms of human visual attention and display resources manipulation. We recall these principles here and put them in the context of bilingual reading. We believe that these principles provide a common language and knowledge basis that can be used at all stages of the design process: They can be used to describe observed behaviors, whether it is to understand the bilingual reading activity or to test design solutions. They can also be used during the design phase to explore different strategies and design solutions.

## 4.1 Visual Selective Attention and Associated Spaces

Visual selective attention is a fundamental mechanism of human vision that is used extensively to read. It can be defined broadly as a selection mechanism, where one “object” in the visual field of the reader is selected and other objects are ignored. We will call this selected object the *focus* of attention, while the position of this object in the visual field or world will be called the *locus* of attention.<fnref n="2" /> While often used as synonyms, these concepts actually refer to elements that belong to different spaces. We will also introduce another notion, the *nexus* of attention, as something that connects two “objects”, permitting or calling for an attention shift from one to the other.

**Locus of Attention and Display Resources.** When reading, the locus of attention is a point on a *display surface* where the text in focus is displayed. For our concerns, it is important to consider the diversity of possible display surfaces and their properties: sheets of paper or electronic paper, LCD screen, desktop or whiteboard surfaces where something is video-projected, etc. The notion of a display surface has become very rich and also includes virtual surfaces such as windows or coherent arrays of displays [5]. It should not be confounded with the close notion of *work surface*, which is a surface in the workspace that can host display surfaces, such as a desktop or wall. Both display and work surfaces will be described as *display resources*.

**Focus of Attention and the Representation Space.** The representation space is the space of all possible focuses of attention. It has therefore 3 axes corresponding to the properties of the object in focus, which belongs to one version of the text defined by its

<footnote n="2">² Our definitions could conflict with other works in HCI or in other fields that are more legitimate to define these notions, such as cognitive sciences.</footnote>
*language*, has a *position* in this text version, and corresponds to a *linguistic level* such as the word, the sentence or the paragraph. By adding a language axis, the representation space generalizes the space-scale diagrams used to discuss information spaces visualization techniques [15], but it uses the position in the text and the linguistic level instead of continuous axes for space and scale.

**Nexus of Attention and Links in the Representation Space.** A nexus of attention connects two possible attention focuses, and can thus be understood as a bidirectional link between two points of the representation space. There are three natural kinds of nexuses: First, consecutive words, sentences and paragraphs are connected by *positional links*, i.e. links between points of the representation space that differ only along the *position* dimension. Second, elements are connected to their containers at the upper linguistic level by *hierarchical links*. And finally, the translation defines *alignment links* between points that differ along the *language* dimension (and potentially also along the other dimensions, e.g. a word can sometime be translated as a full sentence). The text itself can contain links between points that have the same language but can differ along the other dimensions, like footnotes, figures and references. The notion of a nexus of attention can also be extended to include links to elements in other documents, such as hyperlinks or the entry corresponding to a word in a bilingual dictionary.

## 4.2 Using These Spaces in the Design Process

**Analyzing Readers' Behavior.** To analyze readers' behavior, it can be useful to know how the focus of attention changes during the reading, which can be visualized as a path in the representation space. Such information can be hard to obtain, however some works suggest that it can be usefully approximated by the parameters of the rendering process, such as the amount of scrolling [6,20]. They can also be used to compute quantitative values that can be directly compared, such as the proportion of reading time during which a given element was visible.

**Defining the Interaction.** Displaying bilingual texts consists basically in rendering some text elements taken from the representation space at some location in a display surface. It thus requires the definition of a *composition mapping* between the display surfaces and the representation space (see Fig. 2). Interaction can take advantage of the different nexuses of attention, which define possible tasks, as we will see later. In par-

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** The pipeline of visual attention. Gray boxes represent limited resources, and the arrows represent one-to-one mappings between elements of each type of resource. These mappings are defined or controlled by the mechanisms on the left.</figcaption></figure>
ticular, these nexuses can be used to identify opportunities for interactive elements in
the displayed text, like the hyperlinks used in *Doppel Text*.

4.3 Manipulating Limited Resources

In most situations, there is much more text to read than what can be displayed on the
available display surfaces and made visible on the available work surfaces: these display
resources should then be considered as limited. Visual attention can also be considered
as a limited resource since there is only one focus of attention. This limitation in re-
sources motivates the reader's need to interact with the text, including the physical ma-
nipulation of printed books. (The *way* one interacts, however, should be tailored to the
readers' tasks) The management of limited resources (and particularly their reuse) has
generic methods, and we discuss how they apply to attention and display resources.
These methods can be used to describe observed readers' behavior in terms of strategies
used, but can also be used for a design space as we will see later.

**Reuse of Work Surfaces.** Taking into account the need to reuse workspace allows one to highlight the function of some design elements that could otherwise be overlooked. For example, a bookbinding could be seen as only useful to maintain the sheets together, but it is also a mechanical guidance that eases the reuse of work space: when turning a page, the binding forces the new page to take the place of the previous one on the work surface. Similarly in GUIs, dragging is constrained to a single dimension to implement scrolling, while page flipping animations help in understanding how the surfaces are reused.

**Spatial and Temporal Multiplexing.** Visual attention implies a chain of one-to-one mappings depicted in Fig. 3. Attention is bound to a locus on a work surface by eye movements (*gaze mapping*); this locus on a work surface is bound to a locus in a display surface by the physical manipulation of the display surfaces (*workspace mapping*); and then this locus in a display surface is bound to a point in the representation space by the *composition mapping*, which is controlled through the graphical user interface. Moreover, the composition mapping can itself be defined as a chaining of multiple mappings as it can use split panels, each of which correspond to a viewport in a virtual space where another composition process is held. When a space (virtual or real) is split so that a different mapping is used in each sub-space, we refer to *spatial multiplexing*. On the other hand, *time multiplexing* describes a mapping that changes in time. The total mapping must change in time to reuse resources, therefore at least one of the three mappings must change in time.

**Multiplexing in Gaze Mapping.** In *free attention*, eye movements define the locus of attention: the gaze mapping changes in time. The remaining mappings can be fixed, allowing for spatial multiplexing strategies. In *directed attention*, the reader does not control the selection process, which is imposed by the rendering process. Eyes do not have to move (the gaze mapping is fixed), but the remaining mappings evolve in time. Rapid Serial Visual Presentation epitomizes this temporal multiplexing approach [30], although other variants are possible.
**Multiplexing in Composition Mapping.** The printed book uses spatial multiplexing: each page, as a display surface, displays a fixed part of the representation space. On the other hand, the screen of an e-reader relies on temporal multiplexing: it displays different parts of the representation space at different times. However, e-readers use a text container metaphor such as a pagination metaphor that reproduces the pages of a printed book or a scroll metaphor. Their composition mappings are then better described as chains of two mappings: First, the screen is bound to a specific virtual display surface (a virtual page or a viewport on a virtual scroll) using temporal multiplexing. Then, text is placed in this virtual display surface using spatial multiplexing (different parts of the virtual display surface are bound to different words in the text). Fluid documents [7] and fish-eye lenses use a similar mixed strategy, however the time-multiplexed mapping introduces a geometrical deformation in addition to defining a viewport. Temporal and spatial multiplexing can also be used to describe other features: a display surface can be split using spatial multiplexing so that one part of it displays the text at another level of detail (creating an overview) or at another position (e.g. to display footnotes).

# 5 Designing for Bilingual Reading

## 5.1 Four Challenges Highlighted by the Interviews

The bilingual reading mechanisms presented in the previous section offer a complex yet incomplete model of this activity: it ignores emotional aspects and important cognitive mechanisms such as working memory, and oversimplifies user input mechanisms. This framework is however sufficient to describe the following challenges for the design of bilingual reading appliances, which were all derived from an analysis of the concerns spontaneously expressed by the people we have interviewed.

**Secondary Tasks.** The main task in reading is to acquire the text meaning by focusing sequentially on (most of the) successive words. This task only uses one type of nexus corresponding to positional links. Other tasks can involve other types of nexuses and they can also be very important to make the reading experience rich and diversified. Obviously, language links are particularly important for bilingual reading. Hierarchical links are used to define the context, as we learned from some participants of the reading traces experiment: they marked where translated paragraphs should be split to match the original ones. Links to glossaries or to the first occurrence of a proper name were frequently requested, as well as links to external references such as dictionaries or encyclopedias.

**Bi-focal Tasks.** Some tasks require considering simultaneously two elements, which imply to focus alternatively on them with frequent focus switches. An obvious example of such a task would be a comparison task, e.g. comparing the syntaxes used in a sentence and its translation. A more straightforward example for our concerns is a task where the information provided by the focus of attention is better understood in the light of another element, which becomes a second possible focus. The necessity to support this general class of tasks has been recognized for a long time in HCI: many fo-
cus+context techniques have been designed to deal with situations where the context helps to understand the focus [14] and compared using the notions of spatial and temporal multiplexing [10]. These techniques use mainly the hierarchical links to define the context, but language links can also be used for bilingual reading. For instance, the translation of a sentence provides a context that can help understand the meaning of an unknown word in that sentence. We have accordingly observed people using their two index fingers to keep track of the two focuses of attention. Using our framework, we define *bi-focal tasks* as tasks where two points from the representation space need to be accessed quickly from one to the other and back. These two points can be connected by any kind of link defined in the document, e.g. by features such as footnotes, references and figures. They can also be defined on the fly by the reader, e.g. to support light-weight navigation such as *"looking ahead in the text to preview or anticipate"* [24].

**Staying in the Flow.** Many interviewees stressed the fact that when they had to pause their reading because they could not understand a word or sentence, they felt very unsatisfied. The resulting loss of context and the non-satisfaction of their desire to advance in the story cause this feeling, but it may just be a special case of the frustration that arises when somebody's flow of thoughts is interrupted. The challenge of creating interfaces that allow *staying in the flow* (as defined by Csikszentmihalyi) is well known in HCI. Bederson, for example, advise to make interfaces that encourage users to develop their skills; that reduce the need for users to consciously make connections between different interface states; that give users the feeling of being in control; and that let users set clear goals and have feedback on their progression [2]. Reading is considered as a good example of activity that can create a state of flow, and it is easy to see how a book, taken as an interface, follows Bederson's advices. Reading is mostly a navigating in the text: instead of using commands to follow the positional links, the reader rather sets the display resources in a state that allows her to follow these links visually and unconsciously. This principle also appears in the lightweight navigation techniques used by people who read paper magazines [24].

The secondary tasks discussed earlier are often supported explicitly by specific commands such as clicking on a hyper-link. Instead of providing such direct support for some tasks, a strategy that keeps the reader in the flow could then be to let her *navigate* in a continuous space defined by the rendering process. This way, the number of controls required is reduced and the user only needs to manipulate the view until she reaches a setting that allow her to accomplish her task. We thus see an excellent design opportunity in a more pervasive use of navigation as an interaction scheme. Relying on navigation in continuous spaces brings another opportunity, with the design of interaction techniques that change the position of the view simultaneously along multiple dimensions. Such an *integral navigation* has been studied for the dimensions of space and scale, with expected and measured benefits for long-range navigation [1,15].

**Breaking the Container Metaphor.** If we do not want the expression “bilingual reading” to be a paraphrase for “mostly monolingual reading where a different language is sometime used”, we probably need to escape from the box-like container metaphor. It is indeed possible to display some text without making it look like being part of a page or
scroll. For instance, RSVP departs strongly from such layouts [30]. The presentation tool Prezi<fnref n="3" /> has been cited during the interviews because of its non-conventional text flow: it puts all slides in an unbound 2D zoomable space so that they can intersect with or contain other slides. And while these slides are not otherwise different from paper ones, transitions between slides are animated like navigation in the zoomable space, which makes viewing them a quite different experience. In our framework, container metaphors make use of the particular combination of temporal and spatial multiplexing described in section 4.3. From this, a clever designer can devise other strategies by slightly modifying the bindings and spaces involved. For example, *content-aware scrolling* [22] works in a similar way than a scroll metaphor, but defines a continuous path in the document and allows the viewport's size to change to adapt to the document's content. The *horizontal scroll* [4] uses the spatial multiplexing strategy already used in multi-column layouts and mixes it with the temporal multiplexing strategy used by classical scrolling. These techniques illustrate how our framework opens a design space for new text display methods, which need to be explored in search of efficient techniques to display bitexts.

## 5.2 Analysis of Existing Approaches

To better understand the space of solutions to the previous challenges, we have additionally conducted an extended review of the HCI literature, looking for works that addressed at least one of these challenges. These works can be described as using one of three different approaches that we introduce and describe in the following sections and Fig. 3. We discuss the potential benefits and drawbacks of each approach, as well as some related technical issues that need to be addressed.

**Switch on Demand.** Techniques belonging to the *switch on demand* approach use a single display surface. The ones that display text on this surface do it with a standard container metaphor. The techniques support bi-focal tasks by letting the user explicitly select the link between the two focuses, e.g. by clicking on a special element such as a hyper-link. A view that contains the destination of the link is then displayed in a tooltip-like transient visualization, which uses temporal multiplexing for the composition mapping. Another command allows closing that tool-tip and getting back to the original view. This use of transient visualizations is believed to "bring the user into direct and instant involvement with the information representation", use display resources economically, and provide fast access to the information because of its closeness [23]. The drawbacks of this approach are related to its use of an explicit selection of the link that defines the target focus element. Indeed, selecting the link by its source element can be a difficult task for some types of content or interfaces (e.g., using a touch screen). A second problem comes from the fact that a single element may involve multiple nexuses of attention, calling for different targets: this is the "secondary tasks" design challenge. Altogether, this approach might require carefully designed interaction techniques to select at the same time a focus of attention and the type of nexus (or command) to use.

<footnote n="3">³ http://prezi.com/</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Position-focus diagrams for the three discussed approaches. For bilingual reading, the two focuses *f*₁ and *f*₂ are two different languages and these diagrams are a projection of the representation space where the linguistic level is not represented. The blue rectangles represent the set of points displayed on some display surface at a given time. Arrows represent possible navigation actions.</figcaption></figure>

**Synchronized Views.** This approach consists in having one display surface by focus (spatial multiplexing), which are synchronized so that they always display corresponding parts: when the user changes the position or level of detail in one surface, the other one is automatically updated (temporal multiplexing). This approach replaces an explicit selection of the nexus involved in a bi-focal task by a continuous use of similar nexuses, allowing using navigation instead of explicit commands. On the other hand, the approach can only be applied with nexus types that define a correspondence for each subset of the document (which is the case for language nexuses). This approach can also make it difficult to analyze the alignment structure as it is used indirectly in the rendering process. And obviously, it uses more display resources.

Spatial multiplexing can be done in two ways for this approach. First, the two display surfaces can be laid out next to each other, as a split screen or using different devices. This is the strategy used for *parallel texts*: considering the virtual display surfaces formed by a book's open pages, each side corresponds to a different focus. Second, a surface can be laid on top of another, and render the second focus associated with the area hidden below. For instance, *magic lenses* and *magnification lenses* use a lens as the second focus surface; *mobile augmented reality* uses a display surface belonging to a different object held between the original display and the user's eyes.

A major drawback of *parallel texts* is their low granularity, as seen in section 2.1. It makes it hard to locate in the second view the part corresponding to some known element of the first view, e.g. a single word. The searched part being unknown, it can only be identified using the context, which requires some interpretation and often many eye movements back and forth the two views. All techniques using synchronized views suffer from similar issues unless there is a geometrical correspondence between the two views (which is usually the case with mobile AR, but not for bilingual reading). This drawback can however become a benefit if the reader needs to access frequently the second view, as she will memorize the structure and spatial position of previously visited elements.

The approach also has a rendering issue, which is to maintain a coherent second view when the first one is scrolled (see Fig. 4). Indeed, the different ordering of elements can cause some elements that are not relevant anymore to stay in the second view. This view may also have to display more elements than it can contain. Solutions to these problems
<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Coherence issues with synchronized views. Left: paragraph boxes for two versions of a same text in different languages with the corresponding alignment structure. Center: initial views for each language. Right: after scrolling down in view 1, paragraph *a* becomes irrelevant in view 2, but paragraph *c* should be displayed. Two possible renderings for view 2 are provided, with global rescaling (bottom) or ellipsis (top).</figcaption></figure>

may need to break the container metaphor by dynamically changing the size of dis-
played elements and hiding some, e.g. using ellipsis. Techniques inspired by other prop-
erties of paper such as folding could be used, as *Melange* did for bifocal tasks that use
positional or hierarchical links [13].

Magic lenses and Mobile AR provide another opportunity, as they can be used in two
ways: Either the display surface of the second view is kept in a static position and scroll-
ing in the first view triggers updates for the second view, or the display surface is
moved (and eventually resized) relatively to the first view, which will also trigger up-
dates. The second option allows the reader to specify what part of the first view she is
interested in by using a spatial navigation scheme. The corresponding manipulations of
the magic lens or hand-held device are however rather cumbersome and might require
new developments.

**Animated Transition Between Focuses.** With this approach, a single view is used. An element displayed in this view can be selected and, on demand, the whole view content is replaced by another content so that the view now displays the second focus associated to the selected element. Like synchronized views, this approach requires a type of nexus that can be defined on any subset of the document, but uses a temporal rather than spatial multiplexing scheme. When the two focal elements only differ in position or linguistic level, the transition can use animated panning and zooming. Otherwise, there might be no natural way of animating the transition. *Diffamation* [8] and *Gliapse* [12] are notable examples showing that efficient animated transitions can however be devised for some types of textual data. These works use apparition, removal and displacement of text blocs, text lines and other graphics to animate the transition. We refer to this approach as *morphing* as it uses the shapes and positions of visual elements and their hierarchical relations to define the animation. These works highlight the need for advanced techniques such as animating paragraphs reflow, using curved trajectories and stabilizing the view vertically on focal elements. The benefit of these methods is to allow the user to memorize the two documents layouts, and to quickly switch from one document to the equivalent position in the other one. Because animations are deterministic, they can also be memorized to make afterward animations easier to follow. However, animations can also confuse the user if she cannot keep track of what is moving where, so the flow of animation has to be carefully tuned.
## 5.3 Adapting these Approaches to Bilingual Reading

**Switch on Demand.** For bilingual reading, this approach might be the most obvious one and is actually used by current software (see section 2.1). Because selecting text is difficult when done with the usual techniques, one may want to reduce the set of possible selections to full words, full sentences, full paragraphs, etc., taking advantage of the small number of linguistic levels. Yet, the hierarchical organization of such elements prevents the use of simple selection techniques like picking. Existing materials thus only allow the selection of sentences, reducing the granularity of the alignment structure. There is therefore room to design more efficient selection techniques. One possibility would be to reify the various linguistic levels into new in-page interactive elements that do not overlap. Such elements are already used in pop-up books to make elements appear, hide, transform or move in response to user actions. They could be used in similar ways for bilingual reading, e.g. to reveal the translation of a paragraph by raising the (virtual) flap on which it is printed. Simulations of paper folding and bending could also add realism to the space distortions introduced by techniques like fluid documents.

**Synchronized Views.** The synchronized views strategy used by lenses and mobile augmented reality might be inefficient for bilingual reading, as the original and translated texts can have very different layouts. The other strategy offers however many possibilities that are worse exploring, considering the diversity of behaviors that they support (as observed in the reading traces experiment). Notably, using a second device like the reader's smartphone for the second view provides another interaction surface and opens a new space for simple gestures based on the proximity and orientation of this device.

Finally, it is possible to mix the two strategies. For instance, the two views could be defined by splitting a screen along the vertical dimension rather than along the horizontal one like parallel texts do. The top and bottom view would use different languages, and the bottom view would display the text that corresponds to what immediately follows the text displayed in the top view. The reader can then get the translation of what she just read in the bottom view simply by scrolling down, which would move the text read toward the top view where its translation would appear.

**Navigation Between Languages.** Morphing text from one language to another is more complicated than what *Diffamation* and *Gliapse* have done, because there may be no common visual element between the start and final views. There are however many opportunities to create such a morphing: Elements can be morphed differently according to their linguistic level, in the spirit of distinguished visual transitions [34]. The morphing can better use the structure of the document by using successive steps that each address a lower linguistic level. For the animation of changes in abstract trees, this approach allows to easily understand what elements are transformed and how [17]. The last morphing step can then replace words from the first language with their translation.

We also suggest exploring the idea of letting the user control the morphing evolution instead of following a time-controlled animation. This could bring the feeling of navigating between languages. Midway positions could also be valuable, as they expose the linguistic structure of the text, which is sometime enough to relieve ambiguities. The reader can control the animation speed, skip it entirely or go backward, depending on
her needs. By coupling in a single gesture this morphing control together with the initial selection of a word to stabilize, we could get similar benefits than with the “switch on demand” approach: a precise selection of the element whose translation is wanted, at the right linguistic level. On the other hand, the interleaving of words belonging to two distinct languages could also be a source of problems for specific readers (e.g. dyslexic ones) or purposes (e.g. language teaching).

**Mixing Approaches.** These approaches are not necessarily incompatible: they could be used together, or the reader could switch softly from one to another. For example, a tooltip created in a “switch on demand” approach could be turned into a permanent synchronized view. Interactions techniques developed to select text in a “switch on demand” approach could still work for synchronized views, highlighting the part of the second view corresponding to the selection. Similarly, the animation techniques developed for the “navigation between languages” approach could be used to solve the display coherence issues appearing when synchronized views are scrolled.

# 6 Future Work

In our future work, we will follow two parallel tracks. The first one is the continuation of our reflection on the design methods required to better understand how to design for bilingual reading. The second one concerns the realization of prototypes. We hope that the two tracks will converge in the longer term, e.g. in the form of technological probes.

For the realization of prototypes, the three approaches presented in section 5 revealed the usefulness of three types of operations that can be used in combination: advanced text selection operations required for the “switch on demand” approach, space distortion techniques required for the “synchronized views” approach, and structured morphing strategies required for the “navigation between languages” approach. We have already started the development of a framework that provides these operations and helps exploring new solutions.

More importantly, the creation of design methods will require complementing, formalizing and evaluating the design methods that we have started to use, such as the reading traces experiment. One question that we want to answer is whether presenting to potential users the approaches introduced in section 5.3 would trigger their imagination and allow them to provide even more insights about their bilingual reading experiences? Finally, the question of evaluating interfaces for bilingual reading is complex and will need further developments.

# Acknowledgments

This work has been funded by the ANR CONTINT Transread project. We would like to thank Delphine Soriano for her useful and much appreciated help, as well as all the people we interviewed.
References

1. Appert, C., Fekete, J.D.: Orthozoom scroller: 1d multi-scale navigation. In: Proc. of CHI '06. p. 21–30. ACM, New York, NY, USA (2006)

2. Bederson, B.B.: Interfaces for staying in the flow. Ubiquity 2004 (September), (Sep2004)

3. Bederson, B.B., Quinn, A., Druin, A.: Designing the reading experience for scanned multi-lingual picture books on mobile phones. In: Proc. of JCDL '09. p. 305–308. ACM (2009)

4. Braganza, C., Marriott, K., Moulder, P., Wybrow, M., Dwyer, T.: Scrolling behaviour with single- and multi-column layout. In: Proc. of WWW '09. p. 831–840. ACM (2009)

5. Bucher, P., Chatty, S.: Qu'est-ce qu'une surface d'affichage? : une analyse rétrospective. In: Proc. of IHM '09. p. 3–12. ACM, New York, NY, USA (2009)

6. Buscher, G., Biedert, R., Heinesch, D., Dengel, A.: Eye tracking analysis of preferred reading regions on the screen. In: CHI EA '10. p. 3307–3312. ACM, New York (2010)

7. Chang, B.W., Mackinlay, J.D., Zellweger, P.T., Igarashi, T.: A negotiation architecture for fluid documents. In: Proc. of UIST '98. p. 123–132. ACM, New York, NY, USA (1998)

8. Chevalier, F., Dragicevic, P., Bezerianos, A., Fekete, J.D.: Using text animated transitions to support navigation in document histories. In: Proc. of CHI '10. p. 683–692. ACM, New York, NY, USA (2010)

9. Clark, C., Rumbold, K.: Reading for pleasure: A research overview. National Literacy Trust (2006)

10. Cockburn, A., Karlson, A., Bederson, B.B.: A review of overview+detail, zooming, and focus+context interfaces. ACM Comp. Surv. 41(1), 2:1–31 (Jan 2009)

11. Dehaene, S.: Reading in the Brain: The New Science of How We Read. Penguin Group US (2009)

12. Dragicevic, P., Huot, S., Chevalier, F.: Gliimpse: Animating from markup code to rendered documents and vice versa. In: Proc. of UIST '11. p. 257–262. ACM (2011)

13. Elmqvist, N., Henry, N., Riche, Y., Fekete, J.D.: Melange: space folding for multi-focus interaction. In: Proc. of CHI '08. p. 1333–1342. ACM, New York, NY, USA (2008)

14. Furnas, G.W.: A fisheye follow-up: Further reflections on focus + context. In: Proc. of CHI '06. p. 999–1008. ACM, New York, NY, USA (2006)

15. Furnas, G.W., Bederson, B.B.: Space-scale diagrams: understanding multiscale interfaces. In: Proc. of CHI '95. p. 234–241. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1995)

16. Gao, G., Wang, H.C., Cosley, D., Fussell, S.R.: Same translation but different experience: The effects of highlighting on machine-translated conversations. In: Proc. of CHI '13. p. 449–458. ACM, New York, NY, USA (2013)

17. Guilmaine, D., Viau, C., McGuffin, M.J.: Hierarchically animated transitions in visualizations of tree structures. In: Proc. of AVI '12. p. 514–521. ACM, New York (2012)

18. Han, C.H., Yang, C.L., Wang, H.C.: Supporting second language reading with picture note-taking. In: CHI EA '14. p. 2245–2250. ACM, New York, NY, USA (2014)

19. Hinckley, K., Dixon, M., Sarin, R., Guimbretiere, F., Balakrishnan, R.: Codex: A dual-screen tablet computer. In: Proc. of CHI '09. p. 1933–1942. ACM, New York (2009)

20. Hornbæk, K., Frøkjær, E.: Reading patterns and usability in visualizations of electronic documents. ACM ToCHI 10(2), 119–149 (Jun 2003)

21. Hyrskykari, A., Majaranta, P., jouko Räihä, K.: Proactive response to eye movements. In: Proc. INTERACT 2003. p. 129–136. IOS Press (2003)

22. Ishak, E.W., Feiner, S.K.: Content-aware scrolling. In: Proc. Of UIST '06. p. 155–158. ACM, New York, NY, USA (2006)
23. Jakobsen, M.R., Hornæk, K.: Transient visualizations. In: Proc. of OZCHI '07. p. 69–76. ACM, New York, NY, USA (2007)

24. Marshall, C.C., Bly, S.: Turning the page on navigation. In: Proc. of JCDL '05. p. 225–234. ACM, New York, NY, USA (2005)

25. Morineau, T., Blanche, C., Tobin, L., Guéguen, N.: The emergence of the contextual role of the e-book in cognitive processes through an ecological and functional analysis. International Journal of Human-Computer Studies 62(3), 329–348 (2005)

26. Nelson, T.H.: Xanalogical structure, needed now more than ever: Parallel documents, deep links to content, deep versioning, and deep re-use. ACM Computing Surveys 31(4es) (Dec 1999)

27. O'Hara, K.: Towards a typology of reading goals. Tech. rep., Xerox RXRC (1996), http://www.xrce.xerox.com/content/download/16322/117657/ file/EPC-1996-107.pdf

28. O'Hara, K., Sellen, A., Bentley, R.: Supporting memory for spatial location while reading from small displays. In: CHI EA '99. p. 220–221. ACM, New York, NY, USA (1999)

29. Piolat, A., Roussey, J.Y., Thunin, O.: Effects of screen presentation on text reading and re-vising. International Journal of Human-Computer Studies 47(4), 565–589 (1997)

30. Potter, M.C.: Rapid serial visual presentation (rsvp): A method for studying language pro-cessing. New methods in reading comprehension research 118, 91–118 (1984)

31. Rainie, L., Zickuhr, K., Purcell, K., Madden, M., Brenner, J.: The rise of e-reading (april 2012), http://libraries.pewinternet.org/2012/04/04/ the-rise-of-e-reading/

32. Schilit, B.N., Golovchinsky, G., Price, M.N.: Beyond paper: Supporting active reading with free form digital ink annotations. In: Proc. of CHI '98. p. 249–256. ACM Press/Addison- Wesley Publishing Co., New York, NY, USA (1998)

33. Schilit, B., Price, M., Golovchinsky, G., Tanaka, K., Marshall, C.: The reading appliance revolution. Computer 32(1), 65–73 (Jan 1999)

34. Schlienger, C., Dragicevic, P., Ollagnon, C., Chatty, S.: Les transitions visuelles différen-ciées : principes et applications. In: Proc. of IHM '06. p. 59–66. ACM (2006)

35. Shibata, H., Omura, K.: Effects of paper on page turning: Comparison of paper and elec-tronic media in reading documents with endnotes. In: Harris, D. (ed.) Engineering Psy-chology and Cognitive Ergonomics, Lecture Notes in Computer Science, vol. 6781, p. 92–101. Springer Berlin Heidelberg (2011)

36. Subbaram, V.M.: Effect of display and text parameters on reading performance. Ph.D. the-sis, The Ohio State University (2004)

37. Tashman, C.S., Edwards, W.K.: Active reading and its discontents: The situations, prob-lems and ideas of readers. In: Proc. of CHI '11. p. 2927–2936. ACM, New York, (2011)

38. Tiedemann, J.: Bitext Alignment. No. 14 in Synthesis Lectures on Human Language Tech-nologies, Graeme Hirst (ed), Morgan & Claypool Publishers (2011)

39. Tucker, G.R.: A global perspective on bilingualism and bilingual education. Georgetown University Round Table on Languages and Linguistics 1999 p. 332 (2001)

40. Wästlund, E., Norlander, T., Archer, T.: The effect of page layout on mental workload: A dual-task experiment. Computers in Human Behavior 24(3), 1229–1245 (2008)

41. Yu, C.H., Miller, R.C.: Enhancing web page readability for non-native readers. In: Proc. of CHI '10. p. 2523–2532. ACM, New York, NY, USA (2010)

42. Zellweger, P.T., Regli, S.H., Mackinlay, J.D., Chang, B.W.: The impact of fluid docu-ments on reading and browsing: an observational study. In: Proc. of CHI '00. p. 249–256. ACM, New York, NY, USA (2000)