# Aestimo: A Tangible Kit to Evaluate Older Adults' User Experience

Iyubanit Rodríguez, Maria Karyda, Andrés Lucero, Valeria Herskovic

► To cite this version:

Iyubanit Rodríguez, Maria Karyda, Andrés Lucero, Valeria Herskovic. Aestimo: A Tangible Kit to Evaluate Older Adults' User Experience. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.13-32, 10.1007/978-3-030-29381-9_2. hal-02544540

HAL Id: hal-02544540

https://inria.hal.science/hal-02544540

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Aestimo: A Tangible Kit to Evaluate Older Adults' User Experience

Iyubanit Rodríguez<fnref n="1" />[0000-0003-0878-4076], Maria Karyda<fnref n="2" />[0000-0002-3727-3665], Andrés Lucero<fnref n="2" />[0000-0002-7176-2884], and Valeria Herskovic<fnref n="3" />[0000-0002-2650-6507]

**Abstract.** Surveys and questionnaires are commonly used to capture people's experiences with technology. However, some older users experience issues when reading and filling out forms, and inexperienced computer users may not be comfortable with web-based versions. To improve the report of user experience, we designed and implemented *Aestimo*, a tangible interface based on a shortened version of the AttrakDiff questionnaire. The interface was evaluated during a study with 20 older adults (age avg.=65.6). Although completing the *Aestimo* questionnaire took longer than a paper-and-pen version of AttrakDiff, 60% of participants preferred *Aestimo* over AttrakDiff. *Aestimo* was found to be innovative and inviting, and to stimulate the senses and the mind. Participants liked feeling guided by the interface, and also found their experience to be playful and fun. Overall, the evaluation of *Aestimo* was highly positive and suggests that some user groups may benefit from the availability of innovative evaluation experiences.

**Keywords:** User experience · older adults · tangible interface.

## 1 Introduction

An important part of the design of products or systems is to measure user experience (UX) to understand users’ perception about a product [1]. The dominant UX evaluation type are questionnaires [2]. Surveys or questionnaires are used to collect opinions, data, perceptions or feelings about the use of technology in a short period of time [3]. AttrakDiff is one of the questionnaires that have been used to evaluate UX. More specifically, the questionnaire is used to understand the usability and design of an interactive product, and measure hedonic and pragmatic qualities of a product [4].

At what age we humans become *older adults* cannot be universally defined due to cultural and historical differences. In some parts of the world, a person over 50 is considered an older adult [5]. United Nations considers people over 60

<footnote n="1">¹ Universidad de Costa Rica, Costa Rica</footnote>

<footnote n="2">² Aalto University, Finland</footnote>

<footnote n="3">³ Pontificia Universidad Católica de Chile, Chile</footnote>

<footnote>iyubanit.rodriguezramirez@ucr.ac.cr, maria.karyda@aalto.fi</footnote>

<footnote>lucero@acm.org, vherskov@ing.puc.cl</footnote>
to be older adults [6], which coincides with the retirement age in Chile, where this study took place [7]. Partly, this is because the aging process is not uniform, due to differences in lifestyle, genetics and health [8]. As older adults, we may experience issues when asked to answer surveys or questionnaires. Questionnaires can often be long [9], and users with reduced vision may have trouble reading them [10], while users with hand-eye coordination issues may have difficulty writing with a pen [11]. It is claimed that some older adults tend to choose “I do not know” when answering a questionnaire [12], or to respond randomly [11]. Evaluations of technology in this type of situation have occasionally used shorter or adapted versions of standard questionnaires [13] or interviews [10].

Although web-based questionnaires solve some of the accessibility problems of paper-based questionnaires, some potential evaluation participants are inexperienced computer users or lack the necessary equipment. A lack of familiarity in technology use may produce anxiety [14]. Besides the aforementioned contextual differences in identifying when we become older adults, there are similar differences in the use of technology between the Western world and developing countries. In Chile, where this project was developed, some studies suggest that the use of technology by older adults is limited, finding that around 60% have no computer experience [15]. Therefore, a part of the population in the developing world is currently being neglected in their needs with the assumption that most people have easy access to current technology and that anybody can get by using a mobile device.

Multi-touch interaction may be uncomfortable for users with reduced coordination between motor and cognitive skills [16]. Some researchers have recommended the use of tangible user interfaces (TUI), e.g. tangible interfaces with push buttons [17]. Physical interactions may remind users of devices we are all familiar with [18]. In the case of users who are insecure in their abilities to interact with technology, this may allow confidence to increase [19].

In this paper, we aim to explore the following research questions: 1) *Are the results obtained through a TUI similar to those from a paper-and-pen-based questionnaire?*, and 2) *What is the perception of older adults of a TUI for reporting UX?*. To answer these questions, we designed a tangible interface kit called *Aestimo* (based on a prototype described in [20]), which allows users to input their user experience, emotion and overall opinion. Subsequently, we recruited 20 older adults to test our interface. During the evaluation we asked participants to evaluate a simple puzzle application on a tablet device, and then to answer a paper-and-pen questionnaire and a shortened version of the questionnaire on the *Aestimo* device.

This paper extends previous HCI research on the use of tangible technologies to enable data collection [21, 22]. However, this work is focused on improving the way UX is reported by older users. We describe in this paper the *Aestimo* device, a tangible interface that allows UX reporting, as well as our findings regarding how such an interface compares to a paper-based questionnaire, and our insights about the preferences and needs of older adults when recording their user experience. We found that the participants enjoyed that the interface was
built with familiar materials and physical interaction styles, and that in the case
of participants with little experience with technology, an interface such as this
one should be designed to be intuitive and self-explanatory.

2 Related work

**2.1 Older adults and tangible interfaces**

Research on creating technological systems for older adults reports that TUIs
may be more efficient and familiar than current digital technologies. Studies
have focused on creating TUIs with diverse goals, e.g. to improve social com-
munication [23–26], playing [27], entertainment [28], and cognitive training [29,
30].

There are several techniques that may be used to design TUIs, combining
digital interfaces with familiar physical objects. There are several examples of
interfaces designed for a specific population of older adults, e.g. an interface in
which people may compose digital messages in a way that resembles traditional
ways of writing a letter [31], using individual tangible tokens on a digital board
[28], or creating a multimedia book that combines pictures and sounds for rem-
iniscing purposes [29]. During the evaluation of those systems it appeared that
there were often unclear instructions, need for improvement in the feedback or
even physiological characteristics (e.g. tremor) that were not taken into account
when designing those systems. This work is based on the understanding that
utilizing familiar elements in the design of the tangible interface is key when
designing for older adults with little technological experience.

In terms of modalities we can see the aforementioned projects involve projec-
tion or a digital screen complementing a physical prototype. Blossom [25], a TUI
for asynchronous voice messages, is an example of combining digital and physi-
cal modalities. The prototype includes a fake vase with flowers, which resemble
messages waiting for a response. A frame which includes a picture of close family
members complements the physical prototype.

Several works have described their prototypes as tangible when they are ac-
tually touch-screen, mobile, or computer-based [26]. Our work combines several
modalities such as sound, haptic feedback and visual modalities, without any
visible digital information or interaction. A similar prototype, TanCu [32], does
not use screens, rather relying on haptic and visual engagement with the user. In
terms of aesthetics, we believe it is the closest to *Aestimo*, since it avoids screens
and focuses on other modalities.

**2.2 Tangible interfaces to collect people's opinion**

Tangible interfaces have been used to provide support in the collection of data
through physical interaction. The creation of such tangible interfaces has been
used mostly in crowded public spaces. Among these, VoxBox [21] was designed
to collect public opinion of an event in a playful way, finding that the anxiety
of using a novel technology can be mitigated by familiar physical interactions. Another example of an interface to collect information from people is an electronic poster that allows community voting, which was found to be as intuitive as conventional posters, but with the advantages of online surveys [33]. To collect the opinion of communities, a tangible interface was designed with two buttons on the ground (yes/no) to allow voting. This type of interaction attracted the attention of the participants and was easy to use [34]. Sens-Us is an interface that allows collecting personal and sensitive information. The questions are answered by means of sliders, rotary dials, toggle, check buttons and yes/not push buttons, and it was found that people knew intuitively how to interact with the interface despite it being new to them [22]. Tangible interfaces have been used to improve the capacities of attraction and participation to provide information to people, since physical characteristics encourage people to approach these types of devices [35].

In the same way, tangible interfaces have been used to collect the perception of children through surveys in age-appropriate ways. SmallTalk is a tangible survey system designed to be used within a theater space to capture what children thought of a show they just watched [36]. Another example is a paper rating scale for children to provide their answers visually by means of non-adhesive stickers [37]. In general, although tangible interfaces have been created to collect and crowdsource the perception of people, they have not taken us older users into account, with our specific physical characteristics and limitations, and they have also not been used in this way in a more private space to collect user experience information.

# 3 Aestimo

*Aestimo* is based on a previous work (published as late breaking work) focusing on the design of *Aestimo*, which was at the time a non-functional prototype [20]. Instead, this paper presents (1) *Aestimo*'s implementation and (2) the evaluation of the functional prototype with 20 older adults.

*Aestimo* is a tangible interface kit to evaluate UX. The kit includes the evaluation of the user experience, emotional dimensions and the general opinion about the technology that is being evaluated. *Aestimo* uses familiar interactive styles and proposes a playful experience.

## 3.1 Design

The motivation to make *Aestimo* arose from the challenges observed in several evaluations while using questionnaires, in which e.g. some older adults wanted to hear the questions out loud. This section explains the design rationale behind *Aestimo*.

First, we selected AttrakDiff, a validated questionnaire that measures UX, which we have used in several prior evaluations. Based on our experience using AttrakDiff with older adults, we decided to make a shorter version of the
questionnaire to better fit their needs. The number of AttrakDiff questionnaire items was reduced from 28 to 16 questions. In this step the 4 dimensions of AttrakDiff were maintained, but instead of having 7 questions for each dimension, we reduced them to 4 (i.e., two positive and two negative adjectives). Each pair of adjectives was then turned into a question, e.g. *Conservative-Innovative* became "Is the application innovative?". In addition to reducing the number of questionnaire items, we also noticed in previous evaluations that most participants ended up using only three points on the scale instead of the seven available points. Therefore, we simplified the scale to three points to answer the questionnaire: *Yes*, *Neutral* and *No*.

Second, we looked into 27 electronic devices from the 1970s, e.g. Walkman personal stereos, washing machines, and typewriters, to get inspired about classic interaction styles that may seem familiar and playful to older adults. We found 45 types of interaction styles that require physical manipulation. The most commonly used interaction styles were buttons, sliders and knobs.

Third, continuing on the idea of using familiar, approachable elements, we decided to design AttrakDiff as a book and a handset. Each page of the book displays a question with large print. The handset works in conjunction with the book, as it reads the question that the book presents. Combining a book with a handset further served the purpose of providing users with double feedback: visual and auditory, to give support to the person in case that have any limitations in these two areas. A physical knob similar to that of a cooking appliance was selected as the main means to provide an answer to a given question on our three-point scale.

Fourth, we decided to incorporate two more dimensions to report user experience. The first one was an **emotional aspect**, in which the user can indicate how they felt while using the product or service to be evaluated. This decision was inspired by customer feedback that one can normally find in stores and airports <fnref n="4" />, which show 4 faces: *very happy*, *happy*, *sad* or *very sad*. A slider (similar to that of an old radio) was included as a way for the users to select one of the four faces. The second aggregate dimension was **overall feedback** where the person can express in their own words how they felt while using the product or service. A vintage type tape recorder with two buttons was used for this. One button was used to start recording, the other to stop recording. The person can hear and feel the buttons when they are being pressed, which allows them to receive feedback while using the recorder.

Fifth, the interface can guide the user through instructions, which are presented in the form of text engraved on the surface, and are also heard by the handset. Finally, with respect to materials, *Aestimo* was made using laser-cut plywood (for the exterior, buttons, slider and knob) and paper (for the book). The materials were chosen with the purpose of being inviting and manageable.

Table 1 presents the explanation of each dimension.

<footnote n="4">⁴ https://www.happy-or-not.com/en/</footnote>
<table><caption>**Table 1.** *Aestimo* dimensions</caption><thead><tr><th>Dimension</th><th>Interaction</th><th>Description</th></tr></thead><tbody><tr><td rowspan="3">User experience</td><td>Book</td><td>Displays questions.</td></tr><tr><td>Handset</td><td>Plays audio of the instructions and the currently open question in the book.</td></tr><tr><td>Knob</td><td>Allows answering each question, with three options: yes, no, neutral.</td></tr><tr><td rowspan="2">Emotional</td><td>Handset</td><td>Play audio instructions.</td></tr><tr><td>Slider</td><td>Allows to select the emotional state (4 faces).</td></tr><tr><td rowspan="2">Overall opinion</td><td>Handset</td><td>Plays audio instructions.</td></tr><tr><td>Buttons</td><td>Record the user's opinion.</td></tr></tbody></table>

## 3.2 How is *Aestimo* used?

To use *Aestimo*, the user first reads a set of instructions directly from the device, leading them to open a lid, where the user finds a handset and picks it up (see Figure 1). The handset plays the instructions and questions about the user experience, which are synchronized with the questions in the book. After listening to each question, the user is invited to answer by choosing one of three options (yes, neutral, no) using the knob on the right front of the prototype. The system reminds people in case they have not responded to a question from the book that they cannot continue to the next phase until they answer all the questions from the book.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** *Aestimo*: tangible interface kit (33.9 cm, 26 cm, 23.8 cm).</figcaption></figure>
After closing the last page of the book, a new part of the interface is revealed (see Figure 2), in which users are asked to record an emotion and their overall opinion. The emotional dimension presents four faces with different moods and the user has to select the face that identifies the most with his/her feelings by means of a slider. The last part is the general opinion. The user is invited to record their thoughts on the prototype by pressing the *rec* button and then press the *stop* button to complete the recording. When a button is pressed, the corresponding sound of a mechanical tape recorder button is played.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Left: book. Right: emotional aspect and the overall feedback.</figcaption></figure>

## 3.3 Implementation

The interface was implemented using Arduino Mega 2560 R3 as a controller, a real-time clock (DS1307 I2C), classic headset, an 8-relay card with opto-coupled channels, a 12V battery and a SparkFun MP3 Player Shield for sound and information storage. The interface also has an on/off switch (see Figure 3). The implementation of each element of the interface is described next.

**Book** The book presents 16 questions, a shortened version of the UX questionnaire AttrakDiff, which were selected using as a base the version used in [20]. The final version of the questionnaire is presented in Table 2 in Spanish and English.

The book uses magnets so that the system knows which page of the book is open: if a page is making contact on both sides, the page is closed; otherwise, if the contact of the magnets is released the page is open. The system can then provide the corresponding question orally through the handset. The system also recognizes if the question was already answered, indicating to the user to proceed to the next question.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Diagram of the components of Aestimo.</figcaption></figure>

<table><caption>Table 2. Aestimo: book questions.</caption><thead><tr><td>N</td><td>Spanish Question</td><td>English Question</td></tr></thead><tbody><tr><td>1</td><td>¿Es la aplicación complicada de usar?</td><td>Is the application <i>cumbersome</i>?</td></tr><tr><td>2</td><td>¿La aplicación me motiva a usarla?</td><td>Does the application <i>motivate</i> me to use it?</td></tr><tr><td>3</td><td>¿Siento que la aplicación es desagradable?</td><td>Do I feel that the application is <i>disagreeable</i>?</td></tr><tr><td>4</td><td>¿La aplicación es atractiva?</td><td>Is the application <i>attractive</i>?</td></tr><tr><td>5</td><td>¿La aplicación es poco placentera?</td><td>Is the application <i>unpleasant</i>?</td></tr><tr><td>6</td><td>¿La aplicación no me cautiva?</td><td>Does the application not <i>captivate</i> me?</td></tr><tr><td>7</td><td>¿Usar esta aplicación representa un reto para mí?</td><td>Is using this application <i>challenging</i> for me?</td></tr><tr><td>8</td><td>¿La aplicación es convencional?</td><td>Is the application <i>conventional</i>?</td></tr><tr><td>9</td><td>¿La aplicación es innovadora?</td><td>Is the application <i>innovative</i>?</td></tr><tr><td>10</td><td>¿La aplicación es poco profesional?</td><td>Is the application <i>unprofessional</i>?</td></tr><tr><td>11</td><td>¿Siento que esta aplicación me acerca a la gente?</td><td>Do I feel that this application <i>brings me closer to people</i>?</td></tr><tr><td>12</td><td>¿Esta aplicación tiene una apariencia presentable?</td><td>Does this application have a <i>presentable</i> appearance?</td></tr><tr><td>13</td><td>¿La aplicación me aísla de la gente?</td><td>Does the application <i>isolate</i> me from people?</td></tr><tr><td>14</td><td>¿Es la aplicación manejable?</td><td>Is the application <i>manageable</i>?</td></tr><tr><td>15</td><td>¿La aplicación es simple de usar?</td><td>Is the application <i>simple</i> to use?</td></tr><tr><td>16</td><td>¿Fue confuso usar la aplicación?</td><td>Was the application <i>confusing</i>?</td></tr></tbody></table>
A knob is used to answer the questions. The knob was implemented using a Micro Metal Gearmotor HPCB 12V with Extended Motor Shaft, and a Magnetic Encoder Pair Kit for Micro Metal Gearmotors, 12 CPR, 2.7-18V (HPCB compatible). The space of the knob is divided into 4 positions (*yes*, *no*, *neutral* and *start*), which are demarcated by a range. The knob works by using a motor: when the person responds and turns the page, the motor positions the knob at *start* spot, so that the person answers the next question. The motor works in conjunction with the book's magnets.

**Emotional aspect** The emotional state displays four faces (representing four emotions: *very happy*, *happy*, *sad* or *very sad*), and asks the user to select one using the slider. The slider was implemented by means of a linear potentiometer, divided into five positions (*start* and the four emotions). After the user is asked to select a face, the system reads the slider's final position.

**Overall opinion** The recorder uses two push buttons and a recorder (Voice Recorder Pen 8 Gb Digital Usb). The recorder is turned on internally, when the instruction audio of that part of the evaluation starts. The recorder stores the information in its internal memory.

# 4 Method

## 4.1 Participants and Recruitment

We recruited 20 older adults (14F, 6M, 60-83 years, Avg. = 65.6, SD = 5.46). Participants were compensated with a 15 USD gift card for their time. The inclusion criteria was the following: 1) being 60 years of age or older, since the Ministry of Social Development in Chile considers people over 60 to be older adults [7]; 2) not having cognitive problems; and 3) speaking and understanding Spanish. Snowball sampling and signs near the entrance of campus were used to contact people to participate in the study. Out of 20 participants, eight had *none* and five had *low* digital skills, three had *basic* skills, and four had *above basic* digital skills, according to a digital skills questionnaire [38, 39]. 13 people had never previously used a Tablet device. Table 3 provides the participants' information.

**Ethical Considerations** The research protocol was approved by the university ethics committee (170711013). Each participant received oral and written information about the aim of the research, and written consent to the overall study was subsequently provided. Participants were informed that their involvement was voluntary, that anonymity would be guaranteed, and that they could withdraw from the study at any time.
<table><caption>Table 3. Description of study participants</caption><thead><tr><th>P</th><th>Age</th><th>Gender</th><th>Occupation</th><th>Educational level</th><th>Digital skills</th><th>Lives with</th></tr></thead><tbody><tr><td>P1</td><td>70</td><td>F</td><td>Retired</td><td>University</td><td>Above basic</td><td>Family</td></tr><tr><td>P2</td><td>65</td><td>M</td><td>Retired</td><td>School</td><td>Low</td><td>Family</td></tr><tr><td>P3</td><td>62</td><td>M</td><td>Manager</td><td>High School</td><td>Low</td><td>Family</td></tr><tr><td>P4</td><td>65</td><td>F</td><td>Retired</td><td>Technical</td><td>Basic</td><td>Alone</td></tr><tr><td>P5</td><td>60</td><td>F</td><td>Cleaning</td><td>School</td><td>Low</td><td>Family</td></tr><tr><td>P6</td><td>66</td><td>F</td><td>Housewife</td><td>School</td><td>None</td><td>Family</td></tr><tr><td>P7</td><td>63</td><td>M</td><td>Manager</td><td>Technical</td><td>Above basic</td><td>Family</td></tr><tr><td>P8</td><td>62</td><td>F</td><td>Cleaning</td><td>School</td><td>Low</td><td>Family</td></tr><tr><td>P9</td><td>64</td><td>F</td><td>Cleaning</td><td>School</td><td>None</td><td>Family</td></tr><tr><td>P10</td><td>69</td><td>F</td><td>Hairdresser</td><td>Technical</td><td>Low</td><td>Family</td></tr><tr><td>P11</td><td>60</td><td>M</td><td>Security Guard</td><td>High School</td><td>None</td><td>Family</td></tr><tr><td>P12</td><td>83</td><td>M</td><td>Housewife</td><td>None</td><td>None</td><td>Family</td></tr><tr><td>P13</td><td>60</td><td>F</td><td>Chemistry</td><td>University</td><td>Above basic</td><td>Family</td></tr><tr><td>P14</td><td>61</td><td>M</td><td>Chemistry</td><td>University</td><td>Above basic</td><td>Family</td></tr><tr><td>P15</td><td>71</td><td>F</td><td>Housewife</td><td>School</td><td>None</td><td>Family</td></tr><tr><td>P16</td><td>66</td><td>F</td><td>Cook</td><td>Technical</td><td>None</td><td>Family</td></tr><tr><td>P17</td><td>67</td><td>F</td><td>Merchant</td><td>High School</td><td>Basic</td><td>Alone</td></tr><tr><td>P18</td><td>62</td><td>F</td><td>Housewife</td><td>School</td><td>None</td><td>Alone</td></tr><tr><td>P19</td><td>72</td><td>F</td><td>Housewife</td><td>School</td><td>None</td><td>Family</td></tr><tr><td>P20</td><td>63</td><td>M</td><td>Manager</td><td>High School</td><td>Basic</td><td>Family</td></tr></tbody></table>
## 4.2 Study Procedure

This study was carried out at the researchers' university and lasted between 30 and 50 minutes per participant during August 2018.

First, a researcher provided a brief explanation about the purpose of the research and the participant signed the informed consent form, then the demographic data (age, profession and educational level) were collected. Afterwards, the participant completed a digital skills questionnaire [38, 39]. The participants then interacted with a simple puzzle application on a iPad Pro 9.7 [40]. This application was chosen because of its generic design (i.e., no elements specifically designed for children), being simple enough for users without digital skills, and because it allowed us to tailor its content by including photos that would be appealing to adults and not childish. The participants had to put together two different puzzles. The images of the puzzles were buildings or locations in the city they were living in, selected randomly from a set of 10 photographs.

After finishing the game, the participants answered the AttrakDiff [41] questionnaire and the Aestimo device questionnaire (half the participants started with one and the other half with the other). AttrakDiff was completed on paper, and in Spanish and it has four dimensions: pragmatic quality (PQ), is focused on determining how easy the task was completed, hedonic quality-identity (HQ-I), what is that message transmitted to others while the product is used, hedonic quality-stimulation (HQ-S), if the user's abilities are developed when using the product, and attraction (ATT), or the overall charm of the product. Answers are on a scale of -3 to 3 (0 represents neutrality) [4]. The first author then conducted a semi-structured interview to know the perception of the use of Aestimo compared to the traditional paper questionnaire. The interview focused on asking the opinion of the participants (positive and negative aspects) after using both types of questionnaires, then, the participants were asked which method they preferred, and finally, what they would change to the Aestimo interface. The interviews lasted 10 to 15 minutes per participant.

During the evaluation a researcher observed and took notes. Aestimo stored the duration of time each participant used the interface. In the case of the traditional questionnaire a researcher used a stopwatch to measure the time it took for the participant to fill it out. Figure 4 shows some participants during different stages of the study.

In this study, we used a translation of the complete AttrakDiff questionnaire in paper (Spanish) and a short version of AttrakDiff in Aestimo with 16 questions (Spanish). The reason to compare the two questionnaires was mainly to understand whether a shortened tangible version could capture user experience similarly to the full paper-based questionnaire.

## 5 Analysis and Results

A thematic analysis was conducted by one researcher to analyse the interviews and notes made during the evaluation [42]. The first step was to transcribe the
<figure><img src="image_5.png" /><figcaption>Fig. 4. Participants during the study.</figcaption></figure>

interviews. The transcripts of the interviews were made in Spanish, which is the first language of both the participants and the researcher. Then, all the interviews were read to become familiar with the data. As a second step, the initial themes emerged, then the list of topics was taken and classified. In this process, a description of the themes were used to select similar themes and place them in a new sub-list. At this point, we had 9 themes, but some themes were not supported to be candidates (1 was eliminated) and 2 themes were merged with existing themes, resulting in 6 final themes. Quotes from the participants are presented here translated from Spanish.

## 5.1 Are the results obtained through a TUI similar to those from a paper-and-pen-based questionnaire?

We obtained data from *Aestimo* and AttrakDiff, which allows us to compare several aspects: dimensions, questions, preferences and execution time.

**Dimensions** Regarding the dimensions, it can be seen that the short version of AttrakDiff used in *Aestimo* had very similar values in two dimensions: *hedonic quality-identity (HQ-I)* and *attraction (ATT)* with a difference of 0.09 and 0.06, respectively. *Pragmatic quality (PQ)* is very similar with 0.33 difference, and finally, *hedonic quality-stimulation (HQ-S)* presents a greater difference with 1.21, being in both cases the dimension with the lowest scores (see Figure 5).

**Questions** When comparing the score each *Aestimo* question score with the scores received in AttrakDiff, we found that three questions (6, 7 and 8) had scores with a difference greater than 2.4, which is the difference between the two results of those questions in Aestimo and the paper version. The values of AttrakDiff have a range between -3 and 3 (0 is neutral). Therefore, the difference between these two results (Aestimo and paper version) is greater than 2.4.
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Comparison of results according to dimensions: *Aestimo* and *AttrakDiff*.</figcaption></figure>

The questions were formulated following the same form as the long version of AttrakDiff. However, some words did not have a direct translation in Spanish and were therefore used as a negation of another word. This resulted in a final question with strange wording for Spanish speakers. For example: question 6 was phrased with a negative (*Does the application not captivate me?*), which was confusing - some participants said that the application was captivating out loud, but answered yes to the question on the device.

**Preferences** Out of the 20 participants, seven stated they preferred AttrakDiff over the paper questionnaire to report their user experience. The reasons they gave were that it was “simpler and faster” (P7, above basic) and that the AttrakDiff questionnaire is “easier because you can choose between two alternatives” (P11, none), and that “these little dots are clearer, it’s easier for me” (P18, none). Contrarily, 12 participants preferred *Aestimo*, because the questions in it were easier to understand: “it was difficult to understand the differences between both words. The questions are easier here.” (P10, low). Other participants liked that *Aestimo* would give them instructions: “It’s more practical, it tells me what to do. I think this little box is easier.” (P12, none). A participant with visual problems mentioned “It’s more fun, I liked it. Hearing things is easier than reading, since I have problems with my eyesight” (P17, basic). One participant stated that they liked both methods equally.
**Execution time** We measured execution time, i.e. the time in which each participant filled the AttrakDiff and the time in interacting with *Aestimo*.

The average time users spent on AttrakDiff was 4 minutes and 31 seconds, while they spent 8 minutes and 37 seconds on *Aestimo*. Although *Aestimo* took a longer time to answer, none of the participants mentioned that this was a problem. During the experiment, three participants asked the researcher to read the paper questionnaire out loud, which reduced the average time (the average of those 3 participants was 3 minutes and 18 seconds).

## 5.2 What is the perception of older adults of a TUI for reporting UX?

From the interviews and observation, we obtained 6 themes about the experience of the participants after using *Aestimo*.

**Innovation and nostalgia** An interesting aspect of the results is that the participants perceived *Aestimo* as novel, saying that they had never seen something like it, but also as something that seemed handmade and reminded them of previous times. They felt that using it was like going back to the time of their childhood: "it kind of took me back to my times, because in those times we used things like this, made out of wood. It was what was at school or during my studies, these things were manual, we used our hands" (P17, basic). They identified elements that they knew from the past, such as the telephone handset: "There are things that I already know. This [the handset] was the first thing that connected me to others in my childhood" (P13, above basic).

**Guide** The fact that *Aestimo* has instructions and these can be read and heard, guides the participants during the use of the interface: "It guided me, it was friendly. A girl was talking to me from the inside through the phone, she read me the questions, she read me everything I needed to do" (P14, above basic). Also, the interface itself instructed them on how to use it: "... it instructed me to use it, something I never used before. I liked the handset telling me things" (P12, none).

**Inviting** The participants felt motivated and invited to use technology: "I like the way it works, the way it asks the question, it's like a phone and motivates me to work with modern technology" (P8, low), and they do not feel intimidated or nervous: "It's kind of nice, it doesn't make me feel dumb or distant, it invites me to get closer to technology, which I'm currently interested in, not to quit. It invites me to do it and I can do it, it doesn't make me feel dumb or ridiculous, because sometimes I quit because of that" (P13, above basic). Likewise, one participant mentioned that she could use the interface despite her age: "I never expected to use something like this at my age" (P15, none).
**Stimulates senses and mind** The participants mentioned that they felt their senses were activated and challenged by performing the activities: "I feel that here I have hearing, I have touch, reading, seeing - I feel that several senses are involved here and that it activates them. That's what I liked: to hear, touch, see and at the same time develop my brain" (P13, above basic). Participants felt their minds were engaged as well: "It's great because it awakens the imagination, and you try to answer the question correctly, because you have to know how to answer, although the questions are simple... but I'm still clear headed and I can answer it" (P10, low), "You learn and you try to think about what you're doing. I liked the handset and the knob, because I have to use both hands, one for the handset and one for the knob and to turn the page" (P15, none).

**Playful and easy to use** Several participants mentioned that the interface was easy to use due to its characteristics "It's a good size, the knob is easy to find, the font is a big size, so it's perfect... it's easy to use" (P1, above basic). Three participants also stated that Aestimo could be used by people with less physical abilities, low educational levels or people with visual limitations. Although, as previously mentioned, three of the participants requested the paper questionnaire to be read out loud by the researcher, they used Aestimo without help.

The playful aspect was something that arose during the evaluation of Aestimo: "Using the interface was entertaining" (P2, low). After using the interface, some participants said they had expected it was going to be difficult, but rather it was fun. One of the aspects that they liked most was the surprise, when they saw the phone or when the book ended and other questions appeared: "The phone impressed me... what is in there? I was going to put my hand inside and I didn't know what it was, and it was a phone, and it communicates you with someone else. I hadn't thought before that there would be someone else, asking for the answer" (P20, basic).

**Difficulties of using Aestimo** The participants had some difficulties with the book. The problem was caused by having to stick the magnets of the new page to the previous page, since in some occasions the magnet did not make contact, and the user had to press with greater force the area of the magnets: "I would change the manipulation of the magnets, to make it more precise, because the magnet did not get stuck correctly every time, but after awhile I got used to sticking it correctly." (P16, low). Another problem was the position of the buttons on the recorder, since they are vertical and the label was difficult to see. Therefore, many participants asked what each of the buttons did. Some participants mentioned that the inability to change the volume on the telephone was a problem, since the proposed volume was too low for them. Finally, some participants did not read Aestimo's instructions, and began by opening the book (instead of removing the telephone).
# 6 Discussion

What emerged from our study was the enthusiasm of the participants when faced with an interface that they did not perceive to be threatening or complex. One previous study found that some older adults do not easily use previous knowledge when faced with new technology [43]. However, the participants in our study were positive and felt invited to use the interface, similarly to previous studies in which familiar physical interactions mitigated participants' anxiety when using new technology [21]. Some of our participants did not read the provided instructions and therefore did not begin with the first activity, highlighting the importance of designing clear affordances into interfaces, especially tangible ones, to ensure that users will follow the flow of the designed actions. Simpler interfaces have been found to be more intuitive, e.g. for patients with dementia [44]. Thus, a design challenge to consider when designing such interfaces is configuring an interface that is able to control the activities itself and not load this task on the user. For example, in [35], users read the textual information before starting the interaction, since the interaction flow was activated automatically. This problem has been reported in previous studies in the implementation stage or during the evaluation [45].

In the interviews a dichotomy between innovation and nostalgia emerged. Some participants found that our interface was innovative as it was something new for them in terms of function. Aesthetically, it felt familiar: it was made out of wood and paper, and used physical interactions, with which participants felt invited to use it. Wood has been found to attract people to interact with it [35], which was also evident in our experiment.

Regarding the shortened version of AttrakDiff, it was evident that it is important to select the appropriate synonyms and wording so that the same message is expressed in the Spanish and English version, especially in the hedonic quality-stimulation (HQ-S) dimension. This dimension (HQ-S) is composed of questions 6, 7, 8 and 9, out of which three of them (6, 7 and 8) were the questions that had different values between AttrakDiff and Aestimo. Therefore, we believe that this is the reason for the difference in values.

Regarding the results obtained from the comparison of the full version of AttrakDiff on paper and the short version of AttrakDiff in Aestimo, we are aware that, to actually compare paper vs. digital versions of AttrakDiff, the shortened version should be used in both cases. However, in this study we tried to understand if the short version of AttrakDiff could evaluate the user experience in a similar way to the complete questionnaire. The results obtained in this study encourage us to continue with the comparison of this short version of AttrakDiff in both paper and tangible formats. Additionally, the participants mentioned that the questions were simple and easy to understand, which agrees with [10] arguing that older adults respond better to questions rather than statements.

The development of *Aestimo* was based on the specific characteristics of some older adults in developing countries who have had little interaction with technology. *Aestimo* has the potential to be used by other user groups (e.g. people with a disability or people with low digital skills), who would benefit from
this type of technology. However, we propose that *Aestimo* must be modified to better adapt to other groups of users. Additionally, we could study the use of *Aestimo* to collect information in public spaces, to make this a more playful user experience.

*Aestimo* is not designed for all older adults in the world. Our prototype was developed for a subgroup of this population: those older adults who feel rejection, have little knowledge about, and may even fear technology. Therefore, we believe that there should be a greater discussion of how to approach this issue in the HCI community, so as not to neglect this population.

Finally, our results indicate the importance of a playful element when users approach technology that is new to them. The use of different elements (sounds, haptic or text) made the participants enjoy using the interfaces e.g. [29]. These playful elements should be adjusted to the age of the people in ways that are not seen as childish or distant.

We would like to acknowledge the following limitations. The evaluation was conducted with a group of older adults with different digital skills. In addition, the online version of the AttrakDiff questionnaire was not used in this study. Finally, AttrakDiff was compared with *Aestimo* using an application and with a small number of participants, so these results cannot be further generalized.

# 7 Conclusions and future work

In this paper we presented the implementation and evaluation of *Aestimo*, a tangible system to report the user experience of a subgroup of older adults who have had with little experience with technology. The proposed interface had high user acceptance. Our findings indicate that our participants felt that such an interface can be encouraging to introduce technology and that physical interaction can activate the senses and the mind. In the same way, it is evident that an interface such as this one should provide instructions and use familiar elements to support feelings of security and reassurance.

The next steps in this research will be to improve the usability problems found in *Aestimo* e.g. the implementation of the book regarding the use of magnets for the recognition of every page. Likewise, it would be interesting to make a comparison of the *Aestimo* questionnaire, the paper version of AttrakDiff and the online version of AttrakDiff.

# References

1. ISO DIS 9241-210. Ergonomics of human system interaction - part 210. Technical report, Inter. Organization for Standardization, Switzerland, 2010.

2. Javier A. Bargas-Avila and Kasper Hornbæk. Old wine in new bottles or novel challenges: A critical analysis of empirical studies of user experience. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, CHI '11, pages 2689–2698, New York, NY, USA, 2011. ACM.

3. Virpi Roto, Marianna Obrist, and Kaisa Vnnen. User experience evaluation methods in academic and industrial contexts, 2009.
4. J. Isleifsdottir and M. Larusdottir. Measuring the user experience of a task oriented software. In *Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement*, pages 97–101, 2008.

5. World Health Organization. Definition of an older or elderly person. Accessed: 2016-06-06.

6. Francisco J. García-Peñalvo, Miguel Ángel Conde, and Vicente Matellán-Olivera. *Learning and Collaboration Technologies. Technology-Rich Environments for Learning and Collaboration: HCI International*, chapter Mobile Apps for Older Users – The Development of a Mobile Apps Repository for Older People, pages 117–126. Springer International Publishing, Cham, 2014.

7. Ministerio Desarrollo Social. Casen sobre previsin social muestra importante au- mento de la cobertura de la pensin bsica solidaria de vejez en cuatro aos [casen on social welfare shows significant increase in the coverage of the basic solidarity pension for old age in four years], 2017. Accessed: 2019-01-06.

8. S. Singh and B. Bajorek. Defining elderly in clinical practice guidelines for phar- macotherapy. *Pharmacy Practice*, 12(4):489 – 9, 2014.

9. K. Gerling, J. Schild, and M. Masuch. Exergaming for elderly persons: Analyz- ing player experience and performance. In *Mensch & Computer 2011*, Chemnitz, Germany, 2011.

10. M. Heerink, B. Krose, V. Evers, and B. Wielinga. Studying the acceptance of a robotic agent by elderly users. journal of assistive robotics and mechatronics. volume 7, pages 33–43, 2006.

11. M. Harjumaa and M. Isomursu. Field work with older users-challenges in design and evaluation of information systems. *The Electronic Journal Information Systems Evaluation*, 15:50–62, 2012.

12. D.C. Park and A.H. Gutchess. *Cognitive Aging and Everyday Life*. Psychology Press, 2000.

13. Kathrin M. Gerling, Frank P. Schulte, and Maic Masuch. Designing and evaluating digital games for frail elderly persons. In *Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology*, ACE ’11, pages 62:1–62:8, New York, NY, USA, 2011. ACM.

14. Thiago Donizetti dos Santos and Vagner Figueredo de Santana. Computer anxiety and interaction: A systematic review. In *Proceedings of the Internet of Accessible Things*, W4A ’18, pages 18:1–18:10, New York, NY, USA, 2018. ACM.

15. OECD. Skills matter: Further results from the survey of adult skills. In series: OECD Skills Studies, 2016. Retrieved July 2016 from http://www.oecd.org/skills/piaac/Skills-Matter-Chile.pdf.

16. Eliseo Sciarretta, Andrea Ingrosso, Valentina Volpi, Antonio Opromolla, and Roberta Grimaldi. Elderly and tablets: Considerations and suggestions about the design of proper applications. In Jia Zhou and Gavriel Salvendy, editors, *Human Aspects of IT for the Aged Population. Design for Aging*, pages 509–518, Cham, 2015. Springer International Publishing.

17. Blaine A. Price, Ryan Kelly, Vikram Mehta, Ciaran McCormick, Hanad Ahmed, and Oliver Pearce. Feel my pain: Design and evaluation of painpad, a tangible device for supporting inpatient self-logging of pain. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, CHI ’18, pages 169:1–169:13, New York, NY, USA, 2018. ACM.

18. M. Nilsson, S. Johansson, and M. Håkansson. Nostalgia: An evocative tangible interface for elderly users. In *Extended Abstracts on Human Factors in Computing Systems*, CHI EA ’03, pages 964–965, NY, USA, 2003. ACM.
19. J. Häikiö, A. Wallin, M. Isomursu, H. Ailisto, T. Matinmikko, and T. Huomo. Touch-based user interface for elderly users. In *Proceedings of Conference on Human Computer Interaction with Mobile Devices and Services*, MobileHCI '07, pages 289–296, NY, USA, 2007. ACM.

20. Iyubanit Rodríguez, Maria Karyda, Andrés Lucero, and Valeria Herskovic. Exploring tangible ways to evaluate user experience for elders. In *Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems*, CHI EA '18, pages LBW589:1–LBW589:6, New York, NY, USA, 2018. ACM.

21. Sarah Gallacher, Connie Golsteijn, Lorna Wall, Lisa Koeman, Sami Andberg, Licia Capra, and Yvonne Rogers. Getting quizzical about physical: Observing experiences with a tangible questionnaire. In *Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing*, UbiComp '15, pages 263–273, New York, NY, USA, 2015. ACM.

22. Connie Golsteijn, Sarah Gallacher, Licia Capra, and Yvonne Rogers. Sens-us: Designing innovative civic technology for the public good. In *Proceedings of the 2016 ACM Conference on Designing Interactive Systems*, DIS '16, pages 39–49, New York, NY, USA, 2016. ACM.

23. Lisa Ehrenstrasser and Wolfgang Spreicer. kommtui—a design process for a tangible communication technology with seniors. In *Human Factors in Computing and Informatics*, pages 625–632. Springer, 2013.

24. Maria Foverskov and Thomas Binder. Super dots: making social media tangible for senior citizens. In *Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces*, page 65. ACM, 2011.

25. Mingqian Zhao, Zhutian Chen, Ke Lu, Chaoran Li, Huamin Qu, and Xiaojuan Ma. Blossom: Design of a tangible interface for improving intergenerational communication for the elderly. In *Proceedings of the International Symposium on Interactive Technology and Ageing Populations*, ITAP '16, pages 87–98, New York, NY, USA, 2016. ACM.

26. Way Kiat Bong, Weiqin Chen, and Astrid Bergland. Tangible user interface for social interactions for the elderly: A review of literature. *Advances in Human-Computer Interaction*, 2018, 2018.

27. Dagmar Kern, Mark Stringer, Geraldine Fitzpatrick, and Albrecht Schmidt. Curball—a prototype tangible game for inter-generational play. In *null*, pages 412–418. IEEE, 2006.

28. Abdullah Al Mahmud, Omar Mubin, Suleman Shahid, and Jean-Bernard Martens. Designing and evaluating the tabletop game experience for senior citizens. In *Proceedings of the 5th Nordic Conference on Human-computer Interaction: Building Bridges*, NordiCHI '08, pages 403–406, New York, NY, USA, 2008. ACM.

29. A. Huldtgren, F. Mertl, A. Vormann, and C. Geiger. Reminiscence of people with dementia mediated by a tangible multimedia book. pages 191–201, 5 2016. 2nd International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2016), ICT4AWE 2016 ; Conference date: 21-04-2016 Through 22-04-2016.

30. Costas Boletsis and Simon McCallum. Augmented reality cubes for cognitive gaming: preliminary usability and game experience testing. *Int. J. Serious Games*, 3(1):3–18, 2016.

31. Scott Davidoff, Carson Bloomberg, Ian Anthony R. Li, Jennifer Mankoff, and Susan R. Fussell. The book as user interface: Lowering the entry cost to email for elders. In *CHI '05 Extended Abstracts on Human Factors in Computing Systems*, CHI EA '05, pages 1331–1334, New York, NY, USA, 2005. ACM.
32. S. Wolfgang. Tangible interfaces as a chance for higher technology acceptance by the elderly. In *Proceedings Conference on Computer Systems and Technologies*, CompSysTech '11, pages 311–316, NY, USA, 2011. ACM.

33. Vasilis Vlachokyriakos, Rob Comber, Karim Ladha, Nick Taylor, Paul Dunphy, Patrick McCorry, and Patrick Olivier. Postervote: Expanding the action repertoire for local political activism. In *Proceedings of the 2014 Conference on Designing Interactive Systems*, DIS '14, pages 795–804, New York, NY, USA, 2014. ACM.

34. Fabius Steinberger, Marcus Foth, and Florian Alt. Vote with your feet: Local community polling on urban screens. In *Proceedings of The International Symposium on Pervasive Displays*, PerDis '14, pages 44:44–44:49, New York, NY, USA, 2014. ACM.

35. N. Patle. Linkingpark: Design of a physical interface to enhance public engagement in an emerging smart city. Technical report, UCL Interaction Centre, University College London, 2016.

36. Sarah Gallacher, Connie Golsteijn, Yvonne Rogers, Licia Capra, and Sophie Eu-stace. Smalltalk: Using tangible interactions to gather feedback from children. In *Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction*, TEI '16, pages 253–261, New York, NY, USA, 2016. ACM.

37. Cristina Maria Sylla, Ahmed Sabbir Arif, Elena Márquez Segura, and Eva Irene Brooks. Paper ladder: A rating scale to collect children's opinion in user studies. In *Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services*, MobileHCI '17, pages 96:1–96:8, New York, NY, USA, 2017. ACM.

38. A. Ferrari. Digital competence in practice: An analysis of frameworks. Technical report, Research Centre of the European Commission, Seville, Spain, 9 2012.

39. European Commission. Measuring digital skills across the eu: Eu wide indicators of digital competence. Technical report, 2010. Imported from http://ictlogy.net/bibliography/reports/projects.php?idp=2685.

40. Minkov S. Educativos juego puzzles para niños pequeños niñas [educational game puzzles for little boys and girls], 2018.

41. AttrakDiff. http://www.attrakdiff.de, 2018. Accessed: 2018-03-30.

42. V. Braun and V. Clarke. Using thematic analysis in psychology. *Qualitative Research in Psychology*, 3(2):77101, 2006.

43. Simon Lawry, Vesna Popovic, Alethea Blackler, and Helen Thompson. Age, familiarity, and intuitive use: An empirical investigation. *Applied Ergonomics*, 74:74–84, 2019.

44. Li-Hao Chen and Yi-Chien Liu. Affordance and intuitive interface design for elder users with dementia. *Procedia CIRP*, 60:470–475, 2017. Complex Systems Engineering and Development Proceedings of the 27th CIRP Design Conference Cranfield University, UK 10th 12th May 2017.

45. Connie Golsteijn, Sarah Gallacher, Lisa Koeman, Lorna Wall, Sami Andberg, Yvonne Rogers, and Licia Capra. Voxbox: A tangible machine that gathers opinions from the public at events. In *Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction*, TEI '15, pages 201–208, New York, NY, USA, 2015. ACM.