# Vote-for-It: Investigating Mobile Device-Based Interaction Techniques for Collocated Anonymous Voting and Rating

Romina Kühn, Mandy Korzetz, Franz-Wilhelm Schumann, Lukas Büschel, Thomas Schlegel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Romina Kühn, Mandy Korzetz, Franz-Wilhelm Schumann, Lukas Büschel, Thomas Schlegel. Vote-for-It: Investigating Mobile Device-Based Interaction Techniques for Collocated Anonymous Voting and Rating. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.585-605, 10.1007/978-3-030-29381-9_36. hal-02544552

HAL Id: hal-02544552

https://inria.hal.science/hal-02544552

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Vote-for-It: Investigating Mobile Device-Based Interaction Techniques for Collocated Anonymous Voting and Rating

Romina Kühn<fnref n="1" />, Mandy Korzetz<fnref n="1" />, Franz-Wilhelm Schumann<fnref n="1" />, Lukas Büschel<fnref n="1" />, and Thomas Schlegel<fnref n="2" />

**Abstract.** During discussions in collocated work it is necessary to vote for results or to rate them to reach an agreement and continue working. To ensure impartiality and to avoid social embarrassment, the assessment should then be performed anonymously in so far as other groups members should not see directly how a person votes or rates. With a growing number of digital devices in collaboration, this requirement also concerns such kinds of equipment. Our approach of ensuring anonymity of individual votes and ratings submitted on personal mobile phones is to avoid shoulder surfing activities. For this purpose, we designed four device-based interactions that aim at being easy to use and eyes-free to perform to stay in touch with the environment and potential shoulder surfers. We conducted a user study to investigate these interaction techniques and observed seven groups with four participants each while testing the interactions. Participants evaluated usability and User Experience (UX) aspects as well as unobtrusiveness of the four device-based interactions. Furthermore, participants gave valuable user feedback and stated that our proposed interactions help to avoid shoulder surfing.

**Keywords:** Collocated Interaction · Device-Based Interaction · Voting · Rating · Mixed-Focus Collaboration · Shoulder Surfer.

## 1 Introduction

Collocated collaborative group work often comes along with discussions about current progress, results or further proceeding [31, 15]. Especially in case of discussing proposed partial solutions, it becomes crucial to vote on these solutions to continue working. While some votes can benefit from a joint direct discussion, others should happen anonymously to ensure impartiality or to protect a voting person from social embarrassment [20]. Impartiality can be important to guarantee objective voting and rating results that are uninfluenced by opinion

<footnote n="1">¹ TU Dresden, Institute of Software and Multimedia Technology, Dresden, Germany
{romina.kuehn, mandy.korzetz, franz-wilhelm.schumann, lukas.bueschel}@tu-dresden.de</footnote>

<footnote n="2">² Karlsruhe University of Applied Science, Institute for Ubiquitous Mobility Systems
Karlsruhe, Germany
thomas.schlegel@hs-karlsruhe.de</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Collocated collaborative work scenario with shoulder surfer (yellow). Group members apply our anonymous voting and rating interactions with their mobile devices. They either hide their mobile device (blue) from the shoulder surfer, observe the shoulder surfer while interacting (orange), or use a back-of-device interaction (red).</figcaption></figure>

leaders within a group. For example, this can be the case if several employees of a company present their work and need to decide which approach they want to continue with. However, even though people attempt to vote anonymously, some people still try to catch a glimpse of the input or valuations of others. This phenomenon is known as both shoulder surfing [26] and visual hacking [9] and mainly occurs in the context of offices or public places [6]. We use both terms interchangeably although shoulder surfing is often equated only with looking over a victim's shoulder, not from different perspectives. Furthermore, we use these terms to describe the activity of obtaining information from or about others that actually should be kept private. We further refer to this activity as *shoulder surfing*. Independently from specific devices and used analog or digital tools, this issue must be addressed accordingly. Figure 1 illustrates this problem in a collocated collaborative scenario with mobile phones. People are sitting or standing around a table, collaborating and using their mobile phones to assess results. The person standing behind one of the sitting persons on the right side tries to get a look on the screen of the mobile device. Although, the others are not in focus of the shoulder surfer, they evade this problem by hiding their devices and/or keep an eye on their surrounding.

There are already many approaches that investigate shoulder surfing. The plethora of approaches address this problem on an individual level (one shoulder surfer and one “victim”) although consequences of shoulder surfing can also be very big on a group level such as in companies [9]. In this work, we address the shoulder surfer problem in collocated group work by designing device-based interactions for mobile devices especially smartphones. Our approach is not limited to a specific application but applicable for groups in teaching or company domains. Furthermore, it addresses the mixed-focus collaboration activities [15] and is reusable for comparing activities in general. Mobile devices enrich collocated group work in many respects and apply increasingly in spontaneous (ad-hoc) col-
laboration [10]. Smartphones have the advantage that they are small enough to
keep the users' sight unhindered while working digitally, in contrast to laptops.
Addressing digitization strategies in companies and the public sector [27], they
also avoid media breaks by enabling users to directly perform actions digitally, in
contrast to conventional paper-based methods. Additionally, each mobile device
belongs to one user and is therefore mostly personal. Consequently, inhibitions in
using such devices should be low. To use them as voting tool, such devices should
not only include protocols, services, or tools for privacy and security but also
interaction techniques for anonymous valuations that avoid or at least impede
shoulder surfing. We focus on ways to apply smartphones to collocated group
work efficiently by providing interaction techniques that are easy to perform
and support the users' tasks. Thereby, we enable users to act self-determined in
terms of keeping input and information anonymous. The contributions of this
paper are the following:

- *Generalizable design goals* for device-based interactions in collocated groups

- Implemented and evaluated *mobile device-based interaction techniques for anonymous voting and rating* using mobile phones as direct valuation tool

- Recommendations derived from a *user study* that applied the proposed in-
interactions in collocated collaboration and investigated usability and UX

To address the mentioned issues and contributions, we structured the paper as
follows: First, we give an overview on related work that describes approaches
on shoulder surfing and several interactions for voting. Based on this, we derive
design goals concerning ad-hoc collaboration, device-based and eyes-free inter-
action that lead to interactions for anonymous voting and rating. The main part
comprises the user study we performed by means of an interactive prototype.
Our goals were to understand whether the interactions are easy to use, eyes-free
to perform, and suitable for anonymous voting and rating in a collocated setting.
This work concludes with findings, recommendations, and future work.

## 2 Related Work

Device-based interaction techniques have been investigated as additional way to
interact efficiently in certain situations where input and output of data should
be fast, unobtrusive or as easy as possible (e.g., [1, 25, 17, 12]). Ashbrook and
Starner [1] propose *mobile microinteractions* where interactions should take less
than four seconds. Using mobile phones as physical input device that directly
involve touching or moving the device has been analyzed, e.g., by Rico and Brew-
ster [25] and Leigh et al. [17]. Lucero et al. [21] propose design principles (SSI
principles) to embed multiple mobile devices for shared multi-user usage. Korzetz
et al. [12] provide *mobile spaces*, a model for designing lightweight mobile-based
interaction techniques to support individual as well as collaborative usage sce-
narios. Both approaches emphasize tangibility and spatiality as core concepts for
interacting effortless especially during collaborative work. Our approach aims at
supporting mixed-focus collaborative scenarios where people frequently move
between individual tasks and shared work with other group members [7]. Such scenarios require the possibility to vote and rate individual results to continue joint working. Several different approaches have influenced our anonymous voting and rating interactions. We identified three main areas of related work. At first, we comprise the usage of mobile devices in collocated collaborative scenarios in general. Then, we present approaches that enable voting and rating with mobile devices. Third, we describe existing work on avoiding shoulder surfing.

## 2.1 Mobile devices in collocated collaborative scenarios

Collaborative scenarios can take various forms depending on the environmental conditions, the current task and the structure of the group. MobiSurf [29] outlines a collaborative scenario where various integrated personal mobile devices and one shared interactive surface in a home setting support collocated decision-making. The system Ubi-Jector [18] provides a shared information screen as whiteboard utilizing the personal tablets of the group members. Both systems use besides personal devices additional devices to display content. HuddleLamp [24] focuses on ad-hoc multi-device collaboration and uses only mobile devices. However, a lamp with an integrated camera is needed to detect and track the mobile devices.

The need for additional equipment, e.g., displays or tracking devices, is one of the reasons why mobile devices are still less well integrated into collaborative scenarios, although they are an indispensable part of our everyday life. Additionally, it is crucial to join a group spontaneously and start interacting with others in a fast and easy way for participating in mobile collocated interactions [3, 21]. Lucero et al. propose device-based interactions, which only use mobile devices to support ad-hoc collaboration [19, 22]. They focus on sharing content amongst group members. Our approach addresses mobile devices without external technology to overcome these issues and adds further collaborative activities.

## 2.2 Voting and rating in collaborative scenarios

Related work mostly addresses direct voting and rating as part of discussions in collocated collaborative scenarios. McCrindle's et al. approach "t-vote" [23] enables digital collocated decision-making for children in a museum's context by using tangibles on a tabletop. The authors designed the application to rate content directly to foster joint discussions. MobiComics [20] is an application that allows for creating and editing comic panels collaboratively and distributing them among two public displays by performing tangible and spatial interactions according to the SSI principles. The application includes interactions for public voting to encourage discussions: By holding the mobile phone in the air with a thumb up or down image, a user can cast a vote. MobiLenin [28] allows for taking a vote on music videos shown on a public display by selecting an item in the voting menu of a mobile device. Kühn et al. [13] present interactions for mobile devices that support direct voting and rating for ad-hoc collaboration without additional technical equipment. Each mobile device represents one result of a specific task. By moving and arranging the physical devices, they digitize voting
results in real time. The interactions aim at supporting seamless discussions to
find a common rating or voting result.

Since opinion leaders can influence direct valuations it is sometimes necessary to avoid direct voting. To enable unbiased results, groups often use paper-based methods to hide their decision. Collaborators then write their decision on pieces of paper and put them into a box. After group members submitted their piece of paper, they count the results by hand. Then, in case of further digital processing, the results have to be digitized. There are also web-based tools that allow for setting up digital polls easily, e.g., Straw Poll [30]. They focus on providing textual response options for voting. However, although rating can be performed by entering values textually further processing options are missing, e.g., calculating an average value. Approaches for integrating personal mobile devices in collaboration should offer support for anonymous voting and rating to get an overall rating out of each single valuation. We address this issue by providing easy and eyes-free input, so that collaborators can pay attention to the surroundings and other group members while voting and rating. We propose device-based interactions following the SSI principles and mobile spaces without other equipment.

## 2.3 Shoulder surfer problem

Anonymous voting and rating in collocated settings requires techniques to avoid shoulder surfing to hide input from other collaborators. Approaches that address the shoulder surfer problem often focus on entering passwords or PINs (e.g., [2, 32, 33]) and only consider one-to-one scenarios (one shoulder surfer and one “victim”). They often adjust the displayed content on the mobile devices by using handwritten fonts [5] or password grids where users have to locate their password [11]. Instead of adjusting the user interface presentation, De Luca et al. [2] hide the interaction by acting on the device’s backside. Our proposed interactions pick up that idea, as users also can act on the backside. Moreover, users can perform the interactions largely eyes-free, so that they are able to have a look at their surroundings to prevent shoulder surfing.

# 3 Design Goals for Collocated Collaborative Interactions

To design suitable interaction techniques that cover requirements for anonymous voting and rating by preventing shoulder surfing in collocated collaboration, we describe the following design goals to address this issues: (1) provide *unobtrusive and eyes-free* input to facilitate observing the surrounding to prevent shoulder surfing and that collaborators can feel comfortable while entering their vote, (2) provide device-based interaction techniques to enable an *intuitive and seamless* integration of mobile devices in collaboration, and (3) allow a group of collocated people to vote and rate ad-hoc only with their personal devices without further equipment to enable spontaneous collaboration activities. It is our purpose to adopt the easiness but at the same time to overcome the disadvantages of paper-based methods for rating and voting within collaborative sessions. The design
goals focus on providing collocated anonymous voting and rating interactions
but also address collocated collaboration interactions in general.

**3.1 Unobtrusive and eyes-free interactions**

As shoulder surfing is a common issue while voting collocated, we aim at providing anonymous voting and rating interactions that users can perform as easy and unobtrusive as possible. From the users’ perspective, interactions have to be simple and intuitive with little effort. The unobtrusiveness of interactions is twofold. The performance of an interaction technique should be unobtrusive to impede the identification of a particular interaction for shoulder surfers from different perspectives. Otherwise, the interaction input, especially the voting and rating result, should be as unobtrusive as possible in order to prevent the recognition.

Furthermore, to ensure prevention of shoulder surfing activities interactions
should be *eyes-free* to perform. As a result of such an eyes-free interaction, users
can keep an eye on the surrounding instead of looking permanently on the device
screen to feel more safe and comfortable while making assessments. The objective
is to demand less attention for input activities to facilitate additional observing
of the surrounding. This, in turn, allows the user to judge when she can vote
or rate. Users can act accordingly: entering valuations while being unobserved
or hiding input activities with, e.g., hands, table or the backside of the device.
With such simple interactions, we aim at providing a lightweight, fast, and easy
privacy protection as proposed by Eiband et al. [4].

**3.2 Intuitive device-based interaction techniques**

To enable the usage of anonymous voting and rating interactions and to inte-
grate mobile technology in a seamless and unobtrusive way, we use device-based
interaction techniques. Mobile phones are physical devices that facilitate users to
apply familiar interaction concepts. Our interaction techniques follow metaphors
of conventional anonymous voting and rating methods (e.g., paper-based meth-
ods), so that they are easy to learn and to remember and, hence, *intuitive* to
perform. Thus, users are still able to communicate face-to-face with group mem-
bers and to keep an eye on the surrounding because there is low effort in using
such interactions. This aspect addresses anonymous voting and rating interac-
tions as a part of collaboration as well as collaboration interactions in general.

**3.3 Supporting ad-hoc collaboration**

Collaborative groups, e.g., a group of students, often collaborate spontaneously.
Therefore, special equipment, such as tabletops, large displays, or tracking sys-
tems, for collocated collaboration is not necessarily available. In contrast, users
just use their mobile devices without the need for further technical equipment.
One benefit of using only mobile devices is that users can collaborate location-
independently and spontaneously. If users own a mobile device, they can easily
join or leave a group to participate in general collaboration activities as proposed in [15]. We thereby support groups of different sizes, because each person works with an own device. Furthermore, when the user is working digitally, there is no effort afterwards to digitize the results to continue working and users do not have to change permanently between media types (paper vs. digital content). Although, this is a general design goal for collocated collaboration interactions it also addresses the specific requirement of anonymous voting and rating interactions as one collaboration activity by enabling ad-hoc performance.

# 4 Anonymous Voting and Rating Interactions

We build on prior work that presents anonymous voting and rating interactions [14]. We revised and implemented the proposed conceptual interactions in order to evaluate them within a user study. They address the mixed-focus collaboration activity *comparing results* by setting in after group members discussed several interim results to reach an agreement anonymously [15]. We differentiate between rating and voting micro-activities. While voting only allows for accepting or rejecting a proposed solution, rating enables grading and therefore a higher level of granularity. The interactions are designed to be easy and eyes-free to perform and avoid shoulder surfing by enabling users either to observe their surrounding while entering content or easily shielding input from other eyes. Furthermore, they follow the before-mentioned design goals and base on everyday metaphors.

The interactions address the above-mentioned *mobile spaces* model (MSM) [12]. Collocated anonymous voting and rating as part of mixed-focus collaboration innately include individual as well as collaborative usage. Using their built-in sensors (MSM "technological"), mobile phones can identify their spatial arrangement (MSM "spatial"). Furthermore, due to their size and form, users can manipulate the devices physically (MSM "tangible"). Additionally, we offer multimodal feedback in terms of vibrations and enable users to choose between the several voting and rating interactions (MSM "representational").

## 4.1 Anonymous Rating Interactions

For rating solutions, we propose two interactions that can be performed alternatively (Figure 2). The first interaction “Multiple Finger Tap to Rate” (1a, further referred to as *Fingerprint*) uses the fingerprint sensor to rate a result. By tapping the fingerprint sensor multiple times, the counter increases. For example, five taps represent the best rate. Since this interaction occurs on the backside of the device, it hides the input innately and shields it from other eyes. In case there is no fingerprint sensor or to provide an alternative interaction technique, the display acts as input area for ratings (1b, “Multi-Finger Tap to Rate” or *Touch*). The number of fingers tapping the display simultaneously corresponds to the value input. In Figure 2, three fingers are tapping the display and the rating “three” is given. To hide the input from shoulder surfers, users can shield the display, e.g., with their hand or an object, while entering the rating.
<table><caption>Fig. 2. Interactions for anonymous rating and voting using standard mobile phones</caption><thead><tr><th></th><th>Interaction Technique</th><th>Interaction Name</th><th>Short Name</th><th>Used Device Sensors</th></tr></thead><tbody><tr><td rowspan="2">RATING</td><td>(1a)</td><td>"Multiple Finger Tap to Rate"</td><td>Fingerprint</td><td>Fingerprint sensor</td></tr><tr><td>(1b)</td><td>"Multi-Finger Tap to Rate"</td><td>Touch</td><td>Multi-touch display</td></tr><tr><td rowspan="4">VOTING</td><td>(2a)</td><td>"Draw Plus to Accept"</td><td>Draw</td><td>Touch display</td></tr><tr><td rowspan="3">(2b)</td><td>"Draw Minus to Reject"</td><td rowspan="3">Tilt</td><td rowspan="3">Motion sensors (accelerometer, gyroscope)</td></tr><tr><td>"Tilt Forwards to Accept"</td></tr><tr><td>"Tilt Sidewards to Reject"</td></tr></tbody></table>

## 4.2 Anonymous Voting Interactions

To vote on results, group members can use the interactions “Draw Plus to Accept” or “Draw Minus to Reject” (2a, *Draw*) as well as “Tilt Forwards to Accept” or “Tilt Sidewards to Reject” (2b, *Tilt*). To accept a result users either draw a plus on their mobile phone display or tilt their device along the x-axis. If users draw a minus on the display or tilt their device along the y-axis, they reject a result. Users can perform these interactions alternatively depending on their preferences or hardware restrictions. To avoid shoulder surfers, users can either turn the device or shield it with another object while interacting.

# 5 User Study

We conducted a user study to examine the proposed interactions. In the following, we describe the participants, the interactive prototype, our procedure as well as the study design to answer the following questions:

1. How do the interaction techniques perform in terms of usability and UX?

2. How unobtrusive and eyes-free executable are the proposed interaction techniques to address the shoulder surfer issue properly?

## 5.1 Participants

We recruited 28 unpaid participants (11 female) from age 18 to 51 (M = 31.6, SD = 6.7) via e-mail or personally. All participants were familiar with the usage of
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Interactive prototype and study setting. (a) Tutorial describing each interaction textually and visually. (b) Participants try the interactions with given votes or ratings on several images. (c) Participants sat around a table in our lab, the study leader sat in the background. Each session was video recorded from this perspective.</figcaption></figure>

smartphones, although two participants did not own such a device. We divided the participants into 7 groups with 4 people each. Within groups, participants knew each other beforehand as colleagues to establish a more realistic co-working scenario and to avoid inhibitions while interacting with each other.

## 5.2 Interactive Prototype

We implemented all interactions for Android devices with a minimum requirement of Android version 7.0. In addition, to conduct each interaction properly, all devices used for the user study had to provide an accelerometer, a gyroscope sensor, and a fingerprint sensor. We provided two Google Pixel phones and two ZTE Axon 7 devices for the participants to guarantee the availability of the necessary sensors and to control the setting. The devices are quite similar and communicate via Wi-Fi Direct. As shown in Figure 2, both *Touch* and *Draw* use the touch screen to perform the interactions. For *Touch*, the device recognizes the number of touch points during a given time. Performing *Draw*, the touch screen recognizes one or two strokes. For *Fingerprint* the fingerprint sensor identifies a configured fingerprint in short time intervals. Each recognition represents an added rating point. *Tilt* uses both motion sensors - accelerometer and gyroscope. The accelerometer activates the tilt gesture whereas the gyroscope sensor measures how the mobile phone moves along the x- or y-axis.

We provided two Android applications to embed the interaction techniques in a plausible scenario. Both mobile applications use the same technical features. The first one involved a tutorial to become familiar with the anonymous voting and rating interactions (Figure 3a) and the device. Furthermore, the tutorial application included a feedback form to assess first impressions on the interactions. The second application aimed at using the introduced interactions within a realistic but stipulated collaborative scenario. The application enabled to draw simple images, which became the valuation objects later. The second applica-
tion also contained all interactions from the tutorial as well as further feedback
forms. We added instruction text to facilitate the user to work through the app
and the several tasks autonomously. Both applications included a help function
in case users forgot an interaction or the way they can perform the gestures.
Additionally, the second application had the functionality to connect to each
other using Wi-Fi Direct to share working results and valuations. As soon as the
users performed an interaction, the device vibrated in order to give feedback.
Both applications logged the usage of the interactions, input errors, valuation
results, and the completed feedback forms.

5.3 Procedure

After the participants arrived in our lab, we explained the global procedure of the
user study. That included further proceeding and our focus on voting and rating
anonymously. Then, the participants chose one of the provided mobile phones
(Google Pixel or ZTE Axon 7) on their own. We used a square table where the
participants could sit as shown in Figure 3c. We also allowed the participants to
move around within the room and to talk to each other. We emphasized that they
should impede the others from noticing how they rate or vote. Furthermore, we
encouraged the participants to stay curious during the study and try to see how
the others vote and rate. We assume that in collaborative real-world scenarios all
group members would act as shoulder surfer intrinsically motivated while they
collaborate normally. In contrast, by assigning one person to the shoulder surfer
role, we think that this person would have focused on observing and shoulder
surfing instead of performing the collaboration task. That is why we decided
to not appoint one explicit shoulder surfer within the group of participants. To
address the order bias, we permuted the order of interactions. Furthermore, we
recorded each session for retrospective analysis. In addition, the study leader
was present during each session and observed the participants.

The participants started with the tutorial application simultaneously to be-
come familiar with the interactions by perusing an introduction (Figure 3a) and
trying each interaction three times. The tutorial tasks included rating or voting
on images of paintings of known artists (Figure 3b). We predefined how partici-
pants ought to rate or vote in order to see whether they knew what to do or not.
The participants received a positive feedback in form of a green check mark if
the mobile phone recognized the right input value. After performing each inter-
action the participants got a questionnaire concerning the understandability of
the tutorial and first impressions on the interactions including a short version of
the User Experience Questionnaire (UEQ-S). At the end of the tutorial phase,
the participants could try again all interactions independently.

Before starting the second application, the study leader reminded the partic-
ipants to keep the input value anonymous and established a connection between
the four mobile devices. The second application started with the task to draw a
bird using a drawing functionality. The participants prepared content on their
own to create a kind of an emotional link to vote and rate more realistic. After
finishing drawing, every participant could see the other images on the device to
vote and rate using a *stipulated* interaction technique. Then, each device showed the overall rating and voting results. Afterwards, every participant completed a questionnaire concerning the unobtrusiveness of the interactions. In a last run, we asked the participants to draw another image of an animal and to rate and vote on these images using *self-chosen* interactions. Finally, the participants completed the standardized User Experience Questionnaire (UEQ) and a questionnaire regarding demographic data. The study leader concluded by asking several semi-structured questions concerning the interactions and their usage in collaborative settings. Each group took about 70 to 80 minutes for the conduct.

## 5.4 Design

In order to answer the above-mentioned questions, we realized a within-subject design with the several interaction techniques as independent variable. We had two conditions for the interaction techniques: voting interactions (*Tilt* and *Draw*) and rating interactions (*Fingerprint* and *Touch*). We collected quantitative data by means of the standardized User Experience Questionnaire (UEQ) [8, 16] at first after introducing and later after using the interactions several times at the end of the study. The UEQ contains three main scales: attractiveness, pragmatic quality (measuring perspicuity, efficiency, and dependability) and hedonic quality (measuring stimulation and novelty). Pragmatic quality describes goal-directed quality aspects, hedonic quality not goal-directed quality aspects. These three dimensions help us to evaluate usability and UX aspects. All data coming from the UEQs was assessed on 7-point Likert scales. Additionally, we provided questionnaires to investigate the unobtrusiveness and eyes-free performance of the interactions assessed on 5-point Likert scales. Finally, we collected qualitative data from observations during working with the interactive prototype, the prototype protocols as well as during semi-structured interviews.

# 6 Results

We received valuable insights concerning the usability and UX of our interactions as well as on the behavior of groups while rating and voting anonymously. For analyzing our data, we used logging protocols from both applications. These protocols included all voting and rating values, performed interactions and resulting errors as well as completed questionnaires. With 28 participants and 4 different interaction techniques, we received 112 responses to each questionnaire. We assessed and interpreted the following aspects also mentioned in section 5:

- Usability aspects: attractiveness and pragmatic quality (perspicuity, efficiency, dependability)

- UX aspects: hedonic quality (stimulation, novelty)

- Unobtrusive and eyes-free interaction: ease of recognizing voting and rating input as well as the respective interaction, effort of keeping input anonymous
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Mean values (M) and standard deviation (SD) derived from the 5-point Likert scale (5 = best) concerning the unobtrusiveness of the interaction techniques, their pragmatic quality, their attractiveness, the effort of keeping the input anonymous, the perception of recognizing input, and the perception of recognizing each interaction for *Fingerprint* (FI), *Touch* (TO), *Tilt* (TI), and *Draw* (DR).</figcaption></figure>

## 6.1 Usable interactions for anonymous voting and rating

From the protocols, we analyzed the errors that occurred while performing the interactions for the first time. Applying ANOVA showed that for the rating interactions the effect on the number of errors was not statistically significant ($F_{1,27} = 3.847, p > .05$). We could not analyze the voting interactions because the devices only could recognize the correct performance of *Tilt* and, therefore, did not document errors for analyzing. However, from the observation we noticed stronger responses to the voting interactions in case of errors.

After performing the tutorial, we asked the participants to assess each interaction regarding pragmatic quality on a 5-point Likert scale (5 = best). Furthermore, they assessed how unobtrusive the interactions were and how much they liked each interaction (attractiveness). Regarding these three criteria, participants assessed *Touch* as rating interaction better than *Fingerprint* and the voting interaction *Draw* better than *Tilt*. Figure 4 summarizes the mean values (M) as well as the standard deviation (SD) of the interactions. Overall, *Draw* has the lowest standard deviation, which we interpreted as strong consensus especially on pragmatic quality (SD = 0.6) and unobtrusiveness (SD = 0.8).

We compared the evaluation of the questionnaire with the results of the standardized UEQ-S. The UEQ-S as well as the long UEQ queried an assessment of contrasting pairs on a 7-point Likert scale ranging from -3 (worst) to 3 (best). Figure 5 (left) shows the mean values (M) from the UEQ-S completed after the tutorial. Concluding, the UEQ-S confirms the evaluation of the first questionnaire. Participants assessed *Draw* best concerning pragmatic quality and attractiveness. *Touch* was rated better than *Fingerprint*. Although participants assessed *Tilt* worse than *Draw*, this interaction has the highest mean value regarding novelty of interaction (M = 1.5) as part of hedonic quality on
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Mean values from UEQ-S (left) and UEQ (right) concerning attractiveness, pragmatic quality and hedonic quality for *Fingerprint* (FI), *Touch* (TO), *Tilt* (TI), and *Draw* (DR). Values between -0.8 and 0.8 show a neutral evaluation (>0.8 = positive).</figcaption></figure>

the before-mentioned 7-point Likert scale. In contrast, *Draw* only got a neutral rating (M = 0.4) regarding novelty.

We compared both rating and voting interactions respectively by performing a t-test. With an alpha level of α = 0.05, the voting interactions showed significant differences (p < .05) concerning attractiveness, perspicuity, efficiency, dependability, and novelty. Only stimulation showed no significant difference (p = .18). Comparing the rating interactions, the t-test showed significant differences in attractiveness, efficiency, and stimulation. We could not prove significant differences regarding perspicuity (p = .06), dependability (p = .39), and novelty (p = .11). We can derive that the voting interactions *Tilt* and *Draw* differ significantly concerning attractiveness and pragmatic quality, whereas the rating interactions *Fingerprint* and *Touch* only show significant differences in attractiveness. Finally, to substantiate our findings, we used ANOVA to investigate the significance of our results. The rating interactions (*Touch* and *Fingerprint*) only showed statistical significance concerning attractiveness (F₁,₂₇ = 3.007, p < .005). Analyzing our voting interactions, the effect of the respective interaction was statistically significant (p < .0001) concerning unobtrusiveness, pragmatic quality, and attractiveness. Table 1 shows the results in detail.

## 6.2 Eyes-free performance of interactions

Participants ought to state why they looked at the mobile phone while performing the interactions. Half of the responses (56 of 112) comprised that participants did not look at the screen of the mobile device at all or only out of habit. This was especially the case for both *Fingerprint* (20 of 112) and *Draw* (15 of 112). The other half of responses showed that participants had to look at the screen since they either felt insecure (31 of 112) or unspecified unable to interact (25 of
<table><caption>**Table 1.** ANOVA results showing the effect of the interactions on each aspect.</caption><thead><tr><th></th><th>Rating interactions</th><th>Voting interactions</th></tr></thead><tbody><tr><td>Unobtrusiveness</td><td>(F<sub>1,27</sub> = 0.861, p > .05)</td><td>(F<sub>1,27</sub> = 67.747, p < .0001)</td></tr><tr><td>Pragmatic quality</td><td>(F<sub>1,27</sub> = 3.007, p > .05)</td><td>(F<sub>1,27</sub> = 46.023, p < .0001)</td></tr><tr><td>Attractiveness</td><td>(F<sub>1,27</sub> = 10.818, p < .005)</td><td>(F<sub>1,27</sub> = 25.369, p < .0001)</td></tr><tr><td>Effort of keeping input anonymous</td><td>(F<sub>1,27</sub> = 0.096, p > .05)</td><td>(F<sub>1,27</sub> = 19.277, p < .0005)</td></tr><tr><td>Recognized input</td><td>(F<sub>1,27</sub> = 0.213, p > .05)</td><td>(F<sub>1,27</sub> = 22.216, p < .0001)</td></tr><tr><td>Recognized interaction</td><td>(F<sub>1,27</sub> = 0.031, p > .05)</td><td>(F<sub>1,27</sub> = 39.676, p < .0001)</td></tr></tbody></table>

112). Several participants mentioned that they marked these options to express that they needed the screens content to remember the input value that we pre-defined in the tutorial. Additionally, some mentioned that they wanted to see what happens next or “to get a visual feedback” (P4). From these results, we derive that participants do not necessarily need the look at the mobile device during using our anonymous voting and rating interactions. Their look at the screen depended on the given task of the study. Consequently, we can state that the interaction techniques are eyes-free to perform.

## 6.3 Ensuring anonymous input values by unobtrusive interactions

We asked the participants what they did to keep their input anonymous. About half of the responses (57 of 112) showed that participants hid their device under or behind an object, e.g., the table, or used their body to hide the device, e.g., shielded the device’s screen with their hand or forearm (see Figure 3). From our observations during the sessions and the recordings, we can confirm this and even reinforce that most of the participants changed their position when it came to rate and vote. However, about one third renounced keeping the interactions secret. For the *Tilt* interaction, some stated that it would not have helped at all to hide the interaction because it is very noticeable. Participants found the *Draw* interaction unobtrusive enough to omit hiding. The remaining participants either both observed the surrounding and interacted while they felt unobserved, distracted the others by moving, or varied the input rate. Using a 5-point Likert scale, the participants evaluated (4) the *effort* to keep the input anonymous (5 = least effort), (5) the ease of *recognizing the input* value of others (5 = very easy) and (6) the ease of recognizing the *performed interactions* of other group members (see Figure 4). Participants assessed *Draw* better regarding the effort to keep an input anonymous in contrast to *Tilt*. Furthermore, the *Draw* interaction itself makes it hard to recognize input value as well as interaction. Although *Tilt* received a worse rating, the standard deviation is high, which means that the participants’ answers differed most regarding this interaction. *Touch* performs slightly better than *Fingerprint* regarding all three questions. Again, we performed an ANOVA to analyze the significance of the results (see Table 1). For the rating interactions the effect on the effort of keeping input anonymous, the recognized input as well as the recognized interaction was not
statistically significant (*p* > .05). In contrast, the effect of the respective voting interaction on these three aspects was statistically significant.

## 6.4 Using interactions in collocated settings

The last task aimed at using the interactions in a collaborative scenario whereas participants applied self-chosen interactions out of the presented interactions to rate or vote. Two participants used *Tilt* for voting and 26 participants used *Draw*. Comparing the rating interactions, 12 participants applied *Fingerprint*, whereas 16 participants used *Touch*. At the end of the user study, participants gave an overall assessment of all interactions regardless of rating or voting interaction. They confirmed that *Draw* is best concerning pragmatic quality (21 of 28) and unobtrusiveness (16 of 28). Participants also stated that *Draw* was fun (12 of 28). *Touch* also received positive feedback concerning fun factor (10 of 28).

With a concluding UEQ we asked the participants to assess usability and UX of their chosen interactions. Figure 5 (right) summarizes the mean values of the three aspects attractiveness, pragmatic quality, and hedonic quality. Although, the order of interactions at the end of the study is almost identical to the order after the tutorial, the mean values are better at the end. Only *Draw* slightly deteriorates concerning valuation. However, the results of the UEQ confirm the first impressions regarding popularity and functionality of the interactions. Furthermore, we derive a positive learning effect through repeating the interactions.

## 6.5 Qualitative user feedback and further observations

After each session, participants had the opportunity to give additional verbal feedback on the interactions and to express further observations they made. We received user feedback and improvement suggestions concerning the technical implementation, several types of device feedback, and the tutorial.

Overall, the most discussed interactions were *Fingerprint* and *Tilt*. Whereas some participants enjoyed the voting interaction *Tilt* very much (P10 and P24: "Performing the tilt interaction was fun."), others described problems they had while interacting: "I only could reject results. Accepting did not work well." (P9). Concerning obtrusiveness one participant (P25) mentioned, "Tilt was [a] very effusive [gesture] and would benefit from being a little smaller." P18 suggested "it should also be possible to perform this gesture with one hand." Concerning *Fingerprint*, the participants stated that all in all this interaction was quite good to rate anonymously although "it is unfamiliar to handle" (P16). P24 said, "I liked this interaction most because it is the most unobtrusive for me." The main point of criticism was that the fingerprint recognition "took surprisingly long" (P2). We assume that the recognition time led to depreciation. As a result, we will revise the proposed interaction to reduce the recognition time.

Participants further commented on the given feedback possibilities of the interface. We provided vibrations in terms of haptic feedback but participants criticized it as noticeable. Nevertheless, participants highlighted the importance of feedback. They mentioned problems in understanding how they rated and
thus wanted additional (visual) feedback (P5 and P9). Two participants (P3 and P27) also addressed the problem of correcting given input, which we did not enable directly during our tests but what would be crucial in real-world settings.

From the video recordings and the observations of the study leader during the study conduct, we received further insights on avoiding shoulder surfing, discovering new interactions, and acting in collaborative settings while assessing results. We observed that in the beginning of a session the group members sat closely at the table. They discovered the interactions as well as the tasks they had to perform during this time. When they started voting or rating, they mainly sat back in their chair and increased the space between each other. At the same time, they pulled their device closer towards their body and started looking around to observe the others. In this position, the participants primarily voted and rated. We also observed some other reactions. About one third of participants hid their device under the table while bending over. Occasionally, participants shielded their device with their hand (see Figure 3) or with their legs crossed. These findings can be utilized for further improvement of the proposed interactions.

As described in section 5, we decided to provide the participants a tutorial to become acquainted with the interactions. After reading the tutorial, participants tried the interactions the first time. We often observed that they had different kinds of difficulties during their first performances. For example, some participants did not touch the screen long enough performing *Touch*, some had problems with fingerprint recognition, or did not know instantly how fast to perform *Tilt*. Higher error rates coming from the specific logging protocol confirmed that observation. Since some participants mentioned missing information in the tutorial, e.g., execution speed, we recommend to investigate useful description methods for device-based interactions in depth. We further noticed that the group members normally talked to each other during the session. However, when the rating and voting started, the participants became much quieter in order to concentrate but easily resumed after performing the respective interaction. This shows that the participants could continue fluently and that the interactions did not obstruct them while collaborating.

# 7 Findings and Recommendations

From the results, we derived suggestions for improving the tested interactions as well as some further ideas and impressions for voting and rating activities using mobile devices in mixed-focus collaboration.

## 7.1 Device-based interaction techniques and their UX

The focus of the user study was on investigating the proposed interactions for anonymous voting (*Tilt* and *Draw*) and rating (*Fingerprint* and *Touch*) regarding their usability and user experience. The results from standardized UEQs and further questionnaires showed that overall interactions are useful and mostly suitable for anonymous voting and rating. However, there are some minor issues
concerning each interaction. We implemented *Fingerprint* with two interaction
steps to perform: first, tapping the fingerprint sensor and, second, confirming the
input at the end of the rating. From our observations and user feedback, we as-
sume that providing only one interaction step would lead to a better evaluation.
However, participants described this interaction as very unobtrusive especially
if the observation occurs from behind as illustrated in Figure 1. An additional
user study examining the application of the revised *Fingerprint* interaction in
different local arrangements would be beneficial.

The *Touch* interaction received positive feedback, which possibly correlates
with the fact that this interaction also resembles common smartphone interac-
tion. We assume that this is also the case for *Draw*, which participants rated
positively. However, the reason why the number of usage at the end of the per-
formed study is that high remains vague: either they really liked *Draw* the most
or they assessed *Tilt* as not suitable. Although, participants evaluated *Tilt* not
as good as the other interactions the standard deviation had the highest value
and shows that valuations differed most. *Tilt* was the only spatial interaction
in terms of movement and most innovative according to the participants. This
opens up possibilities for further investigations, e.g., on how such interactions
can be better introduced or implemented to meet the users' preferences.

In general, we will revise the interactions in order to make them more precise
in recognition and faster to perform. These are the main issues for designers
and developers to address when implementing interaction techniques that are
easy-free and unobtrusive to perform. Additionally, we need to reconsider device
feedback modalities in terms of visual feedback versus non-visual feedback in
context of shoulder surfing activities. This is due to the need of more intuitive
correction options while voting and rating. Since we found that participants eval-
uated the interactions better at the end of the user study than at the beginning,
we suggest addressing the learning phase of new interactions better even though
they also need to be innately intuitive. We used a common type of mobile tutorial
that showed a non-animated image of a certain interaction with additional text.
Especially, interactions with a higher degree of novelty or more distinct move-
ments could benefit from tutorials that, e.g., use short video sequences instead of
images or more precise movement descriptions. While creating such interactions,
designers should also consider the way of introducing the interaction techniques.

Performing new interactions was sometimes exhausting especially when they
did not meet the expectations. In contrast, the participants stated that the per-
formed drawing task was fun and that they could relax a little bit while drawing
and talking. Therefore, we recommend providing tasks that are entertaining or
easy to perform to prevent overworking.

## 7.2 Limitations and Future Work

Since the number of possible arrangements increases with a growing number of
devices and participants, the user study limited the number of used devices to
four. Of course, also larger groups can perform the interactions and the results
can be fully exploited when more devices are used. Nevertheless, some limitations
require further investigation. Although, we motivated the participants to stay curious and try to catch a glimpse on other devices, we did not appoint one or more explicit shoulder surfers within the group to receive a more realistic scenario. As a result, participants varied in performing shoulder surfer activities. It remains an open issue how one or more explicit shoulder surfers would change the participants' responses. In addition, other local arrangements, e.g., rows of seats instead of a table, should be further taken into account. This could extend the application area to other scenarios, e.g., polls in classrooms. Furthermore, the specifications of the mobile devices can influence the usage of the interaction techniques, e.g., the position of the fingerprint sensor, the device size, or the mobile platform. However, we provide a set of easy and useful interactions for anonymous voting and rating for further investigation.

Additional, further investigations should include a comparison with standard interactions for mobile devices, e.g., visual star rating. We did not include such interactions in our investigation yet because they do not follow our design goal to be performed eyes-free but need screen input. With our results in mind, we currently plan a usability study that examines this question. The results of our study provide the basis for further investigations using the revised interactions.

## 8 Conclusion

In this paper, we investigated interactions for anonymous voting and rating in collocated collaborative settings. The interactions' main aim is to avoid shoulder surfer activities by providing unobtrusive input and eyes-free performance. We introduced three design goals for mobile device-based interactions in collocated groups and described four derived interaction techniques and their implementation. The developed prototype was used in order to evaluate usability and UX aspects within a user study. We provided two Android applications that include all interactions and asked participants to use them for anonymous voting and rating. We collected data from observations, logging protocols, standardized UEQs, and semi-structured interviews.

Overall, participants assessed the interactions as suitable for anonymous voting and rating in terms of unobtrusive performance and eyes-free interaction to prevent shoulder surfing activities. From the questionnaires and interviews, we received some useful insights on revising the proposed interactions, especially *Fingerprint* and *Tilt*, and performing further user studies in collocated collaborative settings. According to the user feedback, we plan to apply our interactions to real-world scenarios with changing group sizes and local arrangements to extend the area of application and to investigate issues in different group settings.

## Acknowledgements

The European Social Fund (ESF) and the German Federal State of Saxony have funded this work within the project CyPhyMan (100268299).
References

1. Ashbrook, D., Starner, T.: MAGIC: A Motion Gesture Design Tool. In: Mynatt, E., Schoner, D., Fitzpatrick, G. (eds.) Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI’ 10). pp. 2159–2168 (2010)

2. De Luca, A., Harbach, M., Zezschwitz, E.V., Maurer, M.E., Slawik, B., Hussmann, H., Smith, M.: Now You See Me , Now You Don’t Protecting Smartphone Au- thentication from Shoulder Surfers. In: Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). pp. 2937–2946 (2014)

3. Dong, T., Churchill, E.F., Nichols, J.: Understanding the Challenges of Designing and Developing Multi-Device Experiences. In: Proc. of the 2016 ACM Conference on Designing Interactive Systems (DIS ’16). pp. 62–72 (2016)

4. Eiband, M., Khamis, M., von Zezschwitz, E., Hussmann, H., Alt, F.: Understanding Shoulder Surfing in the Wild: Stories from Users and Observers. In: Proc. of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. pp. 4254–4265 (2017)

5. Eiband, M., von Zezschwitz, E., Buschek, D., Hußmann, H.: My Scrawl Hides It All: Protecting Text Messages Against Shoulder Surfing With Handwritten Fonts. In: Proc. of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’16. pp. 2041–2048. ACM Press (2016)

6. Goucher, W.: Look behind you: the dangers of shoulder surfing. Computer Fraud & Security **2011**(11), 17 – 20 (2011)

7. Gutwin, C., Greenberg, S.: Design for Individuals, Design for Groups: Tradeoffs Between Power and Workspace Awareness. In: Proc. of the 1998 ACM conference on Computer Supported Cooperative Work (CSCW’98). pp. 207–216 (1998)

8. Hinderks, A., Schrepp, M., Thomaschewski, J.: User experience questionnaire (2019), www.ueq-online.org

9. Institute, P.: Global Visual Hacking Experimental Study: Analysis. Tech. Rep. August, Ponemon Institute (2016)

10. Kim, S., Ko, D., Lee, W.: Utilizing Smartphones as a Multi-Device Single Display Groupware to Design Collaborative Games. In: Proc. of the 2017 ACM Conference on Designing Interactive Systems (DIS ’17). pp. 1341–1352. ACM (2017)

11. Kim, S.H., Kim, J.W., Kim, S.Y., Cho, H.G.: A new shoulder-surfing resistant password for mobile environments. In: Proc. of the 5th International Conference on Ubiquitous Information Management and Communication - ICUIMC ’11. ACM Press (2011)

12. Korzetz, M., Kühn, R., Schlegel, T.: Turn it, pour it, twist it: A model for de-
signing mobile device-based interactions. In: Proceedings of the 5th International
Conference on Human-Computer Interaction and User Experience in Indonesia.
p. 4. CHIuXiD ’19, ACM (2019)

13. Kühn, R., Korzetz, M., Büschel, L., Korger, C., Manja, P., Schlegel, T.: Natural voting interactions for collaborative work with mobile devices. In: Proc. of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 2570–2575. CHI EA ’16, ACM, New York, NY, USA (2016)

14. Kühn, R., Korzetz, M., Büschel, L., Schumann, F.W., Schlegel, T.: Device-based Interactions for Anonymous Voting and Rating with Mobile Devices in Collaborative Scenarios. In: Proc. of the 15th International Conference on Mobile and Ubiquitous Multimedia - MUM ’16. pp. 315–317 (2016)

15. Kühn, R., Schlegel, T.: Mixed-Focus Collaboration Activities for Designing Mobile Interactions. In: Proc. of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI’18). pp. 71–77 (2018)
16. Laugwitz, B., Held, T., Schrepp, M.: Construction and Evaluation of a User Experience Questionnaire. In: HCI and Usability for Education and Work. pp. 63–76 (2008)

17. Leigh, S.w., Schoessler, P., Heibeck, F., Maes, P., Ishii, H.: THAW: Tangible Interaction with See-Through Augmentation for Smartphones on Computer Screens. In: Proc. of the 9th International Conference on Tangible, Embedded, and Embodied Interaction - TEI ’15. pp. 89–96 (2015)

18. Lim, H., Ahn, H., Kang, J., Suh, B., Lee, J.: Ubi-jector: An Information-Sharing Workspace in Casual Places Using Mobile Devices. In: Proc. of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI ’14). pp. 379–388 (2014)

19. Lucero, A., Holopainen, J., Jokela, T.: Pass-Them-Around: Collaborative Use of Mobile Phones for Photo Sharing. In: Proc. of the 2011 Conference on Human Factors in Computing Systems - CHI ’11. pp. 1787–1796 (2011)

20. Lucero, A., Holopainen, J., Jokela, T.: Mobicomics: Collaborative use of mobile phones and large displays for public expression. In: Proc. of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services. pp. 383–392. MobileHCI ’12, ACM (2012)

21. Lucero, A., Keränen, J., Jokela, T.: Social and Spatial Interactions: Shared Co-Located Mobile Phone Use. In: Proc. of the 28th of the international conference extended abstracts on Human factors in Computing Systems. pp. 3223–3228. CHI EA ’10, ACM Press (2010)

22. Lucero, A., Porcheron, M., Fischer, J.E.: Collaborative Use of Mobile Devices to Curate Sources of Inspiration. In: MobileHCI ’16 Proce. of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. pp. 611–616 (2016)

23. McCrindle, C., Hornecker, E., Lingnau, A., Rick, J.: The design of t-vote: a tangible tabletop application supporting children’s decision making. In: Proc. of the 10th International Conference on Interaction Design and Children. pp. 181–184 (2011)

24. Rädle, R., Jetter, H.C., Marquardt, N., Reiterer, H., Rogers, Y.: HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration. In: Proc. of the Ninth ACM International Conference on Interactive Tabletops and Surfaces - ITS ’14. pp. 45–54. ACM Press (2014)

25. Rico, J., Brewster, S.: Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In: Proc. of the 28th International Conference on Human Factors in Computing Systems - CHI ’10. pp. 887–896 (2010)

26. Roth, V., Richter, K.: How to fend off shoulder surfing. Journal of Banking and Finance 30(6), 1727–1751 (2006)

27. Rott, B., Marouane, C.: Digitalization in Schools – Organization, Collaboration and Communication, pp. 113–124. Springer Berlin Heidelberg (2018)

28. Scheible, J., Ojala, T.: MobiLenin Combining A Multi-Track Music Video, Personal Mobile Phones and A Public Display into Multi-User Interactive Entertainment. In: MULTIMEDIA ’05: Proc. of the 13th annual ACM International Conference on Multimedia. pp. 199–208 (2005)

29. Seifert, J., Simeone, A., Schmidt, D., Holleis, P., Reinartz, C., Wagner, M., Gellersen, H., Rukzio, E.: MobiSurf: Improving Co-located Collaboration through Integrating Mobile Devices and Interactive Surfaces. In: Proc. of the 2012 ACM international conference on Interactive tabletops and surfaces - ITS ’12. pp. 51–60. ACM Press (2012)

30. Straw poll - the place to create instant, real-time polls for free. (2019), https://www.strawpoll.me/
31. Turnbull, D.: Rating, voting & ranking: designing for collaboration & consensus. In: CHI '07 Extended Abstracts on Human Factors in Computing Systems. pp. 2705–2710. ACM, New York, NY, USA (2007)

32. Winkler, C., Gugenheimer, J., De Luca, A., Haas, G., Speidel, P., Dobbelstein, D., Rukzio, E.: Glass Unlock: Enhancing Security of Smartphone Unlocking through Leveraging a Private Near-eye Display. In: Proc. of the 33rd ACM Conference on Human Factors in Computing Systems - CHI '15. pp. 1407–1410. ACM (2015)

33. von Zezschwitz, E., De Luca, A., Brunkow, B., Hussmann, H.: SwiPIN - Fast and Secure PIN-Entry on Smartphones. In: Proc. of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 1403–1406 (2015)