# SwipeBuddy: A Teleoperated Tablet and Ebook-Reader Holder for a Hands-Free Interaction

Max Pascher, Stefan Schneegass, Jens Gerken

► To cite this version:

Max Pascher, Stefan Schneegass, Jens Gerken. SwipeBuddy: A Teleoperated Tablet and Ebook-Reader Holder for a Hands-Free Interaction. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.568-571, 10.1007/978-3-030-29390-1_39. hal-02877654

HAL Id: hal-02877654

https://inria.hal.science/hal-02877654v1

Submitted on 22 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# SwipeBuddy

## A teleoperated tablet and ebook-reader holder for a hands-free interaction

Max Pascher¹,²[0000-0002-6847-0696], Stefan Schneegass², and Jens Gerken¹

¹ Westphalian University of Applied Sciences, Human-Computer Interaction Gelsenkirchen, Germany

² University of Duisburg-Essen, paluno Essen, Germany

**Abstract.** Mobile devices are the core computing platform we use in our everyday life to communicate with friends, watch movies, or read books. For people with severe physical disabilities, such as tetraplegics, who cannot use their hands to operate such devices, these devices are barely usable. Tackling this challenge, we propose SwipeBuddy, a teleoperated robot allowing for touch interaction with a smartphone, tablet, or ebook-reader. The mobile device is mounted on top of the robot and can be teleoperated by a user through head motions and gestures controlling a stylus simulating touch input. Further, the user can control the position and orientation of the mobile device. We demonstrate the SwipeBuddy robot device and its different interaction capabilities.

**Keywords:** Self-determined Life · Head-based Interaction · Human-Robot Interaction

## 1 Introduction

Providing a hands-free interaction mode to interact with technology is an important feature which may in particular benefit people with severe physical disabilities, such as tetraplegics, who can not use their hands to interact with technology.

Interaction technologies such as gaze-based interaction and head movement have been explored to operate, e.g. a PC [2–4]. However, today’s ubiquitous technology interaction scenarios are much more tightly integrated in everyday activities. Prime examples might be reading the news on a smartphone or tablet while eating breakfast, or surfing the web on such a device while at the same time watching TV on a shared big screen. Since people with tetraplegia are often relying on human assistants to interact with technology, this parallel interaction and seamless switching between tasks increases the burden on both assistants and users. From our experience and interviews with people with tetraplegia, we know that they may also be reluctant to have their human assistant constantly switch between such small tasks [1]. On the other hand, technological devices that may support e.g. reading, are also bound to certain locations and positions and do not offer the flexibility that people without such disabilities enjoy. In this
demonstration paper, we present SwipeBuddy, a mobile robot with a hands-free Interface enables such flexiblity and enhances a human’s opportunities regarding task and attention switching.

## 2 The SwipeBuddy-System

SwipeBuddy is a physical robotic device that acts as a mobile ebook reader and photo browser and can be controlled by using head movements. Its main tasks are to a) hold the digital device (e.g. Amazon Kindle) b) provide an interaction mechanism with the device to swipe pages and c) flexibly move around so that it does not interfere with parallel activities (e.g. eating). The SwipeBuddy acts as an r-c mobile ebook-reader that can be positioned using head movements.

### 2.1 Hardware platform

The prototype of SwipeBuddy consists of a mobile robotic platform with a continuous track vehicle propulsion (caterpillar track) for high maneuverability, i.e. turning around its own axis. A mobile device (smartphone, Kindle etc.) is placed on a tilting platform that allows the user to easily manipulate the view-angle of the device. Furthermore, the robot has a mechanical swipe mechanism that helps flipping pages or pressing icons. The swiping mechanism consists of a tip of a stylus for capacitive touch displays and a motorized arm which provides contact pressure for swipe and scroll. Using a mechanical swipe mechanism enables a user to activate a swipe action with any application and with any device. A software controlled swipe would be device dependent and thus less flexible. A visual mock-up of the SwipeBuddy is depicted in Figure 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Draft concept of the SwipeBuddy. To increase mobility the robot is equipped with a continuous track vehicle propulsion. Furthermore, it features a tilting platform to change the tilt and therefore view-angle of the mobile device.</figcaption></figure>
## 2.2 Interaction design

The interaction concept consists of a Magnetic-AngularRate-Gravity (MARG) sensor that is mounted on a headband. This setup enables capturing of head motion data along the roll, pitch and yaw axis of the head when a user wears the headband in order to control the swipe arm to switch pages of the device and furthermore steer the robot itself.

Robustness of the measured orientation depends on the underlying method of sensor data fusion since various disturbances introduce errors. In this work, we use a robust quaternion based filter approach to estimate orientation from MARG-sensors that was introduced by Wöhle and Gebhard [5] and is implemented on a standalone micro-controller. Since the orientation is measured relative to the global quantities (gravity and magnetic field), there are no environmental restrictions that would limit usage to a local use-case. We are using a self-designed sensor system that enables wireless communication via WiFi UDP. The orientation data is mapped on the actuators of the system to achieve the defined tasks.

In our interaction design, the user can switch between different modes to steer the robotic platform, change the tilt angle of the electronic device, and perform a forward or backward swipe action. Additionally, an idle mode is available to block all interactions to put the sensor headband on or off. In particular, the user is switching modes by performing a movement along the yaw-axis, like it is shown in Figure 2. Movements in roll and pitch axis are used in each mode different i.a. driving, steering, tilting, and swiping.

To provide a visual feedback to the user about which mode is selected and to help to orient, 25 RGB LEDs are installed and used in a way that supports intuitive insight. For Example in driving mode SwipeBuddy is following a car metaphor. White headlamp and red tail lamps are turned on and the continuous track is illuminated green (see Figure 3). A slightly moving in the roll axis which is used for steering is changing the green continuous track light to a yellow blinker to indicate a change of the steering direction on the respective side. Furthermore, all LEDs are mounted at special positions and on special parts of the robotic system where they could be recognized easily and show which parts are active.

## 3 Conclusion

With the SwipeBuddy we presented a software- and device independent robotic assistant system to people with severe physical disabilities. It does not aim to replace human assistants but support them for very specific tasks where users might not feel comfortable constantly asking for help. Our first informal tests with people with tetraplegia showed that the SwipeBuddy could potentially be a very welcome helper device with positive feedback regarding the use-case as well as the overall interaction design. While our scenario focuses on people who are physically limited in the usage of their hands, the SwipeBuddy could also assist people whose hands are currently tied to other activities, e.g. in an industrial
<figure><img src="image_3.png" /><figcaption>Fig. 2: The modes of SwipeBuddy are arranged in a circular queue and can be selected by movement along the yaw-axis of the head in both directions.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3: The user is steering SwipeBuddy by his head movements. The white headlamps are turned on.</figcaption></figure>

context (industry 4.0) where an operator might need their hands to operate a machine interface and at the same time need to read a digital manual.

## References

1. Baumeister, A., Pascher, M., Thietje, R., Gerken, J., Klein, B.: Anforderungen an die interaktion eines roboterarms zur nahrungsaufnahme bei tetraplegie — eine ethnografische analyse. In: Kongress und Ausstellung zu Alltagsunterstützenden Assistenzlösungen — Tagungsband. pp. 100–101. Karlsruher Messe- und Kongress GmbH (2018)

2. Duchowski, A.T.: Gaze-based interaction: A 30 year retrospective. Computers & Graphics **73**, 59 – 69 (2018). https://doi.org/10.1016/j.cag.2018.04.002, http://www.sciencedirect.com/science/article/pii/S0097849318300487

3. Plaumann, K., Ehlers, J., Geiselhart, F., Yuras, G., Huckauf, A., Rukzio, E.: Better than you think: Head gestures for mid air input. In: Human-Computer Interaction — INTERACT 2015, pp. 526–533. Springer International Publishing (2015). https://doi.org/10.1007/978-3-319-22698-9_36, https://doi.org/10.1007/978-3-319-22698-9_36

4. Roig-Maimó, M.F., MacKenzie, I.S., Manresa-Yee, C., Varona, J.: Head-tracking interfaces on mobile devices: Evaluation using fitts’ law and a new multi-directional corner task for small displays. International Journal of Human-Computer Studies **112**, 1–15 (2018). https://doi.org/10.1016/j.ijhcs.2017.12.003, http://www.sciencedirect.com/science/article/pii/S1071581917301647

5. Wöhle, L., Gebhard, M.: A robust quaternion based kalman filter using a gradient descent algorithm for orientation measurement. In: 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). pp. 1–6 (May 2018). https://doi.org/10.1109/I2MTC.2018.8409593