# FGFlick: Augmenting Single-Finger Input Vocabulary for Smartphones with Simultaneous Finger and Gaze Flicks

Yuki Yamato, Yutaro Suzuki, Shin Takahashi

<figure><img src="image_1.png" /></figure>

## To cite this version:

Yuki Yamato, Yutaro Suzuki, Shin Takahashi. FGFlick: Augmenting Single-Finger Input Vocabulary for Smartphones with Simultaneous Finger and Gaze Flicks. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.421-425, 10.1007/978-3-030-85607-6_50. hal-04291256

HAL Id: hal-04291256

https://inria.hal.science/hal-04291256v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# FGFlick: Augmenting Single-Finger Input Vocabulary for Smartphones with Simultaneous Finger and Gaze Flicks

Yuki Yamato[0000-0002-5288-9329], Yutaro Suzuki, and Shin Takahashi

University of Tsukuba, Tsukuba, Ibaraki, Japan
{yamato,ysuzuki,shin}@iplab.cs.tsukuba.ac.jp

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** FGFlick overview. The user flicks with a finger on a smartphone while he also moves his gaze linearly.</figcaption></figure>

**Abstract.** FGFlick is an interactive technique featuring simultaneous single-finger operation and a gaze. The user flicks a smartphone and moves their gaze linearly. FGFlick thus augments the single-finger input vocabulary. As a result of the evaluation of the FGFlick gestures, we achieved success rates of 84.0%.

**Keywords:** FGFlick · Single-Finger Input Vocabulary · Flick Gesture · Gaze Input · Interaction Techniques · Mobile Devices

## 1 Introduction

Smartphone users favor one-handed operation [4], but it causes the limited thumb reach problem—distant bezels and physical buttons are hard to reach [1]. The one-handed operation also restricts multi-finger inputs. It is necessary to increase the single-finger input vocabulary to allow one-handed operation with a stable grip.

In this work, we developed FGFlick, an interactive technique that simultaneously uses a single-finger operation and an explicit gaze (Fig. 1). The user flicks a smartphone and moves their gaze linearly. By combining finger and gaze flicks, FGFlick augments the single-finger input vocabulary. Here, we briefly describe the design and implementation of FGFlick.
## 2 Related Work

Previous work proposed combined hand and eye movements to improve the manipulation of laptop computers and wearable devices [8,3]. The works of Pfeuffer et al. [7,5,6] forwarded a touch to a gazed point on a tablet; this permitted point selection and cursor manipulation around the point. They sought principally to attain unreachable areas of the screen. However, we sought to augment the input vocabulary.

The work of Elleuch et al. [2] proposed a method to interact with mobile devices using a combination of gazes and static hand gestures; we employed finger flicks instead. Wang et al. [9] developed the BlyncSync gesture set (synchronous blinks and touches); we employed simple gaze gestures.

## 3 FGFlick

FGFlick is an interactive technique featuring simultaneous single-finger operation and a gaze (Fig. 1). The user flicks a smartphone while moving the gaze. We have designated this gaze gesture as a “gaze flick.” Because finger and gaze flicks must be performed simultaneously, they are distinguished from conventional flicks and involuntary eye movements. A gaze flick is a linear gaze. Compared with compound gazes, a gaze flick is rapid and does not require visual guidance. By combining the directions of finger and gaze flicks, FGFlick augments the single-finger input vocabulary of smartphones.

FGFlick has various interaction examples. As an alternative to a physical button, an FGFlick gesture is used to capture screenshots; the user is not required to hold the power button and the volume button simultaneously to get screenshots, as one of the traditional ways. FGFlick can also be employed to manipulate unreachable targets. We assign FGFlick gestures in the same direction to “Back” and “Open.” The user can navigate back to a photo gallery from a screen displaying a selected image by executing *Flick-Right-Gaze-Right*. The user can open the Notification Center by executing *Flick-Left-Gaze-Left*.

## 4 Prototype Implementation

We used an iPhone X and ARKit<fnref n="1" /> to implement eye-tracking and the FGFlick recognition system.

### 4.1 Recognition of FGFlick Gestures

The system starts FGFlick recognition when it detects a finger flick. Figure 2 shows the finger flick and simultaneous gaze flick. When the system detects a finger flick, it notes the start and end times. It then calculates the start and end times of a possible gaze flick: $G_{start} = F_{start} - offset_{start}$, and $G_{end} =$

<footnote n="1">¹ https://developer.apple.com/documentation/arkit</footnote>
<figure><img src="image_5.png" /></figure>

* F_start / F_end: the start and the end position of the finger flick on the screen.

* G_start: the start position of the gaze flick before offset_start from F_start.

* G_end: the end position of the gaze flick after offset_end from F_end.

* offset_start / offset_end: the offset between F_start and G_start and between F_end and G_end.

<caption>Fig. 2. A part of FGFlick recognition parameters.</caption>

F_end + offset_end. Based on preliminary experiments, we set offset_start to 0 ms and offset_end to 66.8 ms.

Next, the system decides whether the finger flick is an FGFlick gesture or a conventional flick, depending on whether the gaze flick is explicit or simply an involuntary eye movement. In preliminary experiments, we found that a gaze flick (an explicit gesture) was longer than involuntary eye movements during conventional flicks. Thus, we distinguish gaze flicks by the length of eye movement during the finger flick (d_gaze; the linear distance travelled from G_start to G_end). If the length is shorter than a threshold (d_th), the system regards the eye movement as involuntary and the finger flick is thus a conventional flick; otherwise, the system regards the eye movement as a gaze flick. We set d_th to 218 pt based on preliminary experiments.

The FGFlick gestures are classified by the directions of the finger and gaze flicks. For finger flicks, the system calculates the angle (θ_finger) of F_start to F_end and assigns it to one of four directions. For gaze flicks, the system calculates the angle (θ_gaze) of G_start to G_end and assigns it to one of four directions.

## 4.2 Preliminary Evaluation

We evaluated the accuracy of the FGFlick recognition system. We enlisted eight volunteers (P1 – P8; mean age, 23.0 years [range, 21 – 25 years]; six males). We asked them to perform 20 types of FGFlick gestures, four finger flicks (up, down, left, right) multiplied by four gaze flicks (up, down, left, right) + “Keep.” All volunteers grasped an iPhone X and performed each FGFlick gesture once in one session; they completed five sessions in total. We thus obtained 800 gestures: $= 8_{volunteers} \times 5_{sessions} \times 20_{gestures}$.

Figure 3 shows the confusion matrix. The success rate was 84.0%. “Down” was often mistaken for “Keep” because many volunteers positioned the device lower than the face. Thus, they initially looked down and lacked the ability to gaze “Down” further. We plan to change the “Down” distance threshold.
<figure><img src="image_6.png" /><figcaption>Fig. 3. The confusion matrix of the FGFlick gestures.</figcaption></figure>

## 5 Conclusion and Future Work

FGFlick is an interactive technique featuring simultaneous single-finger operation and a gaze. The user flicks a smartphone and moves their gaze linearly. FGFlick thus augments the single-finger input vocabulary. We will further evaluate accuracy and usability, then explore whether users develop eye fatigue.

## References

1. Bergstrom-Lehtovirta, J., Oulasvirta, A.: Modeling the Functional Area of the Thumb on Mobile Touchscreen Surfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1991–2000. CHI ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2556288.2557354

2. Elleuch, H., Wali, A., Samet, A., Alimi, A.M.: Interacting with Mobile Devices by Fusion Eye and Hand Gestures Recognition Systems Based on Decision Tree Approach. In: Verikas, A., Radeva, P., Nikolaev, D.P., Zhang, W., Zhou, J. (eds.) Ninth International Conference on Machine Vision (ICMV 2016). vol.
10341, pp. 62–66. International Society for Optics and Photonics, SPIE (2017). https://doi.org/10.1117/12.2269010

3. Nagamatsu, T., Yamamoto, M., Sato, H.: MobiGaze: Development of a Gaze Interface for Handheld Mobile Devices. In: CHI '10 Extended Abstracts on Human Factors in Computing Systems. pp. 3349–3354. CHI EA '10, Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1753846.1753983

4. Ng, A., Brewster, S.A., Williamson, J.H.: Investigating the Effects of Encumbrance on One- and Two- Handed Interactions with Mobile Devices. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1981–1990. CHI '14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2556288.2557312

5. Pfeuffer, K., Alexander, J., Chong, M.K., Gellersen, H.: Gaze-touch: combining gaze with multi-touch for interaction on the same surface. In: Proceedings of the 27th annual ACM symposium on User interface software and technology. pp. 509–518. UIST '14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2642918.2647397

6. Pfeuffer, K., Alexander, J., Chong, M.K., Zhang, Y., Gellersen, H.: Gaze-Shifting: Direct-Indirect input with pen and touch modulated by gaze. In: Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. pp. 373–383. UIST '15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2807442.2807460

7. Pfeuffer, K., Gellersen, H.: Gaze and Touch Interaction on Tablets. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology. pp. 301–311. UIST '16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2984511.2984514

8. Rivu, S., Abdrabou, Y., Mayer, T., Pfeuffer, K., Alt, F.: GazeButton: Enhancing Buttons with Eye Gaze Interactions. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications. pp. 1–7. ETRA '19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3317956.3318154

9. Wang, B., Grossman, T.: BlyncSync: Enabling Multimodal Smartwatch Gestures with Synchronous Touch and Blink. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. pp. 1–14. CHI '20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3313831.3376132