# The Presenter in the Browser: Design and Evaluation of Human Interactive Overlays with Web Content

Maxime Cordeil, Anais Servais, Guillaume Truong, Tim Dwyer, Dhaval Vyas, Christophe Hurter

<figure><img src="image_1.png" /></figure>

## To cite this version:

Maxime Cordeil, Anais Servais, Guillaume Truong, Tim Dwyer, Dhaval Vyas, et al.. The Presenter in the Browser: Design and Evaluation of Human Interactive Overlays with Web Content. Multimodal Technologies and Interaction, 2025, 9 (2), pp.10. 10.3390/mti9020010. hal-05176151

HAL Id: hal-05176151

https://enac.hal.science/hal-05176151v1

Submitted on 22 Jul 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Article

# The Presenter in the Browser: Design and Evaluation of Human Interactive Overlays with Web Content

Maxime Cordeil <fnref n="1" />,*,† ID, Anais Servais <fnref n="2" /> ID, Guillaume Truong <fnref n="3" /> ID, Tim Dwyer <fnref n="4" /> ID, Dhaval Vyas <fnref n="1" /> ID
and Christophe Hurter <fnref n="3" />,<fnref n="5" />,*,† ID

**Abstract:** This research explores the design and evaluation of a webcam-based presentation tool that enables presenters to directly interact with web content via free-hand gestures. Our approach consists of overlaying the webcam video feed on top of web browser content to enable live presentations of any webpage. To support interactive presentations, we designed free-hand gesture interactions with the webpage to enable pointing, clicking, panning, and zooming interactions. We propose three alternatives to enable free-hand clicking: dwell time, modal key control, and a pinching interaction technique. We conducted an exploratory user study of these alternative designs to gather insights on the usability of such systems from a presenter point of view, with a focus on understanding the impact of the three techniques on flow interruptions. The results indicate that the system we propose can be used to deliver presentations effectively and that natural gestures do not disturb the flow of the presentation.

**Keywords:** visualization; augmented reality; pointing; gesture; video

## 1. Introduction

The web is full of rich interactive content for teaching, entertainment, utilities, and many other applications. It has become more and more common that this content is used as material for online presentations, e.g., screen sharing of a GitHub page or a Wikipedia page during a presentation. However, the interactions to support such presentations during online presentations are typically via mouse and keyboard for pointing, scrolling, and navigating the webpage. The problem with such interactions is that they split the viewers' attention between the presenter's video feed and the interaction taking place at the mouse location. Furthermore, apart from the mouse cursor, the interactions are not particularly visible to the audience and are easily missed.

Recent advances in computer vision and the availability of real-time webcam-based segmentation libraries such as MediaPipe [1] enable user hand tracking and therefore support more natural pointing interactions to support more engaging presentations [2,3]. The design space of applications introduced by the combination of a webcam and a real-time tracking library is novel and exciting for designers to build a new range of more

<figure><img src="image_3.png" /></figure>

check for
updates

Academic Editor: Keith Cheverst

Received: 8 October 2024

Revised: 10 December 2024

Accepted: 15 January 2025

Published: 27 January 2025

**Citation:** Cordeil, M.; Servais, A.; Truong, G.; Dwyer, T.; Vyas, D.; Hurter, C. The Presenter in the Browser: Design and Evaluation of Human Interactive Overlays with Web Content. *Multimodal Technol. Interact.* **2025**, *9*, 10. https://doi.org/10.3390/mti9020010

**Copyright:** © 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD 4072, Australia; d.vyas@uq.edu.au</footnote>

<footnote n="2">² GIGA Research—CRC Human Imaging, Liège University, 4000 Liège, Belgium; anais.servais@uliege.be</footnote>

<footnote n="3">³ Fédération ENAC ISAE-SUPAERO ONERA, 31055 Toulouse, France; guillaume.truong@enac.fr</footnote>

<footnote n="4">⁴ Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia; tim.dwyer@monash.edu</footnote>

<footnote n="5">5 IPAL, International Research Lab (IRL2955) CNRS-A*STAR-NUS, Singapore 138632, Singapore</footnote>

<footnote>* Correspondence: m.cordeil@uq.edu.au (M.C.); christophe.hurter@enac.fr (C.H.)</footnote>

<footnote>† These authors contributed equally to this work.</footnote>
immersive, embodied interactions with web content using the webcam, and on widely
available hardware.

Using this technology, recent work in the human–computer interaction community
has contributed tools to enhance the presentation of content in live video-conferencing
applications, specifically for interactive data visualisations [2] or interactive multimedia
content [3,4]. These novel interactive video presentation paradigms augment a remote
video presentation with data visualisations, images, or textual content overlays, and the
presenter can interact with the content directly with hand gestures to support and illustrate
their speech.

The audience, on the other side, benefits from a more engaging presentation experience
with a potentially increased sense of presence as the presenter’s video feed, including their
gestures and facial expressions, is embedded within the content they present [2,3].

While this initial body of work tackles the problem through custom-built application-
specific desktop software, we aim for a more generic solution: we designed and developed
WebPresenter, a webcam-based, online presentation plugin for Chrome and Chromium-
based browsers to integrate any interactive webpage content into a gesture-controlled,
embedded video-stream presentation platform. WebPresenter overlays the live feed of the
presenter’s body onto a webpage and maps hand movements and gestures to mouse events
in the browser. Presenters can point, select, navigate, pan, zoom, and scroll content on
the web to support their presentation. This approach literally replaces the mouse with the
presenter themselves to facilitate the creation of casual online or recorded presentations.

In this paper, our central contribution is to report on our design approach to mapping free-hand gestures and movement-to-mouse interactions that can be interpreted in the browser for presentation purposes. We also design and evaluate two novel interaction techniques, focus on the hand and presenter overlay, that aim to minimize the distraction induced by this type of presentation, a limitation identified in previous research [2,3]. Our design exploration focuses on three main alternatives to map hand gestures to trigger actions on a webpage (e.g., clicking a link or a button): dwell time, modal interaction, and pinch. We conducted a qualitative user study with five participants, all experts in and regular users of online presentations, to assess the relative benefits and drawbacks of each mode. Our study also focuses on how this type of free-hand gesture presentation tool affects and disrupts the flow of a presentation. We report observations and participants’ feedback on the usability, current limitations, and possible ways to improve WebPresenter.

To summarise, this work contributes the following:

* WebPresenter, an open-source web browser plugin that puts the presenter into a webpage to perform presentations;

* A set of novel techniques to support free-hand gesture interactions with web content;

* An initial study on such presentations that focuses on flow interruptions regarding the three different modalities of free-hand interactions, which are dwell time, key-controlled modal interaction, and pinch.

The structure of the paper is as follows. First, we review the literature to contextualize
our work and highlight the need for improved interaction techniques in data presentation.
This section identifies gaps in existing solutions and establishes the motivation for our
proposed approach. Next, we detail the rationale behind our interaction design, describing
the investigated interaction modalities and their intended use in enhancing data presenta-
tion. We then provide a comprehensive overview of our implementation, explaining the
technical aspects and choices made during development. Following this, we present the
use cases we investigated to demonstrate the practical applications of our system and its
potential to address real-world challenges. The subsequent section outlines our user study,
including the methodology, observations, and in-depth analysis of the results. Finally, we
discuss the implications of our findings, their relevance to the field, and directions for future research, concluding the paper with a summary of our contributions.

## 2. Related Work

The design and study of WebPresenter involves designing point-and-click interactions in mid-air, understanding hand gestures for presentations, and integrating interactive content into the video feed. We review the literature in each of these areas and explain why interaction design is needed in the emerging domain of webcam-based free-hand-gesture presentations.

### 2.1. Augmented, Interactive Presentations

Augmenting presentations with multimodal interaction has been an active topic in HCI for more than a decade. Fourney et al. [5] designed Maestro, a gesture-based slide presenter. However, in their work, the presenter interacted on a projected image of the slides, and not in the webcam feed directly. Previous work has also investigated ways to augment presentations by providing an interactive surface with pen interactions for the presenter to draw data visualisations for storytelling [6,7]. These tools were found to be expressive for the presenter and engaging for the audience compared to traditional PowerPoint presentations, but required an interactive surface to support the presentation of visual information. A number of studies have investigated augmenting the presentation directly into the video stream. A famous example is Hans Rosling's Gapminder video (https://www.gapminder.org/videos/how-did-the-world-population-change/) (accessed on 17 January 2025)) presentation where a data visual is overlaid on the video stream and, via scripted movements and gestures, the presenter tells a story about world socioeconomic data. More recent studies have used live movement tracking to directly embed gestures into the video stream. In ChalkTalk [8], Perlin et al. designed a sketching tool for presentations in science education that uses mouse gestures to create illustrations of physical phenomena that are overlaid in the video stream. More recently, Kirilyuk et al. [9] contributed a technique called "visual caption", which suggests and overlays images in the presenter's space based on speech recognition in a video call to illustrate complex ideas. Users use the mouse to organize the content in the video stream.

A more recently emerging class of augmented video presentation tool is based on modern image-based hand tracking APIs such as MediaPipe [1] or Microsoft's Kinect 3D depth camera. RealityTalk [4] uses speech recognition to augment video content, and MediaPipe allows hand tracking and thereby enables interaction with media within the webcam feed, for example, to grab and move images. Saquib et al. [10] designed a kinect-based system to allow users to create and engage with live graphics in 2D videos for more expressive storytelling [11]. Davis et al. [3] designed and studied CLIO, a multimodal webcam-based presentation tool that enables media interaction with voice, gestures, mouse, and touch inputs to enrich presentations. Their study showed that most participants preferred the mid-air gestures as a mode of interaction with content, and provided evidence of audience enthusiasm and engagement with such presentations. The interaction design of the gestures involved a static set of gestures to trigger actions and manipulate content. Our work focuses on studying differences in well-known alternative designs for pointing and clicking, and their impact on the presenter and presentation. Hall et al. [2] contributed "Augmented Chironomia", which focuses on data visualisation storytelling. They designed and studied a system in which the user manipulates, within the webcam feed, data visualizations to tell a story. They found the tool to be engaging for audiences but also identified challenges such as the potential for distraction and ambiguity between intentional and non-intentional gestures.
Most previous studies in this area required the development of bespoke systems (e.g., for science presentations, visualisations, sketching, or multimedia content communication or to suggest images automatically) and rely on a separate preparation phase to select data and visuals or multimedia content to add to the system for presentation. Our work takes a different approach; there is no need for a bespoke system, as WebPresenter overlays the webcam feed directly into the browser to enable interaction with any type of content and focuses on the modality of the interaction to navigate, point, and select elements in content for presentation purposes.

## 2.2. Point-and-Click with Free-Hand Mid-Air Gestures

Point-and-click is a fundamental interaction paradigm in WIMP, with straightforward adaptation to touch and mid-air interfaces [12]. Mouse point-and-click involves the user moving a cursor or pointer to a graphical target and selecting it with a mouse click [13]. Except for touch phones and tablets, point-and-click-input devices continuously track hand movements to point a cursor and include a button to confirm the selection of a graphical target [14]. The button press allows for the decoupled movement of a cursor freely on the graphical user interface and also prevents a user from moving the cursor away from the target to perform the selection, avoiding the Heisenberg selection effect [15].

Pointing and clicking is also a relevant interaction paradigm for systems that leverage mid-air interactions [16]. Free-hand gestures have been mainly used to control displays by pointing at them, and techniques mainly involve raycasting [17,18], gaze+pinch [19], or more relative pointing techniques [16] to support selection. We explore a different interaction style; the user does not raycast but aligns the image they see of their own index finger (pointer) overlaid on top of graphical elements (e.g., button, sliders) in a webpage. This is different from most research, which has focused on distant displays. Thanks to webcams, a low-cost 2D and 3D device to support user interaction, mid-air interactions, and free-hand gestures can be used for pointing, using image-based hand tracking [20]. Modern machine learning techniques allow for even more precise and reliable pointing and more elaborate gesture interactions [21]. Most research in this area demonstrates how image-based hand tracking with a webcam can be used as 3D input, e.g., that of Chunduru et al. [22], who also used MediaPipe's hand tracking to enable 3D interaction with a globe.

Van de Camp et al. [23] proposed a taxonomy of gestures to perform a mid-air 'click', including dwell push/pull, air tap, and grab. Dwell time [24] is a specific modal interaction for selection that requires the user to stay still in a zone for a period of time. Yoo et al. found that dwell time for mid-air pointing was more accurate than push and pull gestures in the context of public display interaction [25]. In a recent study, Dube et al. [26] also identified that push, tap, dwell, and pinch are the most common interactions for selection. Gupta et al. [27] proposed a new selection paradigm called " summon and select " to resolve the Midas touch effect—i.e., triggering unintended interactions while going over targets—as well as ambiguity in gesture to trigger actions and arm fatigue due to continuous gestures. Their study demonstrated that the technique reduces the effort needed to select targets and is faster than point-and-select methods. However, our approach is to allow for continuous movement so the user can leverage direct point-and-click interactions in the intended location in the interface.

To our knowledge, no previous study has focused on the design and evaluation of mid-air gestures to support point-and-click interaction in the browser using the user's overlaid video feed for presentation purposes. Specifically, our design space focuses on dwell time, key-controlled modal interaction, and pinch, as previously identified in mid-air
interaction design studies [23,26], but adapted to the specific case of the user interacting with their overlay in the browser.

## 2.3. Presentation and Flow Interruption

Flow, as defined by Csikszentmihalyi [28], is a psychological state characterized by complete absorption and concentration in an activity, where individuals are intrinsically motivated and fully engaged in what they are doing. In this state, temporal concerns like time and ego-self are typically disregarded, and individuals experience a profound sense of fulfilment and skill, often described as being “in the zone”. When applied to oral presentations, Csikszentmihalyi’s concept of flow can significantly enhance the presenter’s experience and audience engagement. In flow, presenters exude confidence, maintain intense focus on their content, and adapt to the audience’s reactions effortlessly, resulting in a more compelling and impactful presentation. Interruptions [29], particularly in the context of computer-based presentations, can significantly disrupt the flow of an oral presentation and create challenges for the presenter. When using technology for presentations, interruptions can come in various forms, such as unexpected technical issues. These interruptions can be particularly jarring because they disrupt both the presenter’s and the audience’s immersive flow [5]. This phenomenon has been observed with the development of novel interactive presentation technology based on gesture recognition. Gestures play a significant role in oral presentations, improving both oral quality and audience engagement [30]. Nevertheless, failures in gesture recognition can hinder oral presentations by introducing uncontrolled task switching [31], disruptions [32], issues with memorization [33], and limitations in decision-making [34].

Taking these considerations into account, our system aims to provide seamless gesture recognition, enhancing the presentation of online web content. In doing so, we contribute to expanding our understanding and the design of innovative interactive content presentation tools, striking a balance between presenter efficiency and engaging content.

# 3. Interaction Design

We designed WebPresenter with the intention to turn any webpage into support material for presentations. Our main aim was to design hand gesture mapping to mouse actions, such that the presenter can easily interact with web content as their video stream is overlaid on that content. The gestures should be natural and easy to perform for the presenter, but should be clearly visible and seem natural to the audience. Our design also addresses focus and distraction issues that arise with this type of interface. We introduce two novel features to address this: a transition from a presenter-in-side-panel view to a full presenter overlay over the webpage, and a fade-in-fade-out transition of the presenter when they point at an element of the webpage.

## 3.1. Mid-Air Gesture Mapping to Mouse Actions

Pointing is a straightforward mapping from the position of the tip of the index finger to the mouse pointer. This interaction affords smooth, continuous movements similar to pointing with a mouse. Mouse clicks are the main challenge of our design. We explore three main paradigms: dwell time + air tap, modal keystroke, and pinch. We also designed specific gestures for scrolling, panning, and zooming that build on metaphors that echo typical 2D touch surface interactions.

**Pointing:** Pointing is the primary gesture used in most presentations [35,36]. In WebPresenter, pointing is supported by tracking the position and movements of the index finger. Mouseover in the browser is a direct consequence of hovering the tip of the index finger over a graphical target, and it triggers the hover box.
**Dwell time and air tap:** Triggering actions such as pushing buttons, clicking links, or moving a slider are achieved in two steps. The first step consists of locking the target HTML element under the index finger (Figure 1b). A green animated disc appears under the index finger of the user. If the user stays in place, the disc is fully filled after one second and the HTML element is locked. This dwell time length was decided as a trade-off between the intention of target selection and the mitigation of unintended clicks on the page. A leader attached to the finger is displayed as visual feedback for the presenter to indicate that they are leaving the target. To trigger the action, e.g., push a button or follow the link, the user pushes towards the camera in depth. The disc grows in size when the user approaches the camera, and after a distance threshold is reached, a click is triggered (Figure 1c).

<figure><img src="image_4.png" /><figcaption>**Figure 1.** (a) Webcam overlay of the presenter in the webpage. (b) The user points at and remains over a hyperlink for one second. Note that when the user points at a link—or any other object—the focus is on the pointer and the presenter fades into the webpage. (c) The target item is locked and the disc turns green. Then, the user can push towards the screen to follow the link.</figcaption></figure>

**Modal key-stroke:** In this mode, triggering events builds on modal interaction. The user points at HTML elements and presses the keyboard spacebar to trigger a left mouse click action (Figure 2a). Releasing the spacebar simulates a left mouse release action. This approach is based on a bi-manual interaction; the dominant hand is used to point while the non-dominant hand is used to trigger click actions. From a design point of view, it saves the time and potential arm fatigue, which is intrinsic to the dwell-time approach. However, it requires an external device to trigger the clicks.

<figure><img src="image_5.png" /><figcaption>**Figure 2.** (a) Modal key-stroke combines pointing with the index and clicking with a key press. (b) A pinch interaction to click web elements.</figcaption></figure>

**Pinch:** An alternative way for users to trigger actions by is pinching over a webpage element with their index finger and thumb (Figure 2b). The clicking action is triggered when a minimal threshold distance between the tips of the index finger and the thumb is reached.

**Scrolling, Panning, and Zooming:** Scrolling, panning, and zooming are supported by a different set of gestures, which do not require a pointing action. We based our interactions on gestural consensus for mid-air interactions, recently organized in a gesture taxonomy
by Hosseini et al. [37]. Vertical and horizontal scrolling—and panning in pages such as Google Maps—are achieved through a uni-manual interaction that consists of closing the fist (Figure 3a) and moving it up or down (Figure 3b) or left and right, respectively. Zooming uses a bi-manual interaction; the user closes both fists and moves them towards or away from each other, to, respectively, zoom in and out the content in the webpage. Note that, as a side effect in pinch mode, panning can also be achieved by pinching the thumb and the index finger and moving the pinch gestures in the place, which translates as a click and drag interaction in the browser.

<figure><img src="image_6.png" /><figcaption>**Figure 3.** Scrolling is performed by closing the fist (a) and dragging it up or down (b).</figcaption></figure>

## 3.2. *Minimizing Distraction*

Recent research in webcam-based presentations has identified the potential for distraction [2]. We designed two features that aim to minimize distraction and focus on two principles: (1) “focus on the hand” and (2) “presenter in side panel” to “presenter overlay” transition.

**Focus on the hand:** Thanks to the MediaPipe API used in this project, it is straightforward to isolate the image mask that contains the hand of the presenter. When the presenter is not pointing, they remain fully opaque in the frame (Figure 4a). When they point at an element on the webpage and their hands move further away from the rest of their body, an opacity filter is applied in which only the pointing hand remains fully opaque. The presenter gradually fades out (Figure 5b) and occupies less room on the screen to minimize distraction. When the action is performed, the presenter returns to fully opaque, which increases their presence in the presentation again.

<figure><img src="image_7.png" /><figcaption>**Figure 4.** (a) When the user is not pointing at an element, they appear fully opaque. (b) When the user points at an element, they progressively fade into the background to draw attention to where they are pointing.</figcaption></figure>

**Presenter-in-side-panel mode to presenter-overlay mode transition:** To allow the presenter to adjust their presence in the presentation, we provide an option to transition between a picture-in-picture mode, with the presenter's video feed constrained to a side panel, as traditionally found in online web conference software, to the full overlay of the presenter in the page. The transition between these two modes is supported by a keyboard shortcut (ALT + S). When invoked while the presenter is shown in the side panel (picture-in-
picture), the background of the presenter's video fades away, leaving only the segmented video of the presenter's head, torso, and hands, while simultaneously, a translation and scale transition is applied such that the presenter is centered and zoomed to fit the screen.

<figure><img src="image_8.png" /><figcaption>**Figure 5.** A progressive transition (b) between presenter-in-side-panel mode (a) and presenter-overlay mode (c) is performed when the user presses the ALT+S key combination, to emphasize or de-emphasize moments in their presentations.</figcaption></figure>

# 4. Implementation

Our goal was to incorporate free-hand gestures as the primary user interface method in the browser. Additionally, we aimed to seamlessly integrate our system with online content accessible through a web browser, enhancing the user experience. User movement and gestures are captured by a standard webcam. A key priority throughout our development process was to reduce technical constraints, ensuring that our system could operate effectively with minimal setup and configuration. Ultimately, our overarching goal was to maximize the utilization of existing web-based content while minimizing installation requirements and technical barriers to ensure the functional and user-friendly nature of our system.

## 4.1. Overview and Web Browser Integration

While previous studies have explored hardware-dedicated solutions to integrate gestures to control browser navigation [38], our approach aims for easy use with common webcam available on laptops and desktops. For easy integration with as much content as possible, we decided to use a web browser plugin for all Chromium-based browsers, such as Google Chrome and Microsoft Edge. The plugin works as follows: After loading a webpage, the browser launches the plugin. Our plugin augments the current webpage with camera management, content visualization, image segmentation management, gesture recognition, and interaction mapping. We utilized the MediaPipe API (https://developers.google.com/mediapipe (accessed on 17 January 2025)) to manage webcam content and perform image segmentation for detecting user motion and hand positions. While MediaPipe also provides gesture recognition features, we chose to develop our own for more flexibility and fine-tuning of the interaction. Finally, our plugin embeds a simple mouse event mapping, where gesture recognition triggers mouse-based events, making our system highly usable with a wide variety of web content.

## 4.2. Identified System Success and Limitations

While our system has been successfully tested with a large variety of web content, we have identified some limitations and potential incompatibilities. In this section, we provide details of the limitations we have discovered with our plugin technology and discuss possible solutions to overcome them.

Managing video streams with image segmentation, hand movement detection, and image composition (displaying the user within the canvas) demands a considerable amount of computational power. Although our configuration (a standard laptop, i5 core, 32Mo Ram, 1050 graphic card) consistently maintains a frame rate above 15 images per second,
we had to reduce the webcam resolution to 480 × 320 pixels. Higher camera resolutions are feasible, but they may lead to frame rates dropping below 10 images per second, potentially affecting the overall system's interactivity. However, we anticipate that advancements in technology will address this limitation in the near future. In terms of hand tracking, a crucial aspect of our system, we encountered several challenges. Despite our extensive efforts to optimize gesture recognition, the system still occasionally misinterprets gestures. We will delve into this issue further in our discussion section. Nevertheless, our system remains interactive, and gesture recognition, even with some limitations, remains useful for our intended content presentation. Future advancements in gesture recognition may help alleviate this limitation.

From a timing perspective, our plugin takes approximately 2 s to launch, which is reasonable for our use case. However, as a potential limitation, when transitioning from one web content to another, our plugin requires a reload, again taking 2 s. This can disrupt the presentation flow but remains acceptable within our context. As a minor limitation, our plugin primarily operates in Chrome due to our constrained development timeline. Expanding our plugin's compatibility to various web browsers may necessitate additional efforts.

Finally, we implemented a simple mapping from our gestures to mouse events, enabling actions such as page scrolling, zooming, clicking, and selection. More work is needed to maximize compatibility with all web content and to develop possible additional mappings.

# 5. Use Cases

WebPresenter is designed to put the presenter into a webpage and enable interaction directly with hand gestures and different visual elements. Therefore, the use cases and scenarios are vast.

## 5.1. *Programming Lecture*

Education institutions often rely on the flipped classroom model, in which students access learning material before a contact lecture time. Very often, this content involves a lecturer teaching programming concepts in a video. In this use case, WebPresenter is used to teach a functional reactive programming technique in JavaScript as part of a fairly advanced Programming Paradigms unit. The code implements a streaming card counting algorithm (Figure 6a). The code is demonstrated and explored in a web-based integrated development environment (IDE) (https://stackblitz.com (accessed on 17 January 2025)) with a variety of interactions, usually available via mouse or touch interaction, which are replaced with video gestures.

<figure><img src="image_9.png" /><figcaption>**Figure 6.** Three use cases for WebPresenter: (a) programming lecture with web-based IDE, (b) Miro research presentation, (c) casual weather forecast presentation.</figcaption></figure>

The lecturer starts by pointing to the right-hand side of the screen where the app is running in real time. He reaches out to the "Hit Me" button and pushes it with his finger to start the animated card counting algorithm. The presenter looks at the visualization of the cards being drawn to capture the attention of the audience. Next, the lecturer draws
the students' attention to type annotations in the code. Hovering a finger over the code symbol triggers the code editor's usual mouse hover behavior of displaying an info box showing the possible values of the given type—in this instance, the possible card suits. Later, the lecturer explores more complex types in the code, drawing the viewers' attention to different parts of the type annotations to explain the structure of the code. The lecturer switches between the code IDE webpage and illustrations in the course notes, using the scroll gesture to draw the students' attentions to the relevant sections in the notes.

This eight-minute video was recorded and edited together in about an hour and disseminated to approximately 600 students several days before the next class, giving them time to review it. Feedback from students was more positive than usual, with a generally positive reaction to the novel use of technology.

## 5.2. Portfolio Presentation

Researchers often use online portfolio platforms such as Miro, which lets them organize content, relate information, and create a flow and narrative to articulate their thoughts. This tool is particularly useful for students or postdocs driving research and used as presentation material instead of traditional slides. The presenter usually zooms in and out and pans to navigate the space, focusing on items such as paper PDFs, illustrations, videos, post-its, and text content. In this scenario, a PhD student has a meeting with his supervisors to report on their progress in making the state of the art and analyzing gestures for presentations. The PhD student starts WebPresenter in picture-in-picture mode. Then, the student transitions to overlay mode. The PhD student uses pan and zoom to present multiple types of gestures involved in presentations. The student finally navigates to a video and presses the play button to present the content (Figure 6b).

## 5.3. Casual Broadcaster

WebPresenter can be used as a presentation tool for casual broadcasters. Content creators use platforms like YouTube, Twitch, or TikTok to broadcast, very often in casual ways, their content. In this use case, we demonstrate how WebPresenter can be used as a powerful tool to create even more engaging experiences for content creators, for example, a casual weather forecast report (Figure 6c); a broadcaster navigates to a national weather website and uses the visuals and temperature curves in their interactive background to create a similar, but augmented, experience of a weather forecast report. In a related context, WebPresenter can be used on top of one or many news websites too, and be used as a minimal editorial report presentation support. WebPresenter can also be used by influencers to present their content, e.g., a broadcaster can use their Instagram page to showcase their cave diving activities and share their content with the community.

# 6. Study

WebPresenter's design: We invited participants to perform presentations on different types of web content using WebPresenter. We conducted qualitative observations during test sessions with five different participants to gather insights about the potential use cases, user-friendliness, and limitations of WebPresenter. We used three different types of web content, dealing with the same topic (population growth), to evaluate WebPresenter in different scenarios.

## 6.1. Participants

Participants were five volunteers (two women) who frequently give online presentations in their work (two PhD students, one postdoctoral researcher, one educational engineer, and one theoretical knowledge instructor), referred to as P1–5. Participants did not receive compensation. Before their inclusion in the study, participants signed an
informed consent form. The protocol received approval from the Ethics Committee of
Toulouse, France (2023-688), and was conducted according to their guidelines.

6.2. *Training*

We introduced the participants to WebPresenter by describing the functionalities through a guided training session that happened as follows. First, participants completed a tutorial where they had to click on the different buttons which appeared sequentially at nine different locations inside a 3-by-3 grid on the screen. They completed this small practice training session for the three point-and-click modalities that can be used in WebPresenter: air tap, pinch, and modal keystroke. Second, participants were told that WebPresenter is designed to present any content on the web. They were given three scenarios—a Wikipedia (https://www.wikipedia.org/ (accessed on 17 January 2025)) page, a slide presentation on Canva (https://www.canva.com/ (accessed on 17 January 2025)), and an interactive graph on an “Ourworldindata” (https://ourworldindata.org/ (accessed on 17 January 2025))—to present. Participants had time to read the three pages to familiarize themselves with the content. To facilitate the learning of the content, we chose three pages dealing with the same topic, that is, the growth of the world population. While learning the content, WebPresenter was active so that participants could become familiar with the interactions that they would like to use during their presentations. It was also an opportunity for them to test scrolling and zooming actions. Before starting the task, participants had the opportunity to ask any questions about WebPresenter. They were also told that they were allowed to make any comment about their experience with WebPresenter during the task. In summary, the training phase varied between participants, and we ensured that they were adequately prepared before proceeding. This was achieved through two checks: a visual assessment by the experiment facilitator to confirm sufficient mastery of the tools and the participant's own acknowledgment that they felt confident in using each interaction modality.

6.3. *Task*

For each of the three webpages, participants were asked to present the content of each
page in 3 min using WebPresenter. Each presentation was conducted three times, once
using each click modality: air tap, pinch, and modal keystroke. Participants could use the
scroll and zoom on all their presentations.

6.4. *Procedure and Set-Up*

Overall, each study session lasted about 2 h per participant. All participants started by signing the consent form and completing the training session. Then, for each click modality, participants gave 3-min presentations for the three given webpages (3 × 3 design). Note that, to account for fatigue or training effects, but also for systematic effects due to the order, the order in which participants presented the different webpages was counterbalanced. Similarly, the order in which the different click modalities were used to present each of the content was also counterbalanced. After each presentation, participants completed the NASA-TLX questionnaire online (https://nasa-tlxhttps://nasa-tlx://nasa-tlx.firebaseapp.com/tlx (accessed on 17 January 2025)) to evaluate the Task Load Index as well as their subjective level of mental demands, physical demands, temporal demands, frustration, effort, and performance. At the end of the session, after all the presentations, participants were asked to rank their preferences in terms of interactions in a questionnaire. They were also asked to report any comments that they might have about WebPresenter and their user experience. Moreover, after completing the questionnaire, the experimenter discussed the reason for their answers with the participant. The session lasted approximately two hours including the training, the presentations, and the questionnaires.
Participants were comfortably sitting at approximately 80 cm from the 28" screen. A 1080 p webcam was positioned on top of the screen. All webpages were presented in full-screen mode. The distance of the participant from the screen and the screen size were selected to maximize the participant's comfort. Indeed, we noticed during pre-tests that a smaller screen or shorter distance from the screen increases the size of the presenter in the frame, leading to uncomfortable and unnatural arm distortions to perform interactions. During the task, the participants could interact with the web content using the following interactions that they generally perform using their mouse: clicking, scrolling, and zooming in or out.

## 6.5. Self-Reports by the Subjects

On a scale from 1 ("I would not like it at all") to 10 ("I would love it"), on average, the participants would most like to use WebPresenter for future presentations using the modal keypress (M = 5.6; Max = 8; Min = 3), a bit less with pinch (M = 5; Max = 8; Min = 1) and not at all with air tap (M = 2.6; Max = 7; Min = 1). However, we observed discrepancies between participants. While P1 said "Scrolling, air tap, and keystroke were intuitive, but not the pinch. I felt more precise using the air tap as I could sense better what I was pointing at.", P2 reported that "For clicking, the pinch is the most intuitive and the least effortful. Otherwise, the air tap was physically demanding to keep the arms high, while the modal keystroke was mentally demanding to coordinate for hands. However, this is at the cost of higher error rate and bigger lack of precision using the pinch". The modal keystroke was ranked first by all the participants for the interactive graph presentation, while it was ranked first by a majority of participants (n = 3) for the slide presentation, and it fought for first place with the pinch for the Wikipedia presentation. P3 and P5, who consistently ranked the keystroke first, argued that it was more reliable; otherwise, they found the pinch more intuitive and natural, but they wished that it become more efficient after improvement. The air tap was always ranked last by the majority of participants. In general, participants needed more practice for the air tap and they judged it less time-efficient due to the dwell time. Although the air tap was the one that looked natural to perform spontaneously to some participants, notably P3 (they suggested this gesture for clicking even before I had explained the modalities), it is possible that the way it is currently implemented is not optimal. P5 and P4 said that it was difficult to approach the hand from the screen with the right angle to trigger the click. They found it counter-intuitive as they would have naturally aimed towards the position of the information at which they pointed on the screen.

Overall, the participants found the scrolling and zooming gestures more natural, intuitive, and easy to learn than all the modalities that we selected for clicking.

The overall Task Load Index of using WebPresenter had an average weighted rating (https://measuringu.com/nasa-tlx/ (accessed on 17 January 2025)) of 50.83 ± 14.95 across all contents and clicking modalities. Compared to the typical keyboard-and-mouse combination, the load index associated with the use of WebPresenter was considered somewhat high (between 30 and 49) to high (between 50 and 79) compared to previously established thresholds [39]. Differences between clicking modalities or contents can be visualized in Figure 7 for exploratory purpose. Among the six dimensions of the NASA-TLX, we observed that physical demand was globally higher for the modal keystroke and air tap than for the pinch. To explain this effect, P4 reported the need to coordinate for the modal keystroke, and P5 verbalized strong muscle tension to keep the arm high and the finger still. However, the modal keystroke generated less frustration than the other modalities because of the higher sense of control that it provides. While these results have been known for decades [40], and mid-air gestures are known to induce the Heisenberg effect on point
selection with free-hand selection [15], it is still notable that the users were able to perform relatively uninterrupted presentations of good quality in all modes of interaction.

<figure><img src="image_10.png" /><figcaption>**Figure 7.** Average weighted rating of load index on the NASA-TLX for each clicking modality and web content (means and CI95).</figcaption></figure>

# 7. Observations and Discussion

While participants were conducting their presentations for the different contents using WebPresenter (and the three point-and-click interaction paradigms), we observed the fails (i.e., attempts to perform an interaction that did not work as expected) and the unintended actions (i.e., interactions that were triggered involuntarily) to understand the circumstances of their occurrence. We also paid close attention to the position of the projected image of the presenter onto the screen and the possible interruptions in the flow of the presentation. Despite the limitations of the current version of WebPresenter, all participants were able to complete the presentation tasks. It also worth reporting that P3 and P4 demonstrated a proficient use of WebPresenter to deliver a quality presentation, leveraging the potential to create, according to us, more engaging presentations.

**Position of the presenter and occlusion:** All the participants, except P2, behaved as if their image was not on the content. The fading in allowed them to continue pointing at information, even if occluded by their image. They neither moved to change their position on the screen nor scrolled to move the information outside of their silhouette area. The participants globally appreciated the “fading in” effect. P1 was the one who appreciated it the most, as they reported being so immersed in the task that they did not notice their presence anymore. However, P3 would have liked the hands in transparency too because they were hiding information, and P4 complained about an uncomfortable permanent head tilt to not hide certain pieces of information. Similarly, P5 suggested a different position of the presenter in the frame (e.g., on the side, as there is generally less information there than in the center). On the contrary, P2 was highly concerned by their image hiding information. Especially while presenting slides, where scrolling was not available, P2 moved their chair to not appear in front of the presented information. This information is very useful for future work, and opens design opportunities to optimize the position of the presenter in the webpage, e.g., using space-filling techniques to place the user in blank space. While these observations are useful for future work, they go beyond the scope of this paper.

**Fails:** P1: "Most of the time, I was able to perform the actions that I intended to". We observed relatively few fails; P3 and P4 in particular performed presentations of very good quality. The air tap, however, required more training than the other modalities and generated notable fails for slide presentations for P3. P3 and P4 experienced a few failures in the presentation of the interactive graph using the air tap, resulting from the difficulty of keeping the finger still for the complete dwell-time period when the clicking area was small, what P5 described as muscle tension to keep the lifted arm immobile. However, for P2, the keystroke generated many fails, particularly while presenting the interactive graph, as this page required finer and more precise clicks. This could be due to the fact that the keystroke, in contrast to air tap and pinch, does not highlight the information that
is pointed at. Visual feedback seems therefore important. The pinch, on the other hand, generated very few fails (but more involuntary actions; see below), except for P5, who struggled to click on the bottom part of the screen using the pinch. These observations are in line with previous observations [5], as fails create interruptions in the presentation flow. We believe that our interactions, however, induce more forgiving fails (e.g., undesired highlights or small pans) that do not completely deteriorate the presentation.

**Unintended actions:** Three participants faced unintended triggering of actions due to their hand movements. While talking, they tended to bring back both hands in the fist position, leading to unintended scrolls and zooms in or out. Touching the hair (P1), the face (P2), or putting the hand under the chin (P5) also triggered multiple inappropriate zooming and scrolling actions. P1, P3, and P5 performed uncontrolled clicks in the pinch modality, in particular, during the slide presentation where the area sensitive to clicks was large. P4, on the contrary, achieved an excellent level of interaction with WebPresenter. P4 never encountered unintentional actions except for a few clicks on the wrong information (on the interactive graph, P4 clicked by mistake on spatially close check boxes instead of the targeted ones). P2 triggered almost no involuntary actions at all, but their strategy was extreme: they thoughtfully tried to keep their hands below the table to avoid any unintended action. The main drawback of this strategy is the absence of pointing at relevant information, which negatively affected the quality of the presentation. P2 themselves reported: "I would have liked using pointing during the presentations, as I would usually do with my mouse cursor or a laser pointer, but I was afraid of triggering undesirable or uncontrollable actions". That was not the case for the other participants who pointed as they would normally. While we did not anticipate such interactions in our design process, we accept that these observations are valuable and should be included in a follow-up design process to include and interpret, e.g., on-body interactions [41].

**Flow of the presentation:** Modal keystroke was the modality that globally preserved the flow of the presentation the best. The flow was interrupted numerous times in the air tap modality for all the participants, due to systematic short pauses (more or less 1 to 2 s) during the dwell time. More work is needed to find the right timing for the dwell time that strikes the right balance between fast efficient selection to minimize Gorilla's arm effect and unintended triggered actions on elements of the webpage. P4 said "I tried to prepare the clicks in advance, before I actually needed to click, to compensate for the waiting time". The pinch modality was overall one that induced the highest number of interruptions in the flow across participants. This was mostly due to unintended clicks and scrolls causing longer interruptions (up to a few tens of seconds) as the participant ended up lost in front of unexpected information. Notably, we also observed that P4 and P5 systematically stopped talking while concentrating on executing the pinch properly, showing an additional mental load.

**Co-adaptation and catachresis:** During this preliminary study, the experimenter also noticed strategies that participants used to compensate for some of the current limitations of WebPresenter. One of the adaptations concerned P2 and P4, who both kept an immobile fist in the frame to mimic an interaction and trigger the transparency of their body while presenting otherwise occluded information. Another adaptation concerned pointing and the finger hovering over the screen while using the air tap modality. We believe that the following adaptation aimed to decrease a problem described by P3, who reported 11 "is quite pressuring to see the loading wheel while pointing at information, I was afraid that it would trigger unwanted clicks". To solve this issue, P4 and P5 chose unusual strategies to point at information in the air tap condition. P4 pointed using two fingers (the index and the middle finger), while P5 pointed with their entire hand or their middle finger to prevent the dwell time from starting and pressuring them. Finally, all participants pointed with
their finger over the screen, as they would use "mouse over", to highlight information or make annotations appear (e.g., Wikipedia). Even though this is an emerging interaction that we did not plan for, it nicely enhanced the good quality of the presentations.

## 8. Limitations and Perspectives

Being able to control when to enable or disable the ability to interact with the content was reported as a limitation by three participants; P3, P4, and P5 would have preferred to control when they could or not interact with the content (e.g., using a button in the hand that activates the possibility of using the interactions or moving the chair at a farther distance from the camera to disable the interactions—this proxemics [42] interaction will be investigated in the next iteration of WebPresenter). Future development will consider the improvement regarding the position of the presenter in the frame. Specifically, different positions (e.g., on the side rather than in the center) and transparency mechanisms will be tested. In addition, four participants out of five mentioned that they would have appreciated an additional gesture to switch slides, similar to a swipe gesture on touch surfaces. Similarly, after zooming in, two participants reported that the zoom was not natural because it is not centered on the information that is in the center of the hands, and because the current version does not provide the possibility to drag the page. Other zoom types will be investigated to improve the interaction to zoom where the presenter intends to. Finally, in its current form, WebPresenter is able to send mouse events that are mapped to hand gestures on most standard HTML pages. However, when a webpage runs a specific javascript application that has its own state manager to handle mouse interactions, it is necessary to adapt the code within WebPresenter to support such interactions. For example, in Miro, or in Canva's slides, we had to write custom code to propagate pan, zoom, and next-slide events.

Our investigation aligns with the growing emphasis on remote presentation technologies. Key trends in this domain include the integration of AI into remote collaboration systems [43] and the use of natural language processing to enhance collaborative workflows [44]. Furthermore, recent studies have proposed design guidelines aiming to optimize remote collaboration experiences [45]. Novel interactive techniques, such as elasticity-inspired interfaces [46] and visualization-driven coordination tools [47], have expanded the possibilities for interaction design. The integration of augmented reality technologies has further opened pathways for immersive and engaging collaborative experiences [48]. Collectively, these advancements highlight the transformative potential of innovative tools in bridging physical and virtual spaces for seamless remote collaboration and presentations.

## 9. Conclusions

In this paper, we introduced WebPresenter, a webcam-based presentation tool designed to facilitate interactive presentations using web content as the primary material. Our work explored three key free-hand interaction techniques—dwell time, modal keystroke, and pinch gestures—for engaging with webpage elements. Additionally, we proposed two innovative focus techniques, focus on the hand and presenter-in-side-panel mode to presenter-overlay mode transition, aiming to minimize distractions during presentations. Our evaluation highlighted the effectiveness of these techniques, particularly for selection tasks, with the modal keystroke being the most preferred among the methods tested. Initial findings suggest that WebPresenter supports seamless presentations with minimal interruptions to flow. While the results are promising, they would benefit from further validation through larger participant samples and broader task scenarios to assess generalizability across diverse interaction techniques. Despite the inherent challenges of designing robust
free-hand interaction methods, WebPresenter already demonstrates substantial value. For instance, it has been successfully utilized in an educational setting to deliver a lecture to a class of 500 students. By enabling a presenter-overlay presentation style, WebPresenter offers a flexible and accessible tool that can enhance presentation experiences for a wide audience. We believe it represents a significant step toward enriching interaction possibilities for presenters across various domains.

**Author Contributions:** Conceptualization, all authors; methodology, all authors; software, G.T.; validation, A.S.; investigation, all authors; writing—original draft preparation, M.C. and C.H.; writing—review and editing, all authors. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was partially financed by the European Union's Horizon 2020 within the framework SESAR 2020 research and innovation program under grant agreement N. 101114838, Trustworthy intelligent system for remote digital tower, TRUSTY and N. 101114765, Controller adaptive Digital Assistant, CODA.

**Institutional Review Board Statement:** The protocol received approval from the Ethics Committee of Toulouse, France (2023-688) and was conducted according to these guidelines.

**Informed Consent Statement:** Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Source code available here: https://github.com/GuillaumeTruong/Gesture_Ext.

**Conflicts of Interest:** The authors declare no conflicts of interest.

## References

1. Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.L.; Grundmann, M. Mediapipe hands: On-device real-time hand tracking. *arXiv* 2020, arXiv:2006.10214.

2. Hall, B.D.; Bartram, L.; Brehmer, M. Augmented Chironomia for Presenting Data to Remote Audiences. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22), Bend, OR, USA, 29 October–2 November 2022. [CrossRef]

3. Davis, J.U.; Asente, P.; Yang, X.D. Multimodal Direct Manipulation in Video Conferencing: Challenges and Opportunities. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS '23), Pittsburgh, PA, USA, 10–14 July 2023; pp. 1174–1193. [CrossRef]

4. Liao, J.; Karim, A.; Jadon, S.S.; Kazi, R.H.; Suzuki, R. RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22), Bend, OR, USA, 29 October–2 November 2022. [CrossRef]

5. Fourney, A.; Terry, M.; Mann, R. Gesturing in the wild: Understanding the effects and implications of gesture-based interaction for dynamic presentations. In Proceedings of the 24th BCS Interaction Specialist Group Conference (BCS '10), Dundee, UK, 6–10 September 2010; pp. 230–240.

6. Lee, B.; Kazi, R.H.; Smith, G. SketchStory: Telling more engaging stories with data through freeform sketching. *IEEE Trans. Vis. Comput. Graph.* **2013**, *19*, 2416–2425. [PubMed]

7. Brosz, J.; Nacenta, M.A.; Pusch, R.; Carpendale, S.; Hurter, C. Transmogrification: Causal Manipulation of Visualizations. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), St. Andrews, UK, 8–11 October 2013; pp. 97–106. [CrossRef]

8. Perlin, K.; He, Z.; Rosenberg, K. Chalktalk: A Visualization and Communication Language-As a Tool in the Domain of Computer Science Education. *arXiv* 2018, arXiv:1809.07166.

9. Liu, X.B.; Kirilyuk, V.; Yuan, X.; Chi, P.; Olwal, A.; Chen, X.A.; Du, R. Experiencing Augmented Communication with Real-time Visuals using Large Language Models in Visual Captions. In Proceedings of the Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), San Francisco, CA, USA, 29 October–1 November 2023.

10. Saquib, N.; Kazi, R.H.; Wei, L.Y.; Li, W. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), Glasgow, UK, 4–9 May 2019; pp. 1–12. [CrossRef]

11. Riche, N.H.; Hurter, C.; Diakopoulos, N.; Carpendale, S. *Data-Driven Storytelling*; CRC Press: Boca Raton, FL, USA, 2018.

12. Van Dam, A. Post-WIMP user interfaces. *Commun. ACM* **1997**, *40*, 63–67. [CrossRef]
13. Mackinlay, J.; Card, S.K.; Robertson, G.G. A semantic analysis of the design space of input devices. *Hum.-Comput. Interact.* **1990**, *5*, 145–190. [CrossRef]

14. Jacob, R.J.K.; Sibert, L.E.; McFarlane, D.C.; Mullen, M.P. Integrality and Separability of Input Devices. *ACM Trans. Comput.-Hum. Interact.* **1994**, *1*, 3–26. [CrossRef]

15. Wolf, D.; Gugenheimer, J.; Combosch, M.; Rukzio, E. Understanding the Heisenberg Effect of Spatial Interaction: A Selection Induced Error for Spatially Tracked Input Devices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Honolulu, HI, USA, 25–30 April 2020; pp. 1–10. [CrossRef]

16. Vogel, D.; Balakrishnan, R. Distant freehand pointing and clicking on very large, high resolution displays. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, Seattle, WA, USA, 23–26 October 2005; pp. 33–42.

17. Bragdon, A.; DeLine, R.; Hinckley, K.; Morris, M.R. Code space: Touch + air gesture hybrid interactions for supporting developer meetings. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11), New York, NY, USA, 13–16 November 2011; pp. 212–221. [CrossRef]

18. Vatavu, R.D. Point & Click Mediated Interactions for Large Home Entertainment Displays. *Multimed. Tools Appl.* **2012**, *59*, 113–128. [CrossRef]

19. Pfeuffer, K.; Mayer, B.; Mardanbegi, D.; Gellersen, H. Gaze + pinch interaction in virtual reality. In Proceedings of the 5th Symposium on Spatial User Interaction (SUI ’17), Brighton, UK, 16–17 October 2017; pp. 99–108. [CrossRef]

20. Yeo, H.S.; Lee, B.G.; Lim, H. Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware. *Multimed. Tools Appl.* **2015**, *74*, 2687–2715. [CrossRef]

21. Islam, M.M.; Islam, M.R.; Islam, M.S. An efficient human computer interaction through hand gesture using deep convolutional neural network. *SN Comput. Sci.* **2020**, *1*, 211. [CrossRef]

22. Chunduru, V.; Roy, M.; Chittawadigi, R.G. Hand tracking in 3d space using mediapipe and npn method for intuitive control of virtual globe. In Proceedings of the 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC), Bengaluru, India, 30 September–2 October 2021; pp. 1–6.

23. van de Camp, F.; Schick, A.; Stiefelhagen, R. How to click in mid-air. In Proceedings of the Distributed, Ambient, and Pervasive Interactions: First International Conference, DAPI 2013, Held as Part of HCI International 2013, Las Vegas, NV, USA, 21–26 July 2013; Proceedings 1; Springer: Berlin/Heidelberg, Germany, 2013; pp. 78–86.

24. Hansen, J.P.; Johansen, A.S.; Hansen, D.W.; Ito, K.; Mashino, S. Command without a click: Dwell time typing by mouse and gaze selections. In Proceedings of the 10th International Conference on Human-Computer Interaction, Crete, Greece, 22–27 June 2003; Volume 3, pp. 121–128.

25. Yoo, S.; Parker, C.; Kay, J.; Tomitsch, M. To dwell or not to dwell: An evaluation of mid-air gestures for large information displays. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction, Parkville, VIC, Australia, 7–10 December 2015; pp. 187–191.

26. Dube, T.J.; Ren, Y.; Limerick, H.; MacKenzie, I.S.; Arif, A.S. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-Air Selection Methods Augmented with Ultrasonic Haptic Feedback. *Proc. ACM Hum.-Comput. Interact.* **2022**, *6*, 207–225. [CrossRef]

27. Gupta, A.; Pietrzak, T.; Yau, C.; Roussel, N.; Balakrishnan, R. Summon and Select: Rapid Interaction with Interface Controls in Mid-Air. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS ’17), Brighton, UK, 17–20 October 2017; pp. 52–61. [CrossRef]

28. Csikszentmihalyi, M. *Flow: The Psychology of Optimal Experience*; Harper & Row: New York, NY, USA, 1990.

29. Hurter, C.; Girouard, A.; Riche, N.; Plaisant, C. Active Progress Bars: Facilitating the Switch to Temporary Activities. In Proceedings of the CHI ’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’11), Vancouver, BC, Canada, 7–12 May 2011; pp. 1963–1968. [CrossRef]

30. Hostetter, A.B.; Alibali, M.W. Visible embodiment: Gestures as simulated action. *Psychon. Bull. Rev.* **2008**, *15*, 495–514. [CrossRef]

31. Czerwinski, M.; Horvitz, E.; Wilhite, S. A diary study of task switching and interruptions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria, 24–29 April 2004; pp. 175–182.

32. Borst, J.P.; Taatgen, N.A.; van Rijn, H. What makes interruptions disruptive? A process-model account of the effects of the problem state bottleneck on task interruption and resumption. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015; pp. 2971–2980.

33. Oulasvirta, A.; Saariluoma, P. Long-term working memory and interrupting messages in human–computer interaction. *Behav. Inf. Technol.* **2004**, *23*, 53–64. [CrossRef]

34. Speier, C.; Valacich, J.; Vessey, I. The effects of task interruption and information presentation on individual decision making. In Proceedings of the International Conference on Information Systems (ICIS), Atlanta, GA, USA, 14–17 December 1997; Volume 18, pp. 21–36. https://dl.acm.org/doi/10.5555/353071.353080.

35. Ju, S.X.; Black, M.J.; Minneman, S.; Kimber, D. Summarization of videotaped presentations: Automatic analysis of motion and gesture. *IEEE Trans. Circuits Syst. Video Technol.* **1998**, *8*, 686–696. [CrossRef]
36. Matsumoto, D.; Frank, M.G.; Hwang, H.S. *Nonverbal Communication: Science and Applications*; Sage Publications: Thousand Oaks, CA, USA, 2012.

37. Hosseini, M.; Ihmels, T.; Chen, Z.; Koelle, M.; Müller, H.; Boll, S. Towards a Consensus Gesture Set: A Survey of Mid-Air Gestures in HCI for Maximized Agreement Across Domains. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), Hamburg, Germany, 23-28 April 2023. [CrossRef]

38. Shi, Y.; Taib, R.; Lichman, S. *GestureCam: A Smart Camera for Gesture Recognition and Gesture-Controlled Web Navigation*. In Proceedings of the 2006 9th International Conference on Control, Automation, Robotics and Vision, Singapore, 5-8 December 2006; pp. 1-6. [CrossRef]

39. Prabaswari, A.D.; Basumerda, C.; Utomo, B.W. The mental workload analysis of staff in study program of private educational organization. *IOP Conf. Ser. Mater. Sci. Eng.* **2019**, *528*, 012018. [CrossRef]

40. Card, S.K.; Newell, A.; Moran, T.P. *The Psychology of Human-Computer Interaction*; L. Erlbaum Associates Inc.: Mahwah, NJ, USA, 1983.

41. Harrison, C.; Tan, D.; Morris, D. *Skininput: Appropriating the body as an input surface*. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, GA, USA, 10-15 April 2010; pp. 453-462.

42. Ballendat, T.; Marquardt, N.; Greenberg, S. *Proxemic Interaction: Designing for a Proximity and Orientation-Aware Environment*. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '10), New York, NY, USA, 3-6 October 2010; pp. 121-130. [CrossRef]

43. Li, H.; Wang, Y.; Qu, H. Where are we so far? Understanding data storytelling tools from the perspective of human-ai collaboration. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 11-16 May 2024; pp. 1-19.

44. Hoque, E.; Islam, M.S. *Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions*. *Comput. Graph. Forum* **2024**, e15266, online version of record. [CrossRef]

45. Shi, Y.Z.; Li, H.; Ruan, L.; Qu, H. Constraint representation towards precise data-driven storytelling. *arXiv* **2024**, arXiv:2410.07535.

46. Cao, Y.; Kazi, R.H.; Wei, L.Y.; Aneja, D.; Xia, H. *Elastica: Adaptive Live Augmented Presentations with Elastic Mappings Across Modalities*. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 11-16 May 2024; pp. 1-19.

47. Femi-Gege, T.P.; Brehmer, M.; Zhao, J. *VisConductor: Affect-Varying Widgets for Animated Data Storytelling in Gesture-Aware Augmented Video Presentation*. *Proc. ACM Hum.-Comput. Interact.* **2024**, *8*, 133-154. [CrossRef]

48. Tong, W.; Shigyo, K.; Yuan, L.P.; Fan, M.; Pong, T.C.; Qu, H.; Xia, M. *VisTellAR: Embedding Data Visualization to Short-form Videos Using Mobile Augmented Reality*. *IEEE Trans. Vis. Comput. Graph.* **2024**, *early access*.

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.