Behind The Game: Implicit Spatio-Temporal Intervention inInter-personal Remote Physical Interactions on Playing Air
HockeyAzumi Maekawa∗

Information Somatics Lab
RCAST, The University of Tokyoazumi@star.rcast.u-tokyo.ac.jp

Hiroto Saito∗

Information Somatics Lab
RCAST, The University of Tokyo
hiroto.saito@star.rcast.u-tokyo.ac.jp

Shunichi Kasahara

Information Somatics Lab
RCAST, The University of Tokyo
Sony Computer Science Laboratories
kasahara@star.rcast.u-tokyo.ac.jp

Narin Okazaki

Information Somatics Lab
The University of Tokyonarin@star.rcast.u-tokyo.ac.jp

Masahiko Inami

Information Somatics Lab
RCAST, The University of Tokyo
inami@star.rcast.u-tokyo.ac.jp

Figure 1: (a) The physical machine allows a remote player to play against a local player. (b) The remote player side. (c) Graphicaluser interface for the remote player.

ABSTRACT
When playing inter-personal sports games remotely, the time lag
between user actions and feedback decreases the user’s performance and sense of agency. While computational assistance can
improve performance, naive intervention independent of the context also compromises the user’s sense of agency. We propose acontext-aware assistance method that retrieves both user performance and sense of agency, and we demonstrate the method using
air hockey (a two-dimensional physical game) as a testbed. Our
system includes a 2D plotter-like machine that controls the striker
on half of the table surface, and a web application interface that enables manipulation of the striker from a remote location. Using oursystem, a remote player can play against a physical opponent fromanywhere through a web browser. We designed the striker control∗ Both authors contributed equally to this research.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
SIGGRAPH ’21 Emerging Technologies , August 9–13, 2021, Virtual Event, USA© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8364-6/21/08. . . $15.00
https://doi.org/10.1145/3450550.3465348

assistance based on the context by computationally predicting thepuck’s trajectory using a real-time captured video image. With thisassistance, the remote player exhibits an improved performance
without compromising their sense of agency, and both players canexperience the excitement of the game.

CCS CONCEPTS• Human-centered computing → Interaction devices; Webbased interaction.

KEYWORDS
Teleoperation; Sense of agency; Human-machine integration
ACM Reference Format:
Azumi Maekawa, Hiroto Saito, Narin Okazaki, Shunichi Kasahara,
and Masahiko Inami. 2021. Behind The Game: Implicit Spatio-Temporal
Intervention in Inter-personal Remote Physical Interactions on Playing
Air Hockey. In Special Interest Group on Computer Graphics and Interactive
Techniques Conference Emerging Technologies (SIGGRAPH ’21 Emerging Technologies ), August 9–13, 2021, Virtual Event, USA. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3450550.3465348

1

INTRODUCTION

Physical sports play an important role in our daily lives, as they
can facilitate connecting with others and improve our health. With

SIGGRAPH ’21 Emerging Technologies , August 9–13, 2021, Virtual Event, USA

the advancement of information technology, various activities havebecome possible beyond the limitations of location. However, it
is challenging to play inter-personal sports remotely. One of the
problems is the time lag between user actions and feedback, whichis caused by systems connecting remote and local areas. The time
lag decreases the reaction speed and operation accuracy of action
controls and induces various errors. Such a situation can also reduce
the remote player’s sense of agency, the subjective experience ofcontrolling one’s own action, and places them in them an unfavorable position relative to the physical player. This unfair condition
reduces both players’ motivation to play the game.
This study explores how well-balanced inter-personal games canbe achieved between a remote player and a physical (local) player.
In general, although the remote player has a disadvantage, theirperformance can be improved with the assistance of computational
intervention. However, a naive intervention that does not rely on
the user’s intention may also compromise the player’s sense ofagency. In particular, in interpersonal games the user’s intentions
can change dynamically depending on the context, which may
include other players’ action, their’s own actions, and changes inthe environment. Hence, it is necessary to consider an intervention
method optimized for the context rather than focusing only on theuser’s performance.
In this paper, we propose an intervention method called contextaware assistance, in which the computer is aware of the context andintervenes in the user’s actions according to it. We demonstrate themethod using air hockey, a two-dimensional physical game, as atestbed. Air hockey is a popular inter-personal game that requires
temporal and spatial physical control. Players need to predict apuck’s trajectory quickly and accurately hit the puck with a striker.
Using the puck’s movement and the user’s behavior, our systemestimates two contexts, (1) defense and (2) offense, and provides
appropriate individual assistance in response.
The system mainly consists of a 2D plotter-like machine that
controls the movements of a striker attached to its end effector (Figure 1 (a)) and a interface in the form of a web application (Figure 1(b) and (c)). While the remote player controls their striker using
the interface, the system detects the puck and striker from thereal-time image captured from a video feed and intervenes in the
striker motion to prevent missed shots caused by the time lag. Our
system, firstly, estimates the context from the puck’s movements
and the player’s behavior. Then, the system dynamically adjusts
the striker’s position to meet the puck through a method designed
for the context. This assists the player to hit the puck as intended,
generating the sense of agency.
In our demonstration, the remote participant can use a standard
web browser and graphical user interface (GUI) to control the striker
and play against a physical opponent in a local.

2 RELATED WORK
2.1 Teleoperation
Teleoperation systems have a long history, beginning with themaster-slave manipulators developed by Goertz [Sheridan 1989].
While teleoperation using master-slave manipulators enables users
to perform operation intuitively, noise and delays in signal transmission can hinder performance. In 1967, Ferrel and Sheridan proposed

Maekawa and Saito, et al.

Figure 2: Overview of the system framework. Communication between remote and local has a time lag of tens to hundreds of milliseconds.

a method called supervisory control, in which a human observercooperates with an automated machine to complete physical tasks
from a remote location [Ferrell and Sheridan 1967]. Supervisory
control is a pioneering method of remotely performing tasks.
Through it, various methods have been used to perform physicaltasks remotely by leveraging automated machine advantages, suchas collision avoidance in mobile robots [Takayama et al. 2011] andcancellation of the relative motion between surgical instruments
and a beating heart [Bebek and Cavusoglu 2007]. Recently, Rakita
et al. proposed a method for intuitively controlling manipulatorsthat have different kinematics from those of the human arm [Rakita
et al. 2017]. In this method, the constraints of the mapping, fromthe position and orientation of the user’s hand to those of themanipulator’s end effector, are relaxed depending on the motion of
the user’s arm. This allows the user to feel that they are controllingthe manipulator directly, while simultaneously attaining smooth
movement and singularity avoidance in the manipulator. While
the main goal of these studies was to improve performance, ourresearch focuses on not only performance improvement but also
the user’s sense of agency as an important factor in achieving asatisfying experience playing sports games remotely.

2.2

Human-machine collaboration in motor
actionIt has been argued that it is important for a system to understandthe user context in order to compute effectively [Perera et al. 2013].
However, many studies have focused on sensor data or display of
information, and research on understanding the context for the
purpose of intervention in the user’s physical motor actions isstill lacking. Therefore, our research explores important factors forintervention in motor actions in the context of physical sports in aremote environment.
Previous studies have suggested that physical motion assistancecan provide an augmented experience. For example, Kasahara etal. proposed a system that uses electrical muscle stimulation toaccelerate the user’s motion without compromising their sense ofagency [Kasahara et al. 2019]. Maekawa et al. reported that a user’s
performance in throwing with a handheld device can be improved
without the user noticing [Maekawa et al. 2020]. These studies built
their systems for simplified situations, but we focus on physicalsports involving more complex situations.

Behind The Game

3

SIGGRAPH ’21 Emerging Technologies , August 9–13, 2021, Virtual Event, USA

SYSTEM OF BEHIND THE GAME

The system consists of an air hockey table, a camera to capture
images of the hockey table surface, a physical machine on one sideof the table, and a network server (Figure 2). The remote player
watches the live video stream and controls the remote striker. On theother side of the table is the physical opponent in a local location.

3.1

Physical machine for air hokcey gameWe tailored a machine for physical interactions on an air hockey
table by its hardware to generate sufficient acceleration and velocity
to follow the user’s arm motions. The machine has two actuateddegrees of freedom driven by DC motors. We adopted a parallelmechanism using a belt and pulley to minimize the mass of end
effector and ensure sufficient acceleration and velocity. The end
effector is equipped with a striker and can move across half of thetwo-dimensional plane of the table.

3.2

Remote user input and feedback

We built a web application that allows the remote user to control
the physicl machine. The application displays the live video streamof the table surface captured by the camera. The movable rangeof the striker is highlighted with a rectangle overlaid onto thevideo feed. Inside the rectangle, an indicator corresponding to thestriker’s position is displayed. The user can control the striker bymoving this indicator, which can be manipulated with any familiar
peripheral, such as a mouse or touchpad. There is a delay of tens tohundreds of milliseconds both between when the camera captures
the image of the physical table and when that image is displayedon the remote application. The same delay also exists between the
remote user’s input and the physical machine’s motion.

4

Figure 3: The system estimates the context from the movements of the puck and striker. (a) When the system detects
the context, assistance is momentarily activated. (b) Afterthe striker hits the puck, the assistance rate attenuates gradually until the striker is entirely back under the player’s control.

ASSISTANCE DESIGN

The time lag induces various hindrances, such as failing to react
the opponent’s shot or to meet the puck accurately with the striker.
These experiences impede the user’s sense of agency. Hence, we designed computational interventions to reduce hitting errors caused
by delays. The remainder of this section describes the specifics of
our assistance design that we investigated in preliminary experiments.

4.1

Context-aware assistance

From the observation of user performance in air hockey, we foundthat the game can be divided into two main contexts:(1) Defense: the user reacts to a puck hit by an opponent andblocks or returns the puck.
(2) Offense: the user shoots the puck into the opponent’s side.
We also found that the accuracy of horizontal control is critical in air
hockey. If the horizontal control accuracy is low, the player may not
hit the puck as they intend, or they may not stop the puck hit by the
opponent, resulting in a goal being scored against them. Therefore,
we designed a system that optimizes the horizontal position in thetwo contexts of defense and offense.
The assistance in optimizing the horizontal position is triggered
only when the system detects two contexts. When assistance istriggered, the horizontal control is given to the computer (Figure

Figure 4: Context-aware assistance. (a) Defense: the striker’s
position is adjusted to match the estimated contact point according to the puck’s trajectory. (b) Offense: the striker’s position is adjusted to match the position of the puck.
3-a). Even when this intervention is active, the player still has control over the vertical movement. Thus, the player can strategicallydecide to shoot the puck onto the opponent’s side or keep the puckon their side. After the striker hits the puck, the assistance rate
gradually attenuates until it is completely back under the player’s
control (Figure 3-b). This horizontal correction helps the user hit
the puck accurately, even under remote control with a time lag. Because the player’s goal is to hit the puck, this experience contributes
to maintaining a sense of agency.
4.1.1 Defense. The defense context is estimated when the puck
approaches towards the remote player’s side above a certain speed.
In this case, the system adjusts the striker’s horizontal position
to match the intersection between the predicted trajectory of thepuck and the horizontal line through the striker’s current location(Figure 4 (a)). In the defense context, the intervention is triggered
independently of the player’s input. This is because the player’s
reaction based on a delayed video prevents them from hitting theSIGGRAPH ’21 Emerging Technologies , August 9–13, 2021, Virtual Event, USA

puck on time. We found that in many cases, the player felt moreagency than when they move the striker by themselves because of
the apparent congruency between action and feedback.
4.1.2 Offense. The offense context is estimated when the puckstays below a certain speed on the remote player’s side, and thesystem detects the player is moving the striker to hit the puck. In
this case, the system adjusts the striker’s horizontal position tomatch the position of the puck detected by the camera (Figure 4
(b)).

4.2

Haptic feedback

We also implemented haptic feedback as an optional feature. Thisfeature includes two types of haptic feedback: collision feedback
between the puck and the striker, spring force feedback to navigate
to the corrected striker position. Because haptic sensation is one of
the essential factors in motor learning, we believe it can contributeto becoming proficient in manipulating the system. However, thisfeature can also make the presence of interventions apparent.

4.3

Two-dimensional assistanceTo design more performative intervention, we implemented a prototype system that intervenes for vertical and horizontal control.
In this implementation, the system estimates the contact point infront of the current striker position and achieves accurate shooting.
In preliminary experiments, we confirmed that this implementation
enabled the puck to be shot with more power than the horizontal assistance. However, some test users reported that their senseof agency was lower than it was when only horizontal assistance
was activated. This may have been because the intervention generated movements and timings that were different from the player’s
intentions, which may have been caused by an inaccurate estimation of the context. However, we will achieve this two-dimensional
assistance without compromising user agency by implementing
another context estimation method. Specifically, by properly switching between horizontal and two-dimensional assistance based on
the user’s input, the two-dimensional assistance may be able to beincorporated into the system without compromising agency.

5

USER EXPERIENCES

We will demonstrate our Behind The Game system at the SIGGRAPH 2021 Emerging Technology exhibition Our system allows
remote users to control the striker and play against a physical opponent at our demo location. Our demonstration can be achieved
fully online manner. The attendees can join our exhibition over
the web browser with their mouse. The context-aware assistance
we designed increases the user’s performance and sense of agency,
even in teleoperation, in which there is a time lag between theuser’s action and feedback. Thus, the user is able to play fairly witha physical opponent. Even if there is no haptic feedback, it willallow remote participants to experience the concept.
If attendees can physically visit our demonstration, they canexperience the optional haptic feedback and play against remoteattendees. Playing against remote opponents who cooperate witha machine will also provides a new experience for the physicalplayers.

Maekawa and Saito, et al.

6

LIMITATION AND FUTURE WORKS

Because the time lag varies depending on network conditions, there
is a possibility that we will need to adjust the intervention parameters for different networks.
We are planning on conducting a detailed user study. One ofour goals is to clarify how to adjust the parameters to maximize
the user’s sense of agency. Eventually, we hope to provide design
guidelines for solving physical constraints in teleoperation using a
psychological approach.
We aim to develop this project not only as a solution to teleoperation problems, but also as research on human-machine interactions
in motor actions. For instance, we are interested in what behavioral
changes are generated when the intervention is gradually increased
or decreased, and when the user learns a machine-human mutual
motor model.

7

CONCLUSIONS

We demonstrated a computational intervention method for teleoperation of an remote air hockey that retrieves the remote user’s performance and sense of agency, even with a time lag. With contextaware assistance, in which the system is aware of the context andintervenes according to it, the remote player exhibits in improved
performance without compromising their sense of agency, and bothplayers can experience the excitement of the game. We believe thatthe experience of human-machine collaborative actions achievedin this study provides new insights in the research field of humanmachine integration that are beyond the solution framework fortime lags in teleoperation.

ACKNOWLEDGMENTS
This work was supported by JST ERATO Grant Number JPMJER1701, Japan.

REFERENCES
Ozkan Bebek and M Cenk Cavusoglu. 2007. Intelligent control algorithms for roboticassisted beating heart surgery. IEEE Transactions on Robotics 23, 3 (2007), 468–480.
William R Ferrell and Thomas B Sheridan. 1967. Supervisory control of remotemanipulation. IEEE spectrum 4, 10 (1967), 81–88.
Shunichi Kasahara, Jun Nishida, and Pedro Lopes. 2019. Preemptive Action: Accelerating Human Reaction Using Electrical Muscle Stimulation Without Compromising Agency. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, New York, NY, USA, 643:1–643:15.
https://doi.org/10.1145/3290605.3300873
Azumi Maekawa, Seito Matsubara, Sohei Wakisaka, Daisuke Uriu, Atsushi Hiyama,and Masahiko Inami. 2020. Dynamic Motor Skill Synthesis with Human-Machine
Mutual Actuation. In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems (CHI ’20). ACM, New York, NY, USA, 1–12. https://doi.org/10.
1145/3313831.3376705
Charith Perera, Arkady Zaslavsky, Peter Christen, and Dimitrios Georgakopoulos. 2013.
Context aware computing for the internet of things: A survey. IEEE communications
surveys & tutorials 16, 1 (2013), 414–454.
Daniel Rakita, Bilge Mutlu, and Michael Gleicher. 2017. A motion retargeting method
for effective mimicry-based teleoperation of robot arms. In Proceedings of the 2017
ACM/IEEE International Conference on Human-Robot Interaction. 361–370.
Thomas B Sheridan. 1989. Telerobotics. Automatica 25, 4 (1989), 487–507.
Leila Takayama, Eitan Marder-Eppstein, Helen Harris, and Jenay M Beer. 2011. Assisted
driving of a mobile remote presence system: System design and controlled user
evaluation. In 2011 IEEE international conference on robotics and automation. IEEE,
1883–1889.