Teleoperation Method by Illusion of Human Intention and Time
Junki Aoki1, Ryota Yamashina1, and Ryo Kurazume2
Abstract— Shared control, in which teleoperation and autonomous control are combined to move the robot, is expected to
improve the efficiency of the user teleoperation problem. However, a problem exists whereby the user acceptance decreases
owing to the conflict of intention between the teleoperation andautonomous control. In this study, we address this problem byproviding an illusion to humans. We propose a teleoperation
method named the "Illusory Control" that can achieve both
mobility efficiency and user acceptance by implementing acyber-physical system that controls a robot in real space
through robot operations in virtual space. Illusory Control has
two functions: the "Illusion of Intention," which provides theillusion that the robot is operating according to human intention,and "Illusion of Time," which provides the illusion of time to fill
the gap by changing human behavior when the robot positionsin the virtual space and real space diverge. Preliminary teleoperation experiments with subjects demonstrated that thesystem improves the operational efficiency and acceptance ofthe system compared to conventional teleoperation methods,
namely direct teleoperation and shared control.

I. INTRODUCTION
Within the context of the recent labor shortage and increased telecommuting, the use of teleoperation robots, which
can be controlled from a distance, has become popular. A
teleoperation robot may be, for example, a mobile communication robot that is equipped with interfaces such as cameras and microphones.
When a teleoperation robot is controlled from a remotelocation, it is generally equipped with a safety system
whereby, in addition to human operator control, the robot
detects obstacles by using sensors that are mounted on the
robot and autonomously avoids or stops them. This mechanism of the efficient completion of mobile tasks while sharing
human operator control and robotic autonomy has beenstudied as shared control and has been used in practical applications; for example, in the double communication robot of
Double Robotics[1]. However, when the robot is controlled
while observing the camera image, the robot may react to
obstacles that the human operator does not notice, and theoperator may feel strong stress because the robot does not
move according to their commands or it moves in unexpectedmanners. This is owing to the fact that the motion commands
issued by the robot differ from those issued by the humanoperator, as illustrated in Fig. 1 In previous work, this has
been described as the human operator decisions not matching
the robot decisions, which causes the operator to be less accepting of the robot (less trusting in the system) [2].
1
Junki Aoki and Ryota Yamashina are with Ricoh Company, Ltd. junki.aoki@jp.ricoh.com, ryota.yamashina@jp.ricoh.com
2
Ryo Kurazume is with the Faculty of Information Science and ElectricalEngineering, Kyushu University. kurazume@ait.kyushu-u.ac.jp

An ideal solution to the problem of diminishing acceptancewould be for the robot to avoid obstacles of which the humanoperator is unaware automatically, while the operator remains
unaware of the avoidance behavior itself and feels that the
robot is moving as if it were performing the intended actions.
However, provided that the robot has autonomy, this is notpossible in practice, because the feeling that the robot ismoving on its own will always arise when the human operator
command and robot action do not match.
In this study, we attempt to solve this problem by using amethod named the "Illusory Control" (derived from the term
illusion of control [3] in psychology), which provides theillusion that the robot is being moved according to the humanoperator intention. To create the illusion, we use acyber-physical system that includes virtual spaces that are
copies of the real space and the robot moves in each space. In
the virtual space, the robot autonomy is not considered andthe human operator can control the robot according to theirpreference. Subsequently, the robot in the real space moves
autonomously, as if it were chasing a human-operated robot
in the virtual space. When moving in the real space, the robot
is equipped with a function that automatically avoids obstacles and stops when a collision is predicted, to ensure safety
against obstacles and people.
We verify that this approach can be used to ensure task
efficiency and acceptance in robot teleoperation.

Figure 1. In a shared control system, the human operator intention andtrajectory of a robot trying to avoid an obstacle may differ.

II. RELATED WORKS
Numerous studies have been conducted to support teleoperation, and some success has been achieved in improving
the efficiency of user operation. However, shared control
systems have yielded mixed results compared to direct teleoperation systems in terms of whether or not human operators
prefer such a system [4]-[6]. In the study of Roy et al., it wasasserted that users prefer manual control over a control

method with an assistive controller [6]. In response to thesestudies, Brooks et al. attempted to ensure human acceptancein the assistive controller by clarifying the intentions andtrajectories of the assistive controller through augmented
reality visualization [7]. Although this may increase theacceptance of the system, conflicts of intent between humans
and agents can occur in that they share and manipulate asingle control object. Therefore, cases may exist in which
acceptance cannot be ensured.
The control method whereby a human and a robot cooperate with one another to control an object is generally referred
to as “supervisory control” [8]．Within this context, the term
"traded control" refers to the control of a single object by ahuman and a robot while exchanging control authority,
whereas "shared control" refers to the control of a single
object by a human and a robot while combining the control
commands. Traded control includes directing robot behavior
through trajectory sketching [9], delegating control to experts
based on inference uncertainty [10], and supporting humanintervention for disaster rescue [11]. Shared control encompasses the use of probabilistic approaches to estimate goals[12], human-in-the-loop reinforcement learning to obtainhuman-assisted policies [13], and brain signals to obtain
control assistance [14]. Other methods vary widely, such as
those that extend human capabilities without requiring goal
estimation [15]. In both traded control and shared control, theagent and the human control the same object, so it is expectedthat the agent may delegate control authority to the human at
unexpected times, or the robot may perform actions that differ
from the intention of the human. Therefore, the problem of amismatch between the human and the agent intentions is
unavoidable. Although the system in this research is based on
the latter concept of shared control, the proposed Illusory
Control is a new method in which a human (virtual space) andan agent (real space) each control a different object. Humans
can feel that they can control the robot according to theirintention, which is expected to increase the acceptance of thesystem.
Moreover, this research is related to the field of virtual reality (VR)-based teleoperation. In recent years, an inexpensive VR environment for human operator training purposes
has been proposed [16]. Omarali et al. [17], Sebastian et al.
[18], and Whitney et al. [19] used multiple cameras to sense
the work environment and to transfer the point cloud information to the VR environment, thereby enabling the remotecontrol of a realistic robot while viewing virtual images inreal time. These conventional works assumed that the system
can accurately perceive the scene and reflect the user operation in its behavior. Therefore, safety concerns remain, suchas erroneous operations owing to the inability of humans toperceive the environment accurately. In this research, themotions of the human and robot do not necessarily need tomatch. We attempt to ensure both human operability andsafety by enabling the robot to act to compensate for the
problems of the human in the virtual space, or allowing thehuman in the virtual space to act to compensate for the
problems of the robot in the real space.

III. ILLUSORY CONTROLAs noted in Section I, the operators may feel stress when
they perceive that the teleoperation robot is not moving according to their commands. To address this issue, we devised
two methods: (A) not allowing the operator to feel that the
robot is not working as expected, and (B) changing the operator commands themselves without the operator noticing.
First, we propose a function named the "Illusion of Intention" to prevent the operator from feeling that the robot is notmoving as expected. Even if the robot automatically avoids
obstacles and at times comes to a standstill, if the operator is
not informed of this, they will think that the robot is moving
smoothly and as expected. That is, the real robot automatically avoids obstacles or stands still, but the operator is notshown the images of the environment in which the robot islocated; thus, the operator may feel that the robot is moving
according to their intention.
Subsequently, we propose a function named the "Illusionof Time" as a means to change the operator command itself
without the operator noticing. The problem of the Illusion ofIntention is that, over time, a difference will exist between thepositions of the human-operated robot and the robot that is
actually moving. Therefore, to reduce the difference in positions, the operator commands need to be guided unconsciously to fit the system needs over time. The solution to thisproblem is to hijack the operator thoughts by changing theoperator commands and the environment that the operator
perceives without the operator noticing. This can cause theoperator to believe that they are controlling the robot by theirown intention, even though they are actually issuing commands that match the intentions of the system (robot).
In this study, we implemented the Illusion of Intention and
Illusion of Time based on the above concept. In the following,
we describe each implementation in detail.
A. Illusion of Intention
For the proposed system, we constructed a real space and avirtual space, which was a 3D model of the real space.
Moreover, we implemented a set of modules to control therobot in the virtual space and a set of modules to control thereal space. The system was implemented using the robot
operating system (ROS). The concept of the system and thesystem architecture are depicted in Fig. 2.
First, the operator sends control commands to the robot inthe virtual space to cause it to move. While the robot in thevirtual space is being moved by the operator, the systemestimates the current robot position sequentially. Thereafter,
the estimated current position information of the robot in thevirtual space is transmitted to a group of modules in the real
space.
Subsequently, the modules in the real space perform navigation, using the current position information received fromthe virtual space as the goal. The navigation is implemented
using the Navigation Stack in ROS. Given a goal position, thenavigation stack conducts path planning based on Dijkstra's

Figure 2. (A) In conventional systems, an operator and an autonomous agent control the same robot. In the Illusory Control system, the operator controls therobot in the virtual space and the autonomous robot moves in the real space. The robot moves while interacting with the virtual space and real space. (B)System architecture of Illusory Control. Navigation is performed so that the real-world robot approaches the position of the virtual robot that is directly
controlled by the operator. When the positions of the virtual robot and real-world robot diverge, the Illusion of Time function is activated.

algorithm and autonomous movement is achieved while
avoiding obstacles using the dynamic window approach.
While the robot is navigating in the real space, the current
robot position in the virtual space changes constantly owing
to the continuous teleoperation of the virtual space by theoperator. Therefore, based on the amount of robot movementin the virtual space, the new current position is transmitted tothe modules in the real space once the robot has moved beyond a certain threshold from the previously estimated position. When the modules in the real space are provided withthe position information from the modules in the virtual space,
they update the goal point and resume navigation with a newpath plan.
This architecture enables the operator to move autonomously in real space safely, while providing the illusion thatthe robot is operating according to their intention in virtual
space.
B. Illusion of TimeIn the teleoperation using the Illusion of Intention described in (A), the problem of the difference between thepositions of the robot in the virtual space and real space occurs. One possible solution is to force the operator to stop.
However, this may cause strong stress for the operator. Another solution is to guide the operator unconsciously so that
this difference becomes smaller over time. The Illusion ofTime function is activated based on the current distance of therobot in the virtual space and that in the real space. When the
distance between the robots in the virtual space and real spaceexceeds a certain threshold, the Illusion of Time function is
activated. In this study, we devised three methods for realizing the Illusion of Time function and conducted comparison
experiments. The methods are described in detail as follows:

1) Deceleration: This is a method of gradually slowing down
the robot movement in the virtual space according to the
distance between the robot in the virtual space and the robot
in the real space. Decelerating the movement speed of therobot in the virtual space has the effect of waiting for the
robot in the real space to approach the robot in the virtual
space.
2) Blur: In this method, as the distance between the robot inthe virtual space and that in the real space increases, it becomes substantially more difficult for the operator to perceive
the environment in the virtual space. The operator perceives
the virtual environment through the operation user interface(UI) and controls it remotely. When blurring the image of thevirtual space and making it difficult for the operator to perceive the space, it becomes difficult for the operator to navigate between obstacles and to perceive the goal ahead. As aresult, the operator can expect the effect of decelerating theteleoperation of the robot in the virtual space by their own
will, and thus, adjust the time until the distance between thevirtual space and real space is reduced.
3) Obstacle: This method presents an obstacle in front of therobot that is controlled by the operator in the virtual spacewhen the distance between the robot in the virtual space andthat in the real space exceeds a certain threshold. When an
obstacle is created in front of the operator, the operator musttake steps to avoid the obstacle with their own intention.
While taking the steps to avoid this obstacle, the effect iswaiting for the robot in the real space to approach the positionof the robot in the virtual space.
To verify the functional concept of Illusory Control, weconstructed both the real space and virtual space on Gazebo to
implement a prototype. Figure 3 presents the activation ofeach function of the Illusory Control described above.

Figure 3. Illusion of Intention: In (1) to (3), the robot in the real space moves to follow the robot operated in the virtual space. The operator perceives theenvironment of the virtual space through the operation UI. Illusion of Time: (1) By decelerating the speed, the time required for the real-world robot to catch up
with the virtual space can be controlled. (2) The operator operation UI gradually becomes blurred, making it difficult to perceive the environment of the virtual
space. (3) An obstacle is appeared in front of the robot in the virtual space.

IV. PRELIMINARY EXPERIMENTS

used to calculate a path to avoid obstacles and turn the robot
in a direction that is free from obstacles.

We verified the usefulness of the proposed method througha user study in which we compared it with conventional teleoperation methods. We aim to answer the following threequestions:• Can Illusory Control improve task efficiency as well
as shared control?
•

Can Illusory Control improve system acceptancebetter than conventional methods?

•

Which of the three methods using the Illusion of Timeis more acceptable?

A total of nine participants were used as subjects (six men
and three women). Moreover, we divided the subjects into
two groups: those who understood the function of Illusory
Control (five subjects) and those who did not (four subjects).
This was to verify whether the operation time and impressionof the system differed depending on the level of understanding. The tasks to be performed and questionnaire items were
the same in both groups. The difference was whether or not
the function of Illusory Control was explained to the subject
beforehand. In this experiment, the subjects were asked tocontrol a mobile robot remotely and to perform a task according to five control methods.
The five methods were direct teleoperation, shared control,
Illusory Control (deceleration), Illusory Control (blur), andIllusory Control (obstacle).
Conventional methods are based on direct teleoperationand shared control. In direct teleoperation, the robot moves
according to the directional input from the operator. In sharedcontrol, the directional keys of the operator and the local cost
map are used as the input, and the local planner of the ROS is

Figure 4. Experimental environment.

A. Experiment Setup
The subjects were asked to operate the robot with a
PlayStation controller while viewing the operation UI. The
robot started from the start point indicated in Fig. 4 and aimed
at the goal point while avoiding obstacles. The task was
performed once per method. To eliminate order effects, theorder in which the five methods were performed was randomized for each subject. Furthermore, the subjects could
become accustomed to the course as they attempted the fivemethods. For this reason, we randomly applied two different
courses with varying obstacle locations. Moreover, we considered it to be a problem that the robot behavior wouldchange for obstacles that the operator could not perceive
without knowing the reason for the change. Therefore, weplaced one obstacle that the subject could not see and required
them to avoid it. The proposed method, Illusory Control, isbased on the assumption that the operator controls the virtual
space. Therefore, it is possible to pass through obstacles
during the operation of Illusory Control.

Figure 5. (A) The proposed method resulted in a shorter operation time compared to the conventional method. (B) A significant difference was observed only
when the deceleration method was compared with direct teleoperation. (C) The proposed method improved the acceptance compared to the conventional
method. (D) There was no difference in the attention to obstacles between the conventional and proposed methods. (E) and (F) There was no difference between
the three Illusion of Time methods.

An overview of the functions was provided prior to starting
the experiment. We did not explain the Illusion of Timefunction, such as the presentation of obstacles along the way
or the difficulty of perception, to the group that did not understand Illusory Control. To evaluate the pure impression of
the Illusion of Time function when it was activated, the subjects were asked to operate it with no prerequisite knowledge.
B. Measure
Both objective and subjective metrics were used to evaluate the usefulness of the proposed method.
We used the operation time from the start to the goal as an
objective measure to evaluate the improvement in the operation efficiency. In Illusory Control, there is a virtual space inwhich operators operate and a real space in which robots
move autonomously, but in this experiment, we focused only
on the operation time in the virtual space.
As a subjective evaluation, we asked the subjects to answer
a questionnaire after using each method. This enabled us toevaluate the impressions of the system. First, the System
Usability Scale (SUS) [20] was used to evaluate the systemusability. Thereafter, the subject impressions of the systemwere evaluated using four questionnaire items with afive-point Likert scale. Among the four items, the two items
in Table I were common to all five methods and were used toevaluate the acceptance of the methods. The other two questionnaire items in Table II were used to test the acceptance ofthe three Illusion of Time methods.
TABLE I.
No

QUESTIONS FOR ACCEPTANCE OF THE SYSTEM
Question

Q1

I felt that the robot was working according to my will.

Q2

I took great care not to let the robot collide with any obstacles.

TABLE II.
No

Q3
Q4

QUESTIONS FOR ACCEPTANCE OF ILLUSION OF TIME
Question

Before and after the change by Illusion of Time, I felt that I
was able to move the robot according to my will.
After the change by Illusion of Time, I felt inconvenient tooperate the robot.

V. ANALYSIS AND RESULTS
The experiment was completed by all subjects, who wereable to reach the goal point from the starting point for all
methods. The scores of the experimental results are depicted
in Fig. 5. Although we grouped the subjects according towhether or not they understood the function of the Illusory
Control, no significant difference was exhibited. Therefore,
the graph in Fig. 5 summarizes the results for all subjects.
The number of subjects in the preliminary experiments was
small. As a result, some of the measurement results were notnormalized or equivariant. Therefore, one-way analysis ofvariance was conducted for those with normality and equalityof variance, and multiple comparisons were drawn using theDunnett test as a post hoc test, considering the conventional
method as the control group and the three proposed methods
as the experimental group. The Kruskal–Wallis test was used
for items without normality or equal variance. For items thatwere compared to the baseline, the Steel test was used as apost hoc test, and for those that were not, the Steel–Dwass test
was used to determine whether there was a significant difference.
The operation time results demonstrate that the proposed
method was more effective than direct teleoperation in the
scenario with obstacles that the operator could not observefrom the camera. In the direct teleoperation process, thesubjects searched for obstacles and attempted to avoid themby trial and error. Furthermore, the proposed method allowedthe subject to move through obstacles even when obstacles
existed. Therefore, they did not need to be distracted by the
obstacles and the operation time differed.
In terms of usability, a significant difference was observed
only when direct teleoperation and Illusory Control (deceleration) were compared. This may be owing to the fact thatthe subjects were less confused than in the other Illusion ofTime methods, because no change occurred in the appearanceof the environment. This phenomenon can also be attributed
to the fact that certain subjects did not decelerate as much asothers. In Illusory Control (obstacle) and Illusory Control(blur), the subjects could not understand the reason for theappearance of the obstacles or blurred vision, which suggeststhat there was no difference from the conventional method.
For Q1, there was a clear difference between the conventional and proposed methods. The reason for this is that in theconventional method, the subject could not operate the robot
as desired, because the robot stopped moving when it bumped
into an obstacle without understanding the reason or changed
its movement direction when it detected an obstacle.
For Q2, there was no difference from the conventional
method. In this experiment, we used a simulator for both thevirtual space and real space. Therefore, the subject operated
the robot in the virtual space for all of the methods. It is desirable to design experiments so that the subject operates theconventional methods in the real space. If a difference exists
between the operation in the virtual space with the Illusory
Control system and that in the real space with the conventional method, the degree to which the subject is concerned
about obstacles is expected to differ.
For Q3 and Q4, no difference was observed among thethree methods. In all three methods, the subjects noticed thechange in the environment caused by the Illusion of Time;thus, they were surprised by the sudden change and found it
difficult or inconvenient to move the robot as they intended.
Therefore, by applying a method of environmental changethat the subject does not notice, it is expected that differences
will appear in the results. Further improvement of the methodin addition to the three Illusion of Time functions described inthis paper will be necessary in the future.
VI. CONCLUSION
To improve the efficiency of mobile tasks and user acceptance of teleoperated robots, we have proposed a newcontrol method named the Illusory Control, and conducted
experiments on subjects using the system. According to theexperimental results, a significant difference in the operation
time compared to that of direct teleoperation was observed in
the scenario with obstacles that the user could not see fromthe camera. Furthermore, there was a significant difference
between the conventional and proposed methods in terms ofwhether the user could operate the system as desired. In
summary, we have demonstrated that Illusory Control offers
the potential to increase the efficiency of user operation as
much as the shared control and to improve the acceptance ofthe system more than direct teleoperation and shared control.

However, no significant difference was observed in the impression of the Illusion of Time function.
In this study, we have constructed a simulator environmentthat assumes a real space to verify the robot operation. In
future work, we plan to operate the actual robot in real space
and to include a large number of subjects to verify the operation. Furthermore, in this experiment, we used a special
scenario in which we placed obstacles that were difficult forthe subjects to perceive. In future studies, we will conduct
experiments on scenarios with perceivable obstacles andscenarios in which obstacles are placed chaotically.
REFERENCES[1][2]

Double Robotics, “Double 3,” https://www.doublerobotics.com/.
H. Tokushige et al., “Trust Lengthens Decision Time on Unexpected
Recommendations in Human-agent Interaction,” Proceedings of 5thInternational Conference on Human Agent Interaction (HAI), pp.
245-252, 2017.
[3] Langer. E. J, “The Illusion of Control,” Journal of Personality andSocial Psychology, vol. 32, no. 2, pp. 311-328, 1975.
[4] A. D. Dragan et al., “A Policy-Blending Formalism for Shared Control,” The International Journal of Robotics Research, vol. 32, no. 7, pp.
790-805, 2013.
[5] S. Javdani et al., “Shared autonomy via hindsight optimization,”
Robotics: science and systems: online proceedings, vol. 2015, 2015.
[6] Q. Roy et al., “Automation Accuracy Is Good, but High Controllability
May Be Better,” Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI), no. 520, pp. 1-8, 2019.
[7] C. Brooks et al., “Visualization of Intended Assistance for Acceptanceof Shared Control,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11425-11430, 2020.
[8] T. B. Sheridan, “Telerobotics, automation, and human supervisory
control,” MIT Press, 1992.
[9] K. Lee et al., “Development of the Human Interactive Autonomy for
the Shared Teleoperation of Mobile Robots,” 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.
1524-1529, 2016.
[10] Y. Cui et al., “Uncertainty-Aware Data Aggregation for Deep Imitation
Learning,” 2019 International Conference on Robotics and Automation(ICRA), pp. 761-767, 2019.
[11] B. Doroodgar et al., “A Hierarchical Reinforcement Learning BasedControl Architecture for Semi-Autonomous Rescue Robots in Cluttered Environments,” 2010 IEEE International Conference on Automation Science and Engineering (CASE), pp. 948-953, 2010.
[12] S. Gholami et al., “A Probabilistic Shared-Control Framework for
Mobile Robots,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11473-11480, 2020.
[13] S. Reddy et al., “Shared Autonomy via Deep Reinforcement Learning,”
Robotics: Science and Systems: online proceedings, vol. 2018, 2018.
[14] J. Philips et al., “Adaptive Shared Control of a Brain-Actuated Simulated Wheelchair,” 2007 IEEE 10th International Conference on Rehabilitation Robotics (ICORR), pp. 408-414, 2007.
[15] Y. Du et al., “AvE: Assistance via Empowerment,” Neural Information
Processing Systems (NeurIPS), 2020.
[16] L. Pérez et al., “Industrial Robot Control and Operator Training Using
Virtual Reality Interfaces,” Computers in Industry, vol. 109, pp.
114-120, 2019.
[17] B. Omarali et al., “Virtual Reality Based Telerobotics Framework With
Depth Cameras,” 2020 29th IEEE International Conference on Robotand Human Interactive Communication (RO-MAN), pp. 1217-1222,
2020.
[18] S. Kohn et al., “Towards a Real-Time Environment Reconstruction for
VR-Based Teleoperation Through Model Segmentation,” 2018
IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), pp. 6305-6310, 2018.
[19] D. Whitney et al., “ROS Reality: A virtual reality framework using
consumer-grade Hardware for ROS-enabled robots,” 2018 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS), pp.
5018-5025, 2018.
[20] J. Brooke, “SUS: A Quick and Dirty Usability Scale,” Usability
evaluation in industry, CRC Press, 1996.