## 3 The Human Role in Teleoperation
### System Latency  
Latency in teleoperation refers to the elapsed time between a user's action and the consequent system's observed response. Several components of the system can contribute to this overall time lag: user-input device lag, motor actuator lag, video acquisition and display lag, synchronization lag, communication time delay, etc. This occurs in the remote agent control flow and the visual and/or haptic feedback flow. The operator's control actions, such as head and hand motions, are mapped into the robot's commands using pose-tracking devices and hand controllers. These commands are then transmitted through communication links and executed remotely by the robot's actuators. Considerable delays can occur from the command generation to its execution, due to motor actuator lag and or the latency of the network. The execution of the commands is viewed through images of the remote scene acquired by cameras and delivered to the operator using the network link. This transmission adds more time delays because of the network latency and the time required to transfer the data images. The time to send the command itself is influenced by bandwidth limitation, and although it is comprised of a few small data packets, it can take a long time when the emitter is far from the receptor (e.g., 2.56 s in the two way light time latency between the Earth and Moon [63]). A head-mounted display (HMD) can also introduce some delays. Immersive technologies related to VR refer to motion-to-photon latency (\(l_{m2 photon}\)) as the time delay between the movement of the operator's head (\(t_{mow}\)) and the change of the display screen reflecting the operator's movement (\(t_{disp\_move}\)) [34], \(l_{m2 photon}=t_{disp\_move}-t_{mow}\). Lag is a crucial element to allow the operator's brain to think that he/she is physically interacting in the remote place, experiencing the sense of telepresence. For that, immersive interfaces should provide stimuli that trick the sensory system, ensuring consistency between vision, internal sensory information (e.g., vestibular, proprioceptive), and cognitive models. The non-compliance of expectations can break the experience of presence, negatively affecting the task performance and causing motion sickness and nausea [64; 65]. For example, when a user stops rotating his/her head, he/she expects not to see moving images on the display, or when he/she starts head motion, he/she is supposed to see images moving immediately and not after a delay. Research [34] has found that the _just noticeable difference_ (JND) for latency discrimination should remain below 15 ms. The response of the system to head motions should be as fast as the human vestibulo-ocular reflex, the response of which ranges from 7 ms to 15 ms [33]. During head motion, this reflex stabilizes the retinal image at a chosen fixated point by rotating the eyes based on vestibular and proprioceptive information compensating the motion. In fast movements of the eye, such as _saccade movements_ (up to 900\({}^{\circ}\)/s), the retinal image may become blurred. To avoid processing unclear information, the brain has a mechanism, called a _saccadic mask_[66], that temporally suspends the visual processing so that the motion of the eye or the gap in visual perception is unnoticed. _Saccade movements_ enable fovea rendering approaches, optimizing display resources, and additionally, the _saccadic mask_ suggests some tolerance in current VR eye-tracking-to-photon latency [67; 68]. NASA researchers [69] studying HMDs to support synthetic enhanced vision systems in flight decks found that for extreme head motions, such as higher than 100 deg/s, the system latency requirement must be below 2.5 ms. Anyway, for demanding tasks requiring headset displays with a large field of view and high resolutions, the recommended system latency is less than 20 ms. For example, Oculus RIFT developers recommend not exceeding 20 ms for motion-to-photon latency in a VR application [70].