# Augmented Reality-Based Interface for Bimanual Robot Teleoperation
## 2 Previous Work
### Description of the Advanced Bimanual Robot Teleoperation Application  
The advanced bimanual robot teleoperation is based on the _task-priority strategy_[71; 72] and _conventional_ and _non-conventional Sliding Mode Controllers_ (SMCs) [73; 74]. As commented before, the goal of this bimanual robotic application is to perform a human-robot cooperative control loop so that the user operator partially teleoperates two robotic arms to perform a surface treatment operation, whilst the robots automatically ensure the appropriate tool force and orientation; see Figure 1. Thus, the so-called _workpiece robot_ (WR), which consists of a 7R collaborative robot with a workpiece of flat methacrylate fixed to the end-effector using a self-made piece (see Figure 1a), is in charge of holding the workpiece. Meanwhile, the so-called _surface treatment robot_ (STR), which consists of a 6R robotic arm with a Force/Torque (F/T) sensor and a cylinder-shaped tool with a piece of cloth (see Figure 1a), operates with the surface treatment tool on the workpiece. Thus, the user controls the workpiece position and orientation and, simultaneously, controls the 2D tool motion on the workpiece surface using an interface, which consists of a gamepadto command the robots and a visual feedback screen to show the user the robots and the user reference states; see Figure 1a.  
Figure 1b shows the block control diagram for both robots, where subscript \(s\) stands for the STR; subscript \(w\) stands for the WR; subscript \(ref\) stands for the user reference; subscript \(c\) stands for the commanded control action; \(\mathbf{p}=\begin{bmatrix}x&y&z&\alpha&\beta&\gamma\end{bmatrix}^{T}\) is the robot pose, i.e., the linear positions \(\{x,y,z\}\) plus orientation angles \(\{\alpha,\beta,\gamma\}\); \(\overline{\mathbf{p}}_{s}=\begin{bmatrix}x&y\end{bmatrix}^{T}\) is the 2D position of the STR tool on the workpiece surface, i.e., the linear positions \(\{x,y\}\) relative to this surface; \(\mathbf{q}=\begin{bmatrix}q_{1}&\cdots&q_{n}\end{bmatrix}^{T}\) is the robot configuration, with \(n\) the number  
Figure 1: Bimanual application setup and block diagram (for further details, refer to [31]). (**a**) Previous setup used for the real experimentation. (**b**) Block control diagram for both robots (WR and STR).  
of robot joints; and \(\mathbf{F}\) is the vector containing the measured forces and torques. Thus, using the gamepad joysticks, the user is able to send the reference to the WR pose \(\mathbf{p}_{w}\) and, simultaneously, the reference to the 2D position \(\overline{\mathbf{p}}_{s}\) of the STR tool on the workpiece surface. Thus, the high-level controllers of both robots compute the corresponding joint commands \(\mathbf{q}_{c}\) from the user references, the state \(\{\mathbf{q},\dot{\mathbf{q}},\mathbf{p}\}\) of both robots, and the force sensor data \(\mathbf{F}\). These joint commands are then sent to the low-level controllers of both robots, as shown in Figure 1b, in order to complete the teleoperation task. See [31] for further details on the high-level controllers of both robots and the related signals.  
In addition, some constraints are considered for both robots in order to increase the safety of the task: (1) the WR is automatically controlled to maintain the workpiece center inside the allowed region that is modeled as a superellipsoid, which is similar to a rectangular prism with smooth corners; (2) the STR is automatically controlled to keep the center of the treatment tool within the allowed region on the workpiece, which is modeled as a superellipse, i.e., a rectangle with smooth edges.