# A Motion Capture and Imitation Learning Based Approach to Robot Control
## 4 Proposed Approach
For extracting additional information specific to the throwing task, we also track the object thrown and use its pose information to annotate each observation with target coordinates extrapolated from its flight arc and a release timing signal determined by heuristics described in Section 5.2 (pre-processing). The next crucial decision to make is what the input and output spaces of the model will be. Setting aside the additional control parameters (target coordinates and release signal), when provided with a sequence of pose observations, the main options are as follows:  
* Pose-Pose--predict the pose at the next discrete time step from the current observation;
* Pose Derivatives--predict velocities or accelerations as actions;
* Joint state-Joint target--analogous to pose-pose, but in joint space;
* Joint state-Joint velocity--analogous to pose derivatives, but in joint space. Approaches crossing the Cartesian-joint space domain boundary were not considered, as the model would be required to learn the inverse kinematics of the robot. The advantages of having a model operate entirely in joint space would be seen at the next stage--integration with the robot controller--as motion planning to joint targets is trivial. However, training such a model would require mapping the demonstrations to robot configuration space by solving inverse kinematics on them, much the same as with a model operating in Cartesian space. While computationally less expensive at runtime, this approach is more difficult to reason about or explain, complicating the hyperparameter discovery process. Any model obtained this way would also be tightly coupled to a specific robot. Hence, all models were trained in Cartesian space, with the motion planning step left for last. Outputting pose derivatives (linear and angular velocities) is advantageous, as it enables the use of servo controllers, and input-output timing constraints are less stringent than they would be in a sample frequency-based model where a constant time step is used to modulate velocities. However, a model-in-the-loop control scheme is required--forward planning is not possible without a means to integrate model outputs with realistic feedback from the environment. Moreover, for training, the model derivatives of the pose need to be estimated from sequential observations, highly susceptible to discretization error. Predicting the future pose allows for using discrete-time pose observations directly, and, assuming a pose following controller can keep within tolerances--which, in our case, were determined by the requirements for triggering gripper release (see Section 5.4)--forward planning becomes a matter of running the model autoregressively (using its previous outputs for generating a sequence). This approach naturally lends itself to the employment of recurrent model architectures. The main drawback is that the pose derivative information is encoded in the time step, which imposes constraints on observation and command timing, complicating the use of such a model in a real-time feedback manner. It also requires that a method for the accurate time parametrization of Cartesian motion plans is available. Ultimately, the approach selected (see Figure 1) is to use pose-pose policies with footprints given in Equation (16). These predict sequences of states containing position and orientation in Cartesian space. Such sequences can then be used as inputs to Cartesian motion planning algorithms, which produce joint trajectories to control the robot, as described in Section 2.