# A Motion Capture and Imitation Learning Based Approach to Robot Control
## 5 Implementation
### Data Collection  
All demonstrations were recorded using _OptiTrack_ equipment that consists of a set of cameras, highly reflective markers to be attached to trackable objects, and the _Motive_ software package, which handles pose estimation and streaming. As shown in Figure 2, a cage with eight cameras installed serves to hide external sources of specular reflections from the cameras and confine moving objects, such as drones, from leaving the scene. To simulate an industrial robot end effector, a gripping hand tool was equipped with markers. Since the motivating application calls for throwing plastic bottles, one such one was also marked for extrapolating target coordinates and identifying the release point in a demonstration. A start point was laid down on the floor to serve as a datum for automatically separating the recorded demonstrations during pre-processing. Holding the effector stand-in here for at least a second before each demonstration provided a consistent signal that can be identified programmatically.  
Figure 1: A high-level overview of the proposed approach. Demonstrations are recorded with motion capture equipment, a split and cropped data set augmented with target coordinates, and a gripper actuation signal is created. This is used to train neural network models in behavioral cloning. When operating in the open-loop regime, prior model outputs are used to autoregressively predict subsequent states. The resulting sequence can then be fed into a Cartesian motion planner to control a robot.  
Nets were used to catch the thrown bottle and prevent damaging the markers. These were also marked to aid in delimiting the ballistic segment of the thrown object's flight. While in principle it is possible to detect an object in freefall by observing its acceleration, for trajectory extrapolation, this was deemed too prone to error--some impacts radically change the horizontal component of the object's velocity without breaking its fall, and the acceleration of the empty bottles proved to be affected by air resistance to a significant degree, owing to their low terminal velocity.  
In software, rigid body definitions were created for all objects to be tracked. These were then streamed on the local network, relayed over _robot operating system_ (ROS) topics corresponding to each rigid body using a pre-existing package ROS and recorded into a bag file to be converted into a.csv data set. Over the course of this project, two data sets with roughly 150 and 50 demonstrations each were collected. The first was used in the development process but proved to contain throws beyond the capabilities of the robot hardware used. Therefore, a second data set of throws with less pronounced swings was produced to fit within the working volume of the robot arm. A notable feature of the second data set is that initial orientations were randomized to a much lesser degree (\(\pm 30^{\circ}\)) than in the original corpus (\(\pm 90^{\circ}\)), and the range of estimated target coordinates for the throws was also more tightly constrained (1.5...1.8 m along the \(x\)-axis as opposed to 1.6...3.6 m), likely contributing to the better estimated throw accuracy metrics as illustrated in Section 6.