# A Motion Capture and Imitation Learning Based Approach to Robot Control
## 5 Implementation
### Evaluation Metrics  
To draw comparisons between models of different architectures developed as part of our work, trained with differing hyperparameter sets, quantitative evaluations needed to be computed. As this is not a standard task with agreed-upon metrics and benchmarks, some exploratory work was required to arrive at quantifiers that agree with intuitively self-evident characteristics. While in principle it should be possible to judge models based on throw accuracy, this is not very helpful in the early stages of research when most models fall short of attaining the desired objective. Furthermore, limitations imposed by the aforementioned time parametrization issues with Cartesian motion planning in ROS make such a comparison as yet infeasible. Hence, it was decided to compare model outputs with the demonstration data set. As judging against the training data set only informs us to the extent to which the model has been able to overfit, a smaller validation data set was set aside for out-of-distribution comparisons. The outputs to be compared were obtained by executing the models autoregressively on each demonstration's initial state for as many steps as were present in the corresponding recorded demonstration. This can be formally stated as  
\[\mathcal{D}_{eval}=\left((\tau_{1}^{d},\tau_{1}^{g}),...,(\tau_{k}^{d},\tau_{ k}^{g})\right) \tag{17}\]  
\[\tau^{d},\tau^{g}=(\mathbf{s}_{1}^{d},...,\mathbf{s}_{m}^{d}),(\mathbf{s}_{1 }^{g},...,\mathbf{s}_{m}^{g});\mathbf{s}_{1}^{d}=\mathbf{s}_{1}^{g} \tag{18}\]  
where \(\mathcal{D}_{eval}\) refers to the evaluation data set of a single model with respect to either the validation or test demonstration set, but \(\tau_{i}^{d},\tau_{i}^{g}\) are the demonstration and generated trajectories (sequences of state observations \(s_{j}\)) sharing the same initial state respectively. Three broad classes of evaluation metrics were computed: data set-wise (global), step-wise and throw parameters. The first consists of vector similarity measures, such as Pearson's correlation coefficient, cosine similarity and distance metrics applied to the entire data set and the trajectories generated against it as concatenated vectors:  
\[f_{global}(\mathcal{D}_{eval})=f:(\tau_{1}^{d},...,\tau_{k}^{d})\times(\tau_{1 }^{g},...,\tau_{k}^{g})\rightarrow\mathbb{R} \tag{19}\]  
Figure 5: Qualitative performance evaluation—visualization and execution. In both cases, the trajectories were sequences of pose (position, orientation) goals obtained by running the models autoregressively on their outputs at prior time steps. Release timing is represented in (**a**) by the change in marker size. The second class involves applying various measures to the observation/state variables at each time step:  
\[f_{stepwise}(\mathcal{D}_{eval})=f:\left[\sum_{\pi^{d},\pi\in\mathcal{D}_{eval}} \sum_{i=1}^{m}\Bigl{(}g:\mathbf{s}_{i}^{d}\times\mathbf{s}_{i}^{g}\to\mathbb{R }\Bigr{)}\right]\to\mathbb{R} \tag{20}\]  
where the inner function \(g\) corresponds to metrics such as position error, rotation error (quaternion angular distance), or categorical cross-entropy in the release signal. Finally, a release error metric was computed trajectory-wise. To do this, the release point of each trajectory was found, the corresponding position found and the velocity vector estimated. When considering ways to combine these two terms into a single quantitative error estimator, it was decided that a simple ballistic extrapolation and resulting miss distance along the ground plane would serve as a decent first-order approximation of throw accuracy _vis-a-vis_ the training or validation data set.