# A Motion Capture and Imitation Learning Based Approach to Robot Control
## 5 Implementation
### Models  
We studied two classes of parametric models as part of this project--simple feedforward neural networks and RNNs, operating autoregressively. The choice was motivated by two factors:  
* The dynamics of the problem were deemed to be simple enough that even small models would be able to model them adequately--making it possible to quickly train on development machines with low parameter counts, using pre-existing code libraries;
* Given the recent advances in employing sequence-to-sequence models for imitation learning tasks, it was decided that models with broadly similar footprints and characteristics should be used to enable further research in this direction.  
After an initial hyperparameter discovery process, the values in Table 1 were arrived at for both model types, respectively. A range of model sizes and training epoch counts was compared for both models. In the case of the recurrent neural network, performance with different learning rates was also evaluated. As the feedforward models were developed first, performance with and without a time signal in the input data was also compared. For the RNN architecture, a _gated recurrent unit_ (GRU) was selected, as prior research suggests that it outperforms _long short-term memory_ (LSTM) when dealing with small data sets of long sequences [28].  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Parameter** & **Feedforward** & **RNN** \\ \hline Architecture & 2 dense hidden layers, ReLU & GRU, dense linear output \\ Parameter counts & 128–1024 perceptrons per layer & 128–512 perceptrons in the unit \\ Training epochs & 20–100 & 300–1200 \\ Batch size & 64 & 32 \\ Optimizer & Adam & Adam \\ Learning rate & \(10^{-4}\) & \(10^{-3},10^{-4}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Model hyperparameters.  
Figure 4: Estimation of the throw target coordinates. Bottle position data are used to annotate each demonstration—consisting of effector pose observations—with throw target coordinates. Regression on the \(z\)-axis is used to find the ground plane intersection time, at which extrapolated \(x\)- and \(y\)-coordinate values are found.  
The basic model footprint is given as follows:  
\[(\mathbf{r}_{t+1},\mathbf{q}_{t+1},g_{t+1})=\pi_{\Theta}\left(\frac{t}{f_{sample} },\mathbf{r}_{t},\mathbf{q}_{t},g_{t},\mathbf{r}_{t}^{target}\right) \tag{16}\]  
where \(\mathbf{r}_{t},\mathbf{q}_{t},g_{t}\) represent the end effector translation vector, orientation quaternion and gripper actuator signal, respectively, at time step \(t\) in both the input and output. The input is augmented with a time/phase signal (discrete time step divided by the sampling frequency, in this case 100 Hz) relative to the start of the demonstration or generated trajectory, as well as the target coordinate vector \(\mathbf{r}_{t}^{target}\), which corresponds to the extrapolated target coordinates in the demonstration data set and commanded throw coordinates at inference. The time signal was added to the input, as it was found that trajectories generated by feedforward networks were liable to diverge without it, and the data sets thus modified were used for all training thereafter. The gripper control signal has values \(\{0,1\}\) in the training data set.  
For training both types of networks, state transitions \(\left(\left(t,\mathbf{s}_{t},\mathbf{r}_{t}^{target}\right),\mathbf{s}_{t+1}\right)\) are constructed. In the case of the feedforward network, these state transitions are then shuffled and batched independently. For the RNN, pairs of feature-label sequences are formed corresponding to complete demonstrations, and the loss function is computed on the entire predicted output sequence. To hold these variable-length sequences, a ragged tensor is used, which is batched along its first axis and ragged along the second. With both types of networks, the Huber loss function is utilized.