# A Motion Capture and Imitation Learning Based Approach to Robot Control
## 2 Preliminaries  
In this article, an _agent_ can be taken to mean the part of the system that acts based on state or observation vectors \(\mathbf{s}\) in an _environment_, according to a _policy_\(\pi\) which is specified by a parametric _model_--a function \(\pi_{\mathbf{0}}(\mathbf{s})\) with parameters \(\mathbf{\theta}\) tuned in optimization and produces an output _action_\(a\). For all practical intents and purposes, this means that references to the agent, model, and policy are almost interchangeable in most contexts. A formalism common in both reinforcement and imitation learning contexts is the _Markov decision process_ (MDP), formally given by [5]  
\[MDP=(S,A,T,R,I) \tag{1}\]  
where \(S\) is the set of states \(s\), \(A\) is the (formally discrete) set of actions \(a\), \(T:S\times A\to S\) is a state transition function that encapsulates the environment, \(R:S\rightarrow\mathbb{R}\) is a reward function associated with each state and \(I=p(\mathbf{s}_{0}\in S)\) represents the initial state distribution. In many imitation-learning-related cases, a reward function need not be specified or considered. Moreover, if one is willing to break with the strict formal definition and give up the use of mathematical tools defined only on finite probability distributions, continuous action spaces can also be considered. One important characteristic of the MDP is that it is history agnostic--the future state distribution of the system is uniquely defined by its current state. While technically true for physical environments, it is often the case that instead of a complete state representation vector \(\mathbf{s}\), we are instead operating on a more limited _observation_\(\mathbf{o}\) (often interchangeably referred to as \(\mathbf{s}\) for brevity), where each element \(o_{i}\) is given by some function \(f_{i}(\mathbf{s})\). It is therefore possible that historical observations contain information about hidden system state variables, even assuming that the MDP formalism holds for the underlying environment. An example of this is the case when individual observations contain only the current position of an object, but not its derivatives, as in video data. In such situations, it may prove beneficial to break with the formalism further by redefining the policy to operate on sequences of \(k+1\) previous states/observations  
\[t,k,n\in\mathbb{N};\pi_{\Theta}:\{(\mathbf{s}_{n})_{t-k}^{t}\}\to A \tag{2}\]  
which, by allowing for input sequences of variable length, may become a function defined on the entire known state history:  
\[\pi_{\Theta}:\{(\mathbf{s}_{n})_{1}^{t}\}\to A \tag{3}\]  
These adjustments allow for the employment of sequence-to-sequence predictor architectures also studied in other areas of machine learning, such as recurrent neural networks (RNNs) [9] and transformers [10]. Finally, if continuous action spaces are permitted, it is no great stretch to also consider formats where the action corresponds to a predicted future state to be used for static motion planning or in a feedback controller  
\[\pi_{\Theta}:\{(\mathbf{s}_{n})_{1}^{t}\}\to S \tag{4}\]  
which, when running the model on its prior output, is equivalent to sequence-building tasks encountered in domains such as text generation. If the sequence of states contains a sequence of joint state vectors (a _joint space trajectory_) in the form  
\[((\mathbf{y}_{1},t_{1}),...,(\mathbf{y}_{n},t_{n})) \tag{5}\]  
where each _waypoint_\((\mathbf{y}_{i},t_{i})\) consists of a joint space goal \(\mathbf{y}\) and corresponding timestamp \(t\), it could in principle be used to directly produce control inputs  
\[\mathbf{y}(\tau)=\text{interp}(\mathbf{y}_{t_{k}}\mathbf{y}_{t_{l}},\tau) \tag{6}\]  
where \(\mathbf{y}(\tau)\) denotes the interpolated, time-varying joint state setpoint to be used in a feedback controller and \(t_{k},t_{l}\) are timestamps such that \(t_{1},t_{2},...,t_{k}\leq\tau\leq t_{l},t_{l+1}...,t_{n}\).