## 3 Basic Methodology
### Hierarchical Dirichlet Process Vector Auto-Regressive Hidden Markov Models  
HMMs segment sequential data into interpretable hidden states that are assumed to be Markovian. They are limited by its _a priori_ determination of hidden states (and fixed throughout the modeling process) as well as the HMMs limited ability to model complex dynamics due to the assumption that observations are conditionally independent given the state. Instead, this work uses Bayesian non-parametric techniques that model HMMs through priors that can model an unbounded number of hidden states. In particular, we use HDP priors to learn HMMs with infinite hidden states. We also use the switching vector auto-regressive (HDP-VAR-HMM) to model multimodal observation sequences and its element consists of features extracted from an F/T sensor, joint encoders, and a tactile sensor (for a discussion on the used features see Section 6.2). Figure 3 illustrates graphical model representations for the sHDP-HMM and the sHDP-VAR-HMM. Assume that we record \(N\) sequential executions for each skill and expect to learn the common dynamics among them. We use the HDP-VAR-HMM to jointly model those sequences, where sequence \(n\in N\) has multidimensional observation data \(y_{n}=[y_{1},y_{2},\ldots,y_{T}]\) at time \(t\). For instance, \(y_{t}\in\mathbb{R}^{d}\) could be an instant of wrench signatures and end-effector velocity, where \(d\) is the dimensionality of observed features. The HDP-VAR-HMM interprets each observation \(y_{t}\) by assigning it a single hidden state \(z_{t}\). The hidden-state value is derived from: (i) a countably infinite set of consecutive integers \(z_{t}=k\in\{1,2,3,\ldots\}\); (ii) an initial probabilistic state distributions \(\pi_{0}\) generated with Markovian dynamics; and (iii) a transition distribution \(\{\pi\}_{k=1}^{\infty}\). Then, the Markovian structure on the state sequence dictates that for all \(t>1\):  
\[ \begin{split}& z_{t}\sim\pi_{z_{t-1}},\\ & p(z_{1}=k)=\pi_{0},\\ & p(z_{t}=k^{{}^{\prime}}|z_{t-1}=k)=\pi_{kk^{\prime}}.\end{split}  \tag{5}\]Since \(z_{t}\sim\pi_{z_{t-1}}\), we see that \(z_{t-1}\) indexes all observations \(y_{t}\) are assigned with \(z_{t-1}\). We can draw the observation \(y_{t}\) given specific hidden state \(z_{t}=k\) from its corresponding observation likelihood functions. In our case, we consider the first-order auto-regressive Gaussian likelihood. As such, observations \(y_{t}\) are considered to be a noisy linear combination of the previous observation plus additive white noise \(\epsilon\) and can be expressed as:  
\[y_{t}=A_{k}y_{t-1}+\epsilon_{t}(z_{t}=k),\quad\epsilon_{t}(k)\sim\mathcal{N}( 0,\Sigma_{k}). \tag{6}\]  
where each state \(k\) has two dynamic parameters \(\{A_{k},\Sigma_{k}\}\): \(A_{k}\), a \(d\times d\) matrix of regression coefficients that defines the expected value of each successive observation as \(\mathbb{E}[y_{t}|y_{t-1}]=A_{k}y_{t-1}\), and \(\Sigma_{k}\), a \(d\times d\) symmetric positive definite matrix that defines the covariance matrix of state \(k\). Therefore, the Gaussian regression observation model explains observed data pairs of input \(y\) and output \(z\). Each input is a vector of length \(y\in\mathbb{R}^{d}\), while each output is a scalar. In this paper, we assume that the input data \(y\) are fixed dependence on previous observation and focus on a generative model for learning the output data \(z\) which depends on \(y\). Various generative models, such as full-covariance Gaussian and diagonal-covariance Gaussian, are possible for the observed input data. These can directly define the multivariate Gaussian log-likelihood function given specific state \(z_{k}\),  
\[log\ p(y_{t}|\theta_{k},y_{t-1}) =log\ p(y_{t}|A_{k},\Sigma_{k},y_{t-1})=log\ \mathcal{N}(y_{t}|A_{k}y_{t-1},\Sigma_{k}) \tag{7}\] \[=-\frac{d}{2}log(2\pi)-\frac{1}{2}log|\Sigma_{k}|-\frac{1}{2}(y_{ t}-A_{k}y_{t-1})^{T}\Sigma_{k}^{-1}(y_{t}-A_{k}y_{t-1}).\]  
However, it is often the case that we do not know the parameters of this distribution, but instead estimate them.