# Non-Parallel Articulatory-to-Acoustic Conversion Using Multiview-Based Time Warping
## 3 Multiview Temporal Alignment  
The problem we address in this work can be formulated as finding the optimal alignment between two time series \(\mathbf{X}=(\mathbf{x}_{1},\ldots,\mathbf{x}_{T_{x}})\) and \(\mathbf{Y}=(\mathbf{y}_{1},\ldots,\mathbf{y}_{T_{y}})\), where the feature vectors \(\mathbf{x}_{i}\) and \(\mathbf{y}_{j}\) may have possibly different dimensionality. In our case, \(\mathbf{X}\in\mathbb{R}^{d_{x}\times T_{x}}\) is a sequence of feature vectors extracted from the captured biosignal (what-ever type the SSI is using) and \(\mathbf{Y}\in\mathbb{R}^{d_{y}\times T_{y}}\) is the sequence of acoustic speech parameters. We further assume that both sequences encode the same phonetic content (i.e., same words in the same order) but have, possibly, different duration. Mathematically, this involves solving the following minimization problem,  
\[\operatorname*{argmin}_{\mathbf{\phi}^{x},\mathbf{\phi}^{y}}\sum_{t=1}^{T}d(\mathbf{x}_{ \phi_{t}^{x}},\mathbf{y}_{\phi_{t}^{y}}), \tag{1}\]  
where \(T=\max(T_{x},T_{y})\), \(d(\mathbf{x},\mathbf{y})\) is a distance function, \(\mathbf{\phi}^{x}\in\{1:T_{x}\}^{T}\) and \(\mathbf{\phi}^{y}\in\{1:T_{y}\}^{T}\) are warping functions that map the indices of the original time series to a common time axis where both series are aligned. This way, \(\mathbf{x}_{i}\) and \(\mathbf{y}_{j}\) will be aligned if \(\phi_{t}^{x}=i\) and \(\phi_{t}^{y}=j\) for a given \(t\). The goal of the alignment algorithm is to find the warping paths \(\langle\mathbf{\phi}^{x},\mathbf{\phi}^{y}\rangle\) that minimize the total sum of distances among all possible alignments between both time series. To enable the alignment of sequences with different dimensionality, we assume that there exists a pair of transformation functions \(\mathbf{f}:\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{z}}\) and \(\mathbf{g}:\mathbb{R}^{d_{y}}\rightarrow\mathbb{R}^{d_{z}}\), modeled as DNNs in this work, that project the original feature vectors into a common, latent space where the data from both views are maximally similar. Thus, the problem in (1) now becomes,  
\[\operatorname*{argmin}_{\mathbf{\phi}^{x},\mathbf{\phi}^{y}}\sum_{t=1}^{T}d(\mathbf{f}( \mathbf{x}_{\phi_{t}^{x}}),\mathbf{g}(\mathbf{y}_{\phi_{t}^{y}})). \tag{2}\]  
For some fixed mapping functions \(\mathbf{z}^{x}=\mathbf{f}(\mathbf{x};\Theta_{f})\) and \(\mathbf{z}^{y}=\mathbf{g}(\mathbf{y};\Theta_{g})\) (\(\Theta_{f}\) and \(\Theta_{g}\) denote the set of parameters for these functions), the problem of temporal alignment of the latent variable sequences \(\mathbf{Z}^{x}=(\mathbf{z}_{1}^{x},\ldots,\mathbf{z}_{T_{x}}^{x})\) and \(\mathbf{Z}^{y}=(\mathbf{z}_{1}^{y},\ldots,\mathbf{z}_{T_{y}}^{y})\) can be solved efficiently by means of the DTW algorithm. Conversely, for fixed warping paths \(\mathbf{\phi}^{x}\) and \(\mathbf{\phi}^{y}\), the problem in (2) involves optimizing the functions \(\mathbf{f}(\cdot)\) and \(\mathbf{\phi}(\cdot)\) to minimize \(\sum_{t=1}^{T}d(\mathbf{f}(\mathbf{x}_{\phi_{t}^{x}}),\mathbf{g}(\mathbf{y}_{\phi_{t}^{y}}))\). If the mapping functions are modeled as DNNs, as in our case, this latter problem can be solved by back-propagation. Thus, TRANSIENCE algorithm solves (2) by alternating between two phases: (i) finding the optimum set of DNN weights \((\Theta_{f},\Theta_{g})\) by fixing the warping paths, and (ii) applying the DTW algorithm to compute the optimum alignments \(\langle\mathbf{\phi}^{x},\mathbf{\phi}^{y}\rangle\) between the sequences of latent vectors by freezing the DNN weights. The warping paths are initialized by uniformly aligning the sequences, i.e., \(\phi_{t}^{x}=1+\left\lfloor\frac{t-1}{t-1}(T_{x}-1)\right\rfloor\) and \(\phi_{t}^{y}=1+\left\lceil\frac{t-1}{t-1}(T_{y}-1)\right\rceil\) for \(t=1,\ldots,T\) and \(T=\max(T_{x},T_{y})\).