# Non-Parallel Articulatory-to-Acoustic Conversion Using Multiview-Based Time Warping
## 1 Introduction
Therefore, our problem now becomes concerned with aligning sequences obtained by sampling the same physical process with different sensors (e.g., speech and biosignal data). Once the sequences are aligned, standard machine learning techniques could be readily applied to model the relationship between the speech and biosignal data. The problem stated above is similar to what happens in voice conversion (VC) [15; 16; 17], where the goal is to modify the spectral content (and possibly the prosody, too) of the speech of a source speaker in a way that it sounds as it was said by a different speaker (target speaker). To perform this transformation, a machine learning model (e.g., a deep neural network (DNN)) is trained with pairs of speech signals recorded from the source and target speakers while reading aloud a set of sentences. Again, a time-alignment procedure is  
Figure 1: Block diagram of a typical (data-driven) direct synthesis technique. In training stage, a machine learning model is estimated, which represents relationship between feature vectors extracted from speech and nonspech signals. This model is latter used in conversion stage to predict speech (acoustic) parameters from biosignals. necessary to align the source and target speakers' signals because they might have different duration. To this end, the well-known DTW algorithm [18] is normally used. The problem we focus on this work is, in a way, similar to the one described above for VC, that is, aligning feature sequences with the same phonetic content but possibly different duration. However, in contrast to VC, the direct application of DTW might not be possible in our case because the feature vectors extracted from the speech and bio- signals may have different dimensionality (i.e., \(d_{x}\
eq d_{y}\), in general), while the standard DTW algorithm requires that \(d_{x}=d_{y}\). To address this issue, in this work we propose an extension of the DTW algorithm called multiview temporal alignment by dependence maximisation in the latent space (TRANSIENCE) (Code is available at [[https://github.com/joseangl/transience](https://github.com/joseangl/transience)]([https://github.com/joseangl/transience](https://github.com/joseangl/transience)); accessed on 30 December 2021), which is based on the theory of multiview learning [19; 20]. TRANSIENCE attempts to find the optimal temporal alignment between sequences from different views (e.g., speech and biosignal data) by first projecting the feature vectors from the sequences into a common, latent subspace where the resulting embeddings have the same dimensionality and are maximally similar (we will define later what we mean by similarity). After that, the sequences of embedding vectors obtained for the different views are aligned by means of the DTW algorithm. The remainder of this paper is organized as follows. First, in Section 2, the relevant related work is reviewed. The details of the proposed temporal alignment technique are presented in Section 3. Section 4 describes the experimental setup employed to evaluate the performance of the proposed technique. In particular, the proposed algorithm is evaluated on an A2A task involving the conversion of articulatory data captured using permanent magnet articulography (PMA) [21; 22] to speech. Experimental results obtained on this task are shown in Section 5. We conclude in Section 6 and discuss potential future research directions.