# Non-Parallel Articulatory-to-Acoustic Conversion Using Multiview-Based Time Warping
## 1 Introduction  
Silent speech interfaces (SSIs) are devices that enable speech communication in the absence of audible speech by decoding speech from other nonacoustic, speech-related biosignals generated by the human body during the process of speech production [1, 2, 3]. These biosignals can range from the neural activity in the speech and language areas of the brain [4, 5, 6], electrical activity driving the facial muscles captured by surface electrodes (i.e., electromyography (EMG)) [7, 8, 9], or motion capture of the speech articulators by means of imaging techniques [10] or electromagnetic articulography techniques [11, 12, 13, 14]. SSIs have many potential applications, as summarized in [1]. For instance, an SSI could be used to enhance speech communication in noisy environments as the above-mentioned biosignals are mostly immune to acoustic noise. Also, SSIs might be used to enable covert communication to preserve privacy when speaking in public spaces. Yet another potential application, the one that motivates this work, is to restore oral communication to persons with speech impairments [3]. Thus, an SSI could be used to decode the words its user wants to say from the captured biosignals. Because SSIs do not rely on the acoustic speech signal, they offer a radically new form of restoring oral communication to people with speech impairments. To decode speech from the biosignals, two alternative SSI approaches were proposed [3]: silent speech recognition, which involves the use of automatic speech recognition (ASR) algorithms to transform the biosignals into text, followed by text-to-speech synthesis and direct speech synthesis, which transforms the biosignals into a set of acoustic parameters amenable to speech synthesis. In this work, we focus on the latter approach, which is also known as articulatory-to-acoustic (A2A) synthesis when the biosignals encode information about the movements of the speech organs. The most successful direct synthesis techniques so far adopted a data-driven framework, in which supervised machine learning is used to model the mapping \(\mathbf{y}=\mathbf{h}(\mathbf{x})\) between source feature vectors \(\mathbf{x}\) extracted from the biosignals and target feature vectors \(\mathbf{y}\) computed from the speech signals, as shown in Figure 1. To train this function, a dataset \(\mathcal{D}=\{(\mathbf{X}_{1},\mathbf{Y}_{1}),\ldots,(\mathbf{X}_{M},\mathbf{Y}_{M})\}\) with pairs of sequences of feature vectors \((\mathbf{X}_{i},\mathbf{Y}_{i})\) extracted from time-synchronous recordings is used, where \(\mathbf{X}_{i}\in\mathbb{R}^{d_{x}\times T_{i}}\) and \(\mathbf{Y}_{i}\in\mathbb{R}^{d_{y}\times T_{i}}\) (\(M\) is the number of parallel sequences in the dataset, \(d_{x}\) and \(d_{y}\) are the dimensionality of the source and target vector, respectively, and \(T_{i}\) is the length of the \(i\)-th sequence pair). The need for parallel recordings, however, limits the application of direct synthesis techniques to only a few clinical scenarios, as described in [3]. For instance, people who already lost their voices could not use this technology because of the impossibility of recording parallel data to train the machine learning model, as shown in Figure 1.  
A solution to this problem, which we explore in this work, involves the use of previously available speech recordings from a voice donor (e.g., a relative or recordings of the patient's own voice made before the voice loss). Using these recordings, it would be possible (in principle) to obtain the necessary parallel data by asking the patient to repeat the speech recordings in silence while the necessary biosignals are captured, which is similar to karaoke. However, even in this case, it is likely that the captured biosignals will not be perfectly aligned with the speech recordings, since both modalities would have slightly different duration, thus preventing the application of standard, supervised machine learning techniques.