# Temporal Convolution Network Based Joint Optimization of Acoustic-to-Articulatory Inversion
## 2 Methods
### Temporal Convolution Network  
The Temporal Convolution Network (TCN) was first introduced in [25]. This architecture uses dilated convolution layers to transform the input sequence information to the synchronous output sequence. When dealing with the speech signals, we first get the input speech spectrum \(\mathbf{X}:\{x_{0},x_{1},\cdots,x_{t}\}\) as a t-frame sequence where \(x_{t}\in\mathbb{R}^{f}\) represents the feature vector at the \(t\)th frame. And we wish to get an output \(\mathbf{X}^{\prime}:\{x^{\prime}_{0},x^{\prime}_{1},\cdots,x^{\prime}_{t}\}\) as the frame synchronous \(c\)-dimension embedding \(x^{\prime}_{t}\in\mathbb{R}^{c}\) for every single frame. The sequence model network is any function \(f\colon X\Rightarrow X^{\prime}\) that produces the mapping as Equation (1).  
\[x^{\prime}_{0},x^{\prime}_{1},\cdots,x^{\prime}_{t}=f(x_{0},x_{1},\cdots,x_{t}) \tag{1}\]  
A TCN block is comprised of dilated convolution layers and a residual connection. The dilated convolution layers are the key factors in a TCN block to produce the same length output sequence as the input, using the sequence's contextual information. Figure 1 illustrates an example for a 3-stack dilated convolution layers with a kernel size as 3 and the dilation as 2. The kernel size determines the size of the convolution, and the dilation indicates refers to the number of intervals between the points of the convolution kernel. These two parameters, alongside the layer depth n, determines the reception field of the sequence model. The larger the reception field is, the more contextual information can be seen when conducting the convolution at the time-step t. The reception field is computed as Equation (2):  
\[\omega=1+(k-1)\times\frac{d^{n}-1}{d-1} \tag{2}\]where \(\omega\) is the reception field, \(k\) is the kernel size, \(d\) is the dilation base and n represents the layer number.  
Since the reception field is positively correlated with the depth of the convolution layers, we need to train a deeper model. The residual connect is proposed so that a relatively deep network can be trained [26]. A residual connect contains an additional output branch leading to the model \(\mathcal{F}^{\prime}s\) final output unit, converting the output \(o\) to:  
\[o=Activation(x+\mathcal{F}(x)) \tag{3}\]  
As shown in Figure 2, we stack two dilated convolution layers with Rectifier Linear Unit (ReLU) [27] as the activation function. Weight normalization and dropout techniques are introduced into the construction of our final TCN block in order to avoid over-fitting.  
In addition to the TCN layer, the entire model consists of two linear layers to first transform the acoustic feature \(\mathbf{X}\in\mathbb{R}^{t\times f}\) into hidden space with a feature sequence \(\mathbf{X}^{D}\in\mathbb{R}^{t\times h}\), a feed-forward readout layer to get the final 16-dim articulatory trajectories, and finally a post-filter layer for smoothing the output sequences. The Figure 3 illustrate the stacks of the model architecture.  
The post-filter is implemented with reference to [28] by employing a \(1\times 1\) fixed-param convolutional layer. A 5-order low-pass filter is used to smooth the output trajectories in the conv1d layer. This can avoid dispensable computation of error back-propagation due to the jitter of the unsmoothed predicted trajectories during the training stage. The target EMA sensor recordings are smoothed by the same filter at the data preprocessing stage as well to avoid target mismatch.  
Figure 1: The visualization of a stack of dilated convolution layers which transform the input sequence \(\mathbf{x}\) into \(\mathbf{x}^{\prime}\) with kernel size = 3. The dilation_base is set to be 2. 3 stacks of dilated conv-layers acquire a reception field of 15 frames.