## 2 Methods
### Network Model
#### 2.3.2 Attention Mechanism-Based Decoder  
Based on the hidden representation \(h\) of the encoder explained in the section above, we have redesigned the decoder that can predict the output cepstral features.  
In the conventional Seq2 Seq framework, the decoder adopts the output cell state \(c_{t}\) of the last hidden layer as its context vector. However, for esophageal speech conversion, the length of the normal feature sequence is always shorter than that of the esophageal feature sequence. In this kind of sequence, we can observe a lot of singularities, i.e., different ES phonemes can be linked to more than one phoneme at different positions of the target speech.  
To cope with this problem, an attention mechanism is adopted, which more efficiently models these different nonlinear relationships. Therefore, this attention mechanism allows  
Figure 3: Internal structure of the LSTM unit [32].  
for greater flexibility and corrects the alignment by adaptively estimating the decoder output through the constructed self-adaptive context vector.  
In our proposed framework, we suggest that the current state of the decoder is fully connected to all hidden states of the encoder, and each hidden state has a different effect on the estimation of the decoder's current state. Therefore, to estimate the current self-adaptive context of the decoder, all backward and forward hidden states of the encoder are considered simultaneously.  
Our designed decoder is illustrated in Figure 4, with step = 1 for clarity.  
At each \(i\)th decoder step, the attention output (context vector) \(c_{i}\) is computed using a weighted linear combination of the encoder's hidden states, as shown in Figure 4.  
\[c_{i}=\sum_{j=1}^{j=N}\alpha_{ij}h_{j}, \tag{13}\]  
where \(h_{j}\) represents the encoder's hidden state at position \(j\), \(\alpha_{ij}\) are the attention weights as shown by Formula (14).  
\[\alpha_{ij}=\frac{\exp\left(score_{ij}\right)}{\sum_{j=1}^{N}\exp\left(score_{ ij}\right)} \tag{14}\]  
\(score_{ij}\) is the attention score, which calculates the similarity between the encoder's hidden state and the previous cell state of the decoder, computed according to Formula (15).  
\[score_{ij}=att\ ((s_{i-1},h_{j});\theta_{attention}) \tag{15}\]  
where \(att\) is a feed-forward neural network (FFN) that produces the alignment scores between \(h_{j}\) and the previous state of the decoder's output \(s_{i-1}\). \(\theta_{attention}\) are the trainable parameters of the model.  
Once the context vector \(c_{i}\) is produced, the decoder model takes this context vector, the previous decoder hidden state \(s_{i-1}\), and the input \(\hat{y}_{i}\) to generate the current decoder hidden state \(s_{i}\).  
\[s_{i}=LSTM\ ((s_{i-1},\hat{y}_{i},c_{i});\theta_{decoder}) \tag{16}\]  
The concatenation of the last context vector \(c_{i}\) and the output of decoding LSTMs \(s_{i}\) is linearly projected to produce the cepstrum output of the decoder network.  
Thus, the decoder output is calculated according to the previous Equation (16) as:  
\[\hat{y}_{i+1}=fc\ (s_{i},c_{i}) \tag{17}\]  
where \(fc\) is a fully connected FFN that allows mapping of the hidden dimension to the output dimension.  
Figure 4: Illustration of attention layer, where \(a_{i,j}\) = attention weight, \(c_{i}\) = context vector, \(s_{i}\) = attention target state, and \(\bigoplus\) = concatenation.