# Evaluation of Tacotron Based Synthesizers for Spanish and Basque
## 2 Materials and Methods
### Tacotron 2  
To represent the relation between the audio signal and the input text, an acoustic model is required. In this work, we used Tacotron 2 as acoustic model. Tacotron 2 is a model that uses a sequence of characters as input and transforms it into a mel-scale spectrogram. Different open-source implementations of this model are publicly available. In this work, we used the open-source implementation provided by NVIDIA ([[https://github.com/NVIDIA/tacotron2](https://github.com/NVIDIA/tacotron2)]([https://github.com/NVIDIA/tacotron2](https://github.com/NVIDIA/tacotron2)) (accessed on 20 December 2021)) as the basis for our system. The architecture of this model is comprised of an encoder, an attention mechanism and a final decoder. To allow a better understanding on how this model works, a brief description of each module is provided:  
* The encoder handles and transforms the input of the acoustic model. In the encoder the input sentence is converted into a hidden feature representation (that is later consumed by the decoder). First, each character of the input sequence is represented by a 512-dimensional vector using character embedding. The sequence of vectors is then fed into a three-convolutional-layer stack that models the long-term relationships between the input characters. Finally, using the output obtained from the last convolutional layer, a bidirectional Long short-term memory (LSTM) network generates the encoder hidden output (a vector of 512 \(\times\) T, being T the length of the input sequence).  
Figure 2: Distribution of words per utterance in the testing dataset.  
* The attention mechanism consumes the output of the encoder to produce a context vector at each decoding step. This attention mechanism is one of the most important parts of the model. It is at this point where the alignment between the input text and the frame-level acoustic features is learnt. The context vector provides the decoder with the necessary information to refer to the corresponding part of the encoder sequence at each decoding step. Tacotron 2 uses a custom location-sensitive attention mechanism [25], with an additional feature that is computed from the cumulative attention weights of the previous decoding steps.
* The decoder of Tacotron 2 is a recurrent neural network that predicts one frame at each decoding step in an auto-regressive fashion. During training, the decoder makes use of the context vector and the previous ground truth frame to compute the current step frame. This way of training the network is referred to as \"teacher-forcing\", and it is employed to ensure that each predicted frame is correctly aligned with the features of the target audio. During inference, as the ground truth frames are not available, the decoder uses the frame computed in the previous decoding step. The architecture of the decoder consists on a 2 layered pre-net, a 2 layer LSTM network and a convolutional post-net. The prediction from each decoding step is passed through the pre-net and then concatenated to the context vector. The resulting concatenation is fed into the two-layer LSTM. This output is again concatenated to the context vector and then passed through two different projection layers: one that predicts the stop token, and another one that predicts the target spectrogram frame. The final mel spectrogram is a combination of the whole spectrogram and a residual obtained in an ending convolutional post-net.