# Exploring Efficient Neural Architectures for Linguistic-Acoustic Mapping in Text-To-Speech +
## 5 Experimental Setup
### Model Details and Training Setup  
We had three main structures: the baseline MUSA-RNN, QLAD, and SALAD. The RNN took the form of an LSTM network for their known advantages of avoiding typical vanilla RNN pitfalls in terms of vanishing memory and bad gradient flows. Each of the three different models had two configurations, small (Small RNN/Small QLAD/Small SALAD) and big (Big RNN/Big QLAD/Big SALAD). This was intended to show the performance difference with regard to speed and distortion between the proposed model and the baseline, but also their variability with respect to their capacity (RNN, QLAD, and SALAD models of the same capacity have an equivalent number of parameters although they have different connection topologies). Figure 1 depicts all these models' structure, where only the size of their layers (LSTM, embedding, MHA, FFN and CConv1D) changes with the mentioned magnitude. Table 1 summarizes the different layer sizes for all types of models and magnitudes. Additionally, all models have dropout [40] in certain parts of their structure to avoid over-fitting easily to the available training set per speaker. The RNN and QLAD models have it after the hidden LSTM or QRNN blocks. The SALAD model has many dropouts in different parts of its submodules, replicating the ones proposed in the original Transformer encoder [33]. The RNN and QLAD dropouts are 0.5, and SALAD has a dropout of 0.1 in its attention components and 0.5 or 0.2 in FFN and after the positioning codes, depending on the model performance based on a validation criterion. Note that both QLAD and SALAD models also can have \(P\) and \(N\) number of blocks, respectively, which turned out to be \(P=N=3\) for all performed experiments. All models were trained with batches of 32 sequences of 120 symbols. The training consisted of stateful-structured batches, such that we carried the sequential state between batches over time (that is, the memory state in the RNN or QRNN and the position code index in SALAD). To achieve this, we concatenated all training frames into a very long sequence, and then chopped it into 32 long pieces. We then used a non-overlapped sliding window of size 120, so that each batch contained a piece per sequence, continuous with the previous batch. This made the models learn how to deal with sequences longer than 120 outside of train, learning to use a conditioning state different from zero in training. All models were trained for a maximum of 300 epochs, but they triggered a break by early-stopping with the validation data. The validation criterion for which they stop was the mel cepstral distortion (MCD, as discussed in Section 6) with a patience of 20 epochs. Regarding the optimizers, we used Adam [41] for the RNN and QLAD models, with the default parameters in PyTorch (\(\
abla=0.001\), \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), and \(\epsilon=10^{-8}\)). For SALAD, we used a variant of Adam with adaptive learning rate, already proposed in the Transformer work, called Noam [33]. This optimizer is based on Adam with \(\beta_{1}=0.9\), \(\beta_{2}=0.98\), \(\epsilon=10^{-9}\) and a learning rate scheduled as  
\[\
abla=H^{-0.5}\cdot\text{min}(s^{-0.5},s\cdot w^{-1.5}) \tag{6}\]  
where we have an increasing learning rate for \(w\) warmup training batches, decreasing afterwards proportionally to the inverse square root of the step number \(s\) (number of batches). We used \(w=4000\) \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Emb** & **HidRNN** & \(\mathbf{d_{\text{ff}}}\) \\ \hline Small RNN & 128 & 450 & - \\ Small QLAD & 128 & 360 & - \\ Small SALAD & 128 & - & 1024 \\ Big RNN & 512 & 1300 & - \\ Big QLAD & 512 & 1150 & - \\ Big SALAD & 512 & - & 2048 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Different layer sizes of the different models. Emb, linear embedding layer, and hidden size \(H\) for SALAD models in all layers but FFN ones; HidRNN, Hidden LSTM or QRNN layer size; \(d_{\text{ff}}\), dimension of the feed-forward hidden layer inside the FFN. in all experiments.