# Exploring Efficient Neural Architectures for Linguistic-Acoustic Mapping in Text-To-Speech +
## 4 Self-Attention Linguistic-Acoustic Decoder  
Based on the Transformer architecture [33], we also consider a pseudo-sequential processing network that can leverage distant element interactions within the input linguistic sequence to predict acoustic features [34]. This is similar to what an RNN does, but discarding any recurrent connection. This allows us to process all input elements in parallel at inference, hence substantially accelerating the acoustic predictions. In our setup, we do not face a sequence-to-sequence problem as stated previously, so we only use a structure similar to the Transformer encoder, which we call a linguistic-acoustic decoder.  
The SALAD architecture begins with the same embeddings of linguistic and prosodic features from Section 2, followed by a positioning encoding system. As we have no recurrent structure, and hence no processing order, this positioning encoding system allows the upper parts of the network to locate their operating point in time, such that the network knows where it is inside the input sequence [33]. This positioning code \(\mathbf{c}\in\mathbb{R}^{H}\) is a combination of harmonic signals of varying frequency:  
\[c_{t,2i}=\sin\left(t/10000^{\frac{2i}{H}}\right) \tag{5}\] \[c_{t,2i+1}=\cos\left(t/10000^{\frac{2i}{H}}\right),\]  
where \(i\) represents each dimension within \(H\). At each time-step \(t\), we have a unique combination of signals that serves as a time stamp. We can expect this to generalize better to long sequences than having an incremental counter that marks the position relative to the beginning, owing to the cyclic nature of sine functions. Each time stamp \(\mathbf{c}_{t}\) is added to each embedding \(\mathbf{h}_{t}\), and this is input to the decoder core, which can leverage this ordering information even though it does not have recurrent connections.  
The decoder core is built with a stack of \(N\) blocks, depicted within the dashed blue rectangle in Figure 1c. These blocks are the same as the ones proposed in the encoder of [PERSON] et al. [33]. The most salient part of this type of block is the multi-head attention (MHA) layer. This applies \(h\) parallel self-attention layers, which can have a more versatile feature extraction than a single attention layer with the possibility of smoothing intra-sequential interactions. After the MHA comes the feed-forward network, composed of two fully-connected layers. The first layer expands the attended features into a higher dimension \(d_{\text{ff}}\), and this gets projected again to the embedding dimensionality \(H\). Finally, the output layer is a fully-connected dimension adapter such that it can convert the hidden dimensions \(H\) to the desired amount of acoustic outputs. In this case, we may slightly degrade the quality of predictions with this output topology, as recurrence helps in the output layer capturing better the dynamics of acoustic features. Nonetheless, this may suffice for our objective of having a highly parallelizable and competitive system.  
Note that, in this model, contrary to QLAD, we do not even have a state vector \(\mathbf{h}_{t}\) that carries dynamic features over time. As such, it can operate fully parallel during training and inference at all stages. Nonetheless, self-attention modules require computing large matrix dot products which depend on the sequence length. This may impose a limit in the inference speed and even a performance mismatch with the QRNN structures at a certain sequence length.