# Exploring Efficient Neural Architectures for Linguistic-Acoustic Mapping in Text-To-Speech +
## 3 QRNN Linguistic-Acoustic Decoder  
The QRNN was proposed as an efficient alternative to classic RNN cells, such as LSTMs, by removing the affine projections from recurrent connections. The equation that drives classic RNNs is  
\[\mathbf{h}_{t}=\tanh(\mathbf{W}\mathbf{x}_{t}+\mathbf{U}\mathbf{h}_{t-1}+\mathbf{b}) \tag{2}\]  
where matrix \(\mathbf{U}\in\mathbb{R}^{H}\) is used to project each previous time-step state \(\mathbf{h}_{t-1}\) and to mix these features with current new inputs \(\mathbf{x}_{t}\), thus obtaining the current state \(\mathbf{h}_{t}\). This temporal dependency with \(\mathbf{U}\) connections dramatically limit parallelism and make RNNs unwieldy for very long sequences, where \(\mathbf{U}\) matrix projects up to \(T\) times each \(\mathbf{h}_{t}\), being \(T\) the total sequence duration. In contrast to this, QRNNs alternate convolutional layers, applied in parallel across timesteps, with minimalist recurrent pooling functions that apply in parallel across channels. Despite having no trainable recurrent connections (i.e., removing \(\mathbf{U}\)), QRNNs proved to be very effective in different NLP tasks, in comparison to their LSTM counterpart, when enough QRNN layers are stacked [29].  
The first QRNN stage operates with a series of convolutional blocks, which can process all input time-steps in parallel. As we apply the QRNN after the embedding layer introduced in Section 2, we have the tensor \(\mathbf{X}=[\mathbf{h}_{1},\cdots,\mathbf{h}_{T}]\in\mathbb{R}^{T\times H}\) containing the embeddings at all time-steps. We formulate the convolutional projections as  
\[\begin{split}\mathbf{Z}&=\tanh(\mathbf{W}_{z}* \mathbf{X})\\ \mathbf{F}&=\sigma(\mathbf{W}_{f}*\mathbf{X})\\ \mathbf{O}&=\sigma(\mathbf{W}_{o}*\mathbf{X}), \end{split} \tag{3}\]  
where \(\mathbf{W}_{z}\), \(\mathbf{W}_{f}\) and \(\mathbf{W}_{o}\) are convolutional filter banks that predict the cell activation, the forget gate activation, and the output gate activation, respectively. The sizes of these filter banks are \(\mathbb{R}^{K\times H\times M}\), being \(K\) each kernel width, \(H\) the number of input channels, and \(M\) the number of convolutional kernels. Note that \(\mathbf{Z}\), \(\mathbf{F}\), and \(\mathbf{O}\) are of dimension \(T\times M\), thus all \(T\) time-steps are processed in parallel  
Figure 1: Comparison of architectures for: (**a**) RNN/LSTM acoustic model; (**b**) QLAD; and (**c**) SALAD acoustic model. The embedding projections are the same. The QRNN block is stacked \(P\) times in the hidden layers. In SALAD, the positioning encoding introduces sequential information. The SALAD decoder block is stacked \(N\) times to form the hidden structure. FFN, Feed-forward Network; MHA, Multi-Head Attention; CConv1D, Causal 1D convolution.  
by each filter bank. After obtaining these activations, we can pass them to the aforementioned recurrent pooling, defined as  
\[\mathbf{c}_{t} =\mathbf{f}_{t}\odot\mathbf{c}_{t-1}+(1-\mathbf{f}_{t})\odot \mathbf{z}_{t} \tag{4}\] \[\mathbf{h}_{t} =\mathbf{o}_{t}\odot\mathbf{c}_{t},\]  
where \(\odot\) denotes element-wise multiplication. Note that each vector, indexed by \(t\), is a time-slice of each of the previously defined activation tensors \(\mathbf{Z}\), \(\mathbf{F}\), and \(\mathbf{O}\). These operations are similar to the ones defined in RNNs, but note how we have no affine transformation between time-steps in this case. Hence, the key to QRNN efficiency is that all learnable parameters are conveyed in the parallel computation phase. The described operations are depicted in Figure 1b. The QRNN layer is contained within the green dashed line, and we stack up to \(P\) different layers. Finally, the output QRNN, analogous to the RNN version, is also stacked. This architecture can thus be seen as an intermediate step between full recurrence and no recurrence at all, and we name it QLAD.