# Sequence-to-Sequence Acoustic Modeling with Semi-Stepwise Monotonic Attention for Speech Synthesis
## 2 Related Work  
The Tacotron [6] model unified acoustic modeling and duration modeling in a single model, and adopted a simple additive attention mechanism [12] to calculate the attention weights by query and keys. Further, Tacotron2 [7] used a location-sensitive attention mechanism [13], but there were still alignment errors, especially for out-of-domain texts. The alignment errors led to robustness problems in the predicted acoustic features, such as repeating, skipping, and failing to stop. To alleviate these problems, a number of  
Figure 1: The waveform and aligned transcription for an example sentence, where _<pb>_ means phrase boundary. The pink segments indicate the positions of short pauses (sp) in the waveforms. The English translation of this sentence is “The weak schools with low funding, poor conditions and low salaries are unable to retain talents. Some middle schools even need external teachers for Chinese and mathematics classes”. methods have been proposed, including non-autoregressive acoustic models with explicit phone duration modeling [14, 15] and improved attention mechanisms, such as forward attention [8], stepwise monotonic attention (SMA) [9] and location-relative attention [16]. Among them, SMA [9] applied strictly stepwise and monotonic constraints to alignment paths. Its mechanism is described in Algorithm 1, where \(\mathbf{W}\), \(\mathbf{V}\), \(\mathbf{U}\), \(\mathbf{v}\) and \(b\) are trainable weights, \(\mathbf{G}\) denotes weights of convolution kernels, \(\sigma\) is sigmoid function, \(\theta\) is the Heaviside step function, and \(\gamma\) is a trainable weight to control the strength of Gaussian noise. ```
Input: query vector \(\mathbf{q}_{t}\), key vectors \(\mathbf{K}=\{\mathbf{k}_{1},\mathbf{k}_{2},\ldots,\mathbf{k}_{N}\}\), previous attention weights \(\mathbf{a}_{t-1}\), \(\mathbf{mode}\in\{\mathbf{hard},soft\}\) Output: attention weights \(\mathbf{a}_{t}=\{a_{t1},a_{t2},\ldots,a_{tN}\}\), context vector \(\mathbf{c}_{t}\) if\(t=1\)then \(a_{t1}\gets 1;\{a_{t2},a_{t3},\ldots,a_{tN}\}\gets 0\); else \(\mathbf{F}\)=\(\{\mathbf{f}_{1},\mathbf{f}_{2},\ldots,\mathbf{f}_{N}\}\leftarrow\mathbf{G}*\mathbf{a}_{t-1}\); // \(*\) means convolution operation. for\(n\gets 1\) to \(\mathbf{N}\)do \(\mathbf{c}_{t,n}\leftarrow\mathbf{v}^{\top}\tanh(\mathbf{W}\mathbf{q}_{t}+\mathbf{V}\mathbf{k}_{n}+ \mathbf{U}\mathbf{f}_{n})+b\); ifmode = softthen \(\mathbf{p}_{t,n}\leftarrow\sigma(\mathbf{e}_{t,n}+\gamma\mathcal{N}(0,1))\); elseifmode = hardthen \(\mathbf{p}_{t,n}\leftarrow\theta(\mathbf{e}_{t,n})\);  end if  end for \(\mathbf{a}_{t}\leftarrow\mathbf{a}_{t-1}\cdot\mathbf{p}_{t}+\left[0;\mathbf{a}_{t-1,1:N-1} \cdot(1-\mathbf{p}_{t,1:N-1})\right]\); // \(\cdot\) means element-wise product. // \(1:N-1\) means the index range. // \(\mathbf{a}_{t}\) are normalized. end for \(\mathbf{c}_{t}\leftarrow\sum_{t=1}^{N}a_{t,n}\)\(\mathbf{k}_{n}\); return\(\mathbf{a}_{t}\), \(\mathbf{c}_{t}\);
```  
**Algorithm 1**Stepwise Monotonic Attention (SMA). At the first decoding step, the attention weights are manually set as one for the first phone, and zero for the rest phones considering the phone sequence is monotonically aligned with the acoustic feature sequence. Starting from the second decoding step, the attention weights are calculated recursively. The SMA calculates energy value \(\mathbf{e}_{t,n}\) based on query value \(\mathbf{q}_{t}\) in the decoder, key values \(\mathbf{K}=\{\mathbf{k}_{1},\mathbf{k}_{2},\ldots,\mathbf{k}_{N}\}\) in the encoder outputs and location features \(\mathbf{F}\). Then \"selection probability\" \(p_{t,n}\) based on energy value is computed by a sigmoid function. To sharpen the probability of the output, SMA adds Gaussian noise to energy values before feeding them into the sigmoid function. In order to achieve a monotonic and non-skipping attention mechanism, the attention weights \(\mathbf{a}_{t}\) are calculated based on the previous attention weights \(\mathbf{a}_{t-1}\) and \"selection probability\" \(\mathbf{p}_{t}\) as shown in Algorithm 1.