# Sequence-to-Sequence Acoustic Modeling with Semi-Stepwise Monotonic Attention for Speech Synthesis
## 3 Proposed Methods  
We propose to insert a hidden state between every two adjacent phones. To deal with these inserted hidden states, we propose a semi-stepwise monotonic attention (SSMA), as shown in Figure 2. The SSMA mechanism is a modification of the SMA mechanism, that can deal with the situation that there are missing pause labels in phone sequences due to the lack of explicit phrase boundaries. These hidden states are expected to absorb the decoding frames that correspond to pauses. We name it \"semi-stepwise\" because these hidden states are skippable if there is no pause between two adjacent phones, thus the alignment paths are not strictly stepwise. At the \(t\)-th decoding step, \(a_{tn}\) denotes the attention weight of the \(n\)-th phone, and \(l_{tn}\) denotes the attention weight of the hidden state after the \(n\)-th phone. We have \(\sum_{n}(a_{tn}+l_{tn})=1\). As shown in Figure 2, the attention at the \(n\)-th phone of the \((t-1)\)-th decoding step has three choices to derive the attention of the \(t\)-th decoding step, staying at current phone, jumping forward to the next phone, or jumping to the hidden state after current phone. Let \(p_{t,n}^{c\to c}\),\(p_{t,n}^{c\to n}\) and \(p_{t,n}^{c\to h}\) denote the probability of these three choices, respectively, and their sum should be \(1\). Moreover, the attention at a hidden state of the \((t-1)\)-th decoding step has two choices to derive the attention of the \(t\)-th decoding step, staying at current hidden state or jumping to the next phone. Let \(p_{t}^{h\to h}\) and \(1-p_{t}^{h\to h}\) denote the probability of these two choices, respectively. Similar to Algorithm 1 for SMA, the attention weights \(\mathbf{a}_{t}=\{a_{t1},a_{t2},\ldots,a_{tN}\}\) and \(I_{t}=\{l_{t1},l_{t2},\ldots,l_{t,N-1}\}\) are calculated in a recursive way. The detailed pseudo-code is shown in Algorithm 2, where \(\mathbf{W}\), \(\mathbf{V}\), \(\mathbf{v}\), \(\mathbf{b}\), \(\mathbf{W}^{\prime}\), \(\mathbf{V}^{\prime}\), \(\mathbf{v}^{\prime}\), \(\mathbf{b}^{\prime}\), \(\mathbf{U}\), \(\mathbf{V}^{\prime}\) and \(\mathbf{b}^{\prime}\) are trainable weights. When \(t=1\), the attention weights are manually set as one for the first phone, and zero for the rest phones and hidden states. Considering that a phone at the \(t\)-th decoding step has three options to choose the attention for the next decoding step, a two-level prediction strategy based on location-sensitive attention [13] is adopted to calculate the probabilities of these options. Specifically, we first predict the staying probability \(\mathbf{p}_{t}^{c\to c}\) and then predict the proportion of \(\mathbf{p}_{t}^{c\to n}\)./\((\mathbf{p}_{t}^{c\to n}+\mathbf{p}_{t}^{c\to h})\), where \(./\) denotes element-wise division. In order  
Figure 2: Schematic diagram of SSMA with \(t\in\{1,2,3\}\) and \(n\in\{1,2\}\). The dashed circles denote hidden states at each decoding step. The definitions of the symbols in this figure can be found in Section 3.  
to enhance the query effectiveness, a query matrix \(\mathbf{q}_{t}^{\prime}\) is calculated using a deep neural network (DNN). The DNN model accepts three inputs, i.e., the query \(\mathbf{q}_{t}\), the keys \(\mathbf{K}\) and the location vectors \(\mathbf{F}\). The probability \(p_{t,n}^{c\to c}\) is calculated by DNN using \(\mathbf{q}_{t,n}^{\prime}\) and \(\mathbf{k}_{n}\). The probability \(p_{t,n}^{c\to n}/(p_{t,n}^{c\to n}+p_{t,n}^{c\to h})\) is calculated by the DNN using \(\mathbf{q}_{t,n}^{\prime}\) and \(\mathbf{k}_{n+1}\). An end-of-sentence (EOS) embedding vector is taught to replace \(\mathbf{k}_{n+1}\) when \(n\) is index of the last phone. Then, \(\mathbf{p}_{t}^{c\to n}\) and \(\mathbf{p}_{t}^{c\to h}\) can be derived from \(\mathbf{p}_{t}^{c\to c}\) and \(\mathbf{p}_{t}^{c\to n}/(\mathbf{p}_{t}^{c\to n}+\mathbf{p}_{t}^{c\to h})\). In our preliminary experiments, we found that this strategy can obtain higher naturalness of synthetic speech than using Gumbel-Softmax [17].