# Sequence-to-Sequence Acoustic Modeling with Semi-Stepwise Monotonic Attention for Speech Synthesis
## 3 Proposed Methods
To calculate the staying probability on hidden states \(p_{t}^{h\to h}\), we simply use a DNN. The DNN model accepts two inputs, i.e., the query vector \(\mathbf{q}_{t}\) and the hidden state embedding \(\mathbf{k}_{l}\). The DNN first predicts \(e_{t}^{l}\), and then the trick of adding-noise and a sigmoid function are applied again to calculate \(p_{t}^{h\to h}\). Finally, the attention weights \(\mathbf{a}_{t}\) and \(\mathbf{I}_{t}\) are calculated recursively and the context vector \(\mathbf{c}_{t}\) is updated accordingly. ``` Input: query vector \(\mathbf{q}_{t}\), key vectors \(\mathbf{K}\)=\(\{k_{1},k_{2},\ldots,k_{N}\}\), hidden state embedding vector \(\mathbf{k}_{l}\), previous attention weights \(\mathbf{a}_{t-1}\) and \(\mathbf{I}_{t-1}\), \(\mathbf{mode}\in\{\mathbf{hard},soft\}\) Output: attention weights \(\mathbf{a}_{t}=\{a_{t1},a_{t2},\ldots,a_{tN}\}\) and \(\mathbf{I}_{t}=\{l_{t1},l_{t2},\ldots,l_{t,N-1}\}\), context vector \(\mathbf{c}_{t}\) if\(t=1\)then \(a_{t1}\gets 1;\{l_{t1},a_{t2},l_{t2},\ldots,a_{tN}\}\gets 0\); else F=\(\{\mathbf{f}_{1},\mathbf{f}_{2},\ldots,\mathbf{f}_{N}\}\leftarrow\mathbf{G}*\mathbf{a}_{t-1}\); \(e_{t}^{l}\leftarrow(\mathbf{v}^{l})^{\top}\text{tanh}\big{(}\mathbf{W}^{l}\mathbf{q}_{t}+ \mathbf{V}^{l}\mathbf{k}_{l}\big{)}+b^{l}\); for\(n\gets 1\)to\(N\)do \(\mathbf{q}_{t,n}^{\prime}=\mathbf{V}^{\prime\top}\text{tanh}(\mathbf{W}\mathbf{q}_{t}+\mathbf{V} \mathbf{k}_{n}+\mathbf{U}\mathbf{f}_{n})+\mathbf{b}^{l}\); \(e_{t,n}^{s}\leftarrow\mathbf{v}^{\top}\text{tanh}\big{(}\mathbf{q}_{t,n}^{\prime}+\mathbf{ V}\mathbf{k}_{n}\big{)}+b\); \(e_{t,n}^{l}\leftarrow\mathbf{v}^{\top}\text{tanh}\big{(}\mathbf{q}_{t,n}^{\prime}+\mathbf{ V}\mathbf{k}_{n+1}\big{)}+b\); ifmode=softthen \(p_{t,n}^{c\to c}\leftarrow\sigma(e_{t,n}^{c\in}+\mathcal{N}(0,1))\); \(p_{t,n}^{c\to n}\leftarrow(1-p_{t,n}^{c\to c})\sigma(e_{t,n}^{j}+\mathcal{N}( 0,1))\); \(p_{t,n}^{c\to h}\leftarrow(1-p_{t,n}^{c\to c})(1-\sigma(e_{t,n}^{j}+\mathcal{N} (0,1)))\); \(p_{t}^{h\to h}\leftarrow(\sigma(e_{t}^{l}+\gamma\mathcal{N}(0,1))\); elseifmode=hardthen \(p_{t,n}^{c\to c}\leftarrow\sigma(e_{t,n}^{s})\); \(p_{t,n}^{c\to n}\leftarrow(1-p_{t,n}^{c\to c})\sigma(e_{t,n}^{j})\); \(p_{t,n}^{c\to h}\leftarrow(1-p_{t,n}^{c\to c})(1-\sigma(e_{t,n}^{j}))\); \(\{p_{t,n}^{c\to h},\,p_{t,n}^{c\to h}\}\leftarrow\text{binary}(p_{t,n}^{c \to c},\,p_{t,n}^{c\to n},\,p_{t,n}^{c\to h})\); \(p_{t}^{h\to h}\leftarrow\theta(e_{t}^{l})\);  end if  end for \(\mathbf{a}_{t}\leftarrow\mathbf{a}_{t-1}\cdot\mathbf{p}_{t}^{c\to c}+\big{[} 0;\mathbf{a}_{t-1,1:N-1}\cdot\mathbf{p}_{t,1:N-1}^{c\to n}\big{]}+\big{[}0;(1-p_{t}^{h \to h})\mathbf{I}_{t-1}\big{]}\); \(\mathbf{I}_{t}\leftarrow p_{t}^{h\to h}\mathbf{I}_{t-1}+\mathbf{a}_{t-1,1:N-1}\cdot\mathbf{p}_{t,1:N-1}^{c\to h}\); // \(\{\mathbf{a}_{t};\mathbf{I}_{t}\}\) are normalized. end for \(\mathbf{c}_{t}\leftarrow\sum_{n=1}^{N}a_{t,n}\mathbf{k}_{n}+\mathbf{k}_{l}\sum_{n=1}^{N-1}l _maximum value among \(p_{t,n}^{e\to c}\), \(p_{t,n}^{e\to n}\) and \(p_{t,n}^{e\to h}\) is set to 1 and the other two values are set to 0, as indicated by the _binary_ function in Algorithm 2. \(p_{t}^{h\to h}\) also becomes 0 or 1. ## 4 Experiments

### Experimental Setup

A Chinese corpus pronounced by a female speaker was used in our experiments. The scripts were selected from newspapers, and the recordings were sampled at 16 kHz with 16 bits resolution. The total 12,319 utterances (\(\approx\)17.51 h) were split into a training set of 11,608 utterances, a validation set of 611 utterances and a test set of 100 sentences. The training set was used to train acoustic models and the validation set was used to tune hyperparameters. A publicly available implementation of Tacotron2 ([[https://github.com/NVIDIA/tacotron2](https://github.com/NVIDIA/tacotron2)]([https://github.com/NVIDIA/tacotron2](https://github.com/NVIDIA/tacotron2)), accessed on 12 June 2020) was utilized as the basis of our implementation. When training the model, 80-band mel-spectrograms were used as the acoustic features.