# A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs +
## 2 Proposed System
### Modified WaveNet Architecture  
The main building block of our system is based on the WaveNet model and architecture. A key aspect of this model is that it is autoregressive. That is, the prediction at each timestep depends on (a window of) predictions of past timesteps. In our case, a timestep corresponds to a single frame of acoustic features. Additionally, the model is probabilistic, meaning that the prediction is a probability distribution rather than a single value. In order to control the prediction, e.g., by phonetic and musical inputs, the predicted distribution is not only conditioned on past predictions, but also on control inputs. This model is implemented using a powerful, yet efficient neural network architecture. The network we propose, depicted in Figure 2, shares most of its architecture with WaveNet. Like this model we use gated convolutional units instead of gated recurrent units, such as Long Short-Term Memory (LSTM) units, to speed up training. The input is fed through an initial causal convolution which is then followed by stacks of \(2\times 1\) dilated convolutions [7] where the dilation factor is doubled for each layer. This allows exponentially growing the model's receptive field, while linearly increasing the number of required parameters. To increase the total nonlinearity of the model without excessively growing its receptive field, the dilation factor is increased up to a limit and then the sequence is repeated. We use residual and skip connections to facilitate training deeper networks [8]. As we wish to control the synthesizer by inputting notes and lyrics, we use a conditional version of the model. At every layer, before the gated nonlinearity, feature maps derived from control inputs are summed to the feature maps from the layer's main convolution. In our case, we do the same thing at the output stack, similar to [9]. The underlying idea of this model is that joint probability over all timesteps can be formulated as a product of conditional probabilities for a single timestep with some causal ordering. The conditional probability distributions are predicted by a neural network trained to maximize likelihood of a  
Figure 2: Overview of the modified WaveNet network architecture. In this case, the network depicted predicts harmonic spectral envelope features (top-right and bottom), given control inputs (mid-right). observation given past observations. To synthesize, predictions are made by sampling the predicted distribution conditioned on past predictions, that is, in a sequential, autoregressive manner. However, while models on which we base our model like WaveNet, or PixelCNN [10] and PixelRNN [11] before it, perform this factorization for univariate variables (e.g., individual waveform samples or pixel channels), we do so for multivariate vectors corresponding to a single frame,  
\[p\left(\mathbf{x}_{1},\ldots,\mathbf{x}_{T}\mid\mathbf{c}\right)=\prod_{t=1}^ {T}p\left(\mathbf{x}_{t}\mid\mathbf{x}_{<t},\mathbf{c}\right)\, \tag{1}\]  
where \(\mathbf{x}_{t}\) is an \(N\)-dimensional vector of acoustic features \(\left[x_{t,1},\ldots,x_{t,N}\right]\), \(\mathbf{c}\) is an \(T\)-by-\(M\)-dimensional matrix of control inputs, and \(T\) is the length of the signal. In our case, we consider the variables within a frame to be conditionally independent,  
\[p\left(\mathbf{x}_{t}\mid\mathbf{x}_{<t},\mathbf{c}\right)=\prod_{i=1}^{N}p \left(x_{t,i}\mid\mathbf{x}_{<t},\mathbf{c}\right). \tag{2}\]  
In other words, a single neural network predicts the parameters of a multivariate conditional distribution with diagonal covariance, corresponding to the acoustic features of a single frame. The main reason for choosing this model is that, unlike raw audio waveform, features produced by a parametric vocoder have two dimensions, similar to (single channel) images. However, unlike images, these two dimensions are not both spatial dimensions, but rather time-frequency dimensions.