# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 2 Materials and Methods
### Proposed Causal Feature Encoder  
In this paper we propose a new type of encoder, the Causal Feature Extractor (CFE), that can be described as a two-step modification of a traditional CNN. The first change is that, instead of using regular convolutions, causal convolutions are applied; this concept was introduced by [PERSON] et al. (2015). Figure 6a,b show a basic representation comparing a regular and causal neural network, respectively.  
Figure 5: An encoder–decoder architecture with attention mechanism. **X**: Input sequence. **Z**: intermediate code. **Y**: Output sequence. \(y_{i}\): Output at instant \(i\). \(c_{i}\): Context coefficients at instant \(i\). \(h_{i}\): hidden states of the decoder at instant \(i\).  
In a regular one-dimensional convolution, the output for a position \(t\) depends on the input values at \(\dots t-2,t-1,t,t+1,t+2\dots\) Conversely, in a causal convolution, the output depends only on the inputs previous or posterior to that position, but never both. In other words, the causal convolution for \(t\) can use the values \(t,t+1,t+2\dots\) or \(t,t-1,t-2\dots\) This idea can be easily extended to images or, in general, to n-dimensional data.  
However, a convolution defined in such way only captures dependencies in one direction. To solve this important drawback, we propose a second change. To make the CFE bidirectional, in a similar fashion as it is done with LSTMs. Thereby, it contains two independent models that read the input in each direction, and their outputs are concatenated to produce the encoded representation. We depict this idea in Figure 7.  
An additional technique is applied in the proposed encoder. Considering the long sequences that can be found in text normalization (in our datasets, up to 177 characters), we apply dilated convolutions to the convolutional models, as described by [PERSON] et al. [5]. This technique consists in doubling  
Figure 6: Comparison between a regular and a causal convolutional neural network, with dilatation coefficient 2. (**a**) Sample regular convolutional neural networks (CNN). (**b**) Sample of the proposed causal CNN. \(t\): Temporal order of the input sequence. \(l_{1}\): First layer (input sequence). \(l_{2},l_{3},l_{4}\): Subsequent convolutional layers.  
Figure 7: Diagram showing the bidirectionality of the proposed Causal Feature Extractor (CFE) encoder. The intermediate output of the encoder, \(\mathbf{Z}\), is the concatenation of two causal convolutions in the opposite directions (blue and red), each of them generating \(f/2\) features. \(\mathbf{X}\): Input sequence. \(l\): Length of the input sequence. \(v\): Size of the alphabet in the one-hot encoding. \(f\): Total number of intermediate features.  
the dilatation of each convolutional layer as it goes deeper into the structure, as depicted in Figure 6. So, in the first layer, the convolution for \(t\) depends on the input at \(t,t-1,t-2\dots\); in the second layer, it depends on the previous layer at \(t,t-2,t-4\dots\); in the third layer, \(t,t-4,t-8\dots\), and so on. Moreover, the same for the opposite causal direction. By doing that, the actual receptive field of the model (i.e., the initial positions that contribute to the final result) is significantly increased without increasing the number of parameters of the network.  
This new encoder has been designed to solve a problem that many applications of CNNs have with attention mechanisms. In previous experiments, it has been observed that CNNs tend to attend the wrong inputs according to our prior intuition. Namely, they choose the \(i+C\)-th element instead of the \(i\)-th element of the input, where \(C\) is a certain constant. Our intuition is that this could be caused by the padding introduced in each side of the input. By using causal convolutions, the model is forced to choose the outermost elements if it is interested in those.