# Abstract Text Summarization with a Convolutional Seq2 seq Model
## 3 Model Description
### Convolutional Seq2 seq Model  
We share the convolutional layer architecture on both encoder and decoder and they will calculate intermediate states via the input elements. We represent the output of \(l^{\text{th}}\) layer as \(\mathbf{z}^{1}=(z_{1}^{l},\ldots,z_{m}^{l})\) for encoder and \(\mathbf{h}^{1}=(h_{1}^{l},\ldots,h_{n}^{l})\) for decoder. Each layer consists of a one dimensional convolution and a non-linearity. If a decoder has one layer with kernel width being \(k\), then its output \(h_{i}^{1}\) will compress the information of \(k\) input elements. To enlarge the length of input elements, we stack blocks over each other, for example, stacking 6 blocks with \(k=5\) could represent 25 input elements. If needed, non-linearities enable our model to deal with the entire input sequence or only a few elements. The computational advantage of our model is that it conducts a parallel computation which is much more efficient than the traditional RNN model computed one element by one element. As mentioned in Section 1, to represent a sequence with \(n\) words, CNNs only need \(O(\frac{n}{k})\) operations, while RNNs need \(O(n)\) operations. However, our hierarchical CNN model is not that efficient than traditional CNN model as ours has to stack CNN layers to represent a sequence more expressively, while a traditional CNN model only needs one layer to explore a whole sequence.  
In each convolution kernel, parameters are \(W\in\mathbb{R}^{2d\times d}\), \(b_{w}\in\mathbb{R}^{2d}\). The input is represented as \(X\in\mathbb{R}^{o\times d}\), which is a matrix having \(o\) input elements with the dimension being \(d\). Then the input is mapped by the layer to get the output being a single element \(Y\in\mathbb{R}^{2d}\) with its dimension twice of that of the input. Then the \(o\) outputs elements will be fed to the subsequent layers. We leverage the gated linear units (GLU) [9] as non-linearity, which is applied on the output represented as \(Y=[A\ B]\in\mathbb{R}^{2d}\):  
\[v([A\ B])=A\otimes\sigma(B), \tag{1}\]in which the GLU takes \(A,B\in\mathbb{R}^{d}\) as inputs, \(\otimes\) denotes the element-wise multiplication. It is notable that \(Y\in\mathbb{R}^{2d}\) is also twice the size of the output \(v([A\ B])\in\mathbb{R}^{d}\). \(\sigma(B)\) is the gate that controls which inputs \(A\) of the source text are currently dealt with.  
In order to realize deep CNN, we adopt the residual connections [10] which connect the input of each convolution layer with the ouput:  
\[h_{i}^{l}=v(W^{l}[h_{i-o/2}^{l-1},\dots,h_{i+o/2}^{l-1}]+b_{w}^{l})+h_{i}^{l-1}. \tag{2}\]  
For encoder, to ensure that the output of the convolution blocks match the input elements length, we pad the input by \(o-1\) elements at each layer with zero vectors on both the left and the right. Specifically, since no future information should be added to the decoder [35], \(o\) elements at the end of the convolution output will be removed.  
Furthermore, we also apply linear mappings on the \(2d\) convolution outputs so as to fit the embedding with its dimension being \(f\). Such project is used to \(w\) when we feed the input embeddings to encoder. We also use it to encoder output \(z_{j}^{i}\) to compute the attention scores, along with the final layer of decoder before the softmax operation layer \(\mathbf{h}^{\mathbf{L}}\).  
After all the operations above, the distribution on \(T\) possible next target elements \(t_{i+1}\) could be generated as follows:  
\[p(t_{i+1}|t_{1},\dots,t_{i},\mathbf{x})=\text{softmax}(W_{s}h_{i}^{l}+b_{s}) \in\mathbb{R}^{T}, \tag{3}\]  
where \(W_{s}\) denotes the weights in the linear layer and \(b_{s}\) is the bias.