# A Simple Convolutional Neural Network with Rule Extraction
## 3 Proposed Model
### CNN Architecture
#### 3.1.2 Two-Dimensional Convolutional Layer and Max-Pooling Layer  
A key element in CNNs is based on the convolution operator. Given a two-dimensional kernel \(w_{p,q}\) of size \(P\!\!\times\!Q\) and a data matrix of elements \(x_{a,b}\), the calculation of an element \(c_{ij}\) of the convolutional layer is  
\[c_{ij}=f(\sum_{p}^{P}\sum_{q}^{Q}w_{p,q}\cdot x_{i+p,j+q}+b_{p,q}); \tag{1}\]  
Figure 1: The Convolutional Neural Networks (CNN) architecture used in this work. From left to right are shown a two-dimensional input layer, a convolutional layer, a max-pooling layer, and an output layer which is fully connected.  
Figure 2: Representation of text by a matrix of real numbers. A word is represented on the horizontal axis by word embeddings, while vertically several words are expressed.  
with \(f\) a transfer function and \(b_{p,q}\) the bias. As a transfer function we use a hyperbolic tangent:  
\[\tanh(x)=\frac{\exp(2x)-1}{\exp(2x)+1}. \tag{2}\]  
We define \(S_{p}\) and \(S_{q}\) as the stride parameters along the horizontal and vertical axis between two successive convolutions. In this work \(S_{p}=S_{q}=1\). Moreover, we require the kernel to be completely inside the sample matrix (without zero padding). As an example, with data samples of size 6 \(\times\) 6 and \(P=Q=3\), the resulting convoluted map has size 4 \(\times\) 4.  
Figure 3 illustrates a particular case of convolution with respect to a text matrix of size 7 \(\times\) 5 and a kernel of size 4 \(\times\) 5. This kernel moves over the text matrix, carrying out an element-wise multiplication with the part of the data it is currently on. This is repeated by sliding down the kernel by one position, vertically. Hence, the result of convolution is a vector of four components. \"Wide convolution\" is the denotation when the horizontal size of the kernel is equal to the horizontal size of the data matrix. With wide convolution, the size of a kernel is defined as its vertical size. For instance, in Figure 3 the kernel size is equal to four.  
A remarkable relationship is fulfilled between the size of the kernels and the number of consecutive words taken into account by the convolution operator. Since the size of a kernel is the number of components on the vertical axis, it corresponds to the number of consecutive words processed at any position in a sentence. As an example, three consecutive words are denoted as 'trigrams' and can be detected by kernels of size three. Similarly, two consecutive words are called 'bigrams' and can be taken into account by kernels of size two. Finally, to emphasize single words it is possible to define filters of size one.  
The max-pooling layer reduces the size of a vector or a matrix by applying a \"Max\" operator over non-overlapping regions. Figure 4 illustrates in the left a number of vectors obtained after convolution. From each vector the maximal value is extracted and concatenated in a new layer, which enables n-gram position invariance.  
Figure 3: Wide convolution: The number of columns of the kernel is equal to the number of columns of the text matrix. The result of convolution (showed after the arrow) is a vector.