# Best Practices of Convolutional Neural Networks for Question Classification
## 2 Materials and Methods
### Question Classification Model
#### 2.2.2 Convolutional Neural Network Architecture  
A classical CNN architecture was used here for associating questions with labels. However, here the architecture was not fixed, but was implemented with freely adjustable settings.  
A convolution was firstly applied to \(\mathbf{X_{emb}}\), by using a single channel, with no padding and stride \(1\), as recommendable in text classification context. Filters of different sizes may be employed, therefore, if there were \(N\)s different sizes and for each size a number \(N\)f of filters, the total number \(N\)tot of filters was:  
\[Ntot=Nf\cdot N\text{s}. \tag{6}\]  
The sizes and the total number of filters were degrees of freedom, and they were considered in the following ranges:  
\[Fs\in[1,10], \tag{7}\]  
\[Ntot\in[50,500]. \tag{8}\]  
Each filter of a certain size consists in a matrix \(\mathbf{W}_{\text{conv}}^{i}\), with \(i\) = 1,...,\(N\)tot made of \(De\) columns and \(Fs\) rows. The result of the convolution was a vector \(\mathbf{x_{conv}}^{i}\) with dimension \(Lmax-Fs+1\), whose components \(x_{\text{conv}}^{i,j}\), with \(j\) = 1,...,\(Lmax-Fs+1\), can be written as:  
\[x_{\text{conv}}^{i,j}=\sum_{j=1}^{Fs}\sum_{d=1}^{De}\Bigl{(}\mathbf{X_{emb}}[ j+jj-1][d]\cdot\mathbf{W}_{\text{conv}}^{i}[jj][d]\Bigr{)}. \tag{9}\]  
Then a bias term \(b_{\text{conv}}^{i}\) was added to each component, and an activation function \(f\) was applied, to get each component \(x_{\text{act}}^{i,j}\), with \(j\) = 1,...,\(Lmax-Fs+1\), of the vector \(\mathbf{x}_{\text{act}}^{i}\), which was the final result of the convolution by the given filter \(\mathbf{W}_{\text{conv}}^{i}\):  
\[x_{\text{act}}^{i,j}=f\Bigl{(}x_{\text{conv}}^{i,j}+b_{\text{conv}}^{i}\Bigr{)}. \tag{10}\]  
Of course, vectors \(\mathbf{x}_{\text{act}}^{i}\) with the same size were obtained by using filters with the same \(Fs\), while vectors of different sizes were obtained by differently sized filters. However, \(N\)tot vectors were obtained, and they constitute the convolutional layer.  
The activation function to use for convolution was a degree of freedom of the proposed implementation. The following functions were used, whose meaning is shown in Figure 2:  
\[f=\left\{\begin{array}{c}\text{eLU}\\ \text{Identity}\\ \text{ReLU}\\ \text{sigmoid}\\ \text{softplus}\\ \text{softsign}\\ \text{tanh}\end{array}\right.. \tag{11}\]The following operation was the pooling, which was implemented coherently with the common choice, i.e., the 1-max pooling strategy. In fact, using the max function was forced by the padding with zeros the input representation of questions shorter than \(Lmax\), and the choice of only one maximum element was certainly enough, due to the big number of filters employed. Therefore, the pooling layer was constituted by a horizontal vector \(\mathbf{p}\) with dimension \(Ntot\), whose elements \(p^{i}\), with \(i=1,\ldots,Ntot\), were:  
\[p^{i}=\text{max}\mathbf{x}_{\text{act}}^{i}. \tag{12}\]  
The following fully connected layer was constituted by \(K\) neurons, where \(K\) was also the number of classes. In the considered case, \(K=9\). The vector of class activations \(\mathbf{y}\) was computed by multiplying \(\mathbf{p}\) by a matrix of weights \(\mathbf{W}_{\text{fc}}\) with \(Ntot\) rows and \(K\) columns, and adding a bias vector \(\mathbf{b}_{\text{fc}}\):  
\[\mathbf{y}=\mathbf{p}\mathbf{W}_{\text{fc}}+\mathbf{b}_{\text{fc}}. \tag{13}\]  
The final output layer was made of only one node, which contains the position of the class with the highest activation:  
\[output=\text{argmax}. \tag{14}\]