# Best Practices of Convolutional Neural Networks for Question Classification
## 2 Materials and Methods
### Question Classification Model
#### 2.2.3 Learning Procedure  
The described model includes many parameters that were initialized randomly and have to be trained, i.e., \(\mathbf{W}_{\text{conv}}^{i}\) with \(i=1,\ldots,Ntot\), \(b_{\text{conv}}^{i}\) with \(i=1,\ldots,Ntot\), \(\mathbf{W}_{\text{fc}}\), and \(\mathbf{b}_{\text{fc}}\), for a total of \(De\cdot Fs\cdot Ntot+Ntot\cdot K+K\). Moreover, \(\mathbf{W}_{\text{emb}}\) can be initialized by pre-trained WE or randomly, but in both cases they were fine-tuned if \(W\)_Etuning_ = Dynamic, bringing other \(V\cdot De\) parameters.  
These parameters were adapted on data by a learning procedure summarized as follows.  
Firstly, the training dataset was divided in batches composed of a certain number of examples. In this work, the batch size was a degree of freedom, studied in its whole range:  
\[batch\in[1,900]. \tag{15}\]  
All examples of a batch were used as input of the model, but during the training, in order to learn separately different parts of the network, the pooling layer was modified by the dropout function, which randomly transforms each component \(p^{i}\) multiplying it by zero with probability (\(1-P_{keep}\)),  
Figure 2: The activation functions employed.  
and by \(1/P_{kcp}\) with probability \(P_{kcp}\), so that the expected sum remains unchanged. Here, the dropout was a degree of freedom, variable in the following interval:  
\[P_{kcp}\in(0.0,1.0]. \tag{16}\]  
For each input \(b\), the loss was calculated by the cross entropy function (19), where \(p_{k}\) (17) was the softmax transform of the \(k\)th component of computed vector \(\mathbf{y}\), and \(c_{k}\) (18) was 1 for the position of the true label \(k_{\text{True}}\), 0 otherwise:  
\[p_{k}=\frac{e^{y_{k}}}{\sum\limits_{\kappa=1}^{K}e^{y_{\kappa}}}. \tag{17}\]  
\[c_{k}=\left\{\begin{array}{l}1\text{ if }k=k_{\text{True}}\\ 0\text{ otherwise}\end{array}\right.. \tag{18}\]  
\[loss_{b}=-\sum\limits_{k=1}^{K}c_{k}\cdot\log(p_{k}). \tag{19}\]  
After a batch, the whole associated loss was calculated as:  
\[loss=\frac{1}{batch}\sum\limits_{b=1}^{batch}loss_{b}+l2\cdot\left(\frac{| \mathbf{W_{\text{fc}}}|^{2}}{2}+\frac{|\mathbf{b_{\text{fc}}}|^{2}}{2}\right). \tag{20}\]  
The regularization parameter \(l2\), used to prevent big values of fully connected layer weights, was a degree of freedom here, studied in the following interval:  
\[l2\in[1.0,5.0]. \tag{21}\]  
The loss gradient was used for updating network weights by a backpropagation approach based on Stochastic Gradient Descent (SGD) algorithm, which implies a stochastic approximation [36] of the basic gradient descent algorithm. Since it reduces the computational complexity, achieving faster iterations in trade for a lower convergence rate [37], it was recognized as a very effective learning algorithm in machine learning [38]. A variant of the updating rule was freely chosen among the following ones:  
\[optimizer=\left\{\begin{array}{l}\text{Adadelta}\\ \text{Adagrad}\\ \text{Adam}\\ \text{Ftrl}\\ \text{GradientDescent}\\ \text{Momentum}\\ \text{ProximalAdagrad}\\ \text{ProximalGradientDescent}\\ \text{RMSProp}\end{array}\right.. \tag{22}\]  
While in case \(optimizer=\text{Momentum}\), the momentum parameter was fixed to 0.1, according to previous findings, the learning rate, which was a further parameter common to all the algorithms, was the last considered degree of freedom, studied in the following wide range, enlarged with respect to previous works [19; 21]:  
\[\eta\in[0.01,10]. \tag{23}\]  
The intent in this work was to get the best possible model, therefore the number of epochs was not taken as an adjustable setting.