# Best Practices of Convolutional Neural Networks for Question Classification
## 2 Materials and Methods
### Question Classification Model
#### 2.2.5 Model Optimization  
In order to optimize the QC model, its classification accuracy was studied by analyzing different experiments, corresponding to respective configurations of settings (factors). Since each experiment was the set of 10 training and testing phases constituting a 10-fold cross-validation, the accuracy of an experiment was the average of the accuracies gained by the 10 trained models on the respective test set. The accuracy on a test set was calculated as a percentage, by averaging \(c_{output}\), which was 1 if the network output (14) was equal to the position of the true label associated with the \(b\)th input of the test set (\(output=k_{\text{True}}\)), 0 otherwise:  
\[Acc=\frac{1}{\textit{Ntest}}\sum_{b=1}^{\textit{Ntest}}c_{output}\cdot\!100\%. \tag{24}\]  
The factors here considered to analyze their influence on the model accuracy are summarized in Table 3. For categorical factors, all the possible values were considered, while for quantitative ones the considered admitted ranges were based on previous literature findings. Since considering all the possible interactions among factors would involve an unfeasible experimentation, some factors were analyzed in the following one by one, since they were hypothesized to have negligible interactions with the others, while some sets of factors were studied together to verify potential interactions. For each factor or set of factors, their individual influences and interactions (effects) were evaluated in a chosen range by performing a set of experiments. Most sets of experiments were planned according to full factorial designs, which comprise all the combinations of factors levels. This approach needs more numerous experiments, but minimizes the risk of confounding different effects. The range of each factor, and the fixed values of other settings, relative to factors not being evaluated in a set of experiments, since were hypothesized to not interact, were chosen according to findings of previous works [19; 20; 21; 33], or to preliminary experiments. Due to the random initialization of weights and to some other sources of randomness in the learning procedure (splitting training data in batches, dropout function, and SGD algorithm), each run, and thus each whole experiment, gave different results if repeated. Therefore, some repetitions were performed, to estimate the experimental variance \(\sigma\)2, which was used to evaluate the experiments reproducibility. The intrinsic variance in the measurement of the experiment performance implies that a deterministic functional dependence between factors and model accuracy does not exist. Therefore, in order to analyze the effects on the QC accuracy, an approximate function was extrapolated from each set of experimental results:  
\[acc=c_{0}+c_{1}x_{1}+c_{2}x_{2}+c_{12}x_{1}x_{2}+\dots \tag{25}\]  
where \(x_{1}x_{2}\),... represent the individual factors, \(x_{1}x_{2}\),... represent their interactions, and coefficients \(c_{0}\),\(c_{1}\),\(c_{2}\),\(c_{12}\),... were used to linearly combine these (also nonlinear) effects to predict experimental accuracy. The significance of effects was evaluated in terms of the respective coefficients [39]. Indeed, each estimated coefficient belongs to a respective confidence interval, corresponding to the interval comprising the true coefficient value with 95% probability, that was calculated as follows. Given the estimated experimental variance \(\sigma^{2}\), calculated with a certain number of degrees of freedom, the variance of each coefficient can be estimated as \(\sigma^{2}\)/\(N\), where \(N\) is the number of experiments of the full factorial design. Therefore, the width of the coefficient confidence interval can be calculated as \(\sigma\)/\(N^{1/2}\cdot t_{0.975}\), where \(t_{0.975}\) is the value of a t-student distribution with the same degrees of freedom corresponding to 0.975 cumulative probability (two tails t-test).