# Cocrystal Prediction Using Machine Learning Models and Descriptors
## 3 Results
### Model Comparison  
We prepared three independent datasets \(D^{1}_{total}\), \(D^{2}_{total}\), and \(D^{3}_{total}\) by shuffling the instances of the dataset \(D_{total}\); so \(|D_{total}|=|D^{1}_{total}|=|D^{2}_{total}|=|D^{3}_{total}|=1476\). We conducted 10-fold cross validation for each of these independent datasets, and computed averaged accuracy, precision, recall, and F1 scores. The optimal parameter settings of the machine learning models are found by a grid searching using a small portion (e.g., 10%) of the training set as a validation set. The parameter settings are summarized in Table 3; the parameter settings are obtained during the experiments. The ANN has a shallow structure (i.e., one hidden layer of 25 nodes) because we found that it gives better performance than other complex structure; the reason might be the small size of dataset that high model complexity will cause overfitting problem. Table 4 summarizes the accuracy of machine learning models; note that we focus on the experimental comparison between the models here, but not the feature selection algorithms. The accuracy values are computed by averaging results with three independent datasets (e.g., \(D^{1}_{total}\), \(D^{2}_{total}\), and \(D^{3}_{total}\)). The ANN gives the best accuracy (e.g., 0.833) among the models. The XGB is comparable to the ANN, and it is the best when \(F_{S}=300\). As a model works faster when the feature dimension is small, the XGB might be preferable if we want better efficiency (i.e., fast prediction) without losing much accuracy. \begin{table}
\begin{tabular}{c l} \hline \hline
**Model** & **Setting** \\ \hline Random forest & Number of estimators = 100 \\ (RF) & No limitation of depth \\  & Minimum samples for splitting = 2 \\ \hline Support vector machine & Kernel = Linear \\ (SVM) & C = 1.0 \\ \hline Extreme gradient boosting & Number of estimators = 100 \\ (XGB) & Learning rate = 0.3 \\ \hline Artificial neural network & \# of hidden layers = 2 \\ (ANN) & Hidden layer sizes (\# of nodes of each layer) = 25 \\  & Activation function = Relu function [57] \\  & Optimizer = Adam [58] \\  & \# of epochs = 50 with early stopping \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Parameter settings of machine learning models. Figure 3: Averaged accuracy using feature selection algorithms, where the vertical axis is the accuracy and the horizontal axis means the number of selected features. One might argue that the model is not useful if its sensitivity is not high enough. Tables 5 and 6 are per-label precision and recall. The ANN gives the best recall of'success' label (e.g., 0.800) without losing much precision (e.g., 0.838). In terms of the precision, the XGB seems the best as its precision of'success' label is 0.841, but we might say that the ANN would be chosen if we need to find as many promising candidates of compound pairs as possible. Table 7 shows per-label F1 scores, and the ANN is turned out to be the best amongst the models. This result is reasonable as the ANN is known to be effective in finding underlying patterns and gives significant performance improvement in many other classification tasks (e.g., malware detection [59], chatbot intent prediction [60]). We believe that the performance will be further improved if we collect more qualified data. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**All Features (\(F_{ALL}\) = 2207)**} & \multicolumn{2}{c}{\(F_{\text{S}}\) = 1103**} & \multicolumn{2}{c}{\(F_{\text{S}}\) = 900} \\ \cline{2-7}  & **Fail** & **Success** & **Fail** & **Success** & **Fail** & **Success** \\ \hline RF & 0.840 & 0.773 & 0.890 & 0.836 & 0.856 & 0.807 \\ SVM & 0.806 & 0.667 & 0.778 & 0.800 & 0.792 & 0.733 \\ XGB & 0.847 & 0.773 & 0.889 & 0.867 & 0.861 & 0.813 \\ ANN & 0.835 & 0.800 & 0.808 & 0.808 & 0.827 & 0.806 \\ \hline \hline \end{tabular}
\end{table}