# Modified Conditional Restricted Boltzmann Machines for Query Recommendation in Digital Archives
## 6 Experiments
### Experiments on ARC-UPD Dataset  
For the ARC-UPD, we evaluated the models with the following metrics:  
1. Mean square error (MSE), which measures the effectiveness of the training method of the proposed M-CRBM model;
2. Recall@k, precision@k, and F1@k to measure the model performance in a recommendation task;
3. Ablation experiments that compared the conventional RBMs with M-CRBMs, which took different extra information in the conditional layer(s);
4. Recommendation examples, including the predictions of all the models by inputting the same query and typical bad cases. Experimental Results  
Figure 5 shows the MSE scores through the training epochs. The MSE score increased in the early epochs and then dropped sharply until it converged. The MSE score's objective was to indicate the training rule defined by us was effective. RBMs and CRBMs are mostly used as generative models. One of the general testing methods of the generative task is to input the original test data without masking part of them. This is to see whether the model has the ability to reconstruct the input data and also to measure whether the hidden units extract meaningful features that are related to the configurations. A characteristic existing in our dataset was that many of the items had only been searched by one query word. If they were masked in the test data, then the input would be null. Therefore, in this experiment, we did not mask part of the input data and saw whether the model could predict it but used the aforementioned method to calculate the recall, precision, and F1 scores of the test data. Then, we show some recommendation examples to prove that our proposed model found meaningful related queries. In Figure 6, the recall, precision, and F1 scores for @1, @5, @10, and @20 are shown. We found that, although the precision@1 and F1@1 were not stable in the early training epochs, they became stable and outperformed the other metrics when the model converged. One of the reasons that the precision@1 and F1@1 scores were high in the experiments might also be that many of the inputs in the test data only included a single query, which means there could only be one hit. Despite that, this result still proves the ability of the proposed model for learning the important features of an input query (or queries) well. Figure 5: MSE through 25 training epochs of the proposed M-CRBMs that take three types of extra information in the conditional layers. The ablation experiments were conducted to answer the questions \"Is the extra information useful?\" and \"What kind of extra information will be more useful?\". In the experiments, for all types of M-CRBM models (M-CRBM models utilize different extra information), the training data and test data that were clamped to the visible layer were the same, but different types of M-CRBM models take different extra information that corresponds to the visible layer. All types of M-CRBM models were trained with the same parameter settings, except for the number of hidden units. Because one hidden unit extracted one configuration among different input data, it should have been changed when the input changed. Especially when the model took genre information into account, it became hard to train and often had the gradient explosion problem. This might be because genre information is too weighted and does not contribute to detecting the configurations. We set the number of hidden units as the most appropriate for each model. All type of M-CRBM models were trained for 25 epochs. We compared the recall@1, precision@1, and F1@1 of the M-CRBM models that utilized different extra information. The results for the metrics are listed in Table 1.  
The results show that, compared with the RBM model, adding the artist or series name information contributed to the results, but the genre information made the results worse. The series name information contributed the most to all kinds of metrics.