Question: How do the authors evaluate the DL and ML methods in the first experiment?
Evidence: In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE. For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training. For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing.
Table TABREF28 presents the results of the first experiment. The DL model with GloVe word embeddings achieved better results on three datasets, with 82.80% Accuracy on SNLI, 78.52% Accuracy on MultiNLI, and 83.62% Accuracy on Quora. Logistic Regression achieved the best Accuracy of 98.60% on Clinical-RQE. We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98.61% Accuracy.
In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions. Table TABREF29 presents the results of this experiment. Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73.18% Accuracy.
To validate further the performance of the LR method, we evaluated it on question similarity detection. A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions. We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA. The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:
 INLINEFORM0 
The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 ). Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets. The hybrid method (LR+IR) provided the best results on both datasets. On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80.57% Accuracy and 77.47% MAP (official system ranking measure in SemEval-cQA). On the cQA-2017 test data, the LR+IR method obtained 44.66% MAP and outperformed the cQA-2017 best system in Accuracy with 67.27%.
In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE. For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training. For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing.
Answer: In the first experiment, the authors evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE. For the datasets that did not have a development and test sets, the authors randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training. For MultiNLI, the authors used the dev1-matched set for validation and the dev2-mismatched set for testing.