SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2 shows the original selection model's performance on different sizes of candidate response sets. The direct predict model is run as it does not need further tuning. Threshold approach, especially with softmax probability as threshold, will need separate rounds of tuning on the threshold. Table 3 shows the complete results for all models on the test set, both for 2 candidates and for 10 candidates. Here, the average F1 is averaged on all 4 F1 scores. For each model architecture, the best performing setting for each metric is in bold.
50% NOTE Test Results (%)
R@10	R@2	N@10	N@2	N F1@10	N F1@2	G F1@10	G F1@2	Average F1
Selection model trained with original data
Direct Predict	56.12	66.77	61.48	78.00	52.82	80.22	67.46	75.21	68.93
+Logistic Regression on Top of Logits	55.98	66.72	87.81	88.19	86.96	87.26	88.56	88.99	87.94
+Logistic Regression on Top of Softmax	50.94	51.93	74.30	74.33	74.46	74.38	74.15	74.29	74.32
Logits Threshold (=0.5)	50.10	55.72	64.28	73.25	62.84	76.73	65.61	68.56	68.43
+Logistic Regression on Top	62.81	77.70	80.45	79.95	80.49	79.92	80.42	79.99	80.20
Softmax Threshold (=0.55)	48.76	48.76	60.10	70.67	59.69	75.63	60.50	63.17	64.74
+Logistic Regression on Top	63.64	69.47	78.50	78.54	80.17	80.20	76.52	76.57	78.36
Selection model trained with data containing NOTA
Direct Predict	55.43	65.03	63.07	78.37	54.28	80.91	69.03	75.04	69.81
+Logistic Regression on Top of Logits	40.66	47.90	78.19	77.45	78.80	78.02	77.53	76.85	77.80
+Logistic Regression on Top of Softmax	51.63	53.90	77.94	78.00	78.21	78.15	77.67	77.85	77.97
Logits Threshold (=2.0)	48.44	55.99	61.32	71.31	57.75	74.35	64.32	67.46	65.97
+Logistic Regression on Top	60.73	76.12	79.22	78.03	79.11	77.85	79.33	78.21	78.62
Softmax Threshold (=0.5)	48.18	48.18	59.06	70.16	57.32	75.19	60.67	62.56	63.94
+Logistic Regression on Top	61.08	68.45	78.01	78.00	79.75	79.74	75.94	75.93	77.84
Pairwise Model
Direct Predict	35.73	40.91	61.72	68.25	63.54	75.07	59.72	56.30	63.66
+LogReg on Top of Logits	35.64	40.73	94.08	94.14	93.72	93.79	94.40	94.46	94.09
+LogReg on Top of Softmax	25.42	27.14	85.06	85.02	85.41	85.34	84.69	84.67	85.03
Logits Threshold (=1.0)	41.64	48.57	61.50	70.01	57.77	74.36	64.62	63.88	65.16
+LogReg on Top	51.58	73.33	77.15	77.27	76.74	76.88	77.55	77.64	77.20
Softmax Threshold (=0.4)	39.70	40.05	54.96	65.90	51.83	72.30	57.70	55.66	59.37
+LogReg on Top	52.00	63.79	74.40	74.33	76.43	76.41	71.99	71.85	74.17
Dropout Model
Direct Predict	28.57	93.47	50.13	62.42	1.48	45.50	66.61	71.32	46.23
+LogReg on Top of Logits	19.21	77.20	66.89	66.72	61.87	61.59	70.74	70.65	66.21
+LogReg on Top of Softmax	21.73	29.37	50.49	54.83	56.37	63.73	42.79	40.15	50.76
Logits Variance Threshold (=0.1)	13.73	22.11	51.89	50.27	57.15	59.13	45.15	36.51	49.48
+LogReg on Top	20.87	60.78	56.13	55.86	40.18	39.29	65.37	65.32	52.54
Softmax Variance Threshold (=0.001)	22.22	36.75	50.03	54.56	38.98	57.64	57.69	50.99	51.32
+LogReg on Top	23.84	26.07	57.21	56.79	60.87	66.47	52.81	39.23	54.85
Table 3: @10 and @2 represent metrics on 10 and 2 candidates respectively.  $R$  represents recall,  $N$  represents binary NOTA classification accuracy,  $NF1$  represents the F1 score on the NOTA class, and  $GF1$  represents the F1 score on the ground-truth-present class. Average F1 is obtained on the 4 F1 scores.