SlowGuess/NAACL_2022_OCR huggingface dataset:
Yes	ProbYes
P	R	F1	P	R	F1
Majority Baseline	0.42	1.00	0.59	0.00	0.00	0.00
RoBERTa without context and tuned with ...
MNLI	0.34	0.38	0.32	0.00	0.00	0.00
BoolQ	0.49	0.52	0.50	0.00	0.00	0.00
Circa	0.51	0.69	0.58	0.33	0.02	0.04
MNLI+BoolQ	0.28	0.34	0.32	0.00	0.00	0.00
MNLI+Circa	0.40	0.43	0.42	0.17	0.06	0.09
BoolQ+Circa	0.15	0.49	0.22	0.00	0.00	0.00
MNLI+BoolQ+Circa	0.48	0.52	0.50	0.00	0.00	0.00
SwDA-IA_Q	0.38	0.48	0.45	0.17	0.19	0.18
SwDA-IA_A	0.58	0.62	0.60	0.29	0.30	0.29
SwDA-IA_QA	0.60	0.64	0.62	0.30	0.34	0.31
MNLI+SwDA-IA_QA	0.56	0.60	0.58	0.19	0.3	0.25
BoolQ+SwDA-IA_QA	0.55	0.59	0.58	0.15	0.49	0.22
Circa+SwDA-IA_QA	0.59	0.63	0.60	0.22	0.30	0.27
MNLI+BoolQ+SwDA-IA_QA	0.60	0.64	0.62	0.30	0.35	0.32
MNLI+Circa+SwDA-IA_QA	0.60	0.64	0.62	0.24	0.30	0.29
BoolQ+CircaQ+SwDA-IA_QA	0.61	0.65	0.64	0.15	0.49	0.22
MNLI+BoolQ+Circa+SwDA-IA_QA	0.52	0.56	0.54	0.28	0.32	0.30
RoBERTa with full context and tuned with ...
MNLI	0.22	0.11	0.12	0.00	0.00	0.00
BoolQ	0.28	0.35	0.32	0.00	0.00	0.00
Circa	0.54	0.56	0.55	0.28	0.02	0.03
MNLI+BoolQ	0.28	0.32	0.30	0.00	0.00	0.00
MNLI+Circa	0.18	0.22	0.20	0.29	0.02	0.02
BoolQ+Circa	0.48	0.53	0.50	0.22	0.28	0.24
MNLI+BoolQ+Circa	0.38	0.29	0.34	0.25	0.30	0.28
SwDA-IA_Q	0.59	0.54	0.57	0.30	0.05	0.08
SwDA-IA_A	0.50	0.56	0.53	0.21	0.23	0.22
SwDA-IA_QA	0.58	0.59	0.59	0.22	0.26	0.24
MNLI+SwDA-IA_QA	0.50	0.58	0.54	0.33	0.41	0.38
BoolQ+SwDA-IA_QA	0.56	0.62	0.59	0.18	0.22	0.20
Circa+SwDA-IA_QA	0.62	0.56	0.58	0.29	0.07	0.11
MNLI+BoolQ+SwDA-IA_QA	0.57	0.54	0.56	0.12	0.23	0.18
MNLI+Circa+SwDA-IA_QA	0.54	0.60	0.58	0.24	0.18	0.20
BoolQ+Circa+SwDA-IA_QA	0.55	0.61	0.58	0.22	0.26	0.24
MNLI+BoolQ+Circa+SwDA-IA_QA	0.59	0.54	0.56	0.24	0.16	0.19
RoBERTa with annotator-selected context and tuned with ...
MNLI	0.28	0.33	0.30	0.00	0.00	0.00
BoolQ	0.18	0.23	0.20	0.00	0.00	0.00
Circa	0.59	0.58	0.58	0.30	0.02	0.04
MNLI+BoolQ	0.28	0.32	0.30	0.00	0.00	0.00
MNLI+Circa	0.57	0.59	0.58	0.10	0.18	0.14
BoolQ+Circa	0.59	0.58	0.58	0.28	0.03	0.06
MNLI+BoolQ+Circa	0.54	0.60	0.56	0.22	0.26	0.24
SwDA-IA_Q	0.52	0.56	0.54	0.20	0.24	0.22
SwDA-IA_A	0.56	0.60	0.58	0.26	0.22	0.24
SwDA-IA_QA	0.58	0.64	0.62	0.20	0.28	0.24
MNLI+SwDA-IA_QA	0.59	0.65	0.62	0.30	0.32	0.31
BoolQ+SwDA-IA_QA	0.54	0.62	0.58	0.26	0.34	0.30
Circa+SwDA-IA_QA	0.56	0.60	0.58	0.18	0.14	0.16
MNLI+BoolQ+SwDA-IA_QA	0.58	0.62	0.59	0.20	0.28	0.24
MNLI+Circa+SwDA-IA_QA	0.62	0.66	0.64	0.38	0.43	0.40
BoolQ+Circa+SwDA-IA_QA	0.60	0.65	0.62	0.26	0.29	0.27
MNLI+BoolQ+Circa+SwDA-IA_QA	0.63	0.66	0.64	0.38	0.32	0.30
Table 8: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the Yes and Probably Yes labels. These results complement Table 3 in the main paper.
Middle
P	R	F1
Majority Baseline	0.00	0.00	0.00
RoBERTa without context and tuned with ...
MNLI	0.22	0.18	0.14
BoolQ	0.00	0.00	0.00
Circa	0.35	0.02	0.05
MNLI+BoolQ	0.00	0.00	0.00
MNLI+Circa	0.00	0.00	0.00
BoolQ+Circa	0.00	0.00	0.00
MNLI+BoolQ+Circa	0.13	0.48	0.20
SwDA-IA_Q	0.14	0.50	0.22
SwDA-IA_A	0.50	0.29	0.40
SwDA-IA_QA	0.29	0.35	0.33
MNLI+SwDA-IA_QA	0.39	0.46	0.44
BoolQ+SwDA-IA_QA	0.30	0.35	0.32
Circa+SwDA-IA_QA	0.19	0.31	0.24
MNLI+BoolQ+SwDA-IA_QA	0.24	0.35	0.30
MNLI+Circa+SwDA-IA_QA	0.24	0.29	0.28
BoolQ+Circa+SwDA-IA_QA	0.38	0.50	0.45
MNLI+BoolQ+Circa+SwDA-IA_QA	0.28	0.38	0.34
RoBERTa with full context and tuned with ...
MNLI	0.23	0.18	0.20
BoolQ	0.00	0.00	0.00
Circa	0.00	0.00	0.00
MNLI+BoolQ	0.00	0.00	0.00
MNLI+Circa	0.22	0.26	0.24
BoolQ+Circa	0.00	0.00	0.00
MNLI+BoolQ+Circa	0.20	0.04	0.08
SwDA-IA_Q	0.22	0.28	0.26
SwDA-IA_A	0.25	0.27	0.26
SwDA-IA_QA	0.31	0.32	0.32
MNLI+SwDA-IA_QA	0.14	0.18	0.16
BoolQ+SwDA-IA_QA	0.27	0.21	0.23
Circa+SwDA-IA_QA	0.33	0.27	0.3
MNLI+BoolQ+SwDA-IA_QA	0.14	0.18	0.16
MNLI+Circa+SwDA-IA_QA	0.19	0.15	0.18
BoolQ+Circa+SwDA-IA_QA	0.24	0.28	0.26
MNLI+BoolQ+Circa+SwDA-IA_QA	0.18	0.22	0.20
RoBERTa with annotator-selected context and tuned with ...
MNLI	0.19	0.01	0.02
BoolQ	0.00	0.00	0.00
Circa	0.30	0.02	0.04
MNLI+BoolQ	0.00	0.00	0.00
MNLI+Circa	0.29	0.02	0.04
BoolQ+Circa	0.32	0.03	0.06
MNLI+BoolQ+Circa	0.00	0.00	0.00
SwDA-IA_Q	0.10	0.16	0.12
SwDA-IA_A	0.38	0.44	0.40
SwDA-IA_QA	0.39	0.41	0.40
MNLI+SwDA-IA_QA	0.35	0.28	0.30
BoolQ+SwDA-IA_QA	0.36	0.33	0.35
Circa+SwDA-IA_QA	0.48	0.52	0.50
MNLI+BoolQ+SwDA-IA_QA	0.26	0.34	0.30
MNLI+Circa+SwDA-IA_QA	0.40	0.44	0.42
BoolQ+Circa+SwDA-IA_QA	0.38	0.44	0.40
MNLI+BoolQ+Circa+SwDA-IA_QA	0.35	0.40	0.37
Table 9: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the Middle label. These results complement Table 3 in the main paper.
No	ProbNo
P	R	F1	P	R	F1
Majority Baseline	0.00	0.00	0.00	0.00	0.00	0.00
RoBERTa without context and tuned with ...
MNLI	0.00	0.00	0.00	0.00	0.00	0.00
BoolQ	0.00	0.00	0.00	0.00	0.00	0.00
Circa	0.37	0.34	0.35	0.25	0.01	0.03
MNLI+BoolQ	0.00	0.00	0.00	0.00	0.00	0.00
MNLI+Circa	0.23	0.16	0.14	0.25	0.02	0.04
BoolQ+Circa	0.18	0.50	0.24	0.00	0.00	0.00
MNLI+BoolQ+Circa	0.30	0.02	0.03	0.21	0.04	0.04
SwDA-IA_Q	0.20	0.16	0.18	0.15	0.48	0.22
SwDA-IA_A	0.50	0.38	0.31	0.18	0.05	0.09
SwDA-IA_QA	0.43	0.44	0.44	0.15	0.06	0.08
MNLI+SwDA-IA_QA	0.56	0.30	0.36	0.14	0.21	0.16
BoolQ+SwDA-IA_QA	0.34	0.38	0.36	0.28	0.04	0.06
Circa+SwDa-IA_QA	0.35	0.37	0.36	0.15	0.48	0.21
MNLI+BoolQ+SwDA-IA_QA	0.28	0.34	0.32	0.12	0.30	0.18
MNLI+Circa+SwDA-IA_QA	0.28	0.33	0.31	0.21	0.23	0.22
BoolQ+Circa+SwDA-IA_QA	0.38	0.44	0.42	0.22	0.26	0.24
MNLI+BoolQ+Circa+SwDA-IA_QA	0.33	0.34	0.34	0.19	0.21	0.20
RoBERTa with full context and tuned with ...
MNLI	0.00	0.00	0.00	0.00	0.00	0.00
BoolQ	0.00	0.00	0.00	0.00	0.00	0.00
Circa	0.15	0.25	0.20	0.00	0.00	0.00
MNLI+BoolQ	0.28	0.10	0.14	0.00	0.00	0.00
MNLI+Circa	0.24	0.12	0.16	0.28	0.03	0.05
BoolQ+Circa	0.28	0.05	0.07	0.28	0.13	0.16
MNLI+BoolQ+Circa	0.30	0.06	0.10	0.00	0.00	0.00
SwDA-IA_Q	0.22	0.26	0.24	0.20	0.01	0.02
SwDA-IA_A	0.34	0.40	0.38	0.20	0.10	0.12
SwDA-IA_QA	0.38	0.42	0.39	0.10	0.18	0.14
MNLI+SwDA-IA_QA	0.40	0.36	0.38	0.19	0.10	0.14
BoolQ+SwDA-IA_QA	0.40	0.34	0.36	0.00	0.00	0.00
Circa+SwDa-IA_QA	0.34	0.40	0.38	0.00	0.00	0.00
MNLI+BoolQ+SwDA-IA_QA	0.30	0.34	0.32	0.11	0.10	0.10
MNLI+Circa+SwDA-IA_QA	0.29	0.32	0.31	0.20	0.07	0.11
BoolQ+Circa+SwDA-IA_QA	0.24	0.32	0.28	0.28	0.10	0.14
MNLI+BoolQ+Circa+SwDA-IA_QA	0.16	0.20	0.18	0.20	0.14	0.16
RoBERTa with annotator-selected context and tuned with ...
MNLI	0.10	0.18	0.14	0.00	0.00	0.00
BoolQ	0.00	0.00	0.00	0.00	0.00	0.00
Circa	0.32	0.40	0.38	0.00	0.00	0.00
MNLI+BoolQ	0.20	0.25	0.21	0.00	0.00	0.00
MNLI+Circa	0.36	0.29	0.33	0.00	0.00	0.00
BoolQ+Circa	0.29	0.02	0.04	0.28	0.02	0.04
MNLI+BoolQ+Circa	0.00	0.00	0.00	0.00	0.00	0.00
SwDA-IA_Q	0.20	0.26	0.23	0.08	0.16	0.11
SwDA-IA_A	0.33	0.39	0.36	0.12	0.20	0.17
SwDA-IA_QA	0.40	0.44	0.42	0.11	0.14	0.12
MNLI+SwDA-IA_QA	0.36	0.40	0.38	0.10	0.16	0.12
BoolQ+SwDA-IA_QA	0.44	0.39	0.41	0.16	0.13	0.14
Circa+SwDa-IA_QA	0.45	0.55	0.51	0.14	0.11	0.12
MNLI+BoolQ+SwDA-IA_QA	0.34	0.40	0.37	0.18	0.20	0.19
MNLI+Circa+SwDA-IA_QA	0.40	0.43	0.42	0.16	0.17	0.16
BoolQ+Circa+SwDA-IA_QA	0.41	0.40	0.40	0.13	0.22	0.17
MNLI+BoolQ+Circa+SwDA-IA_QA	0.36	0.44	0.41	0.13	0.17	0.15
Table 10: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the No and Probably No labels. These results complement Table 3 in the main paper.
All labels	Yes	ProbYes	Middle	ProbNo	No
P	R	F1	F1	F1	F1	F1	F1
Majority Baseline	0.22	0.40	0.28	0.61	0.00	0.00	0.00	0.00
RoBERTa without context and tuned with ...
MNLI	0.14	0.06	0.08	0.30	0.00	0.02	0.00	0.14
BoolQ	0.22	0.16	0.20	0.36	0.00	0.00	0.00	0.12
Circa	0.23	0.32	0.27	0.55	0.03	0.00	0.04	0.20
MNLI+Circa	0.24	0.14	0.18	0.30	0.05	0.08	0.00	0.14
MNLI+BoolQ+Circa	0.20	0.18	0.20	0.43	0.00	0.00	0.00	0.18
SwDA-IA_Q	0.20	0.25	0.18	0.30	0.24	0.06	0.00	0.10
SwDA-IA_A	0.31	0.37	0.32	0.56	0.13	0.04	0.00	0.34
SwDA-IA_QA	0.28	0.37	0.34	0.58	0.19	0.18	0.11	0.30
MNLI+SwDA-IA_QA	0.29	0.36	0.34	0.56	0.12	0.29	0.00	0.40
BoolQ+Circa+SwDA-IA_QA	0.26	0.39	0.35	0.58	0.16	0.22	0.12	0.32
MNLI+BoolQ+Circa+SwDA-IA_QA	0.31	0.32	0.30	0.50	0.19	0.21	0.10	0.20
RoBERTa with full context and tuned with ...
MNLI	0.17	0.13	0.10	0.12	0.00	0.15	0.00	0.00
BoolQ	0.18	0.16	0.09	0.25	0.00	0.00	0.00	0.00
Circa	0.24	0.18	0.20	0.42	0.09	0.14	0.00	0.04
BoolQ+Circa	0.20	0.30	0.22	0.44	0.22	0.00	0.10	0.07
MNLI+BoolQ+Circa	0.18	0.22	0.20	0.30	0.24	0.05	0.00	0.10
SwDA-IA_Q	0.25	0.15	0.16	0.36	0.09	0.00	0.03	0.12
SwDA-IA_A	0.28	0.32	0.30	0.43	0.17	0.23	0.21	0.18
SwDA-IA_QA	0.34	0.38	0.32	0.54	0.18	0.24	0.10	0.21
MNLI-SwDA-IA_QA	0.32	0.37	0.32	0.54	0.22	0.12	0.11	0.23
BoolQ+Circa+SwDA-IA_QA	0.29	0.33	0.30	0.44	0.18	0.23	0.22	0.18
MNLI+BoolQ+Circa+SwDA-IA_QA	0.20	0.27	0.25	0.40	0.10	0.18	0.10	0.15
RoBERTa with annotator-selected context and tuned with ...
SwDA-IA_QA	0.27	0.35	0.33	0.55	0.20	0.12	0.11	0.23
BoolQ+SwDA-IA_QA	0.36	0.34	0.34	0.52	0.24	0.18	0.08	0.30
MNLI+Circa+SwDA-IA_QA	0.39	0.37	0.38	0.55	0.24	0.30	0.08	0.35
MNLI+BoolQ+Circa+SwDA-IA_QA	0.30	0.35	0.32	0.48	0.17	0.23	0.18	0.24
Table 11: Out-of-domain evaluation, i.e., testing with 200 questions from MRDA (Section 5.1.1). These results complement Table 3 in the main paper, which present in-domain evaluation (i.e., testing with SwDA-IA).
Error Type	%	Gold	Pred.
Long sentences	21.20	ProbYes	Yes
15.51	Yes	ProbYes
9.49	Yes	Middle
6.01	ProbNo	No
3.80	ProbNo	Yes
3.80	Yes	No
2.37	Yes	ProbNo
Explicit yes in context	11.87	Middle	Yes
7.59	ProbNo	Yes
Negation in ... question or ind. answer	7.59	Yes	No
3.48	Yes	ProbNo
in context	3.80	Yes	No
3.80	Yes	ProbNo
Table 12: Qualitative analysis of the errors made by our best model (Table 3, second row from the bottom). We indicate the frequency of the most common combinations of gold and predicted labels for each error type. Context here does not include the yes-no question and indirect answer. This table complements Section 6 in the main paper.