SlowGuess/ACL_2020_OCR huggingface dataset:
CONT-1-AGT VERB-1-★ NUM-1 ATTR-1 ENT-1 .
CONT-1-AGT VERB-1-POS NUM-1 ATTR-1 ENT-1 and CONT-2-AGT VERB-1-POS NUM-2 ATTR-1 ENT-1 .
CONT-1-AGT VERB-1-POS NUM-1 ATTR-1 ENT-1 and NUM-2 ATTR-2 ENT-2 .
CONT-1-AGT VERB-1-POS NUM-1 ATTR-1 ENT-1 , but VERB-2-NEG NUM-2 ATTR-2 ENT-2 .
CONT-1-AGT VERB-1-POS NUM-1 ATTR-1 ENT-1 in ATTR-2 CONT-2-ENV .
CONT-1-AGT VERB-1-NEG NUM-1 of the ATTR-1 ENT-1 .
CONT-1-AGT had NUM-1 ATTR-1 ENT-1 , CONT-2-AGT had NUM-2 ATTR-1 ENT-1 , and CONT-3-AGT had NUM-3 ATTR-1 ENT-1 .
NUM-1 ATTR-1 ENT-1 , NUM-2 ATTR-2 ENT-2 , and NUM-3 ATTR-3 ENT-3 were VERB-1-POS in ATTR-4 CONT-1-ENV .
There were NUM-1 ATTR-1 ENT-1 and NUM-2 ATTR-2 ENT-2 in ATTR-3 CONT-1-ENV .
There were NUM-1 ATTR-1 ENT-1 in ATTR-2 CONT-1-ENV .
CONT-1-AGT VERB-1-NEGTRN NUM-1 ATTR-1 ENT-1 to CONT-2-AGT .
CONT-1-AGT VERB-1-POSTRN NUM-1 ATTR-1 ENT-1 from CONT-2-AGT .
Table 8: Sentence templates for synthetic textual examples.
Reasoning	Templates
Selection	How many ATTR-1 ENT-1 were in CONT-1-ENV? How many ATTR-1 ENT-1 did CONT-1-AGT VERB-POS?
Intra-entity difference	How many more ATTR-1 ENT-1 were in CONT-1-ENV than ATTR-2 ENT-2 ? How many more ATTR-1 ENT-1 did CONT-1-AGT have than ATTR-2 ENT-2 ?
Intra-entity subset	How many ENT-1 of CONT-1 were ATTR-1 ENT-1 ? How many ENT-1 of CONT-1 were not ATTR-1 ENT-1 ?
Inter-entity comparison	Were there {more | less} ATTR-1 ENT-1 in CONT-1-ENV or in CONT-2-ENV ? Who had {more | less} ATTR-1 ENT-1, CONT-1-AGT or CONT-2-AGT ?
Inter-entity superlative	Who had the {highest | lowest} number of ATTR-1 ENT-1 in total ?
Intra-entity superlative	What was the {highest | lowest} number of ATTR-1 ENT-1 VERB-POS in CONT-1-ENV ? What is the {highest | lowest} number of ATTR-1 ENT-1 CONT-1-AGT VERB-POS ?
Inter-entity sum	How many ATTR-1 ENT-1 were in CONT-1-ENV (, CONT-★-ENV) and CONT-2-ENV {in total | combined} ? How many ATTR-1 ENT-1 did CONT-1-ENV (, CONT-★-ENV) and CONT-2-ENV have {in total | combined} ?
Table 9: Templates for questions about generated synthetic passages, testing for numerical reasoning. The template placeholders are filled-in with values from the world state obtained after generating the synthetic passage.
pre-training	finetuning
lr	bsz	epochs	lr	bsz
GENBERT	-	-	-	3e-5	16
GENBERT+ND	6e-5	800	60	3e-5	16
GENBERT+ND-LM	4e-5	440	60	3e-5	16
GENBERT+ND-LM-DT	4e-5	440	60	-	-
GENBERT+ND-LM-RS	4e-5	440	60	3e-5	16
GENBERT+TD	1e-5	240	5	3e-5	14
GENBERT+ND+TD	1e-5	240	5	3e-5	14
Table 10: Hyperparameters used for pre-training GENBERT and finetuning it on DROP. lr=leaning rate, bsz=train batch size. Common params: seed=42, optimizer=Bert-Adam, linear-lr-warm-up=0.1, num epochs for finetuning=30, weight-decay=0.01, max-grad-norm=1.0.
Error category	Example
Counting	q: How many people were the heads of the business?
Sorting	q: Which nationality was the fourth largest?
Complex calculation	q: How many percent of people were either Black Hispanic, of Sub-Saharan African origin, or of West Indian or Afro-Caribbean American origin?
Complex semantics	q: By how many points did the Pistons lose their closest game?
Not numerical	q: Who defeated the Kievan Rus at the Battle of the Alta River?
Longer span	q: Where there more people in the peninsula pre-war or at the time of the first census?
a: pre-war
p: pre-war population
Shorter span	q: Was the life expectancy in 2015 higher for males or females?
a: females
p: female
Imprecise number prediction	q: How many more estimated Chinese Americans lived in California compared to Massachusetts?
a: 1130100
p: 110100
Table 11: Error categories of  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  on the development set of DROP, based on a manual error analysis of 85 random examples. The upper part shows categories which are not not covered by our pre-training tasks or do not require numerical skills. The lower part shows categories of inaccurate model predictions. The letters q, a and p denote the question, gold answer and model prediction, respectively.