SlowGuess/NAACL_2022_OCR huggingface dataset:
Our model consists of approximately 440 million parameters. To reduce the training time, the parameters that are shared with BERT are initialized using BERT. The other parameters are initialized randomly. The model is trained via iterations over Wikipedia pages in a random order for seven epochs. To stabilize the training, we update only those parameters that are randomly initialized (i.e., fixed the parameters initialized using BERT) at the first epoch, and update all parameters in the remaining six epochs. We implement the model using PyTorch (Paszke et al., 2019) and Hugging Face Transformers (Wolf et al., 2020), and the training takes approximately ten days using eight Tesla V100 GPUs. We optimize the model using AdamW. The hyper-parameters used in the training are detailed in Table 4.
B Details of Fine-tuning on CoNLL Dataset
The hyper-parameters used in the fine-tuning on the CoNLL dataset are detailed in Table 5. We select these hyper-parameters from the search space described in Devlin et al. (2019) based on the accuracy on the development set of the CoNLL dataset. A document is split if it is longer than 512 words, which is the maximum word length of the BERT model.
C Details of ED Datasets
The statistics of the ED datasets used in our experiments are provided in Table 6.
Name	Value
number of hidden layers	24
hidden size	1024
attention heads	16
attention head size	64
activation function	gelu
maximum word length	512
batch size	2048
learning rate (1st epoch)	5e-4
learning rate decay (1st epoch)	none
warmup steps (1st epoch)	1000
learning rate	5e-5
learning rate decay	linear
warmup steps	1000
dropout	0.1
weight decay	0.01
gradient clipping	1.0
adam β1	0.9
adam β2	0.999
adam ε	1e-6
Table 4: Hyper-parameters used for training on Wikipedia entity annotations.
Name	Value
maximum word length	512
number of epochs	2
batch size	16
learning rate	2e-5
learning rate decay	linear
warmup proportion	0.1
dropout	0.1
weight decay	0.01
gradient clipping	1.0
adam β1	0.9
adam β2	0.999
adam ε	1e-6
Table 5: Hyper-parameters during fine-tuning on the CoNLL dataset.
D Example of Inference by Confidence-order Model
Figure 3 shows an example of the inference performed by our confidence-order model fine-tuned on the CoNLL dataset. The document is obtained from the test set of the CoNLL dataset. As shown in the figure, the model starts with unambiguous player names to recognize the topic of the document, and subsequently resolves the mentions that are challenging to resolve.
Notably, the model correctly resolves the mention "Nigel Walker" to the corresponding former rugby player instead of a football player, and the mention "Matthew Burke" to the correct former Australian rugby player born in 1973 instead of
"Campo has a massive following in this country and has had the public with him ever since he first played here in 1984," said Andrew, also likely to be making his final 20: Twickenham appearance. On tour, 17: Australia have won all four tests against 46: Italy, 47: Scotland, 48: Ireland and 45: Wales, and scored 414 points at an average of almost 35 points a game. League duties restricted the 28: Barbarians' selectorial options but they still boast 13 internationals including 44: England full-back 16: Tim Stimpson and recalled wing 22: Tony Underwood, plus 12: All Black forwards 25: Ian Jones and 14: Norm Hewitt. Teams: 27: Barbarians - 15 - 7: Tim Stimpson (31: England); 14 - 50: Nigel Walker (36: Wales), 13 - 1: Allan Bateman (32: Wales), 12 - 10: Gregor Townsend (39: Scotland), 11 - 4: Tony Underwood (34: England); 10 - 17: Rob Andrew (33: England), 9 - 2: Rob Howley (35: Wales); 8 - 15: Scott Quinnell (37: Wales), 7 - 8: Neil Back (38: England), 6 - 19: Dale McIntosh (41: Pontypridd), 5 - 24: Ian Jones (51: New Zealand), 4 - 11: Craig Quinnell (40: Wales), 3 - 5: Darren Garforth (42: Leicester), 2 - 18: Norm Hewitt (52: New Zealand), 1 - 3: Nick Popplewell (49: Ireland). 43: Australia - 15 - 53: Matthew Burke; 14 - 9: Joe Roff, 13 - 26: Daniel Herbert, 12 - 20: Tim Horan (captain), 11 - 23: David Campese; 10 - 29: Pat Howard, 9 - Sam Payne; 8 - Michael Brial, 7 - 30: David Wilson, 6 - 13: Owen Finegan, 5 - 21: David Giffin, 4 - Tim Gavin, 3 - Andrew Blades, 2 - Marco Caputo, 1 - 6: Dan Crowley.
Order of Inference by Confidence-order Model:
Allan Bateman  $\rightarrow$  Rob Howley  $\rightarrow$  Nick Popplewell  $\rightarrow$  Tony Underwood  $\rightarrow$  Darren Garforth  $\rightarrow$  Dan Crowley  $\rightarrow$  Tim Stimpson  $\rightarrow$  Neil Back  $\rightarrow$  Joe Roff  $\rightarrow$  Gregor Townsend  $\rightarrow$  Craig Quinnell  $\rightarrow$  All Black  $\rightarrow$  Owen Finegan  $\rightarrow$  Norm Hewitt  $\rightarrow$  Scott Quinnell  $\rightarrow$  Tim Stimpson  $\rightarrow$  Australia  $\rightarrow$  Norm Hewitt  $\rightarrow$  Dale McIntosh  $\rightarrow$  Tim Horan  $\rightarrow$  David Giffin  $\rightarrow$  Tony Underwood  $\rightarrow$  David Campese  $\rightarrow$  Ian Jones  $\rightarrow$  Ian Jones  $\rightarrow$  Daniel Herbert  $\rightarrow$  Barbarians  $\rightarrow$  Barbarians  $\rightarrow$  Pat Howard  $\rightarrow$  David Wilson  $\rightarrow$  England  $\rightarrow$  Wales  $\rightarrow$  England  $\rightarrow$  England  $\rightarrow$  Wales  $\rightarrow$  Wales  $\rightarrow$  England  $\rightarrow$  Scotland  $\rightarrow$  Wales  $\rightarrow$  Pontypridd  $\rightarrow$  Leicester  $\rightarrow$  Australia  $\rightarrow$  England  $\rightarrow$  Wales  $\rightarrow$  Italy  $\rightarrow$  Scotland  $\rightarrow$  Ireland  $\rightarrow$  Ireland  $\rightarrow$  Nigel Walker  $\rightarrow$  New Zealand  $\rightarrow$  New Zealand  $\rightarrow$  Matthew Burke
Figure 3: An illustrative example showing the inference performed by our fine-tuned confidence-order model on a document in the CoNLL dataset. Mentions are shown as underlined. Numbers in boldface represent the selection order of the confidence-order model.
Name	#mentions	#documents
CoNLL (training)	18,448	946
CoNLL (development)	4,791	216
CoNLL (test)	4,485	231
MSNBC	656	20
AQUaint	727	50
ACE2004	257	36
CWEB	11,154	320
WIKI	6,821	320
Table 6: Statistics of ED datasets.
the former Australian rugby player born in 1964. This is accomplished by resolving other contextual mentions, including their colleague players, in advance. These two mentions are denoted in red in the figure. Note that our local model fails to resolve both mentions, and our natural-order model fails to resolve "Matthew Burke."