SlowGuess/NAACL_2021_OCR huggingface dataset:
In our method, we applied both pre-training and fine-tuning to the target neural parser because it is the conventional way to improve neural network-based models. However, this might be different from the usual way of re-training models based on traditional supervised learning in a semi-supervised fashion. To investigate whether the approach with both pre-training and fine-tuning is effective, we compared other training methods, specifically, pretraining alone with the CNN and with both the CNN and the RST-DT, and the comparison results are shown in Table 6. The scores of Nucleus, Relation, and Full were not statistically significantly different from each other, which indicates that the difference between the two methods is minimal.
<sup>9</sup>We utilized Morey's code, available at https://github.com/irit-melodi/duce/.
Model	S	N	R	F
Pretrain w/ RST-DT	85.9	72.4	59.3	58.8
Pretrain w/o RST-DT	86.3	72.3	59.1	58.5
SBP+AST (Ave.)	86.8	74.7	62.5	61.8
SBP+AST (Ens.)	87.1	75.0	63.2	62.6
Two-stage parser	86.0	72.4	59.7	58.8
Table 6: Comparison of the training methods (pretraining alone v.s. pre-training and fine-tuning)
Furthermore, compared with the performance of the two-stage parser, i.e., our teacher parser, it is confirmed that our silver data provides adequate quality. Compared with the fine-tuned models, it is also confirmed that fine-tuning improves performance.
Table 7 shows the hyperparameters of SBP+AST.
Computing Infrastructure	Nvidia TITAN RTX
Training duration (Pre-train)
with lmin=9	20 hours
Training duration (Fine-tune)	30 minutes
Hyperparameters
number of epochs (Pre-train)	5
number of epochs (Fine-tune)	10
batch size (# of documents)	10
embedding	GloVe and ELMo
hidden size	[250, 500]
dropout	0.4
learning rate scheduler	Exponential decay
Scheduler reduction factor	0.99
optimizer	Adam
learning rate	0.001
gradient clipping	5.0
validation criteria	Micro-averaged F1 of Relation
lmin	[5, 6, 7, 8, 9, 10]
lmax	240
Table 7: SBP+AST search space. Values in bold indicate best assignments.