SlowGuess/NAACL_2021_OCR huggingface dataset:
Shuo Zhang and Krisztian Balog. 2018. Ad hoc table retrieval using semantic similarity. In Proceedings of WWW 2018, WWW '18, page 1553-1562.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, and Hai Zhao. 2020a. Sg-net: Syntax-guided machine reading comprehension. AAAI.
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2020b. Retrospective reader for machine reading comprehension. arXiv preprint arXiv:2001.09694.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
Both the MRC and RCI training was carried out using the pytorch transformers toolkit made available by Huggingface. Table 9 gives the number of parameters for each introduced model.
Model	Parameters
MRC	25M
MRCxxl	235M
RCIinter	50M
RCIrepr	75M
Table 9: Number of parameters for introduced models
A.1 MRC Model Training
Models were trained and decoded using single GPU training on machines with 32GB Tesla V100 GPUs with 16-bit precision. This results in processing
Hyperparameter	Setting
Max query tokens	64
Max answer tokens	30
Batch size	32
Optimizer	Adam
ε	1-8
β1	0.9
β2	0.999
Warmup ratio %	10%
Learning rate warmup	Linear
Peak learning rate	3e-5
Epochs	3
Document stride	128
Max sequence length	512
speeds of  $\sim 35 - 75$  features per second (depending on whether it's MRC or  $\mathrm{MRC}_{\mathrm{xxl}}$ ). Note: due to the 512 sequence length limitation, each query-table example may be split into multiple feature vectors as per (Devlin et al., 2019) and subsequent work in building MRC models using transformer networks.
SQuAD 2.0 (SQ2) Training
Table 10 lists the hyperparameters used in line with (Lan et al., 2020) for training MRC on SQ2.
Natural Questions (NQ) Training
Table 11 lists the hyperparameters used in line with (Pan et al., 2019; Alberti et al., 2019b) for training MRC on NQ. Note that the NQ training requires a few modifications from the default SQuAD style QA task to account for the dataset's additional answer types [short, long, yes, no, null]6:
1. A dense layer  $(\mathbf{W}_3 \in \mathbb{R}^{5 \times D_e})$  is added which operates only on the contextualized representation of the [CLS] token to produce a likelihood prediction for each answer type:  $\alpha_l = \text{softmax}(\mathbf{W}_3 \mathbf{h}_{[CLS]})$ .
2. At training time an additional cross entropy loss term is added between the true answer type labels and the predicted answer type likelihoods  $(\alpha_{l})$
3. At prediction time the final score is a simple weighted average of the short answer likeli
Table 10: SQ2 MRC Hyperparameter Configurations
Hyperparameter	Setting
Max query tokens	18
Max HTML spans(top level)	48
Max answer tokens	30
Batch size	48
Optimizer	Adam
ε	1-8
β1	0.9
β2	0.999
Warmup ratio	10%
Learning rate warmup	Linear
Peak learning rate	1.6E-5
Weight decay	0.01
Gradient clipping (norm)	1.0
Epochs	1 (All Questions) + 1 (Table Questions)
Document stride	192
Max sequence length	512
Negative Subsampling(Answerable Questions)	4%
Negative Subsampling(Un-answerable Questions)	1%
Table 11: NQ MRC Hyperparameter Configurations
hood score  $(\alpha_{l = short})$  and the SQUAD like max span score based on  $\alpha_{b}^{t_{j}}$  and  $\alpha_{e}^{t_{i}}$
Table Data Sets
An additional epoch was trained using the same configurations as in table 11 using the datasets discussed in the evaluation section.
A.2 RCI Model Training Hyperparameters
All models were trained with the same settings for Adam, Gradient clipping and Weight decay. The max sequence length for interaction models was 512, while for representation models it was 256 for both question and column sequence representation. We did not test variations on these hyperparameters. Table 12 gives the constant hyperparameters and the range tested for the others.
We manually tuned learning rate (LR), batch size, number of training epochs (E), and fraction of training instances for warmup (Warm) on the development set for each dataset. We also report the number of development runs for each model. The final hyperparameters were selected as those that maximize ROC on the row or column subtask. Table 13 shows these hyperparameters. The  $93\%$
As in (Alberti et al., 2019b), yes and no questions are not handled by the model as they are less than  $2\%$  of the data and long answers are predicted by looking up the top level HTML span which contains the predicted short span.
Hyperparameter	Setting
Optimizer	Adam
ε	1-8
β1	0.9
β2	0.999
Learning rate warmup	Linear
Weight decay	0.01
Gradient clipping (norm)	1.0
Max sequence length
interaction	512
representation	256
Hyperparameter	Range
Learning Rate	2.5e-6 to 5e-5
Batch Size	64, 128
Epochs	2 to 6
Warmup	0% to 93%
Table 12: RCI Hyperparameter Ranges
Model	LR	Batch	E	Warm	Runs
WikiSQL
Row	5e-5	128	2	0.93	8
Colinter	2e-5	64	3	0.13	4
Colrepr	1e-5	64	4	0.11	9
TabMCQ
Row	1e-5	64	2	0.02	7
Colinter	4e-5	64	4	0.09	8
Colrepr	1e-5	64	6	0.10	3
WikiTableQuestions
Row	5e-5	128	2	0.00	3
Colinter	1e-5	64	4	0.00	2
Colrepr	1e-5	64	6	0.00	4
Table 13: RCI Tuned Hyperparameters
warmup instances for WikiSQL Row was selected by adding one too many zeros to the number of warmup instances while aiming for around  $10\%$  warmup. However, this turned out to be the best run.
Training was done on a single machine with four P100 GPUs. For all models, training time was between 70 and 80 instances per second, giving training times for WikiSQL of around two and a half hours per epoch for rows and one hour per epoch for columns.