SlowGuess/NAACL_2022_OCR huggingface dataset:
Ricardo Usbeck, Ria Hari Gusmita, Muhamad Saleem, and Axel-Cyrille Ngonga Ngomo. 2018. 9th challenge on question answering over linked data (qald-9). In Joint proceedings of the 4th Workshop on Semantic Deep Learning (SemDeep-4) and NLIWoD4, Vol-2241, pages 58-64.
Rik van Noord and Johan Bos. 2017. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. arXiv preprint arXiv:1705.09980.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS 2017).
Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6456-6466.
Yiren Wang, Lijun Wu, Yingce Xia, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2020. Transductive ensemble learning for neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), pages 6291-6298.
Andrew Gordon Wilson and Pavel Izmailov. 2020. Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv:2002.08791.
Qingrong Xia, Zhenghua Li, Rui Wang, and Min Zhang 2021. Stacked amr parsing with silver data. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4729-4738.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving amr parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2501-2511.
Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2021. XLpt-amr: Cross-lingual pretraining via multi-task learning for zero-shot amr parsing and text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 896–907.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Knowledge distillation for improved accuracy in spoken question answering. In ICASSP 2021.
Yang Ze, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2020. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In The Thirteenth ACM International Conference on Web Search and Data Mining.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. Amr parsing as sequence-to-graph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 80-94.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b. Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJC-NLP), pages 3784-3796.
Jiawei Zhou, Tahira Naseem, Ramon Fernandez As-tudillo, and Radu Florian. 2021a. Amr parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585-5598.
Jiawei Zhou, Tahira Naseem, Ramon Fernandez Astudillo, Young-Suk Lee, Radu Florian, and Salim Roukos. 2021b. Structure-aware fine-tuning of sequence-to-sequence transformers for transition-based amr parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6279-6290.
A Greedy-Select Ensemble Algorithm
We start with  $n$  parses from  $n$  heterogeneous parsing models, where the minimum number of parses is 3. For each input sentence, we compute sentence-level Smatch scores between any two parses across all  $n$  parses, for a total of  $n(n - 1) / 2$  Smatch scores (lines 2-3). Subsequently, we pick the two parses  $\mathrm{AMR}_i$  and  $\mathrm{AMR}_j$  with the highest Smatch score, where  $\mathrm{AMR}_i$  denotes the AMR parse from the system  $i$  (line 4) For each of the two parses,  $\mathrm{AMR}_i$  and  $\mathrm{AMR}_j$ , we choose the parse with the higher Smatch score against the rest of the parses as the best parse (lines 5-11). When the scores are tied, we select the first parse output (equivalent to a random choice of fixed seed).
We incorporate an optional parse selection criterion into Algorithm 1, indicated as Optionally Require and specified in lines 12-15. The bestAMR for input sentence is selected for data augmentation if the Smatch score  $smatch(\mathrm{AMR}_a, \mathrm{AMR}_b)$  is greater than or equal to the pre-specified value  $\theta$ .
Models	AMR2.0	AMR3.0
Structured-BART	34,156	33,200
SPRING1	25,129	29,407
SPRING2	17,866	17,830
APT	10,235	6,949
Total	87,386	87,386
Table 8: Distribution of individual model parses from MBSE greedy-select distillation with silver $^1$  dataset in Table 1
Model	Param	AMR2.0	AMR3.0
sep-voc	src voc size	50,265	50,265
tgt voc size	42,344	42,784
# param	493,011,968	493,913,088
joint-voc	joint voc size	57,912	58,673
# param	414,121,984	414,901,248
Table 9: Vocabulary and parameter sizes of Structured-BART with MBSE distillation on silver $^{1+2+3}$  dataset from Table 1
B Model Structures and Parameter Size
Pre-trained BART and mBART share the same model configurations except for the vocabulary size. There are 12 encoder/decoder layers, 16 heads per layer, 1024 model dimension and 4096 feed forward network (FFN) size. BART includes  $\sim 50\mathrm{K}$  and mBART,  $\sim 250\mathrm{K}$  task vocabulary.
When using separate vocabulary (sep-voc), Structured-BART and Structured-mBART use the same vocabulary as BART and mBART, respectively, for the source. For the target, they create embedding vectors for action symbols and the target vocabulary size varies according to the training data. When using joint vocabulary (joint-voc), Structured-BART shares the same vocabulary between the source and the target, a combination of BART vocabulary and the additional embedding vectors for some action symbols.
Vocabulary and parameter sizes for Structured-BART and Structured-mBART trained with MBSE distillation are shown in Table 9 and Table 10, respectively.
Our models are implemented with FAIRSEQ toolkit (Ott et al., 2019), trained and tested on a single NVIDIA Tesla A100/V100 GPU with 40-80GB memory. We use fp16 mixed precision training and all models are trained on 1 GPU.
For all English AMR parsing models with silver data, we use the Adam optimizer with  $\beta_{1} = 0.9$
Languages	voc size	# param
DE (German)	34,689	681,894,912
ES (Spanish)	34,881	682,288,128
IT (Italian)	33,681	679,830,528
ZH (Chinese)	59,473	732,652,544
Table 10: Target vocabulary (sep-voc) and parameter sizes of Structured-mBART with MBSE distillation on silver $^{1+2+3}$  dataset from Table 1. Source vocabulary size is 250,027 across all languages.
Lgs.	vocab	base model	ens. model
EN	joint-voc	60min	60min
DE	sep-voc	23min	42min
ES	sep-voc	24min	44min
IT	sep-voc	22min	40min
ZH	sep-voc	30min	60min
Table 11: Inference time for AMR2.0 test set. Base models are trained on AMR2.0 treebank only and ens. models are trained on AMR2.0 treebank plus silver $^{1+2+3}$ .
and  $\beta_{2} = 0.98$ . Batch size is set to 1024 maximum number of tokens with gradient accumulation over 8 steps. Learning rate schedule is the same as Vaswani et al. (2017) with 4000 warm-up steps and  $1\mathrm{e} - 7$  warm-up initial learning rate and the maximum learning rate  $1\mathrm{e} - 4$ . Dropout rate is 0.2 and label smoothing rate is 0.01. These hyper parameters are fixed and not tuned for different models and datasets. All models are trained for 10 epochs and the best 5 checkpoints are selected based on the development set Smatch from greedy decoding. Model parameters are averaged over the top 3 and top 5 models. The model that produces the highest development set score, after beam search decoding with beam size  $= 1$ , 5 and 10, is selected as the final model. Training with MBSE greedy-select silver $^{1 + 2 + 3}$  takes 48-72 hours, and all other models with less silver data take less time to train.
For cross-lingual AMR parsing, maximum learning rate is always set to  $3\mathrm{e} - 5$ . Baseline models trained only on AMR2.0 corpus are trained up to 80 epochs whereas models with silver<sup>1</sup> (and AMR-to-text) is trained up to 30 epochs and models with silver<sup>1+2+3</sup>, up to 15 epochs. Model parameters are updated after gradient is accumulated for 8192 tokens. Dropout rate, label smoothing rate and model selection criteria are the same as the English parsers. Training baseline models takes about 10 hours. Training with silver<sup>1</sup> takes about 24 hours. Training with silver<sup>1+2+3</sup> takes about 96-120 hours.
In order to reduce the vocabulary size, which subsequently reduces the model parameter size and memory requirement, we prune out singleton target vocabulary for training with silver data.
Inference time for AMR2.0 benchmark test set is shown in Table 11, where beam size=10 and batch size=64 for all languages. EN is decoded on NVIDIA Tesla A100 and all other languages, on NVIDIA Tesla V100.