SlowGuess/NAACL_2022_OCR huggingface dataset:
scores on most domains, comparing to both pretrained QA models and self-training baselines. The results showed that the RGX model achieves 7.7 and 3.0 average F1 improvement over ELECTRA, the SOTA pretrained language model for QA, by pretraining on NQ and SQuAD respectively. The improvement over previous SOTA self-training QA methods, QAGen2S and SynQA, is also significant on both pretraining corpora, although SynQA applies complicated adversarial QA annotation. The largest gain we got is adapting NQ model to TextbookQA domain, increasing 18.0 EM and 19.4 F1 scores. Note that our model still outperforms all baselines without MMI. The performance on the DROP benchmark drops since DROP requires multi-step reasoning, but the synthetic generation model tends to generate safe question-answer pairs. We also found that without selecting harder questions with SEM in RGX, the performance is significantly lower. These facts indicate that the QA model needs hard training examples for better performance, and explains the good performance of SynQA on DROP. For the same reason, the performance drop led by removing EM from RGX is significantly larger when the QG model is pretrained on SQuAD, since SQuAD questions are more coherent with the context than NQ, and selecting simple questions for RGX training will encourage the model to generate trivial questions, which is harmful for the QA training.
4.4.1 Answer Entity Recognition
We first compare the performance of different AER models and strategies by setting NQ as the source domain and SQuAD 1.1 as the target domain in Table 4. The results showed that the choice of AER model and strategy significantly influences the final QA performance. The low performance of the NER model trained on CONLL shared task suggests the importance of our AER module. We notice that the improvement from the cooperative learning over the pretrained models is higher in the zero-annotation setting than the semi-annotated task. The observation indicates that the model trained with RGX is more robust against the automatically recognized answer entities. More details about the AER methods are shown in Appendix C.
The AER method also enables and improves the maximum mutual information (MMI) inference in test time. Table 2 shows that MMI achieves the
Model
Domain	BioASQ
Bio	TextbookQA
Book	RACE
Exam	RelExt.
Wiki	DuoRC
Movie	DROP
Wiki	Avg
EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
Source Domain: NaturalQuestionswiki, Method: Extraction
ELECTRA	41.9	59.0	31.9	41.5	32.4	43.4	67.7	81.8	40.0	48.5	39.3	51.1	42.2	54.2
QAGen2S	43.2	64.1	39.9	51.7	33.7	45.5	71.6	84.4	43.8	53.2	24.2	37.1	42.7	56.0
RGX (Ours)	50.3	70.1	49.9	60.9	40.3	52.4	76.1	87.2	47.8	58.4	27.6	42.1	48.7	61.9
- w/o MMI	49.7	69.1	49.4	60.6	39.7	51.5	75.4	86.7	46.9	57.5	27.1	41.7	46.8	61.2
- w/o EM	48.2	67.9	47.4	59.8	38.3	50.5	74.1	86.2	46.6	56.9	26.1	40.9	46.8	60.4
- w/o CST	45.4	66.4	41.9	53.8	35.1	47.2	72.7	85.4	45.5	54.9	24.6	37.9	44.2	57.6
Source Domain: SQuADwiki (SQuAD+AQA+Wiki for SynQA), Method: Extraction
ELECTRA	58.7	73.1	43.0	53.6	38.3	52.5	79.0	88.4	53.1	64.2	48.3	60.8	53.4	65.4
QAGen2S	56.8	71.7	48.0	56.5	43.4	54.9	73.4	84.8	53.3	64.6	42.2	54.5	52.8	64.5
SynQA	55.1	68.7	41.4	50.2	40.2	54.2	78.9	88.6	51.7	62.1	64.9	73.0	55.3	66.1
RGX (Ours)	60.3	74.8	51.2	61.2	44.9	58.7	79.2	88.6	57.4	66.2	47.6	60.9	56.8	68.4
- w/o MMI	59.2	73.6	50.1	60.4	46.3	57.6	78.9	88.5	56.2	65.7	46.9	60.6	56.3	67.7
- w/o EM	52.1	64.0	50.6	58.9	35.4	48.3	75.6	85.9	55.6	64.9	40.7	53.2	51.7	62.5
- w/o CST	57.5	72.1	48.6	57.0	43.8	55.2	74.3	85.3	53.9	65.3	43.0	55.1	53.5	65.0
Source Domain: SQuADwiki, Method: Prompt Tuning + Seq2seq Generation
T5	54.6	71.1	37.9	61.9	15.0	53.1	74.5	86.5	48.2	65.2	40.4	51.9	45.1	64.9
T5 + RGX	55.1	71.6	41.1	64.2	15.5	55.1	75.9	87.1	49.5	66.2	42.9	53.8	46.7	66.3
Table 2: The QA performance evaluation on the out-of-domains of the MRQA benchmark. All models used are pretrained on the human-labeled training set from the source domains, and the QA models are finetuned on synthetic data generated based on the unannotated passages of the target domains. The finetuned QA models are evaluated on human-generated evaluation data for each target domains with the exact match (EM) and F1 scores. MMI stands for maximum mutual information inference, EM stands for involving difficult questions with EM selection, and CST stands for cooperative self-training.
QAGen2S	SynQA	RGX
Pretraining	XQ	SQ+AQA	XQ
Synthesis	Target	Wikipedia	Target
Finetuning	XQ+Syn	SQ+AQA+Syn	XQ+Syn
AER Model	None	None	ELECTRA
Coop. ST	No	No	Yes
QA Num.	1M	1.5M	0.3M
best performance, and we also show that the MMI accuracy is hurt without AER. Table 5 shows that MMI grounded on AER constantly outperform the ELECTRA model, but grounding on top-k seriously hurts the EM scores. Some invalid answer predictions leads to low question generation perplexities, which makes MMI inference noisy. Table 6 shows that the QG model generated more diverse questions based on the AER outputs.
4.4.2 Synthetic QA Selection with EM
Previous experiments showed that selecting nontrivial synthetic QA pairs is essential for RGX to achieve high performance. Table 2 shows that the performance of cooperative self-trained RGX is much lower than the pretrained baseline without
Table 3: Comparison of different self-training methods. XQ stands for "NaturalQuestions (NQ) or SQuAD (SQ)". QA Num. stands for the number of synthetic QA pairs used for self-training.
Models	EM	F1
Source domain: NQ, Target domain: SQuAD
Pretrained NQ	67.8	80.3
RGX + NER	27.4	35.4
RGX + AER-Tag	71.4	82.4
RGX + AER-LM	72.7	85.9
RGX + AER-EM	79.2	89.4
Supervised ELECTRA-large	89.7	94.9
Table 4: Comparison of different AER strategies. NER stands for the BERT named entity recognition model trained on the CONLL 2003 shared task. AER-Tag stands for a BIO-based tagging strategy, AER-LM means selecting synthetic QA pairs with lowest QAE losses. AER-EM is the EM-based QA selection strategy applied in our full model.
EM. If selecting QA pairs with low perplexities instead of EM, the QA diversity is significantly lower as shown in Table 6, thus makes the QAE model overfit to simple training cases and hurts the QA accuracy. We show questions about the same answer entity being classified into simple, challenging, and difficult types by EM in figure 3. The data points in the plot represent the losses of synthetic QA pairs and the predicted QA type. Based on the highlighted answer entity, question 1 and 2 are predicted as correct questions, while question 3, which has a relatively high QAE loss,
ELECTRA	Top-k+MMI	AER+MMI
EM	F1	EM	F1	EM	F1
BioASQ	58.7	73.1	57.8	72.9	59.9	74.0
TextbookQA	43.0	54.6	44.6	54.9	45.3	55.4
RACE	38.3	52.5	38.1	52.4	39.7	54.1
RelExt	79.0	88.4	78.6	88.3	79.2	88.6
DuoRC	53.1	64.2	52.6	64.3	53.8	65.1
DROP	48.3	60.8	46.7	60.8	49.7	61.5
Table 5: Comparison between maximum mutual information inference performance grounded on AER results and top-k ( $k = 20$ ) predictions of the QA model.
Models	Mean Len.	Std Len.	Vocab
Ground-truth	11.29	3.72	988703
Semi-anno. RGX	10.54	1.91	923191
-w/o Coop. ST	10.49	2.48	919105
Zero-anno. RGX	10.53	1.94	873300
-w/o Coop. ST	10.57	2.63	789924
-w/o AER	10.60	1.87	743454
-w/o EM	10.18	1.62	692301
is regarded as a wrong question. Note that we only generate one question for each span recognized by the AER model, but different questions might be re-directed to the same AE after QAE fine-graining.
4.4.3 Cooperative Self-training
We found that the cooperative self-training method improves domain adaptation ability of self-trained QA models by increasing both accuracy and diversity of QA synthesis.
Accuracy. We also evaluate the quality of the generated QA pairs without a downstream task by assessing the answer entity hit rate and the BLEU scores of generated questions using the evaluation sets of each domain. The results are shown in
Table 6: The vocabulary sizes and lengths of Annotated and generated questions on SQuAD under both semi- and zero-annotated settings in unseen domains
Domain	RGX w/o Coop. ST	RGX
Hit	BLEU	Hit	BLEU
BioASQ	68.1	5.9	75.8	12.7
TextbookQA	43.7	7.5	58.2	13.2
RACE	8.3	5.2	12.3	6.8
RelExt.	47.4	2.8	54.2	3.3
DuoRC	53.5	6.7	60.0	7.5
DROP	73.5	12.3	75.3	9.3
Table 7: Evaluation of the answer hit rates and question BLEU scores of the synthetic data. Hit rate stands for the percentage of human-labeled answer entities in the evaluation passages that are successfully covered by the selected synthetic data generated by RGX.
Context: Despite differences in the spectrum of mutations in CN or CyN, type or localization of mutation only partially determine the clinical phenotype.
Q1: What determines the clinical phenotype of a person with a mutation?
Q2: What determines the clinical phenotype of a mutation?
Q3: What is the only way to determine the clinical phenotype of a mutation?
Figure 3: Generated questions about the same answer entity classified into different types by EM. Questions Q1 is answered by the QAE model confidently, while the Q2 is considered more challenging than Q1 since less information is provided. Q3 is an unanswerable questions given the context passage.
Table 7, indicating that RGX find mores human-annotated answer entities, and the generated questions have higher BLEU scores on all domains. The evaluation results show that the synthetic QA pars generated by RGX covers more human annotated answer entities, and the generated questions are more similar to human annotations than the pretrained question generation model. We also found that tuning the generation model for more than 1 iterations does not result in further improvement, since keeping training language models with their own outputs leads to difficult optimization.
Diversity. We compare the lengths and vocabulary sizes of the questions and summarize the statistics in Table 6, which shows that the ground-truth questions are longer and more diverse in vocabulary than the generated ones. However, the cooperative self-training, together with AER and EM, improves the vocabulary diversity. We observe a correlation between the vocabulary size and the QA performance reported in Table 1 and 4, presumably because the QAE model requires diverse knowledge for training. Thus, we believe generating more diverse QA pairs with good quality will be a critical next step to improve RGX.
Case Study. An example of a SQuAD passage is shown in Table 8. We list the annotated and generated question-answer pairs by different models. The table shows that the models can recognize rea
Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.
Annotated	Pretrained	RGX
Saint Bernadette Soubirous	a Marian place of prayer and reflection	a Marian place of prayer and reflection
To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?	what is the grotto at st bernadette's?	what is the grotto in st bernadette school?
a copper statue of Christ	the grotto at Lourdes, France	Venite Ad Me Omnes
What is in front of the Notre Dame Main Building?	where is the grotto located at st bernadette school?	what is the message on the statue in front of st bernadette school?
the Main Building	Immediately behind the basilica is the Grotto	1858
The Basilica of the Sacred heart at Notre Dame is beside to which structure?	what is the grotto in st peter's school?	when was the grotto at lourdes built?
a Marian place of prayer and reflection	copper statue of Christ with arms upraised	a simple, modern stone statue of Mary
What is the Grotto at Notre Dame?	what is it a statue of christ?	what is the statue at st bernadette school?
a golden statue of the Virgin Mary	a replica	the grotto at Lourdes, France
What sits on top of the Main Building at Notre Dame?	is the grotto at st bernadette school in paris a replica of which European landmark?	what is the replica of st bernadette's school in paris?
Table 8: An example of a passage in the training set of the SQuAD corpus. We list the annotated question-answer pairs, and the question-answer pairs generated by the models pretrained on NQ and finetuned by RGX. The bold texts are annotated or recognized answer entities. Adapting from NQ is difficult since the questions in NQ do not strictly coherent with a given context. More generation examples are shown in Appendix D.
sonable answer entities other than the annotated ones, and RGX generates more natural QAs.
We propose a cooperative self-training framework, RGX, consisting of an answer entity Recognizer, a question Generator, and an answer eXtractor, for question generation and answering. We also introduce in the framework an expectation-maximization method that measures the quality of generated questions for reinforced finetuning of the question generation models. Experiments show that RGX significantly outperforms pretrained and self-trained model baselines while adapted to unseen domains, suggesting that RGX is a promising framework for making extractive question answering methods more scalable and less dependent on human annotation.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for read
ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2104.08678.
Oliver Bender, Franz Josef Och, and Hermann Ney. 2003. Maximum entropy models for named entity recognition. In Proceedings of CoNLL-2003, pages 148-151. Edmonton, Canada.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866-874.
Hao Fei, Yafeng Ren, and Donghong Ji. 2020. Retrofitting structure-aware transformer language model for end tasks. arXiv preprint arXiv:2009.07408.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13.
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, GP Bhargav, Dinesh Garg, and Avirup Sil. 2019. Span selection pretraining for question answering. arXiv preprint arXiv:1909.04120.
Ian Goodfellow. 2016. Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27:2672-2680.
Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in neural information processing systems, pages 2149-2159.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788.
Matthew Henderson, Inigo Casanueva, Nikola Mrksić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2019. Convert: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. 2021. Question answering infused pre-training of general-purpose contextualized representations. arXiv preprint arXiv:2106.08190.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Sham M Kakade. 2001. A natural policy gradient. Advances in neural information processing systems, 14:1531-1538.
Tassilo Klein and Moin Nabi. 2019. Learning to answer by learning to ask: Getting the best of gpt-2 and bert worlds. arXiv preprint arXiv:1911.02365.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2016. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182.
Dong Bok Lee, Seanie Lee, Woo Tae Jeong, Donghwan Kim, and Sung Ju Hwang. 2020. Generating diverse and consistent qa pairs from contexts with information-maximizing hierarchical conditional vaes. arXiv preprint arXiv:2005.13837.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019a. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019b. Unsupervised question answering by cloze translation. arXiv preprint arXiv:1906.04980.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 million probably-asked questions and what you can do with them. arXiv preprint arXiv:2102.07033.
Jiwei Li and Dan Jurafsky. 2016. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou. 2020. Tell me how to ask again: Question data augmentation with controllable rewriting in continuous space. arXiv preprint arXiv:2010.01475.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Hongyin Luo, Lan Jiang, Yonatan Belinkov, and James Glass. 2019. Improving neural language models by segmenting, attending, and predicting the future. arXiv preprint arXiv:1906.01702.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26:3111-3119.
Igor Mordatch and Pieter Abbeel. 2018. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. arXiv preprint arXiv:2002.09599.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 629-640.
Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. End-to-end synthetic data generation for domain adaptation of question answering systems. arXiv preprint arXiv:2010.06028.
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. 2018. Straight to the tree: Constituency parsing with neural syntactic distance. arXiv preprint arXiv:1806.04168.
Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2020. Structformer: Joint unsupervised induction of dependency and constituency structure from masked language modeling. arXiv preprint arXiv:2012.00857.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms.
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687-10698.
Silei Xu, Sina J Semnani, Giovanni Campagna, and Monica S Lam. 2020. Autoqa: From databases to qa semantic parsers with only synthetic training data. arXiv preprint arXiv:2010.04806.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
Representation learning has been an important topic in NLP area since neural language models were proposed (Bengio et al., 2003). Based on word co-occurrence, Mikolov et al. (2013) and Pennington et al. (2014) proposed language embedding algorithms to model word-level semantics. Recent studies have focused on pretraining contextualized word representations with large-scaled corpora (Peters et al., 2018). State-of-the-art representation models are pretrained with the masked language modeling task (Devlin et al., 2018; Liu et al., 2019; Clark et al., 2020) using the Transformer architecture (Vaswani et al., 2017).