SlowGuess/NAACL_2022_OCR huggingface dataset:
Machine Translation We experiment with two language pairs from the WMT 2020 news translation task (Barrault et al., 2020): Chinese  $\rightarrow$  English (WMT20 ZH-EN) and English  $\rightarrow$  German (WMT20 EN-DE). We use outputs from all submitted translation systems. These two language pairs have expert, rubric-based scores (MQM) from Freitag et al. (2021) for a subset of 10 submitted systems, including the top-performing systems and human translations. Each output sentence is evaluated by three professional translators. Following Freitag et al. (2021), the three scores are averaged to get an instance-level score.
We use all human translations available as a reference set for reference-based metrics. Concretely, every test instance in WMT20 ZH-EN has two translations provided by different human translation services: Human-A and Human-B (Barrault et al., 2020). In addition to Human-A and Human-B, WMT20 EN-DE provides a translation that is created by linguists who are asked to paraphrase Human-A and Human-B as much as possible (Human-P, Freitag et al., 2020). These paraphrased translations are shown to increase corre
lations with human judgments by mitigating the translationese effect and diversifying the reference when the generation quality is measured by reference-based metrics (Freitag et al., 2020).
Along with all submitted generators in WMT20 ZH-EN and WMT20 EN-DE, we train three transformer baselines with the fairseq library (Ott et al., 2019) and place them in our BILLBOARDS: transformer-base, transformer-large, and transformer-large-ensemble with similar hyperparameters (e.g., 6-layer encoder and decoder) to the ones trained on the WMT16 EN-DE data in Vaswani et al. (2017). These baselines allow researchers to compare their translation models without resource-intensive techniques such as backtranslation (Sennrich et al., 2016a), model ensembling, and deep encoders (Kasai et al., 2021a). These techniques are all used in top-performing systems of WMT20 (Wu et al., 2020a; Kiyono et al., 2020) but might be infeasible in many research settings. See Appendix B for a list of all hyperparameters for the baselines.
Summarization We use the CNN/DailyMail corpus (CNNDM, Hermann et al., 2015; Nallapati et al., 2016). We use the standard train/dev./test split and 24 models from Fabbri et al. (2021). 100 test articles are annotated with 10 summaries written by humans (Kryscinski et al., 2019). For those 100 articles, rubric-based, expert evaluations for 18 generators, including human-written highlights, are provided by Fabbri et al. (2021). Each output summary is evaluated by three experts along four dimensions: coherence (collective quality of all summary sentences), consistency (factual alignment with the article, penalizing for hallucinations), fluency (quality of the individual sentences), and relevance (selection of important content). An instance-level score is computed by averaging scores over all these categories and the three experts. Note that this aggregation method can be modified, depending on the downstream task of interest (Kasai et al., 2022). All 10 human-written summaries are used
$^{7}$ Data and models are available at https://github.com/jungokasai/billboard/tree/master/baseline.
Some of the outputs are lowercased and/or tokenized. In these cases, we apply the NLTK detokenizer (Bird et al., 2009) and/or the Stanford CoreNLP truecaser (Manning et al., 2014). We encourage, however, future model developers to provide clean, untokenized output to improve the reproducibility and transparency of evaluation results (Post, 2018; Kasai et al., 2022).
$^{6}$ https://www.statmt.org/wmt20/translation-task.html.
Dataset	|G|	|M|	Top Gen.	Single Metrics	Ensemble of Metrics
Top Metric	Corr.	Linear Combination	Corr.
WMT20 ZH-EN	19	15	Huoshan	COMET	0.55	1.72·COMET-QE+1.48·COMET+1.21·BLEURT	0.61
WMT20 EN-DE	17	11	Tohoku	COMET	0.49	1.19·COMET+0.36·COMET-QE+0.02·Prism-ref	0.51
CNNDM	26	15	Lead-3	COMET	0.41	2.85·COMET+0.26·COMET-QE+0.01·BERTScore	0.29
MSCOCO	4	15	VinVL-large	RefCLIP-S	0.45	2.08·RefCLIP-S+1.51·RefOnlyC+0.82·CIDEr	0.45
Table 1: Summary of BILLBOARDs as of Jan. 10, 2022. Huoshan: Wu et al. (2020a); Tohoku: Kiyono et al. (2020); VinVL-large: Zhang et al. (2021); COMET, COMET-QE: Rei et al. (2020); BLEURT: Sellam et al. (2020); Prism-ref: Thompson and Post (2020); BERTScore: Zhang et al. (2020b); RefCLIP-S: Hessel et al. (2021); RefOnlyC: Kasai et al. (2022). COMET-QE is a referenceless metric. BLEURT is specifically trained to evaluate into-English translations. RefCLIP-S uses image features unlike most metrics for image captioning. RefOnlyC removes image features from RefCLIP-S and only uses reference text features from CLIP (Radford et al., 2021).
as the reference set for reference-based metrics.
Image Captioning We use the MSCOCO dataset (Lin et al., 2014) that consists of everyday-scene photos sampled from Flickr. Every image is annotated with five captions written by crowdworkers (Chen et al., 2015). We apply the standard Karpathy split (Karpathy and Fei-Fei, 2015). For each of 500 test images, rubric-based evaluations (THUMB 1.0) are available for five systems, including one caption from a crowdworker (Kasai et al., 2022). Similar to machine translation and summarization, we use all five crowdworker captions as a reference set for reference-based metrics.
3.2 Mixed-Effects Models
Our mixed-effects model analyzes how much every automatic metric overrates machines over humans (§2.3). This means that we need to free up one human generation per instance to measure its scores in the reference-based metrics. For machine translation, we score Human-B using the reference set of Human-A (WMT20 ZH-EN) or Human-A and Human-P (WMT20 EN-DE). For CNNDM, we use concatenated highlights as human-generated summaries and use the 10 human-written summaries from Kryscinski et al. (2019) as the reference. We follow Kasai et al. (2022) for MSCOCO and score their randomly-selected Human caption using the other four as the reference. As the distinction between the reference and human generation (e.g., Human-A vs. Human B on WMT20 ZH-EN) is arbitrary, we found that swapping the roles would still lead to similar results (See Appendix E).
Here we discuss the current results and make several key observations about the state of language generation evaluation. Table 1 summarizes the four BILLBOARDs. It is particularly noteworthy that COMET, a metric designed for machine translation, achieves the best correlation on the CNNDM summarization task as well. COMET evaluates the similarity between the crosslingual representations from XLM-RoBERTa (Conneau et al., 2020) for input text and its translation candidate. But these crosslingual representations can, of course, be used monolingually for English summarization. This illustrates an additional benefit of BILLBOARDs that centralize different generation tasks and find surprising task transferability of learning-based metrics. See Appendices B and C for lists of all participating generators and metrics.
Ensemble Metric The rightmost section of Table 1 shows the chosen metrics and their coefficients in the ensemble (§2.2). On the machine translation tasks, the ensemble metric outperforms the top individual metric. $^{10}$  In particular, we see a substantial gain of 0.06 points in WMT20 ZH-EN. The referenceless metric of COMET-QE is selected both for WMT20 ZH-EN and WMT20 EN-DE, suggesting complementary effects of diverse metrics. To further test this hypothesis, we perform ablations that drop one out of the three metrics at a time (Table 2). We see that only dropping COMET-QE would result in a decrease in the correlation score. This implies that the reference
<sup>10</sup>We found a major reason for the anomaly in CNNDM; an outlier generator that does not use the standard CNNDM training data (the GPT-2 zero-shot model; Ziegler et al., 2019) has a disproportionately large effect on the regression models. The ensemble metric outperformed the top individual metric of COMET when the zero-shot model was removed.
Prior work used a concatenation of author-written highlights as a reference, but here we do not add it to the reference set. This is because these highlights are sometimes noisy (e.g., containing URLs) or lack coherence (Fabbri et al., 2021).
less metric provides important information that the others do not.
Removed Metric	-	COMET	COMET-QE	BLEURT
Correlation	0.61	0.61	0.57	0.61
Mixed-Effects Models Seen in Table 3 are the results from our analysis that measures how much metrics overrate machines over humans (§2.3). We see that the fixed-effect coefficient  $\beta_0$  is significantly positive in most cases. Referenceless metrics tend to have smaller coefficients. This can be due to the more diverse nature of human text than machine-generated text; reference-based metrics give a low score to human text that differs from the references even if it is of high quality. The conventional n-gram overlap-based metrics (BLEU, ROUGE, and CIDEr) have particularly large coefficients. These results suggest that the evaluation practice should be regularly updated as our generation models become stronger (and perhaps, more similar to human generation) in the future. Note that unlike the other tasks, "human-generated text" for CNNDM summarization is an automatic concatenation of author highlights, which contains substantial noise (Fabbri et al., 2021). This might explain the neutral and negative coefficients.
Table 2: Ensemble ablation studies on WMT20 ZH-EN. Only removing COMET-QE leads to a correlation drop. See Appendix D for the other datasets.
ZH-EN	COMET
0.27±0.02	COMET-QE
0.13±0.01	BLEURT
0.32±0.02	BLEU
0.62±0.02
EN-DE	COMET
0.08±0.03	COMET-QE
-0.17±0.02	Prism-ref
0.44±0.02	BLEU
0.33±0.03
CNNDM	COMET
-0.17±0.12	COMET-QE
0.02±0.11	BERTScore
-0.04±0.12	ROUGE-L
0.33±0.13
COCO	RefCLIP-S
0.09±0.06	RefOnlyC
0.24±0.06	CIDEr
0.43±0.06	CLIP-S
-0.04±0.05
Table 3:  $\beta_0$  fixed-effect coefficients from the linear mixed-effects models, quantifying how much automatic metrics overrate machines over humans, relative to human raters.  $\beta_0 = 0$  is neutral, and statistical significance is indicated by red (positive) or blue text (negative). The subscripts indicate  $90\%$  confidence intervals. Three metrics that correlate best with the human judgments are shown as well as one popular metric. COMET-QE and CLIP-S are referenceless. See §E for the other metrics.
Figure 3: Correlations with varying numbers of references. In all cases, one reference is not sufficient to outperform the referenceless COMET-QE metric. The default ROUGE assumes English input.
Effects of the Number of References Fig. 3 plots correlations over varying numbers of references. COMET was the top-performing reference-based metric regardless of the number of references, but we observe that it underperforms the referenceless metric when only one reference is given. Model performance in machine translation and summarization is commonly measured by applying reference-based metrics against one reference per instance in the research community. Our finding thus raises a further concern about the current evaluation practice. Finally, we see that popular choices of BLEU and ROUGE metrics have much lower correlations than the recent metrics over various numbers of references, in line with the recent studies (Mathur et al., 2020a, inter alia).
5 Related and Future Work
Related Benchmarks WMT organizes the metric competition track in parallel with the translation task every year (Mathur et al., 2020b; Barrault et al., 2020, inter alia). Participants submit automatic scores for the translation outputs from the parallel translation task. Unfortunately, most of these new metrics are not used by subsequent machine translation work, perhaps because they are tested solely against the concurrent translation submissions and it is up to model developers to execute or even implement new metrics. The GEM workshop (Gehrmann et al., 2021) conducts extensive analysis of models and evaluation methods over a wide set of generation tasks. BILLBOARDs ease the burden through standard leaderboard experience where generator developers only need to upload generation outputs for the test split. BILL-BOARDs also offer automatic ensembling of metrics and quantify the diversity that a new metric adds. The human-in-the-loop GENIE leaderboard (Khashabi et al., 2021) centralizes crowdsourced
evaluations for generation tasks. The current BILLBOARD setup is based on rubric-based, expert evaluation data from previous work, but future work can explore ways to improve crowdsourced evaluations and use them to update BILLBOARDs.
From Bidimensional to Multidimensional
BILLBOARDS lend themselves to a natural extension: multidimensional leaderboards. In particular, generation models have more aspects than generation quality, such as training and inference efficiency, sample efficiency, and robustness. These aspects are often ignored in the current leaderboard paradigm but are important to better serving practitioners' needs (Schwartz et al., 2019; Ethayarajh and Jurafsky, 2020; Mishra and Arunkumar, 2021). There are ongoing modeling and benchmarking efforts especially for efficient machine translation (Heafield et al., 2020; Peng et al., 2021; Kasai et al., 2021b, inter alia). We leave this extension to future work and specifically target the gap between generation modeling and evaluation.
We introduced BILLBOARDs, a simple yet powerful generalization of leaderboards that bridges the gap between generation modeling and evaluation research. We established and released four BILLBOARDs on machine translation, summarization, and image captioning tasks. We demonstrated that their built-in analysis of metric ensembling and mixed-effects modeling revealed key insights into the current state of natural language generation and its evaluation methods. BILLBOARDs allow for a standard leaderboard experience both on the modeling and evaluation sides. We invite submissions from researchers through our website.
We thank Kyunghyun Cho, Elizabeth Clark, Jesse Dodge, Saadia Gabriel, Michal Guerquin, Daniel Khashabi, Swaroop Mishra, Jamie Morgenstern, Phoebe Mulcaire, Hao Peng, Sofia Serrano, the ARK group at UW, the Mosaic team at the Allen Institute for AI, and the anonymous reviewers for their helpful feedback on this work. We also thank Xinyan Yu for WMT Chinese-English metaevaluations and Muyun Yang and Shujian Huang for sharing with us the CWMT training data. This work was supported in part by the DARPA MCS
program through NIWC Pacific (N66001-19-2-4031) and Google Cloud Compute.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In Proc. of ECCV.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proc. of CVPR.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubesic, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proc. of WMT.
Douglas Bates, Martin Machler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software.
Rachel Bawden, Giorgio Maria Di Nunzio, Cristian Grozea, Inigo Jauregi Unanue, Antonio Jimeno Yepes, Nancy Mah, David Martinez, Aurélie Néveol, Mariana Neves, Maite Oronoz, Olatz Pérez de Viñaspre, Massimo Piccardi, Roland Roller, Amy Siu, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Dina Wiemann, and Lana Yeganova. 2020. Findings of the WMT 2020 biomedical translation shared task: Basque, Italian and Russian as new additional languages. In Proc. of WMT.
Steven Bird, Evan Klein, and Edward Loper. 2009. *Natural Language Processing with Python*. Cambridge University Press.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references. In Proc. of EMNLP.
Léo Bouscarrat, Antoine Bonnefoy, Thomas Peel, and Cécile Pereira. 2019. STRASS: A light and effective method for extractive summarization based on sentence embeddings. In Proc. of ACL.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proc. of NeurIPS.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proc. of WMT.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proc. of WMT.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In Proc. of EACL.
Tanfang Chen, Weiwei Wang, Wenyang Wei, Xing Shi, Xiangang Li, Jieping Ye, and Kevin Knight. 2020. DiDi's machine translation system for WMT2020. In Proc. of WMT.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dólar, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proc. of ACL.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Hahuong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proc. of ACL.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proc. of ACL.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proc. of ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proc. of NeurIPS.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. BanditSum: Extractive summarization as a contextual bandit. In Proc. of EMNLP.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proc. of ACL.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboards. In Proc. of EMNLP.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proc. of NAACL.
Alexander R Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. TACL.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. TACL.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proc. of EMNLP.
Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proc. of GEM.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proc. of EMNLP.
Ulrich Germann. 2020. The University of Edinburgh's submission to the German-to-English and English-to-German tracks in the WMT 2020 news translation and zero-shot translation robustness tasks. In Proc. of WMT.
Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk.
Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, and Jeffrey P. Bigham. 2020. Twitter a1ly: A browser extension to make twitter images accessible. In Proc. of CHI.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proc. of ACL.
Jeremy Gwinnup and Tim Anderson. 2020. The AFRL WMT20 news translation systems. In Proc. of WMT.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mengnan Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic Chinese to English news translation.
Kenneth Heafield, Hiroaki Hayashi, Yusuke Oda, Ioannis Konstas, Andrew Finch, Graham Neubig, Xian Li, and Alexandra Birch. 2020. Findings of the fourth workshop on neural generation and translation. In Proc. of WNGT.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proc. of NeurIPS.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proc. of EMNLP.
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proc. of ACL.
Yichen Jiang and Mohit Bansal. 2018. Closed-book training to improve summarization encoder memory. In Proc. of EMNLP.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proc. of CVPR.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021a. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In Proc. of ICLR.
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A. Smith. 2021b. Finetuning pretrained transformers into RNNs. In Proc. of EMNLP.