SlowGuess/NAACL_2022_OCR huggingface dataset:
racy of  $23.69\%$  to  $39.00\%$ , suggesting a relative gain of around  $65\%$ . These results demonstrate that GenMC is a more effective usage of pre-trained encoder-decoder models than existing ones.
Moreover, we interestingly find that the decoder-free baseline Text2Textenc outperforms Text2Textvanilla on over half of the experiments. This indicates that the decoder's general language knowledge gained from pre-training is largely wasted by only using it as a classifier, which may further explain the superior performance of our model because GenMC can exploit the pre-trained decoder more effectively. In addition, all LARGE models significantly outperform their BASE counterparts. This suggests that the embedded knowledge gained from pre-training is critical to MCQA tasks, strengthening our point to make full use of pre-trained encoders and decoders.
5.2 Comparison with Other Models
UnifiedQA Existing methods that rely on external documents or corpora have achieved state-of-the-art performance on several MCQA datasets. However, to enable a fair comparison, we only compare with models that adopt the same setting as ours, where a question and its options are the only input to the model. Among these models, UnifiedQA (Khashabi et al., 2020) is the current best model. While UnifiedQA reports the best score using its T5-11B version, since for T5 we experiment with its BASE and LARGE versions, we only report and compare under  $\mathrm{T5}_{\mathrm{BASE}}$  and  $\mathrm{T5}_{\mathrm{LARGE}}$ . Note that instead of training on each dataset separately, UnifiedQA converts a line of popular QA datasets with four formats (e.g., retrieval-based QA, MCQA) into a unified format, and trains a single model over all training data, while GenMC only uses each
CSQA	OBQA	ARC-Easy	ARC-Challenge	QASC
dev	test	dev	test	dev	test	dev	test	dev	test
BASE
RoBERTa	56.51 (±0.34)	58.91 (±0.79)	58.67 (±1.03)	49.67 (±0.76)	56.56 (±0.91)	52.32 (±0.70)	38.64 (±0.90)	34.85 (±2.20)	55.28 (±0.12)	34.38 (±1.72)
ALBERT	53.16 (±0.58)	53.95 (±0.49)	54.53 (±1.10)	49.20 (±2.27)	48.32 (±0.88)	45.84 (±1.94)	34.80 (±1.53)	30.21 (±1.74)	40.99 (±1.78)	24.55 (±1.23)
UnifiedQA\(T_{5} *\)	-	45.00 (±0.00)	-	59.00 (±0.00)	-	53.00 (±0.00)	-	42.40 (±0.00)	-	25.80 (±0.00)
UnifiedQA\(T_{5} \)	41.02 (±0.00)	44.80 (±0.00)	59.20 (±0.00)	59.60 (±0.00)	54.85 (±0.00)	53.66 (±0.00)	44.75 (±0.00)	42.58 (±0.00)	17.94 (±0.00)	25.70 (±0.00)
UnifiedQA\(T_{5}\)-FT	56.81 (±0.49)	62.35 (±0.80)	60.80 (±0.72)	58.47 (±0.64)	54.97 (±0.20)	53.88 (±0.39)	45.31 (±0.39)	42.43 (±0.47)	55.57 (±0.58)	43.20 (±0.57)
GenMC\(T_{5}\)	60.65 (±0.47)	63.45 (±0.29)	62.07 (±1.01)	61.67 (±0.58)	62.38 (±0.67)	58.82 (±0.37)	43.62 (±0.52)	39.00 (±0.30)	58.93 (±1.76)	41.72 (±1.18)
LARGE
RoBERTa	68.92 (±0.76)	71.88 (±0.26)	67.80 (±1.22)	64.47 (±1.41)	65.73 (±0.80)	62.40 (±0.89)	38.08 (±1.99)	35.97 (±1.74)	67.32 (±0.58)	50.22 (±1.88)
ALBERT	60.62 (±0.57)	59.32 (±0.91)	54.50 (±1.40)	49.27 (±0.64)	54.03 (±0.45)	53.77 (±1.81)	33.90 (±1.22)	31.19 (±3.79)	51.11 (±1.72)	33.12 (±1.24)
UnifiedQA\(T_{5} *\)	-	60.90 (±0.00)	-	68.40 (±0.00)	-	65.90 (±0.00)	-	54.40 (±0.00)	-	43.30 (±0.00)
UnifiedQA\(T_{5} \)	55.28 (±0.00)	61.34 (±0.00)	70.40 (±0.00)	68.40 (±0.00)	69.31 (±0.00)	66.43 (±0.00)	56.61 (±0.00)	54.33 (±0.00)	29.24 (±0.00)	43.74 (±0.00)
UnifiedQA\(T_{5}\)-FT	69.00 (±0.51)	73.60 (±0.45)	70.53 (±0.23)	68.80 (±0.69)	69.72 (±0.71)	66.92 (±0.85)	56.84 (±0.39)	54.42 (±0.15)	66.63 (±1.56)	58.71 (±0.90)
GenMC\(T_{5}\)	71.10 (±0.41)	72.67 (±1.02)	71.60 (±0.92)	66.87 (±1.33)	72.49 (±0.77)	69.01 (±1.97)	49.83 (±2.06)	47.41 (±2.00)	67.61 (±1.14)	58.06 (±0.92)
Table 3: Comparison with other models. (* indicates the results reported by Khashabi et al. (2020).)
CSQA	OBQA	ARC-Easy	ARC-Challenge	QASC
dev	test	dev	test	dev	test	dev	test	dev	test
BASE
UnifiedQAT5-FT	56.81 (±0.49)	62.35 (±0.80)	60.80 (±0.72)	58.47 (±0.64)	54.97 (±0.20)	53.88 (±0.39)	45.31 (±0.39)	42.43 (±0.47)	55.57 (±0.58)	43.20 (±0.57)
GenMCTS-U	61.24 (±0.45)	63.45 (±0.76)	62.33 (±0.81)	59.20 (±1.91)	61.73 (±0.35)	59.35 (±0.43)	45.54 (±0.20)	43.98 (±0.36)	60.16 (±0.07)	45.43 (±0.87)
LARGE
UnifiedQAT5-FT	69.00 (±0.51)	73.60 (±0.45)	70.53 (±0.23)	68.80 (±0.69)	69.72 (±0.71)	66.92 (±0.85)	56.84 (±0.39)	54.42 (±0.15)	66.63 (±1.56)	58.71 (±0.90)
GenMCTS-U	71.58 (±0.25)	72.26 (±0.31)	71.67 (±0.46)	69.00 (±0.69)	73.90 (±0.47)	72.87 (±0.50)	59.55 (±1.09)	55.97 (±0.62)	68.55 (±0.81)	58.75 (±0.56)
Table 4: Comparison with UnifiedQA after unifying training sets.
dataset's own training data.
RoBERTa and ALBERT In addition, we compare with two encoder-only models, RoBERTa (Liu et al., 2019) and ALBERT (Lan et al., 2020), which have served as the basis of many MCQA models.
All models are of comparable model size to ours.
The results in Table 3 show that  $\mathrm{GenMC}_{\mathrm{T}5}$  significantly (with p-value  $< 0.01$ ) outperforms the two encoder-only strong baselines RoBERTa and ALBERT. More interestingly,  $\mathrm{GenMC}_{\mathrm{T}5}$  also performs better than UnifiedQA on most datasets. Moreover, for UnifiedQAFT, which further finetunes the model on the training set of the target dataset, GenMC $_{\mathrm{T}5}$  outperforms it on the test sets of CSQA, OBQA, and ARC-Easy for the base models and ARC-Easy for the large models. It also achieves comparable results on the remaining datasets. These results are impressive because UnifiedQA uses more datasets (i.e., eight different QA datasets) for training. The promising results of GenMC further reveals that our model can learn to effectively extract knowledge from pre-trained encoder-decoders with limited training data.
As a fairer comparison in Table 4, by unifying the training sets of all the five datasets, our  $\mathrm{GenMC}_{\mathrm{T5 - U}}$  outperforms UnifiedQA $_{\mathrm{T5 - FT}}$  on all datasets except for CSQA with large models.
5.3 Ablation Study: Influence of Clues
Our main results in Section 5.1 have demonstrated the effectiveness of our model. To better understand its superior results and the influence of our clue generation, we compare with two variants.
5.3.1 Variants of GenMC
Weak Clue We train this variant only using the classification loss  $\mathcal{L}^{\mathrm{READ}}$ , so only the encoder part is updated, while the decoder part is left untouched from pre-training. Intuitively, under this setting, the generated clue is weaker than GenMC which learns how to generate a clue with supervision.
Token Clue In this setting, we separately train a clue generator and a reader. We first collect the generated clue text  $C$  (instead of its representation) from the decoder. We then directly concatenate  $C$  with  $Q$  and  $O_{i}$  to compute a score for  $O_{i}$  using the model's encoder part stacked with an MLP layer. This variant is indeed very similar to Liu et al. (2020c), which also adopts a pipeline framework to first generate a token-level evidence and then use the evidence to expand the question.
5.3.2 Results
Table 5 shows that masking out generation loss leads to substantial performance drops across all datasets, demonstrating that fine-tuning the decoder with generation loss  $\mathcal{L}^{\mathrm{GEN}}$  helps to derive useful clues from pre-trained encoder-decoder models. We also observe that the performance of using token-level clues lags much behind GenMC. This
CSQA	OBQA	ARC-Easy	ARC-Challenge	QASC
dev	test	dev	test	dev	test	dev	test	dev	test
BARTBASE
GenMC	54.82 (±0.61)	56.40 (±0.61)	58.53 (±0.31)	57.53 (±2.91)	59.38 (±1.60)	56.80 (±0.28)	38.64 (±0.90)	33.82 (±1.66)	57.70 (±0.43)	35.96 (±1.70)
Weak Clue	53.96 (±1.01)	54.35 (±1.97)	55.53 (±1.27)	54.27 (±0.92)	57.20 (±1.80)	55.42 (±1.26)	39.89 (±0.20)	32.62 (±0.31)	54.05 (±0.21)	25.99 (±0.82)
Token Clue	45.53 (±1.28)	46.41 (±1.79)	54.07 (±1.72)	52.93 (±1.10)	48.97 (±0.91)	48.87 (±1.29)	31.19 (±0.59)	27.64 (±0.69)	49.06 (±0.39)	21.31 (±1.03)
T5BASE
GenMC	60.65 (±0.47)	63.45 (±0.29)	62.07 (±1.01)	61.67 (±0.58)	62.38 (±0.67)	58.82 (±0.37)	43.62 (±0.52)	39.00 (±0.30)	58.93 (±1.76)	41.72 (±1.18)
Weak Clue	58.80 (±0.70)	60.88 (±1.89)	61.47 (±0.95)	59.73 (±0.90)	58.97 (±0.54)	57.10 (±0.72)	42.26 (±2.21)	37.54 (±0.64)	57.37 (±1.40)	36.29 (±1.66)
Token Clue	50.55 (±0.44)	48.79 (±0.87)	56.00 (±1.25)	54.93 (±1.63)	46.50 (±0.83)	46.65 (±0.54)	32.66 (±0.20)	26.01 (±1.28)	43.69 (±1.52)	27.50 (±1.56)
BARTLARGE
GenMC	69.57 (±0.89)	72.26 (±0.70)	68.93 (±1.17)	68.07 (±1.70)	72.43 (±0.54)	68.68 (±0.34)	48.93 (±0.98)	45.52 (±1.54)	68.39 (±0.68)	55.90 (±0.92)
Weak Clue	67.28 (±2.39)	69.64 (±2.76)	66.20 (±0.53)	64.47 (±1.40)	70.66 (±1.50)	65.71 (±1.47)	27.80 (±2.06)	24.92 (±2.06)	65.68 (±1.31)	52.02 (±1.44)
Token Clue	53.85 (±0.47)	55.23 (±0.62)	61.20 (±3.14)	59.20 (±0.69)	58.02 (±0.98)	54.22 (±1.27)	41.81 (±1.19)	37.60 (±0.90)	48.65 (±1.23)	32.47 (±1.11)
T5LARGE
GenMC	71.10 (±0.41)	72.67 (±1.02)	71.60 (±0.92)	66.87 (±1.33)	72.49 (±0.77)	69.01 (±1.97)	49.83 (±2.06)	47.41 (±2.00)	67.61 (±1.14)	58.06 (±0.92)
Weak Clue	68.33 (±1.62)	71.66 (±1.28)	69.27 (±0.42)	65.87 (±0.90)	69.66 (±0.77)	66.24 (±0.79)	47.57 (±2.04)	46.24 (±1.29)	64.99 (±0.74)	53.35 (±1.35)
Token Clue	59.47 (±0.08)	60.74 (±0.29)	62.80 (±1.44)	57.73 (±1.10)	48.85 (±1.62)	48.36 (±2.15)	37.97 (±0.90)	30.50 (±1.46)	49.22 (±0.62)	38.77 (±1.74)
Table 5: Influence of clues.
Clue Type	Percentage	Example
Instance	Clue
Irrelevant	23.60%	Which would you likely find inside a beach ball?
(A) cheese (B) steam (C) water (D) air	a squid
Relevant but unhelpful	52.40%	What may have been formed by a volcano?
(A) Mt. McKinley (B) Lake Pontchartrain (C) The great lakes
(D) Niagara Falls	a lake
Helpful	24.00%	Where would there be an auditorium with only a single person speaking?
(A) lights (B) crowd (C) university campus (D) theater (E) park	school
Table 6: Distribution of clue types in negative cases with examples. Bold underline indicates the correct answer, and italic indicates the predicted label.
demonstrates that naively using explicit knowledge in plain text, instead of using implicit clues from the decoder's hidden state, is inferior as it may unnecessarily bring information loss and noise.
We analyze the clues generated by GenMC using T5Large with a focus on instances that are correctly predicted by the baseline in our main experiments (i.e., T5Large + Text2Text_vanilla), while our GenMC fails. The intuition is that in these negative cases, the clues generated by GenMC may play a negative role. By studying these potentially negative clues, we can gain more insights into how GenMC fails and discuss venues for future improvement.
Specifically, we randomly sample 50 negative cases from  $\mathrm{T}5_{\mathrm{LARGE}} + \mathrm{GenMC}$  for each dataset. We show six graduate students of computer science<sup>4</sup> an instance along with the generated clue, correct answer, and predicted answer. We then ask them to categorize clues into the following families:<sup>5</sup>
- Irrelevant: The clue is off topic or is not
understandable.
- Relevant but unhelpful: Though relevant, the clue makes a factually incorrect statement, often on the contrary of the main question, or the clue contributes relevant but insufficient knowledge for prediction, such as repetition of the question or other distractors.
- Helpful: The clue adds helpful information to answer the question.
To ensure the annotation quality, we aggregate annotated results from three students for every dataset using majority vote. If all three students annotate differently from each other for an instance, we introduce a fourth student to arbitrate.
Table 6 shows the percent of each clue type across all datasets with an example for each type. Figure 4 breaks down by dataset. Though the majority of our clues are relevant (i.e.,  $76.4\%$  of them are relevant across all datasets), which seems positive, only  $24\%$  of the clues are deemed as helpful. This suggests a great room for improvement. In our future research, we will focus on how to generate more helpful clues from questions.
<sup>4</sup>They are volunteers recruited from the contact author's research group. They know and agree that their annotations will be used for error analysis in a research paper.
We follow a similar definition by Shwartz et al. (2020).
CSQA	OBQA	ARC-Easy	ARC-Challenge	QASC
T5BASE
Text2Textvanilla	0.040 (±0.007)	0.035 (±0.002)	0.035 (±0.002)	0.039 (±0.004)	0.035 (±0.002)
UnifiedQA	0.059 (±0.041)	0.089 (±0.047)	0.097 (±0.055)	0.129 (±0.075)	0.068 (±0.027)
GenMC	0.069 (±0.019)	0.107 (±0.046)	0.113 (±0.060)	0.121 (±0.053)	0.072 (±0.027)
T5LARGE
Text2Textvanilla	0.077 (±0.008)	0.083 (±0.012)	0.081 (±0.012)	0.084 (±0.014)	0.078 (±0.011)
UnifiedQA	0.108 (±0.037)	0.178 (±0.096)	0.190 (±0.107)	0.257 (±0.127)	0.130 (±0.052)
GenMC	0.105 (±0.027)	0.178 (±0.078)	0.219 (±0.120)	0.242 (±0.112)	0.127 (±0.048)
Table 7: Inference time for answering a question (seconds).
Figure 4: Distribution of clue types in negative cases on each dataset.
5.5 Inference Time and Model Size
Table 7 shows the inference time for answering a question. GenMC is slower than Text2Text<sub>vanilla</sub>, but their inference time has the same scale, suggesting that GenMC is more cost-effective considering its superior accuracy. GenMC and UnifiedQA are comparable in inference time.
Among  $\mathrm{T5}_{\mathrm{BASE}}$  based models, Text2Textvanilla and UnifiedQA have  $223\mathrm{M}$  parameters, while GenMC is slightly larger with  $234\mathrm{M}$  parameters. Among  $\mathrm{T5}_{\mathrm{LARGE}}$  based models, Text2Textvanilla and UnifiedQA have  $738\mathrm{M}$  parameters, while GenMC has  $757\mathrm{M}$  parameters.
We present GenMC, a simple yet effective model which tailors pre-trained encoder-decoders for MCQA tasks. Compared with existing usages of pre-trained encoder-decoders for MCQA, our model fully exploits the pre-trained encoder-decoders' NLG capabilities to generate a clue from the input question, which facilitates deep understanding of question-option pairs. Experimental results further verify the superiority of GenMC over existing usages. Notably, our model achieves promising results without using any provided documents or external corpora, showing an interesting application of PLMs by directly inducing either commonsense or scientific knowledge from them
through clue generation.
In the future, we will focus on how to further improve the clue generation quality, which remains a bottleneck of GenMC. We hope this work will spur more research in how to better use pre-trained encoder-decoders for not only MCQA, but also beyond; for tasks with divergent structures from the pre-training, a smarter use of PLMs can boost the performance significantly.
This work was supported in part by the NSFC (62072224) and in part by the Beijing Academy of Artificial Intelligence (BAAI).
Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. Convolutional spatial attention model for reading comprehension with multiple-choice questions. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6276-6283. AAAI Press.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Zixian Huang, Yulin Shen, Xiao Li, Yuang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, and Yuzhong Qu. 2019.
Geosqa: A benchmark for scenario-based question answering in the geography domain at high school level. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5865-5870. Association for Computational Linguistics.
Zixian Huang, Ao Wu, Yulin Shen, Gong Cheng, and Yuzhong Qu. 2021. When retriever-reader meets scenario-based multiple-choice questions. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November; 2021, pages 985-994. Association for Computational Linguistics.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423-438.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1896-1907. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8082-8090. AAAI Press.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2019. What's missing: A knowledge gap guided approach for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2814-2828. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Veronica Latcinnik and Jonathan Berant. 2020. Explaining question answering models through text generation. CoRR, abs/2004.05569.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Xiao Li, Gong Cheng, Ziheng Chen, Yawei Sun, and Yuzhong Qu. 2022. Adalogn: Adaptive logic graph network for reasoning-based machine reading comprehension. CoRR, abs/2203.08992.
Xiao Li, Yawei Sun, and Gong Cheng. 2021. TSQA: tabular scenario based question answering. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13297-13305. AAAI Press.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020a. RikiNet: reading Wikipedia pages for natural question answering. In ACL, pages 6762-6771.
Frederick Liu, Siamak Shakeri, Hongkun Yu, and Jing Li. 2021. Enct5: Fine-tuning T5 encoder for non-autoregressive tasks. CoRR, abs/2110.08426.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020b. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3622-3628. ijcai.org.
Ye Liu, Tao Yang, Zeyu You, Wei Fan, and Philip S. Yu. 2020c. Commonsense evidence generation and injection in reading comprehension. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020, pages 61-73. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4089-4100. Association for Computational Linguistics.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2381-2391. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2463-2473. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4932-4942. Association for Computational Linguistics.
Qiu Ran, Peng Li, Weiwei Hu, and Jie Zhou. 2019. Option comparison network for multiple-choice reading comprehension. CoRR, abs/1903.03033.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. CPT: A pre-trained unbalanced transformer for both chinese language understanding and generation. CoRR, abs/2109.05729.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4222-4235. Association for Computational Linguistics.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4615-4629. Association for Computational Linguistics.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long