SlowGuess/NAACL_2022_OCR huggingface dataset:
Zixian Huang and Ao Wu and Jiaying Zhou
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
{zixianhuang, awu, jyzhou}@smail.nju.edu.cn
Yu Gu
The Ohio State University, Columbus, USA
gu.826@osu.edu
Yue Zhao and Gong Cheng
yuezhao@smail.nju.edu.cn, gcheng@nju.edu.cn
A trending paradigm for multiple-choice question answering (MCQA) is using a text-to-text framework. By unifying data in different tasks into a single text-to-text format, it trains a generative encoder-decoder model which is both powerful and universal. However, a side effect of twisting a generation target to fit the classification nature of MCQA is the underutilization of the decoder and the knowledge that can be decoded. To exploit the generation capability and underlying knowledge of a pre-trained encoder-decoder model, in this paper, we propose a generation-enhanced MCQA model named GenMC. It generates a clue from the question and then leverages the clue to enhance a reader for MCQA. It outperforms text-to-text models on multiple MCQA datasets.
Multiple-choice question answering (MCQA) aims at selecting the correct answer from a set of options given a question. This long-standing challenge in natural language processing (NLP) requires machines to have a wealth of knowledge, such as commonsense knowledge (Talmor et al., 2019; Mihaylov et al., 2018) and scientific knowledge (Clark et al., 2018; Khot et al., 2020; Huang et al., 2019; Li et al., 2021), and have reasoning skills such as multi-hop reasoning (Khot et al., 2019) and logical reasoning (Yu et al., 2020; Liu et al., 2020b; Li et al., 2022).
MCQA has made great progress with the development of pre-trained language models (PLMs). Basically there are two types of PLMs that are suitable for different tasks. BERT (Devlin et al., 2019)
and its variants such as RoBERTa (Liu et al., 2019) and ALBERT (Lan et al., 2020) are encoder-only models, being more suitable for natural language understanding (NLU) tasks including MCQA and other classification and regression tasks. T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) are encoder-decoder models, being more suitable for natural language generation (NLG) tasks. However, encoder-decoder models can also be applied to MCQA (Khashabi et al., 2020; Zhou et al., 2021). This is enabled by the text-to-text framework, which transforms data in different tasks into a unified text-to-text format so that knowledge spanning many and various tasks can be learned, aggregated, and used by a single model.
Research Question To fit MCQA, existing implementations of the text-to-text framework take all the options as input and are trained to generate one of the options, i.e., to copy some tokens from the input. However, this is inconsistent with how encoder-decoder models are pre-trained so that their underlying knowledge may not be sufficiently exploited. Indeed, Liu et al. (2021) have found that in classification and regression tasks, the decoder layer is often under-utilized. One research question is how to apply pre-trained encoder-decoder models in a more natural way to MCQA, in particular, to exploit their NLG capabilities.
Our Contribution Our idea is inspired by human behavior. When reading a question, humans are sometimes triggered to associate the question with their background knowledge to form some clues even before reading the options. For simple questions, a clue may be exactly the correct answer,
Human Language Technologies, pages 3272 - 3287
A company makes notebooks for college courses, so their main material is?
(A) Chips (B) Water (C) Grass (D) Trees
Clue: Paper
while for complex questions, clues may play an auxiliary role to help humans connect the question with the correct answer. For example, for the question shown in Figure 1, the clue "paper" forms an intermediate concept between "notebook" in the question and "tree" in the correct answer.
With this idea, we propose to employ a pretrained encoder-decoder model to generate a clue from the question by exploiting its underlying knowledge, without seeing and being strictly confined to the options as in the text-to-text framework. The clue representation is then leveraged by an encoder-based model to read the options and make prediction. We refer to this generation-enhanced MCQA model as GenMC. It significantly outperforms comparable models, in particular, text-to-text models, on five MCQA datasets.
Outline We discuss related work in Section 2, introduce GenMC in Section 3, describe the experimental setup in Section 4, report the results in Section 5, and conclude in Section 6.
Code Our code is available on GitHub<sup>1</sup> under the Apache Licence 2.0.
2.1 Text-to-Text Paradigm for MCQA
Recently, the text-to-text paradigm has achieved breakthrough results on many NLP tasks (Raffel et al., 2020; Lewis et al., 2020). As illustrated in Figure 2a, adopting this paradigm for MCQA, the question  $Q$  and all the options  $\{O_1,O_2,O_3,O_4\}$  are spliced into a text as input, and the correct answer  $O_{1}$  is used as the generation target. One benefit is that extensive training data can be shared across different tasks. Using such a framework, UnifiedQA (Khashabi et al., 2020) integrates 20 QA datasets into a unified format for training, and achieves state-of-the-art results on multiple MCQA datasets. Similarly, CALM (Zhou et al., 2021)
(a) Text-to-Text
Figure 1: An example MCQA task and a generated clue. Bold underline indicates the correct answer.
(b) Encoder-Only
Figure 2: Paradigms for MCQA.
learns concept-centric knowledge from text for commonsense QA.
However, it might be debatable whether it is appropriate to train a classification task via a generation target. Liu et al. (2021) point out that the decoder layers of T5 are under-utilized when finetuning on classification and regression tasks. Therefore, they propose a method to reduce the number of T5 parameters to improve efficiency without reducing accuracy. By contrast, we address this issue from a different perspective of how to exploit the NLG capability of pre-trained encoder-decoder models for MCQA to improve accuracy.
Some other works propose new pre-trained models for unified generation and classification tasks by designing universal encoders and task-specific decoders (Shao et al., 2021; Sun et al., 2021). They are orthogonal to our work as we leverage existing pre-trained encoder-decoder models instead of pre-training new models at an additional cost.
2.2 Encoder-Only Paradigm for MCQA
Benefiting from the powerful NLU capabilities of BERT-style PLMs (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020), the encoder-only paradigm has been popular for MCQA. As illustrated in Figure 2b, in this paradigm, the question  $Q$  and each option in  $\{O_1,O_2,O_3,O_4\}$  are interacted to calculate a score, and the option with the highest score is chosen as the answer. Building on this, some works study how to design better attention-based models to identify evidence (Chen et al., 2019; Zhang et al., 2020; Zhu et al., 2020). Other efforts mimic human behavior of reading evidence and answering questions (Ran et al., 2019; Tang et al., 2019; Sun et al., 2019). There, evidence is derived from the given passage or retrieved from external corpora. By contrast, we aim at exporting clues from pre-trained models without resorting to extra sources.
https://github.com/nju-websoft/GenMC
Figure 3: Architecture of GenMC. To make the prediction  $O_{p} \in \mathbb{O}$ , the clue generator first takes  $Q$  as input and outputs a clue representation  $\mathbf{H}^{QC}$  which is indicative of the correct answer. The enhanced reader then relies on the generated clue representation to better attend to options from  $\mathbb{O}$  and makes the final prediction. The whole model is trained in an end-to-end manner with both the generation loss  $\mathcal{L}^{\mathrm{GEN}}$  and the classification loss  $\mathcal{L}^{\mathrm{READ}}$ .
2.3 Knowledge in PLMs
Recently, PLMs have been used as knowledge bases (Petroni et al., 2019), and the knowledge in parameters can be exported via methods such as Prompt (Jiang et al., 2020; Shin et al., 2020). Exploiting the knowledge in PLMs for QA tasks has come into play in many forms including question expansion (Mao et al., 2021) and question generation (Shwartz et al., 2020).
There is also research on MCQA trying to exporting knowledge from PLMs before answering. Rajani et al. (2019) propose CAGE as a framework for generating explanations for commonsense QA. However, CAGE relies on explanations annotated by humans, which are not available in many real scenarios and datasets. Latcinnik and Berant (2020) propose a joint generator-classifier model where the generator produces a human-readable textual hypothesis. Although it somewhat improves the explainability of MCQA, in terms of accuracy of MCQA there is little advancement. CEGI (Liu et al., 2020c) is probably the most similar work to ours. It first uses a generative model to generate evidence, and then uses a reading model to incorporate the evidence and predict the answer, both using answer supervision. However, the generative model and the reading model are separate steps in a pipeline and are connected only via the evidence text. Such token-level interaction can lead to significant losses in accuracy as we will see in our experiments, where our representation-level interaction exhibits better performance.
3 GenMC Model
In MCQA, a question  $Q$  is given together with a set of  $n$  options  $\mathbb{O} = \{O_1,\dots ,O_n\}$  with exactly one option being the correct answer. The key to finding the correct answer is to capture and deeply understand the connection between  $Q$  and each  $O_{i}\in \mathbb{O}$ , which oftentimes is beyond the lexical level and requires a non-trivial entailment process. We follow the trend of building on a pretrained encoder-decoder model and use the encoder to jointly encode  $Q$  and each  $O_{i}$ . However, previous works directly use the decoder to generate an option in  $\mathbb{O}$ , i.e., using the decoder as a classifier, which may have under-exploited the model's NLG capability (Liu et al., 2021). Moreover, a simple joint encoding of  $Q$  and each  $O_{i}$  can only enable lexical-level reasoning (Zellers et al., 2019) which is insufficient for MCQA tasks.
Our proposed model GenMC overcomes these limitations. Building on a pre-trained encoder-decoder model, GenMC firstly generates a clue which is indicative of the correct answer, thereby exploiting the NLG capability and underlying knowledge of the pre-trained encoder-decoder model. Then GenMC employs the generated clue representation as intermediate knowledge connecting the question and the correct answer to interact with and enhance a reader for solving MCQA. Our model design mimics how humans solve an MCQA task, i.e., after reading a question, humans may firstly associate it with some of their background knowledge (i.e., looking for clues) that helps them to later identify the correct answer.
The overall architecture of GenMC is shown in
Figure 3. The clue generator (Section 3.1) first generates a clue representation only given  $Q$ . Then the enhanced reader (Section 3.2) uses the generated clue to augment question-option understanding.
3.1 Clue Generator
The clue generator takes the question  $Q$  as input and autoregressively outputs a clue  $C = c_{1},\ldots ,c_{|C|}$  using a pre-trained encoder-decoder model. Note that not the clue text  $C$  but its representation  $\mathbf{H}^C$  will be used in our model, although one could output  $C$  as evidence for explainability.
Specifically, we obtain the question representation  $\mathbf{H}^Q\in \mathbb{R}^{d\times |Q|}$  and the clue representation  $\mathbf{H}^C\in \mathbb{R}^{d\times |C|}$  from the last layer of the encoder and of the decoder, respectively, where  $d$  denotes the representation dimension.  $\mathbf{H}_j^C$  , denoting the representation of the  $j$  -th token  $c_{j}\in C$  , is computed as follows:
$$
\mathbf {p} _ {j} ^ {C}, \mathbf {H} _ {j} ^ {C} = \operatorname {D e c o d e r} \left(c _ {j - 1}, \mathbf {H} _ {<   j} ^ {C}, \mathbf {H} ^ {Q}\right), \tag {1}
$$
where Decoder  $(\cdot ,\cdot ,\cdot)$  takes the last token  $c_{j - 1}$  the representation for the decoding history  $\mathbf{H}_{< j}^{C}$  and  $\mathbf{H}^Q$  as input, and outputs the hidden state  $\mathbf{H}_j^C$  together with the probability distribution  $\mathbf{p}_j^C$  over the decoding vocabulary at the  $j$  -th step.
To encourage the tokens in  $C$  to thoroughly interact with each other and with  $Q$ , we strengthen the clue representation by passing it to a transformer layer (Vaswani et al., 2017) and obtain  $\mathbf{H}^{QC}$ :
$$
\mathbf {H} ^ {Q C} = \operatorname {T r a n s f o r m e r} \left(\left[ \mathbf {H} ^ {Q}; \mathbf {H} ^ {C} \right]\right), \tag {2}
$$
where  $[\cdot ;\cdot ]$  denotes concatenation.  $\mathbf{H}^{QC}$  carries the information of  $C$  which can be helpful to better understand and answer  $Q$
3.2 Enhanced Reader
Previous works often directly model the relevance of each  $O_{i} \in \mathbb{O}$  to  $Q$  via joint encoding using a pretrained encoder, which largely performs superficial lexical reasoning (Zellers et al., 2019). By contrast, we use the previously generated clue representation to enhance our reader for a deeper understanding of each question-option pair.
Specifically, we first concatenate  $Q$  and each  $O_{i}$  independently<sup>3</sup> and feed the concatenated input into the pre-trained encoder (which is shared with our clue generator) to obtain  $O_{i}$ 's contextualized
representation  $\mathbf{H}_i^{QO}$ , which constitutes a column of  $\mathbf{H}^{QO} \in \mathbb{R}^{d \times n}$  where  $n = |\mathbb{O}|$ .
Next, based on the clue representation  $\mathbf{H}^{QC}$ , our model intensively reads each question-option pair and obtains the matching signal between the clue and the option. Specifically, inspired by Huang et al. (2021), we first use dual-attention (Liu et al., 2020a) to fuse information from  $\mathbf{H}_i^{QO}$  to  $\mathbf{H}^{QC}$  and from  $\mathbf{H}^{QC}$  to  $\mathbf{H}_i^{QO}$ . Then we perform max-pooling to aggregate the matching features:
$$
\left(\hat {\mathbf {H}} _ {i} ^ {Q O}, \hat {\mathbf {H}} _ {i} ^ {Q C}\right) = \text {D u a l A t t e n t i o n} \left(\mathbf {H} _ {i} ^ {Q O}, \mathbf {H} ^ {Q C}\right),
$$
$$
\mathbf {f} _ {i} ^ {Q O} = \operatorname {M a x - P o o l i n g} \left(\hat {\mathbf {H}} _ {i} ^ {Q O}\right), \tag {3}
$$
$$
\mathbf {f} _ {i} ^ {Q C} = \operatorname {M a x - P o o l i n g} \left(\hat {\mathbf {H}} _ {i} ^ {Q C}\right).
$$
To obtain the final score  $s_i$  for each  $O_i$ , we concatenate the dual matching features  $\mathbf{f}_i^{QO}$  and  $\mathbf{f}_i^{QC}$  and feed them into a two-layer multi-layer perceptron (MLP):
$$
s _ {i} = \operatorname {L i n e a r} \left(\operatorname {R e L U} \left(\operatorname {L i n e a r} \left(\left[ \mathbf {f} _ {i} ^ {Q O}; \mathbf {f} _ {i} ^ {Q C} \right]\right)\right)\right). \tag {4}
$$
We select the option with the highest score as the predicted answer, denoted as  $O_{p}$ .
We jointly train the clue generator and the enhanced reader in an end-to-end fashion with a combined loss:
$$
\mathcal {L} = \mathcal {L} ^ {\mathrm {G E N}} + \mathcal {L} ^ {\mathrm {R E A D}}. \tag {5}
$$
Generator Loss For  $\mathcal{L}^{\mathrm{GEN}}$ , assuming that  $O_{t}\in$  0 is the correct answer containing  $m$  tokens  $a_1,\ldots ,a_m$ , we first use  $O_{t}$  as the target to calculate our clue generator loss with teacher forcing:
$$
\mathbf {p} _ {j} ^ {O _ {t}}, \mathbf {H} _ {j} ^ {O _ {t}} = \operatorname {D e c o d e r} \left(a _ {j - 1}, \mathbf {H} _ {<   j} ^ {O _ {t}}, \mathbf {H} ^ {Q}\right),
$$
$$
\mathcal {L} ^ {\text {G E N}} = - \frac {1}{m} \sum_ {j = 1} ^ {m} \log \mathbf {p} _ {j, a _ {j}} ^ {O _ {t}}, \tag {6}
$$
where  $\mathbf{p}_j^{O_t}$  denotes the probability distribution over the decoding vocabulary at the  $j$ -th step, and  $\mathbf{p}_{j,a_j}^{O_t}$  is the probability of token  $a_{j}$ .
Reader Loss For  $\mathcal{L}^{\mathrm{READ}}$ , we simply calculate a cross-entropy loss given the correct answer  $O_{t}\in \mathbb{O}$  as follows:
$$
\mathcal {L} ^ {\text {R E A D}} = - \log \frac {\exp \left(s _ {t}\right)}{\sum_ {i = 1} ^ {n} \exp \left(s _ {i}\right)}. \tag {7}
$$
Note that we update the encoder using the joint loss  $\mathcal{L}$ , while we do not allow  $\mathcal{L}^{\mathrm{READ}}$  to be backpropagated to the decoder part to reduce the memory consumption.
2For efficiency, we decode the clue greedily without performing beam search.
$^3$ A delimiters "\n" is inserted between  $Q$  and each  $O_i$ .
Train set size	Dev set size	Test set size	Option number	Question average length	Option average length
CSQA	8,500	1,241	1,221	5	13.38	1.52
OBQA	4,957	500	500	4	10.65	2.85
ARC-Easy	2,241	567	2,365	4	19.36	3.73
ARC-Challenge	1,117	295	1,165	4	22.30	4.93
QASC	7,320	814	926	8	8.12	1.64
Table 1: Dataset statistics. For CSQA and QASC, their official dev sets are used as our test sets, and our dev sets are in-house split from their official training sets.
The above training objective exploits the double properties of the correct answer  $O_{t}$  in MCQA: as a text and as an index. We use  $O_{t}$  as a text to supervise our clue generator, and as an index (i.e., classification label) to supervise our enhanced reader. Such usage is more natural than the text-to-text paradigm (Khashabi et al., 2020; Zhou et al., 2021), thus having the potential to outperform.
We conducted experiments on five popular MCQA datasets spanning from commonsense questions to scientific questions. The former requires commonsense knowledge and reasoning, and the latter requires inference over scientific facts.
Datasets CSQA (Talmor et al., 2019) and OBQA (Mihaylov et al., 2018) are two commonsense MCQA datasets created by crowd workers based on commonsense facts. Each question is given with 5 options in CSQA and 4 options in OBQA. ARC-Easy and ARC-Challenge, denoting two disjoint subsets of ARC (Clark et al., 2018), contain natural grade-school science questions with 4 options, where ARC-Challenge comprises difficult questions which require more advanced reasoning. QASC (Khot et al., 2020) is collected from elementary and middle school level science with 8 options for each question.
Train-Dev-Test Split For OBQA, ARC-Easy, and ARC-Challenge we used their official train, dev, and test sets. For CSQA and QASC, since the correct answers in the official test set are not public, we took their official dev set as our test set for experiments and randomly held out an in-house dev set from the training set. The dataset statistics are shown in Table 1.
External Knowledge For all these datasets, our experiments did not rely on any provided documents or external corpora; a question was solely
provided with its options to form the input. It means that pre-trained models were used as the primary source of knowledge in the experiments.
We used two popular encoder-decoder models as a basis, BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). For each model, we experimented with its BASE and LARGE versions.
We used PyTorch 1.7. We used the Adam optimizer and set warmup fraction  $= 0.1$ , weight decay  $= 0.01$ , maximum source length  $= 64$ , maximum target length  $= 32$ , epoch  $= 30$ , and early stop training when there was no better result on the dev set after 5 epochs. For each model, we searched for the best learning rate from  $\{1e - 4, 5e - 5, 1e - 5\}$ , and for the best batch size out of  $\{8, 64\}$ .
Because neural models are known to be sensitive to different random seeds, especially when the training set is small, we performed multiple experiments for all models with different random seeds, and reported the mean and standard deviation. For CSQA, OBQA, ARC-Easy, and QASC, we used three random seeds  $\{1, 10, 20\}$ . For the smallest dataset ARC-Challenge, we used five random seeds  $\{1, 10, 20, 30, 40\}$ .
All the experiments were performed on a GeForce RTX 3090 with 24G memory.
4.3 Evaluation Metric
For each model, we reported its proportion of correctly answered questions in each dataset.
5.1 Main Results: Comparison with Text-to-Text Models
To empirically evaluate GenMC in terms of whether it better exploits the potential of pretrained encoder-decoder models for MCQA, we compare GenMC with a standard text-to-text implementation and with a variant thereof for analysis.
CSQA	OBQA	ARC-Easy	ARC-Challenge	QASC
dev	test	dev	test	dev	test	dev	test	dev	test
BARTBASE
Text2Textvanilla	51.62 (±0.04)	53.26 (±0.57)	54.93 (±0.83)	52.73 (±1.00)	51.55 (±1.38)	50.51 (±1.82)	30.05 (±1.25)	24.95 (±1.10)	46.72 (±1.21)	26.78 (±1.21)
Text2Textenc	50.63 (±0.66)	52.22 (±1.64)	55.87 (±1.10)	51.00 (±1.83)	49.03 (±1.86)	49.94 (±1.49)	32.32 (±4.87)	26.24 (±2.01)	48.08 (±1.35)	17.06 (±0.39)
GenMC	54.82 (±0.61)	56.40 (±0.61)	58.53 (±0.31)	57.53 (±2.91)	59.38 (±1.60)	56.80 (±0.28)	38.64 (±0.90)	33.82 (±1.66)	57.70 (±0.43)	35.96 (±1.70)
T5BASE
Text2Textvanilla	57.59 (±0.81)	60.93 (±0.73)	59.53 (±0.81)	57.53 (±0.70)	52.20 (±0.31)	51.75 (±0.89)	29.38 (±2.63)	23.69 (±2.47)	54.55 (±1.01)	37.94 (±1.47)
Text2Textenc	58.96 (±1.21)	59.49 (±1.41)	60.67 (±2.86)	57.07 (±3.03)	56.55 (±1.17)	52.92 (±0.29)	29.49 (±5.13)	26.09 (±0.23)	56.84 (±0.84)	39.60 (±2.38)
GenMC	60.65 (±0.47)	63.45 (±0.29)	62.07 (±1.01)	61.67 (±0.58)	62.38 (±0.67)	58.82 (±0.37)	43.62 (±0.52)	39.00 (±0.30)	58.93 (±1.76)	41.72 (±1.18)
BARTLARGE
Text2Textvanilla	65.58 (±2.72)	66.91 (±2.14)	62.66 (±1.18)	61.46 (±1.74)	63.49 (±1.89)	62.81 (±2.15)	29.94 (±2.32)	28.55 (±4.97)	64.57 (±2.21)	47.80 (±2.22)
Text2Textenc	65.00 (±0.66)	67.35 (±0.90)	63.80 (±1.44)	62.47 (±1.53)	68.20 (±2.04)	65.33 (±1.74)	35.37 (±6.07)	31.13 (±5.86)	65.07 (±0.94)	47.19 (±0.71)
GenMC	69.57 (±0.89)	72.26 (±0.70)	68.93 (±1.17)	68.07 (±1.70)	72.43 (±0.54)	68.68 (±0.34)	48.93 (±0.98)	45.52 (±1.54)	68.39 (±0.68)	55.90 (±0.92)
T5LARGE
Text2Textvanilla	67.53 (±0.43)	70.63 (±0.74)	66.80 (±0.87)	63.53 (±1.10)	65.61 (±0.18)	62.55 (±0.54)	43.05 (±1.69)	42.83 (±2.00)	64.13 (±1.47)	57.74 (±0.82)
Text2Textenc	68.41 (±0.73)	70.30 (±0.82)	65.93 (±1.03)	63.67 (±0.46)	69.61 (±0.20)	66.65 (±0.34)	30.73 (±3.15)	28.76 (±4.85)	65.27 (±1.55)	55.65 (±0.45)
GenMC	71.10 (±0.41)	72.67 (±1.02)	71.60 (±0.92)	66.87 (±1.33)	72.49 (±0.77)	69.01 (±1.97)	49.83 (±2.06)	47.41 (±2.00)	67.61 (±1.14)	58.06 (±0.92)
Table 2: Comparison with text-to-text models.
Text2Text_vanilla The vanilla usage of pre-trained encoder-decoders for MCQA is to reform the input and output in a way that can be directly processed by a encoder-decoder model. Specifically, following Raffel et al. (2020), we concatenate the input question with all candidate options, where each option is also preceded by its option ID, and then preprocess the sequence with a dataset name. The concatenated sequence is fed into the encoder part to get a joint representation for the question and all options. Based on the joint representation, the decoder finally outputs an option ID. In this setting, the decoder is basically used as a classifier.
Text2Text<sub>enc</sub> Similar to Liu et al. (2021), we use only the encoder part of a pre-trained encoder-decoder model. Each option is independently paired with the question to obtain a joint representation using the encoder. Then the representation is fed into a scorer (i.e., an MLP) to obtain a matching score for each question-option pair. The model then predicts the option with the highest score. In this setting, the decoder is totally unused. Though Liu et al. (2021) find that their encoder-only model performs comparably to using the decoder as a classifier, we argue that the decoder part can further improve the performance, if being properly used.
The main results (see Table 2) show that GenMC consistently and significantly (with p-value  $< 0.01$ ) outperforms Text2Text_vanilla and Text2Text_enc on all datasets. For several settings, GenMC even obtains an absolute gain of over  $10\%$ . For example, on the test set of the challenging scientific MCQA dataset ARC-Challenge, T5_BASE + GenMC improves T5_BASE + Text2Text_vanilla from an accu