SlowGuess/ACL_2020_OCR huggingface dataset:
Alexandre Tamborrino*, Nicola Pellicanò*, Baptiste Pannier*, Pascal Voitot, Louise Naudin
Samsung Strategy and Innovation Center
{a.tamborrino,n.pelicano,b.pannier,p.voitot,l.naudin}@samsung.com
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks (Devlin et al., 2019). Most of the existing approaches rely on a randomly initialized classifier on top of such networks. We argue that this fine-tuning procedure is sub-optimal as the pre-trained model has no prior on the specific classifier labels, while it might have already learned an intrinsic textual representation of the task. In this paper, we introduce a new scoring method that casts a plausibility ranking task in a full-text format and leverages the masked language modeling head tuned during the pre-training phase. We study commonsense reasoning tasks where the model must rank a set of hypotheses given a premise, focusing on the COPA (Gordon et al., 2012), Swag (Zellers et al., 2018), HellaSwag (Zellers et al., 2019) and CommonsenseQA (Talmor et al., 2019) datasets. By exploiting our scoring method without fine-tuning, we are able to produce strong baselines (e.g.  $80\%$  test accuracy on COPA) that are comparable to supervised approaches. Moreover, when fine-tuning directly on the proposed scoring function, we show that our method provides a much more stable training phase across random restarts (e.g  $\times 10$  standard deviation reduction on COPA test accuracy) and requires less annotated data than the standard classifier approach to reach equivalent performances.
Recent advances in natural language processing have been made using sequential transfer learning over large pre-trained transformer models. From these models, most NLP tasks can be addressed by adding a classifier on top of the transformer embedding outputs (Devlin et al., 2019; Liu et al., 2019).
In this paper, we tackle a subset of NLP tasks consisting in plausibility ranking. Such tasks can be formalised as follows: given a unique premise  $p$  and a set of hypotheses  $H = \{h_i\}_{i=1\dots n}$ , the task consists in returning the appropriate hypothesis  $h^* \in H$  that matches  $p$  (see Section 3 for more details). A natural task that fits into this problem formulation is commonsense reasoning. Thus, it will be the main focus of the present paper.
Traditionally, this problem is solved by jointly classifying each pair  $(p,h_i)_{i = 1\dots n}$ . For instance, assuming a Masked Language Modeling (MLM) model is used, an example from the COPA dataset (Gordon et al., 2012) is commonly casted into two distinct examples:
- [CLS] The man broke his toe. [SEP] He dropped a hammer on his foot. [SEP]  $\rightarrow$  correct
- [CLS] The man broke his toe. [SEP] He got a hole in his sock. [SEP] → incorrect
The special token [CLS] (used for sentence level tasks) is then provided to a classifier in order to predict the label of the given example; [SEP] is a special separator token. This format will be referred to as separated-sentence. For such a task, the use of the randomly initialized head can appear sub-optimal since the pre-trained model does not integrate any prior on the specific classifier label. To validate this intuition, we cast the MLM model inputs into a full-text format. Thus, the separation token is dropped and potentially replaced by conjunction words that are fully specific to the task. The previously illustrated correct example will be turned into: [CLS] The man broke his toe because he dropped a hammer on his foot [SEP]. Using this input format, we apply a new bidirectional word-level scoring function that leverages the MLM head (Devlin et al., 2019)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878-3887 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tuned during the pre-training phase (see Figure 1 for an overview of the proposed approach). This method produces strong zero-shot $^{1}$  baselines on the COPA (Gordon et al., 2012), Swag (Zellers et al., 2018), HellaSwag (Zellers et al., 2019) and CommonsenseQA (Talmor et al., 2019) datasets. Then, we fine-tune this new scoring function with a margin-based loss as proposed in (Li et al., 2019). Using RoBERTa $_{LARGE}$ , our results reveal that this new training procedure leads to better accuracy and much more stable training trajectories which is an important feature since large MLM models are known to be unstable on several tasks (Devlin et al., 2019; Phang et al., 2018). Finally, we find that a progressive decrease of the training dataset size results in a progressive increase of the accuracy gap between our proposed method and the standard classifier ones. This makes our method advantageous in small dataset context.
In (Trinh and Le, 2018), researchers have shown that a RNN Language Model pretrained on a large amount of data can be used to efficiently score sentences in a zero-shot setting. They used the Winograd Schema Challenge (WSC-273) dataset (Levesque et al., 2012) which mostly consists of a pronoun disambiguation task that requires commonsense reasoning. In their approach, the pronoun to disambiguate is replaced by the different candidates. Then, each version of the sentence is scored using the likelihood of the sequence under the forward autoregressive factorization. They showed that targeting the likelihood of the tokens placed after the candidate words performs better than a full-sentence likelihood estimation. This result highlights the fact that the choice of the targeted sub-sequence for the likelihood estimation has an important impact on the overall performance of the model. More recently, analysis of relational knowledge contained in pre-trained BERT models has been the subject of different studies (Petroni et al., 2019; Poerner et al., 2019). Results have shown evidences that BERT models memorize reasoning about entity names and commonsense knowledge, making MLM models appropriate candidates to commonsense oriented tasks.
From a supervised learning perspective, (Li et al., 2019) proposed to replace the traditional
cross-entropy loss with a margin-based one one the COPA dataset. The authors argued that cross-entropy based methods are not adapted for plausibility ranking tasks since they force the scores to adopt extreme values (near 0 or 1). In contrast, a margin-based objective function appeared to be a natural way to rank a set of hypotheses. Both approaches were compared using the [CLS] token of the BERT-base model and a separated-sentence input format. The margin-based objective function surpassed the cross-entropy one by increasing the Test set accuracy from  $73.4\%$  to  $75.4\%$ .
Adopting a token level scoring approach (Koci-jan et al., 2019) used a BERT model with a mixture between a margin-based and a MLM loss on WSC-273 to score the different pronouns to disambiguate. This approach allows the authors to improve the previous state of the art by  $8.8\%$ . Despite being the closest method to the one proposed in this paper, our approach differs from three points:
- We generalize the scoring method by targeting different contiguous sub-sequences for the likelihood estimation. To do so, different datasets are recasted in a full-text format.
- We also focus on targeting the premise avoiding inner statistical biases of different hypotheses (e.g. word frequencies, punctuation, variable sequence lengths etc...).
- The objective of the present paper is to propose a direct comparison in terms of accuracy and training stability across random restarts between the proposed method and standard classifiers.
Given an input premise  $p = (p^{(1)}, p^{(2)}, \ldots, p^{(L_p)})$ , and a set of candidate hypotheses:
$$
H = \left\{h _ {i} = (h _ {i} ^ {(1)}, h _ {i} ^ {(2)}, \ldots , h _ {i} ^ {(L _ {i})}) \right\} _ {i = 1 \ldots n},
$$
we aim to identify the fitting hypothesis  $h^{*} \in H$  which correctly matches  $p$ . The values  $L_{p}$  and  $\{L_{i}\}_{i = 1\dots n}$  are the sequence lengths of premise and hypotheses respectively. In a commonsense settings, such problem corresponds to find premise-hypothesis implications by exploiting some prior commonsense knowledge. Since
For the following of our paper, we will note as zero-shot setting the use of the pre-trained model without fine-tuning.
Figure 1: Overview of the proposed method for the task  $t = \mathrm{COPA}$ . Two full-text sequences (Section 3.1),  $\mathbf{s}_{\text{true}}$  and  $\mathbf{s}_{\text{false}}$ , are given as input (gold and distractor premise/hypothesis pairs respectively). Circled numbers explicitly mark input and output of five different versions of a given sentence, where each has a different premise word masked. The output probabilities  $\mathcal{P}_i^{(k)} = P(p^{(k)} \mid \mathbf{s}_i^{\backslash p^{(k)}})$  contribute to the score computation (target premise score  $S_i^p$  in this example, see Section 3.2). When fine-tuning on the task is performed, gold and distractor scores are used for margin-based loss computation (Section 3.3).
our scoring method consumes input sequences in a full-text format (see Section 3.2), our method is formulated on a commonsense task but not limited to it.
3.2 Sequence Scoring Method
The proposed Sequence Scoring Method (SSM), takes as input a pair  $\langle p, h_i \rangle$  returns a score representing the likelihood of  $h_i$  of being implied by  $p$ .
First, a transform operator  $\mathcal{T}$  converts  $\langle p,h_i\rangle$  pair into a full-text input. Such operator, in it's simplest form, just concatenates the two sequences. However, in general  $\mathcal{T}$  can be constrained on the task  $t$ .
$$
\mathbf {s} _ {i} = \mathcal {T} ^ {t} (p, h _ {i}) = (c _ {l} ^ {t}, p, c _ {m} ^ {t}, h _ {i}, c _ {r} ^ {t}), \qquad (1)
$$
where  $\mathbf{s}_i$  is the resulting full-text input, while  $c_l^t$ ,  $c_m^t$ , and  $c_r^t$  are left, middle and right conjunction sequences of the task. For example, Swag will have no conjunction, since the correct hypothesis is the natural continuation of the premise, while COPA will have because/so middle conjunctions due to its cause/effect nature (see Section 4).
Given the full-text input, the scorer aims to exploit the pre-training task of word masking in or-
der to compute its result. Let us consider the masking of a word  $w$  which contributes to make sense of the matching between  $p$  and  $h_i$ . The intuition is that the confidence of the network in recovering such word is directly related to the score of  $\langle p, h_i \rangle$ . Let us define, inspired by the notation of (Song et al., 2019),  $\mathbf{s}_i^{\backslash w}$  as the sentence  $\mathbf{s}_i$  with the tokens of  $w$  replaced by the [MASK] token.
The target premise score is calculated as follows:
$$
S _ {i} ^ {p} = \sum_ {k = 1} ^ {L _ {p}} \log \left[ P \left(p ^ {(k)} \mid \mathbf {s} _ {i} ^ {\backslash p ^ {(k)}}\right) \right], \tag {2}
$$
where premise words are masked one by one in order to compute their relevance with respect to the given hypothesis. Masked word probability is estimated from direct inference on a model pretrained on MLM task. The computational complexity of such method grows linearly with  $L_{p}$  (requiring  $L_{p}$  examples per forward pass). Alternatively, the target hypothesis score is computed as:
$$
S _ {i} ^ {h} = \frac {1}{L _ {i}} \sum_ {k = 1} ^ {L _ {i}} \log \left[ P \left(h _ {i} ^ {(k)} \mid \mathbf {s} _ {i} ^ {\backslash h _ {i} ^ {(k)}}\right) \right]. \tag {3}
$$
The target hypothesis score needs normalization by  $L_{i}$  in order to allow comparison between variable candidate hypothesis length. The best hypothesis will be taken as the one maximizing the target premise (or hypothesis) score:
$$
h ^ {*} = h _ {j} \in H s. t. \max  _ {i = 1 \dots n} S _ {i} ^ {p} = S _ {j} ^ {p}. \quad (4)
$$
As demonstrated in Section 5.2, the target premise score allows for a fairer comparison between different hypotheses. In fact, they present inherent differences in terms of statistical frequency of words, sequence length or may exhibit more or less strong inter-dependency between words (e.g. composite words reinforce each other confidence). Such variance could introduce a bias in the relative significance of each hypothesis alone (independently from the premise). On the opposite, different probabilities on the same target premise word can only be affected by the change of hypothesis context.
N-grams sequence scoring
We can extend the proposed SSM by scoring the reconstruction not only of single words, but of entire n-grams. Adding n-grams probabilities to the logarithmic mean combination not only robustifies the scoring methods, but helps to better model the joint probability of (dependent) close words, especially in a zero-shot setting. Let us note as  $p^{(u:v)}$  as the sub-sequence of  $p$  spanning between indexes  $u$  and  $v$  (included). The partial target premise score for g-grams (i.e. mask windows of size  $g$ ) can be expressed as:
$$
S _ {i} ^ {p, g} = \sum_ {k = 1} ^ {L _ {p} - g + 1} \log \left[ P \left(p ^ {(k: k + g - 1)} \mid \mathbf {s} _ {i} ^ {\backslash p ^ {(k: k + g - 1)}}\right) \right].
$$
By definition the target premise score in Equation 2 is equivalent to 1-gram partial target premise score (i.e.  $S_{i}^{p} \triangleq S_{i}^{p,1}$ ). The n-gram sequence scoring accumulates masked language model probabilities from every gram size till  $n$ .
$$
S _ {i} ^ {p, [ 1, n ]} = \sum_ {g = 1} ^ {n} S _ {i} ^ {p, g}. \tag {5}
$$
3.3 SSM-based fine-tuning
The proposed score function, since it does not imply any addition of a head module, can be directly applied without any retraining (see Section 5.2). It can also be directly used when fine-tuning on the
task. The different masked inputs needed to compute the target premise score,  $\left\{\mathbf{s}_i^{\backslash p^{(j)}}\right\}_{j = 1..L_p}$ , are batched together in order to compute score  $S_{i}^{p}$  in one forward pass. The model acts as a siamese network that performs independent computation of target premise score for each hypothesis  $h_i$ .
As already noted in (Li et al., 2019), multiple choice tasks (e.g. COPA) are more naturally expressed as learning to rank problems. For this reason we adopt as objective function a margin-based loss in contrast to cross-entropy loss. Given ground truth sentence index  $i^{*}$ , the loss is specified as:
$$
\mathcal{L} = \frac{1}{n}\sum_{\substack{i = 1\\ i\neq i^{*}}}^{n}\max \left(0,\eta -S_{i^{*}}^{p} + S_{i}^{p}\right),\qquad (6)
$$
where  $\eta$  is a margin threshold hyperparameter.
According to our preliminary experiments, we do not add a second MLM component in the general loss (as in (Kocijan et al., 2019)), since it always leads to a decrease of the model performance for various weighted contributions of the MLM term.
The commonsense reasoning datasets that we focus on are COPA (Gordon et al., 2012), Swag (Zellers et al., 2018), HellaSwag (Zellers et al., 2019) and CommonsenseQA (Talmor et al., 2019). All these datasets share the premise-hypothesis task format. Table 1 shows examples of full-text format and separated-sentence format for all datasets.
COPA (Choice of Plausible Alternatives) (Gordon et al., 2012) is a commonsense causal reasoning task where two candidate hypotheses are given. COPA itself is composed of two sub-tasks: effect samples and cause samples. The effect and cause samples have respectively implies and implied by relation with the correct hypothesis. The full-text format of COPA is built by using the conjunction words because (resp. so) as middle conjunctions for cause (resp. effect) samples. Concerning the separated-sentence format, we reverse the premise and hypothesis order for cause samples in
Dataset	Full-text format	Separated-sentence format
COPA (effect)	[CLS] I knocked on my neighbor's door so my neighbor invited me in. [SEP]	[CLS] I knocked on my neighbor's door. [SEP] My neighbor invited me in. [SEP]
COPA (cause)	[CLS] The man broke his toe because he dropped a hammer on his foot. [SEP]	[CLS] He dropped a hammer on his foot. [SEP] The man broke his toe. [SEP]
CommonsenseQA	[CLS] Q: Where on a river can you hold a cup upright to catch water on a sunny day? A: waterfall [SEP]	[CLS] Q: Where on a river can you hold a cup upright to catch water on a sunny day? [SEP] A: waterfall [SEP]
Swag	[CLS] We notice a man in a kayak and a yellow helmet coming in from the left. As he approaches, his kayak flips upside-down. [SEP]	[CLS] We notice a man in a kayak and a yellow helmet coming in from the left. [SEP] As he approaches, his kayak flips upside-down. [SEP]
HellaSwag	[CLS] A man is standing in front of a camera. He starts playing a harmonica for the camera. He rocks back and forth to the music as he goes. [SEP]	[CLS] A man is standing in front of a camera. He starts playing a harmonica for the camera. [SEP] He rocks back and forth to the music as he goes. [SEP]
Table 1: Examples of full-text format and separated-sentence format for gold premise-hypothesis pairs. Left conjunction  $c_{l}^{t}$  is highlighted in italic blue, middle conjunction  $c_{m}^{t}$  in bold red.
order to convert all cause samples into effect samples. This has the benefit to present a unique task to the model, and our experiments show that this give better results than keeping cause samples and effect samples unmodified. We choose the SuperGLUE split (Wang et al., 2019).
CommonsenseQA
CommonsenseQA (Talmor et al., 2019) is a multiple-choice commonsense question answering dataset where each question has one correct answer and four distractor answers. To create the full-text format, we preprocess  $\mathsf{Q} : \bot$  to the question,  $\mathsf{A} : \bot$  to the answer, and then concatenate the question and the answer (  $\bot$  stands for space character). For the separated-sentence format, we also use the  $\mathsf{Q} : \bot$  and  $\mathsf{A} : \bot$  prefixes to follow the best recommendation from the FairSeq repo on how to fine-tune RoBERTa on CommonsenseQA  ${}^{2}$  . Since the benchmark Test set is private,for our zero-shot and fine-tuning stability studies we have split the original validation set evenly, treating last 611 samples as Test set  ${\text{Test}}^{ * }$  .
Swag and HellaSwag
Swag (Situations With Adversarial Generations) (Zellers et al., 2018) is a multiple choice commonsense dataset about grounded situations. Each premise is a video caption with four answer choices about what might happen next in the scene. The correct answer is the video caption for the next event in the video. The other negative an-
svers are created via Adversarial Filtering: generated by language modeling models and filtered by discriminator models. HellaSwag (Zellers et al., 2019) is an evolved version of Swag using better generators and discriminators models for Adversarial Filtering. Since the benchmark test set is private, we evaluate our zero-shot setting on the Val set (we do not perform a fine-tuning study on Swag and HellaSwag as explained in Section 5.3).
In this section we first apply our scoring method in a zero-shot setting on the four aforementioned datasets. Then we fine-tune our scoring method while varying the percentage of the training data used and compare it to approaches that use a randomly initialized classifier head. We use RoBERTaLARGE (Liu et al., 2019) for our pretrained model as RoBERTaLARGE fine-tuned with a classification layer on top has very competitive results on those datasets. Our implementation uses PyTorch and the HuggingFace Transformers library (Wolf et al., 2019).
5.1 Task probing
Before assessing our zero-shot and fine-tuning results, we perform a task probing by evaluating the zero-shot score we obtain by removing the premise from the input and only scoring the hypotheses. If the score is significantly better than a random baseline, it means that the task is not actually solved by commonsense reasoning, but by using statistical biases in the hypotheses. This probing method has been already used on several
$^{2}$ https://github.com/pytorch/fairseq/tree/master/examples/roberta/commonsense_qa
Dataset	Mode	Acc1(%)
COPA	hyp-only	54.6
random	50.0
CommonsenseQA	hyp-only	22.0
random	20.0
Swag	hyp-only	60.6
random	25.0
HellaSwag	hyp-only	50.8
random	25.0
datasets to show that the underlying task was not really solved by the top-performing models (Niven and Kao, 2019; Zellers et al., 2019).
The results of the task probing evaluation are reported in Table 2. While COPA and CommonsenseQA have a hypothesis only score close to the random baseline, the score of both Swag and HellaSwag are significantly higher than their random baseline (more than twice). This confirms the study from (Zellers et al., 2019) that shows that Swag's false hypotheses were generated using a weak generator, therefore the authors argue that the fine-tuning process on a BERT model on Swag learns to pick up the statistical cues left by the weak generator. Our results show that RoBERTa  $LARGE$  can leverage these distributional biases without the fine-tuning phase. We argue that the human-written pre-training corpora of RoBERTa biases it to give better score to human-written language rather than model-generated sentences. As shown in (Holtzman et al., 2019), there is indeed still a strong distributional differences between human text and machine text. Furthermore, our result also highlights that HellaSwag still exhibits a strong bias due to its generation scheme when evaluated with RoBERTa  $LARGE$ .
5.2 Zero-shot Results
For both COPA and CommonsenseQA, the best performing scoring method uses the target premise and 4-grams settings as shown in Tables 3 and 4. Targeting the premise gives better results than targeting the hypothesis, which reinforces our argument that targeting the hypothesis may be harder as the differences between the hypotheses make the score comparison noisier. Also, more grams
Table 2: Commonsense reasoning task probing. hyponly stands for hypothesis only, random for random baseline.  ${}^{1}$  COPA is evaluated on Test,CommonsenseQA is evaluated on Test*, Swag and HellaSwag are evaluated on Val (see Section 4).
Target	Grams	Test Acc (%)
premise	1	74.0
hypothesis	1	69.8
premise	2	76.2
premise	3	79.0
premise	4	80.0
premise	5	79.4
Table 3: COPA zero-shot results.
Target	Grams	Test* Acc (%)
premise	1	47.8
hypothesis	1	37.4
premise	2	53.2
premise	3	53.7
premise	4	56.1
premise	5	55.2
Table 4:CommonsenseQA zero-shot results.
give increasingly better results but the trend inverts after 4-grams, which may be due to the fact that masked models are not trained to mask large chunks of text. It is interesting to note that our zero-shot result is significantly better than a BERT $_{LARGE}$  cross-entropy model fined-tuned on the COPA training set (80.0% vs. 70.6% accuracy) (Wang et al., 2019), while being comparable for CommonsenseQA $^3$ . Moreover, when we intentionally switch the so and because conjunction words on COPA to make the samples erroneous, the accuracy drops significantly (64.4%). We reckon this is an indicator that our scoring method effectively reuse the pre-learned representation the full-text format of the task.
Concerning Swag and HellaSwag, the target hypothesis mode is significantly better than the target premise mode (see Table 5), as expected from our task probing work in Section 5.1. For example, on HellaSwag, the target hypothesis mode is only  $8\%$  better than the hypothesis only mode ( $58.8\%$  versus  $50.8\%$ ), which confirms that on this setting our zero-shot method is mainly taking advantage of the bias in the hypotheses. Therefore we refrain from doing more zero-shot experiments on both datasets.
5.3 Fine-tuning Results
Following the strong bias of Swag and HellaSwag that was shown in Section 5.1 using our scoring method with RoBERTa  $L_{RAGE}$ , we decide to not
<sup>3</sup>https://www.tau-nlp.org/csqa-leaderboard
Figure 2: COPA fine-tuning results on Test set. The whole training set corresponds to 400 examples.
(a)
(b)
Figure 3: CommonsenseQA fine-tuning results on Test* set. The whole training set corresponds to 9741 examples.
Dataset	Target	Val Acc (%)
Swag	premise	48.3
Swag	hypothesis	72.5
HellaSwag	premise	37.1
HellaSwag	hypothesis	58.8
Table 5: Swag/HellaSwag zero-shot results (1-Gram).