Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions
Authors: Jad Kabbara
Jackie CK Cheung
Venue: EMNLP 2023 Findings (2023)
Keywords: Presupposition
natural language inference
discourse
pragmatics

Abstract: We explore the connection between presupposition, discourse and sarcasm and propose to leverage that connection in a transfer learning scenario with the goal of improving the performance of NLI models on cases involving presupposition. We exploit advances in training transformer-based models that show that pre-finetuning—--i.e., finetuning the model on an additional task or dataset before the actual finetuning phase—--can help these models, in some cases, achieve a higher performance on a given downstream task. Building on those advances and that aforementioned connection, we propose pre-finetuning NLI models on carefully chosen tasks in an attempt to improve their performance on NLI cases involving presupposition. We notice that, indeed, pre-finetuning on those tasks leads to performance improvements. Furthermore, we run several diagnostic tests to understand whether these gains are merely a byproduct of additional training data. The results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks plays a role in the performance improvements.

Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions
Jad Kabbara'
Jackie Chi Kit Cheung² McGill University & Mila
Massachusetts Institute of Technology Cambridge, MA, USA jkabbaraêmit edu
Montreal, QC, Canada jcheungêcs mcgill. ca
Abstract
importance in understanding meaning and preva- lence everyday speech, scholars studied how pre- supposition fits into different contexts and aspects of language. Specifically, they investigated how presupposition is intricately related to discourse (Heim, 1992; Kamp and Reyle, 1993) and essen- tial for understanding not only the meaning of an utterance but also essential for the coherence of the semantic relations between sentences forming dis- course (Domaneschi, 2016). Similarly, numerous studies underline how presupposition is linked to sarcasm, dating back to two millennia ago with the work of the Roman rhetorician Quintilian and more recent investigations (Grice, 1975; Camp, 2012).
We explore the connection between presuppo- sition, discourse and sarcasm and propose to leverage that connection in a transfer learning scenario with the goal of improving the per- formance of NLII models cases involving presupposition. We exploit advances in train- ing transformer-based models that show that pre-finetuning- i.e. finetuning the model on an additional task or dataset before the actual finetuning phase
help these models, in some cases, achieve higher performance on a given downstream task. Building on those advances and that aforementioned connection, we propose pre-finedt rreffinetuning NLI models on care- fully chosen tasks in an attempt improve their performance on NLI cases involving presuppo- sition. We notice that, indeed, neefinntuning those tasks leads to performance improve- ments. Furthermore, we run several diagnos- tic tests understand whether these gains are merefy byproduct of additional training data. The results show that, while additional training data seems to helping on its own some cases, the choice of the tasks plays role in the performance improvements.
In this work, we set to exploit this connection between presupposition on one hand and discourse and coherence models and sarcasm on the other hand. We choose a popular NLU task, natural lan- guage inference (NLI) (Dagan et al., 2005; Mac- Cartney and Manning, 2008; Bowman al., 2015) as a testbed for exploring this connection and ask the following question: Could we leverage learning tasks that center around discourse coherence and sarcasm to improve the ability of NLI models at making inferences involving presupposition?
1 Introduction
To this end, we propose exploring how such learning tasks can be leveraged in a transfer learn- ing scenario that could benefit the training of NLI models for the final goal of improving their perfor- mances on NLI cases involving presupposition. To do so, we utilize prefinetuning, a training strategy that has received attention in the context trans- former (Vaswani et al., 2017) models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020) and GPT-3 (Brown et al., 2020). Generally, prefinctuning pretrained model is to finetune it on task/dataset before the actual finetuning stage on the downstream task of inter- est. The effect of pre-finetuning varies depending on different factors, such as whether (and to what extent) a pre-finetuning task is relevant to the fine-
In linguistics and the philosophy of language, pre- suppositions are beliefs that are mutually shared between discourse participants. They are assump- tions that are taken for granted and that play a crucial role in understanding the proper meaning being conveyed in a certain context. For example. in the statement "Roger Federer won the match". the fact that Roger Federer played match (which he ended up winning) is not explicitly stated and as- sumed, hence, we say is presupposed. The study presupposition could be traced to more than a century ago with Frege's 's work (1892). Given its
Most of the work was completed during Jad's PhD McGill. "Canada CIFAR Al Chair.

---

tuning task. Pre-finetuning has been explored in the context of question answering (Tafjord al., 2019), named entities (Shwartz et al., 2020) and broader multi-task learning scenarios (Gururangan et al., 2020; Aghajanyan et al., 2021).
performance improvement. Our contribution is three-fold. First, we draw this connection between presupposition, discourse and sarcasm, showing its effects in neural models which could be beneficial for future investigations in the computational pragmatics area. Second, we present a case study pointing promising direction train transformer-basse models to bet- ter handle hard NLI cases involving pragmatic phe- nomena. Third, through an extensive array of exper- iments and diagnostic tests, we show that the type of learning tasks used in pre-finetuning, namely discourse- based and pragmatics- based tasks, could play a crucial role in improving the performance of NLI models on cases involving presupposition and show that the performance improvements are not solely due to additional training data, but rather influenced by the learning signal coming from the pre-finetuning task.
We believe this to be an interesting study for two reasons. First, previous work (Williams et al., 2018; Jeretic et al., 2020) suggested that MNLI (Williams et al., 2018), a popular NLI benchmark, includes some instances of presupposition triggers, thus raising the question of whether transformer- based models trained on MNLI can generalize to other pragmatic inferences. However, previous work (Jeretic et al., 2020; Kabbara and Cheung, 2022) found that BERT does not perform well on many types of presupposition. This underlines how hard it is to design computational models of prag- matic reasoning. In our case, it is thus interesting to explore ways to improve the performance of NLI models on those hard cases of presupposition. Sec- ond, given the aforementioned intricate connection between the notions of presupposition, discourse and sarcasm, it would be interesting to investigate whether such tasks-cxsppp expected to be beneficial in prineiple-would be useful, in practice, for NLI involving presuppositions.
2 Related Work
2.1 Presupposition, Discourse and Sarcasm
The philosophy of language and linguistics liter- ature covered amply the connection between dis- course, sarcasm and presupposition. While earlier work on presupposition focused on the pragmatic meaning at the level of utterances (Strawson, 1950; Austin, 1962; Stalnaker, 1973), in recent decades, linguists started looking at presupposition from the angle of dynamic semantic theories and how this pragmatic phenomenon fits within representa- tions of language structure that model the change in information in the course of a discourse (Heim, 1992; Kamp and Reyle, 1993; Potts, 2015). This (re) from the level of singular utterances to the level of discourse structure was crucial in high- lighting how presuppositions are essential for un- derstanding not only the meaning of an utterance but also essential for the coherence of the semantic relations between sentences forming discourse (Domaneschi. 2016). In this light, several works presented discourse modeling architectures with the goal of presupposition resolution (Kasper et al., 1999; Davis, 2000). Similarly, the connection be- tween sarcasm and presupposition has been inves- tigated as far back as two millennia ago with the Roman rhetorician Quintilian explaining sarcasm as speech that could be understood as the opposite
Beyond the motivating reasons for this specific study, finctuning is still a very popular learning paradigm despite the recent rise of other paradigms like prompt-based methods (Brown et al., 2020) and recent works continue to investigate ways to improve and apply this paradigm (Gira al., 2022: Zhang al., 2023; Tinn al., 2023). Accordingly, we see our effort as complementing the body of work attempting to explore the efficacy and appli- cability of this learning paradigm.
Our investigation pursues these questions in the context of three pre-finetuning tasks: discourse relation classification, coherence modeling and sar- casm detection. For each of the tasks, we run ex-
periments on two different datasets. We find that,
indeed, pre-finetuning on the selected tasks leads to an improvement in the performance of these mod- els on NLI cases involving presupposition given in the ImpPres dataset (Jeretic et al., 2020). Fur- thermore,
several diagnostic tests to under- stand whether these performance improvements are merely byproduct of additional training data. The results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks seems to be playing a role in the
of what is actually being said. In his authoritative work Logic and Conversation (1975), Grice argues that a speaker speaking sarcastically exploits a pre-

---

supposition: i.e., a mutually shared assumption. that they could not have plausibly meant what they said. On the other hand, Camp (2012) holds the view that sareasm does not need to involve a maxim violation (which is usually the case in the Gricean view) and argues that sarcasm is speech that presup- poses a normative scale where the speaker pretends to evoke commitment to this scale but communi- cates an inversion of that evoked scale. According Camp, the speaker pretends to mean the inverse of what being expressed or presupposes someone else to have meant that meaning.
ing the different types of presupposition shown in Table 1) contains 1900 samples.
2.3 Pre-finetuning Transformer-basedd Learning Models
In (Tafjord et al., 2019), the authors found that an additional pre-fineunnin stage on a different mul- tiple choice question (MCQ) dataset improves the model's generalization to another MCQ dataset, suggesting that the pre-finetuning stage seems to have helped the model Iearn some representa- tions about MCQs in general. In (Shwartz et al., 2020) which investigates the robustness of lan- guage models name swaps, it was found that a pre- ure-tinetnning stage on a different dataset (in some cases pertaining to a different NLP task) led to small performance improvements on the main task of interest (Winogrande (Sakaguchi et al., 2020) or SQUAD (Rajpurkar et al., 2018)) but seemed to increase the model's robustness with respect name swaps. Gururangan al. (2020) consider different domains and classification tasks and show that an intermediate stage (between pre-training and finetuning) of further pre-training on a relevant task leads to a performance boost for their model (RoBERTa). Finally, Aghajanyan et al. (2021) present an extensive study on the benefits of pre- finetuning with 50 different tasks. They show that pre-finetuning consistently improves performance for pre-trained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on wide range of tasks (sentence prediction, commonsense reason- ing, machine reading comprehension, etc.), while also significantly improving sample efficiency dur- ing fine-tuning. Their experiments show that a small number of pre-finetuning tasks does not al- ways help but that for 15+ tasks (up to 50), the improvements become consistent and linear in the number of tasks. Given that pre-finetuning not always helpful (Aghajanyan et al., 2021), one con- tribution of this work is to highlight that a carefully chosen task that exploits the connection between presupposition, discourse and sarcasm could im- prove the performance of NLI models on NLI cases involving presupposition. Other works, c.g., (Pruk- sachatkun et al., 2020; Poth al., 2021), while not specifically focused on the notion of pre-finetuning, also showed that further (pre-)training on a care- fully chosen can also improve performance on downstream tasks.
2.2 Natural Language Inference
Early work on natural language inference (NL.) such as (Dagan et al., 2005; Manning, 2006; Mac- Cartney and Manning, 2008) framed the task as one concerned with learning the "directional re- lation" between two statements and h such that would entail h if, typically, a human reading 1 would infer that h is most likely true (Dagan et al., 2005). Contemporary work on NLI (Bowman et al.. 2015; Williams al., 2018) formulates the task as learning the relationship between a premise and a hypothesis such that the relationship is one of entailment, contradiction or neither (in which the two statements are considered to be neutral with respect to each other). Many NLI datasets have been introduced including SNLI (Bowman al., 2015), MNLI (Williams al., 2018), MPE (Lai al.. 2017), XNLI (Conneau et al., 2018), Sci- Tail (Poliak et al., 2018), JOCI (Zhang et al., 2017) and others. In our work, we investigate the ca- pabilities pre- trained transformer-basee models finetuned on MNLI (Williams et al., 2018) and their performance on pragmatic inferences in the ImpPres dataset (Jeretic et al., 2020). MNLI is crowdsourced dataset of 433k sentence pairs that follows the structure of the original SNLI dataset (Bowman al., 2015) but is richer as it covers a range of genres spoken and written text and tailored towards a cross-genre generalization evalu- ation. The ImpPres dataset is a collection of 25.5k semi-automaticalll generated sentences also struc- tured following SNLI/MNLI with the goal of eval- uating how well NLI-trained models recognize sev- eral classes of presupposition and implicature. In this work, we focus on the presupposition part of ImpPres and show in Table an example from each sub-dataset to make the later discussions clearer. By construction, each of the 9 sub-datasets (target-

---

Type
Premise
Hypothesis
A
All six roses that bloomed died. Both flowers that bloomed died. Rene might have hidden. It might be Becky who rescarched Jesus. is Sandra who disliked Veronica. Susan only writes.
Exactly six roses bloomed.
Both
Exactly two flowers bloomed.
Change State
Rene hid.
Cleft Existence
Someone rescarched Jesus,
Cleft Uniqueness
Exactly one person disliked Veronica. Susan writes.
Only
Possessed Definites
Alice's light did vanish.
Alice has light.
Question
Did Bill wonder when Omar hunted?
Omar hunted.
Table 1: Examples showing the different presupposition types in the Impprre dataset.
3 Proposed Method
3.1.2 Closed-Domaie Coherence Modeling We focus here on another discourse-based task: Co- herence Modeling (CM), the task concerned with classifying passage as being coherent or incoher- ent. The most popular task used to test a coherence model in NLP is sentence ordering', for example, to distinguish between a coherently ordered list of sentences and random permutation thereof. In the closed-domain setup (Tien Nguyen and Joty, 2017), training and testing are done on the same do- main. Similar to previous work (Tien Nguyen and Joty, 2017; Xu et al., 2019), we use use the WSJ portion of the PTB dataset (Marcus et al., 1993). We also use the Wiki-A dataset (Xu et al., 2019) (which uses domain/topic "Person" and splits all the different categories therein randomly into training/testing parts).
The learning framework consists of using a pre- trained transformer-based model to be finetured on the MNLI dataset. This last stage is preceded by pre-finetuning stage in which the model is trained another task different than NLI. The model is finally evaluated on the ImpPres presupposition dataset whose goal is test NLI models' ability to recognize different classes of presupposition. Given that the final evaluation concerned with the model's ability to make pragmatic inferences, we believe an interesting case study would be to use pre-finetuning tasks that deal with aspects of dis- course and pragmatics. That way, we could attempt understand whether relevant knowledge learned from the pre-finetuning stage is passed down the
latter stages.
3.1.3 Sarcasm Detection
In the last task, we focus on task whose successful execution requires understanding of the under- lying pragmatics conveyed in the text conversa- tion. The task is sarcasm detection (SD) in which model is to classify a text as being sarcastic or not. For this task, we use the following two commonly used datasets: The Irony and Sarcasm dataset (Fi- latova, 2012) which is a collection of Amazon re- views annotated as being sarcastic/ironic not, and the News Headlines Dataset (Misra and Arora, 2019) which is dataset consisting of headlines of articles from the Huffington Post (labeled as non- sarcastic) and from the satirical online media outlet. The Onion (labeled as sarcastic).
3.1 Pre-finetuning Tasks and Datasets
3.1.1 Discourse Relation Classification
Discourse relation classification (DRC) is a com- monly used task for evaluating the understanding of discourse relations. The task focuses on character- izing the relation between two adjacent text spans (either clauses or sentences) which could be related by an explicit or implicit relation. In our work, we focus on the top-level (L1) classification sce- nario (according to the PDTB 2.0 structure (Prasad et al., 2008)) in which discourse relation takes one of 4 labels: Comparison, Contingency, Expan- sion and Temporal. We use two datasets for this task: PDTB3.0 (Prasad al., 2019) and the TED Multilingual Discourse Bank (TED-MDB) (Zeyrek al., 2019) which is a PDTB-style multilingual dataset consisting of TED-talks that are annotated at the discourse level in 6 languages. We use in our work the English portion of the dataset only.
*We use this task as is still the most popular for the evaluation of coherence models. However, we note that some recent work, e.g., (O' Connor and Andreas, 2021; Laban et al., 2021; Jeon and Strube, 2022) has pointed out the limitations of this task in effectively evaluating coherence models (e.g.. simplicity, reliance on local word co-occurrences but not word order). And so. for an investigation that is solely focused on coherence modeling, it is advised to also use other tasks for evaluation purposes. See the cited work for suggestions.

---

4 Experiments
Depending on the pre-finetuning task, the base BERT model followed by a corresponding lin- ear layer and softmax for classification (2-way or 4- way classification depending on the task) or none in the case of the PRETR AIN experiment (where the data is used to further pre- pre-train the model and not to predict labels). For the cussifiiaaioon-basee pre- finetuning stages and the MNLI finetuning stage (common all experiments), the model is trained to minimize the standard cross-entropy cost with Adam (Kingma and Ba, 2015) as the optimizer. For the PRETRAIN experiments, in the pre-finetuning stage, the model is (further pre-)trained using a masked language modeling objective. For the MNLI finetuning stage, similar to previous work using BERT for NLI, we concatenate the premise and hypothesis separated by the [SEP] token, with the special [CLS] token preceding them.
In this work, we attempt to answer whether pre- finetuning NLI model on learning tasks that deal with aspects of discourse and pragmatics would help that model, when finetuned on MNLI, per- form better on pragmatic NLI cases (the ImpPres dataset). To do so, we carry out three different experiments.
1. The main setup (MAIN): We pre-finetune the model on each dataset of the three learning tasks (6 in total Section 3). Following fine- tuning on MNLI, the model is evaluated on the ImpPres dataset.
2. Randomizing the labels (RANDOM): To un- derstand whether any performance improve- ments are due to extra training data or some actual learning signal coming from the pre- finetuning stage, we randomly shuffle the la- bels (using a uniform distribution) each of the datasets used in the pre-finetuning stage. The rationale behind this experiment is to see whether, given corrupted data, the perfor- mance would drop. This was can be thought of as "sanity check".
Following the recommended ranges for fine- tuning hyperparameters in the BERT paper (De- vlin et al., 2019), our preliminary experiments showed that, for finetuning BERT only (i.e. no pre-finetuning), the optimal performance on the dev set is reached for batch size of 8, learning rate of 2e-5 and weight decay of 0.01. Our model achieves a dev set accuracy of 85. 14% (comparable to that reported in (Jeretic et al., 2020) and (De- vlin et al., 2019). We fix those hyperparameters for the finetuning stage across all experiments and vary the learning rate and weight decay for the pre-finetuning stage across the following ranges re- spectively: (1e-5, 2e-5, 3e-5, 4e-5) and (0.1, 0.01, 0.001 All other parameters are kept as default. All models are implemented in Py PyTorch (Paszke et al., 2019). All input data is tokenized by Hug- gingFace's 's BERT tokenizer (bert-lagg-unncased). All experiments were run on a Quadro RTX 6000 GPU.
3. Using the datasets for further pre-training (PRETRAIN): Here, we use the input sam- ples from each dataset to further pre-train the BERT model using a Masked Language Mod- cling objective. This is to further understand whether extra pre- pre-trainin trainininng would lead to simi- lar changes in the performance (thus possibly suggesting that perfomance gain is due to ex- tra training data and not to the learning tasks used in the pre-finetuning stage).
Finally, we also run an experiment (MULTI) where we pre-finetune the model on all three tasks, one after the other. Due to limited computational re- sources, do not run all possible combinations of datasets. We chose one order at random be pre- sented proof of concept. The chosen sequence of tasks/datasets is: DRC/PDTB then CM/WSJ then SD/Reviews.
Results
Table 2 shows the accuracy results of the three experiments MAIN, RANDOM and PRETRAIN. MAIN: In the first sub-table, we notice that the addition of pre-finetuning stage leads an im- provement accuracy in the vast majority of cases across pre- mreffintunnng tasks/datasets through the dif- ferent sub-datasets of ImpPres-at least 6 out of the 9 sub-datasets of ImpPres for all tasks/datasets and
4.1 Implementation details
The main building block of our experiments is HuggingFace's 's bert-large-uncased implementation (Wolf et al., 2019) of BERT that was trained on lower-cased English text.
*We include in the appendix the exact learning rate and weight decay found for optimal performance on dev set for the pre-finetuning stage for each of the experiments.

---

Discourse Relation
Coherence Modeling
Sarcasm Detection
MAIN
Classification
Vanilla
PDTB
TED-MDB
WSJ
Wiki
Reviews
Hcadlines
Possessed definites existence
70.61
71.85
71.78
72.06
71.27
71.32
71.48
Question
66.42
65.23
64.90
67.33
67.33
65.18
63.34
Cleft Existence
62.97
62.13
61.97
61.55
65.44
61.66
60.24
Only
62.24
62.24
61.26
60.56
62.76
62.87
60.03
All n
43.49
49.21
46.69
44.85
44.59
45,80
46.01
Both
32.62
34.30
33.91
33.25
24.32
27.78
38.66
Change of state
30.43
31.93
33.63
32.62
34.19
31.57
32.62
Possessed definites uniqueness
23,35
33.04
25.33
35.71
30.36
31.57
33,98
Cleft uniqueness
11.10
14.60
16.51
20.90
16.91
13.24
13.92
Average
44.80
47.17
46.22
47.42
46.35
45.67
46.70
Discourse Relation Classification
Coherence Modeling
Sarcasm Detection
RANDOM
Vanilla
PDTB
TED-MDB
WSJ
Wiki
Reviews
Headlines
Possessed definites existence
70.61
61.71
68.33
68.96
66.86
69.17
69.07
Question
66.42
62.71
63.92
63.81
63.66
64.97
65.39
Cleft Existence
62.97
60.56
60.77
60.61
60.14
60.87
60.82
Only
62.24
61.03
55.46
60.29
56.51
60.98
60.98
All
43.49
44.91
42.12
38.76
43,80
41.91
43.80
Both
32.62
25.16
22.48
22.95
27.63
24.63
26.58
Change state
30.43
26.21
27.21
31.25
28.36
31.36
29.57
Possessed definites uniqueness
23.35
24.74
34.09
23.21
18.59
25.95
27.21
Cleft uniqueness
11.10
12.39
14.71
14.02
10.77
12.18
12.50
Average
44.80
42.16
43.23
42.65
41.81
43.56
43.99
Discourse Relation Classification
Coherence Modeling
Sarcasm Detection
PRETRAIN
Vanilla
PDTB
TED-MDB
WSJ
Wiki
Reviews
Headlines
Possessed definites existence
70.61
69.36
66.6
72.87
71.94
65,96
70.16
Question
66.42
64.48
65.97
65.46
65.63
62.91
64.64
Cleft Existence
62.97
62.76
58.77
61.33
61.92
62.07
63.33
Only
62.24
63.39
62.87
60.49
59.72
57.39
59.70
All n
43,49
43.80
40.91
40.04
42.02
38.41
42.56
Both
32.62
29.46
30.93
30.26
24.68
20.48
27.37
Change state
30.43
31.88
32.35
31.02
30.57
29.84
28.84
Possessed definites uniqueness
23.35
20.41
11.45
24.79
35,59
11.04
39.14
Cleft uniqueness
11.10
14.44
13.39
10.17
12.55
9.70
15.27
Average
44.80
44.44
42.58
44.04
44.96
39.76
45.67
Table 2: Results showing accuracy performance on the various sub-datasets of ImpPres for the three experiments: MAIN, RANDOM and PRETRAIN. Vanilla refers to no pre-fince pre-finetuning. For each column, we emphasize in bold the cases where the accuracy is higher than the corresponding in the Vanilla case. As reminder, all sub-datasets have the same number.

---

Vanilla
Multi
performance. Comparing these results to those of the MAIN experiment, we observe that using si- multaneous tasks not necessarily beneficial as the
Possessed definites existence
70.61
71.36
Question
66.42
67.58
average accuracy performance in this case is lower than most scenarios seen in the MAIN experiment.
Cleft Existence
62.97
65.34
Only
62.24
63.15
All n
43.49
42.03
5.1
Discussion
Both
32.62
25.11
Change of state
30.43
35.77
While the margin of improvement is mostly small between a given case (pre-finetuning task/dataset x ImpPres sub-dataset) and the corresponding vamilla case, we note this margin is comparable to what is reported in related studies using pre- pre-finetuning (e.g., (Aghajanyan et al., 2021; Shwartz al., 2020)). Moreover, we notice that for 4 of the 9 sub-datasets (namely, the All n, Change of state, Possessed definites uniqueness and Cleft unique- ness datasets), there is full consistency in terms of performance improvement across all pre-finetuning tasks and corresponding datasets (that is, for these 4 sub-datasets, pre- rreffinetuned models outperform the vanilla model in all tasks/corresponding datasets).
Possessed definites uniqueness
23.35
27.38
Cleft uniqueness
11.10
14.83
44.80
45.84
Table 3: Accuracy results for the MULTI experiment.
going up to 8/9 sub-datasets in the case of the Co- herence Modeling task with the Wiki dataset. More- over, in all 6 scenarios firetiint task/dataset), we notice that there is an improvement in the aver- age accuracy performance compared to the average performance in the Vanilla case. This suggests that pre-finetuning stage using suitable tasks/datasets could help the performance MNL.-trained mod- els on pragmatic NLJ cases.
The results from the MAIN experiment on their own are quite interesting because previous work (Aghajanyan et al., 2021) I) that ran an extensive em- pirical study on 50 different rre-finetuning tasks showed that using small number of pre-finetuning tasks does not necessarily lead to a performance gain on downstream NLI-based tasks and, in fact, could sometimes hurt the performance. The results presented here suggest that a single suitable pre- finetuning task could indeed help MNLI-trained models perform better on pragmatic NLI cases.
RANDOM: To further validate that hypothesis, we notice in the RANDOM experiment (sccond sub- table) that when randomize the labels the aver- age performance drops in all 6 scenarios compared the Vanilla scenario. Moreover, the performance drops the vast majority of cases (pre-finetuning tusk/dataset x ImpPres sub-dataset).
PRETRAIN: On the other hand, when use the training data from the pre-finetuning stage for fur- ther pre-training (the PRETRAIN experiment) in- stead of the corresponding learning tasks, we no- tice that the average performance drops in of the 6 scenarios (iretfinet task/dataset) along with a drop in most individual cases (pre-finctuning task/dataset x ImpPres sub-dataset) This suggests that, in certain cases, the use of extra pre-training data could help in boosting the performance. How- ever, the performance gains due to further pre- training is far from consistent compared what we saw as a result of using a pre-finetuning stage (i.c. the MAIN experiment).
Beyond accuracy improvements, what is even more interesting is what the two diagnostic tests RANDOM and PRETRAIN-Combined with the accu- racy improvements seen in the MAIN experiment- seem to be suggesting. The suggestion is that the choice of the learning task in the me-finetuning stage is playing role in the performance improve- ments and that those improvements are in most cases not result of simply extra training data. Moreover, given that these pre-finetuning tasks are heavily based on aspects of discourse and prag- matics, the nature of these tasks is relevant to the pragmatic nature of the NLI cases that make up the ImpPres dataset. We thus believe that the perfor- mance improvement is due in large part to cross- task knowledge transfer from the pre-finetuning stage the later stages: This because the learn- ing model was exposed to little focus on notions of pragmatics when it was (originally) only finetuned
MULTI: Table 3 shows the accuracy results ob- tained when using three simultaneous tasks in the pre-finetuning stage. We notice that using a pre- finetuning stage with 3 tasks leads an increase in the accuracy performance in 7 out of 9 sub-datasets as well as an improvement in the average accuracy

---

Vanilla
POS
NER
PDTB
Wiki
Reviews
No-P
W/P
No-P
W/P
No-P
W/P
Poss. def. existence
70.61
70.17
58,93
neutral
0
55
0
58
0
52
Question
66.42
61.34
63.87
entailment
46
Cleft Existence
62.97
60.77
64.71
contradiction
10
0
2 0 6
0
Only
62.24
59.19
58.51
All n
43,49
39.55
32.77
Table 5: Distribution of predictions for the Possessed
Both
32.62
23.48
27.84
Definites Existence sub-dataset for the samples that were incorrectly predicted by the Vanilla model (no pre-finetuning- No-P) but correctly predicted by the pre- pre-finetuned model (W/P). The columm shows the distribution of predictions that were incorrectly made by the Vanilla model (No-P) and how their distribution became when the predictions (on the same samples) correctly made by the pre-finetuned model (W/P).
Change of state
30.43
31.14
32.04
Poss. def. uniqueness
23.35
22.15
18.80
Cleft uniqueness
11.10
15.70
18.99
44.80
42.61
41.83
Table Accuracy results for the experiments involving pre-finetuning POS tagging and NER.
an improvement in the performance of transfomer models NLI cases involving presupposition.
on MNLI and was now being made richer through the aforementioned cross-task knowledge transfer. Regarding the MULTI experiment, the fact that the performance of the model pre-finetuned on simultaneous tasks is comparable or lower in some cases is in line with previous work (Aghajanyan et al., 2021) that found that performance improve- ments become linear with the number of tasks only starting from 15 pre-finetuning tasks. Also, as already mentioned, due to computational resources, the hyperparameters (learning rate, weight decay) were fixed for all 3 pre-finetuning tasks and not optimized for each task on its own which could ve been sub-optimal thus affecting the performance.
Sample Analysis
To understand better the effect of the pre- nreffinetuning stage, we analyze the outputs of the model with and without pre- pre-finetuning. Specifically, we look at the models that are pre-finetuned on PDTB (Discourse Relation Classification), Wiki (Coherence Model- ing) and the Reviews dataset (Sarcasm Detection). As a case study, we choose the Possessed Definites Existence sub-dataset of ImpPres as the models performance is the highest on this sub-dataset. We investigate the samples that were incorrectly pre- dicted by the Vanilla model (no pre- pre-finctunin)) but correctly predicted by the pre-finetuned model for each of these 3 cases. In all 3 cases, there were less than 60 such samples. Table shows the results.
5.2 Pre-finetuning on other tasks
One question that might arise is whether other pre- finetuming tasks that are not directly related to no- tions of discourse or pragmatics would help in a similar fashion to what we've seen in the main experiments (Section 4). To further explore this, we carry out two experiments with different pre- finetuning tasks: POS tagging and Named Entity Recognition (NER). For the first experiment, we use WSI-PTB (Marcus et al., 1993) with the stan- dard section splits. For the second experiment, we use the English portion of the CoNLL-2003 NER dataset (Tjong Kim Sang and De Meulder, 2003). We follow the same training setup as in Section 4.1. Results in Table 4 show drop in overall average performance for both experiments along with drop in performance for the vast majority of ImpPres sub-datasets, further strengthening the hy- pothesis that pre-finetunin frnettnttnnng on tasks specifically centering on discourse and pragmatics leads to
In all 3 pre- pre-finctuning tasks, we notice that, in the vast majority of samples (>95%), the correct label is neutral. While the ImpPres dataset is struc- tured such that the neutral cases are generally easier than the other cases (e.g. the contradiction cases generally the hardest), the model still got these cases wrong, instead predicting them as entailment in most cases and contradiction in others. How- ever, with pre-finetuning the models correctly pre- dicted those cases as neutral instead. For exam- ple, in cases predicted incorrectly as contradiction, we notice the presence of negation in the premise. The model might have been using this (incorrectly) as indicator for contradiction. One example is the following: The premise is "That actor's poli- tics didn't upset Nancy", the hypothesis is "Ken- neth has politics." 'The Vanilla model predicted this sample to be one of contradiction whereas the pre- finetuned model predicted it correctly as neutral.

---

7 Conclusion
sentences often describe highly unlikely scenarios or might include combinations of lexical items that make the sentence sound unnatural).
Pre-finetuning transformer- based models has been explored in recent literature as a means to im- prove the performance of these models on various tasks. In our work, we investigate the use of a pre-finetuning step in attempt improve the per- formance of MNL.Itrained models on pragmatic NLI cases. Particularly, we examine three pre- finetuning tasks: discourse relation classification, coherence modeling and sarcasm detection. We fo- cus on these tasks because they are discourse-based or require some notion of pragmatic understanding of the text. We notice that, indeed, pre- reefinntunnng on the selected tasks leads to an improvement in the performance of these models on NLI cases involv- ing presupposition. Furthermore, we run several diagnostic tests to understand whether these per- formance improvements are merely a byproduct of additional training data. The results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks seems to be playing role in the performance im- provement. In future work, we intend investigate other pre-finetuning tasks for the final goal of eval- uating NLI models on NLI cases involving other types of pragmatic phenomena (e.g., implicature). Moreover, we are interested in exploring different learning strategies for making transformer-basee models better at pragmatic NLI cases.
A Acknowledgements
The authors would like to thank the reviewers for their valuable comments. Jad Kabbara was sup- ported by IVADO PhD fellowship. This work is partially supported by the Canada CIFAR AI Chair Program held by Jackie Chi Kit Cheung at Mila.
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer. and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
John Langshaw Austin. 1962. How to do things with words. Clarendon Press.
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. A large amno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
8 Limitations
Evidently, one limitation of this work that results are shown for the English language only. In princi- ple, we expect that cross-task knowledge transfer from tasks involving notions of discourse and prag- matics to help in the performance of learned models on downstream tasks that also involve notions of pragmatics (as is the case in this work). However, what other languages this generalizes to (and to what extent) especcally languages that are morpho- logically richer than reglishs a question for further exploration. Another limitation is that the ImpPres dataset which is the focus of evaluation in our work is a dataset of autimattomat gen- erated sentences. The generation process involves pre-specified templates that are to be filled with constituents sampled from limited vocabulary (3000 lexical items). Thus large-scale quality check process would be required to validate the quality of the entirety of the dataset (c.g., is men- tioned in (Jeretic et al., 2020) that the generated
Elisabeth Camp. 2012. Sarcasm, pretense, and the semantics/pragmatics distinction. Noñs, 46(4):587- 634. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Proceedings of

---

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entail- ment challenge. In Proceedings of the First In- ternational Conference on Machine Learning Chal- lengex: Evaluating Predictive Uncertainty Visual Ob- ject Classification, and Recognizing Textual Entail- ment, MLCWW05, page 177-190, Berlin, Heidelberg. Springer-Verte
Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020, Are natural language infer- ence models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 8690-8705, Online, Association for Computational Linguisties.
Jad Kabbara and Jackie Chi Kit Cheung. 2022. Inves- tigating the performance of transformer-bast based NLI models on presuppositional inferences. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 779-785, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Paul C Davis. 2000. Presupposition resolution with discourse information structures.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Hans Kamp and U Reyle. 1993. From discourse to logic: introduction to modeltheoretic semantics of natural language, formal logic and discourse representation theory. Part 1. Springer Science & Business Media.
Filippo Domaneschi. 2016. Introduction: Presupposi- tions philosophy, linguistics and psychology. Topoi, 35(1):5-8.
Robert T. Kasper, Paul C. Davis, and Craige Roberts. 1999. An integrated approach to reference and presupposition resolution. In The Relation of Dis- course/Dialogue Structure and Reference.
Elena Filatova. 2012. Irony and sarcasm: Corpus gener- ation and analysis using crowdsourcing. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation (LREC' 12), pages 392 -98, Istanbul, Turkey, European Language Re- sources Association (FLRA).
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceeding of the 2015 International Conference on Learning Representation (ICLR 2015), San Diego, California.
Philippe Laban, Luke Dai, Lucas Bandarkar, and Marti A. Hearst. 2021. Can transformer models mea- sure coherence in text: Re-thinking the shuffle test. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Lan- guage Processing (Volume 2: Short Papers), pages 1058- 1064, Online. Association for Computational Linguisties.
Gottlob Frege 1892. Über Sinn und Bedeutung. Zeitschrift für Philosophic und philosophische Kritik, 100:25-50.
Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality. Diversity and Inclusion, pages 59-69, Dublin, Ireland. Association for Computational I Linguistics.
Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Con- ference on Natural Language Processing (Volume I: Long Pupers). pages 100-109, Taipei. Taiwan. Asian Federation of Natural Language Processing.
Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
Suchin Gururangan, Ana Marasovie, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't i stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8361 8360, Online. Association for Computational Linguisties.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Trene Heim. 1992. Presupposition projection and the semantics of attitude verbs. Journal of semantics, 933):183-2211
Sungho Jeon and Michael Strube. 2022. Entity-based neural local coherence modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 7787-7805, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint 190iv 1692.

---

Bill MacCartney and Christopher Manning, 2008. Modeling semantic containment and exclusion in nat- ural language inference. In Proceedings of the 22nd International Conference on Computational Linguis- lics (COLING 2008), pages 521-528, Manchester, UK. COLING 2008 Organizing Committee.
pages 2961 Marrakech, Morocco. European Language Resources Association (ELRA).
Rashmi Prasad, Bonnie Wehher, Alan Lee, and Aravind Joshi. 2019. Penn Discourse TreeBank Version 3.0. LDC2019105.
Christopher Manning. 2006. Local textual inference: It's hard to circumscribe but you know it when you see it and NLP needs it.
Yada Pruksachatkun, Jason Phang. Haokun Liu. Phu Mon Huut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020, Intermediate- -task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 5231-5247, Online. Association for Computational Linguistics.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics. 19(2):313-330
Rishabh Misra and Prahal Arora. 2019. Sarcasm de- tection using hybrid neural network. arXiv preprint (00::7908.07411
Pranav Rajpurkar, Robin Jia, and Percy Liang, 2018. Know what you don't know: Unanswerable ques- tions for SQUAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
Joe O' Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 851-864, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga- vatula, and Yejin Choi. 2020. WinoGrande: An ad- versarial Winograd Schema Challenge at scale. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pages 8732-8740.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, perfooerformecce deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 4024-8035. Curran Associates, Inc.
Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. "You are grounded!": Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 1861,
Online. Association for Computational Linguistics.
Robert Stalnaker. 1973, Presuppositions. Journal of
philosophical logic, 2(4):447-457.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In Proceedings of the Seventh Joint Confer- ence on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Peter F. Strawson. 1950.
On referring.
Mind,
59(235):320-344 344.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNLP) pages 5941-5944 Hong Kong. China. Association for Computational Linguistics.
Clifton Poth. Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Dat Tien Nguyen and Shafiq Joty. 2017. A neural local coherence model. In Proceedings of the 55th Anmual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 1320-1330, Vancouver, Canada, Association for Computational Linguistics.
Christopher Potts. 2015. Presupposition and implica- ture. The handbook of contemporary semantic theory.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008 The Penn Discourse TreeBank 2.0. Proceedings of the Sixth International Conference on Language Resources and Evaluation 08).
Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xi- aodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2023. Fine-tuning large neural lan- guage models for biomedical natural language pro- cessing. Patterns, 4(4).

---

Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: I sanguge--indeentte
named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, pages 5998-6008. Cur- ran Associates, Inc.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long Papers), pages 11 112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- iez. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao. Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Hug- gingface's 's transformers: State-of-the-ae natural lan- guage processing. ArXiv, abs/1910.03771.
Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019, A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 678-687, Florence, Italy. Association for Computational Linguistics.
Deniz Zeyrek, Amália Mendes, Yulia Grishina, Mu- rathan Kurfalı, Samuel Gibbon, Maciej Ogrod- niczuk. 2019. Ted multilingual discourse bank (ted- mdb): parallel corpus annotated in the PDTB style. Language Resources and Evaluation, pages 1-27.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan. Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. I.LaMA-Adapte Efficient fine- tuning language models with zero-init attention. arXiv preprint arXiv:2303.1e. 16199.
Sheng Zhang. Rachel Rudinger, Kevin Duh, and Ben- jamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Com- putational Linguistics, 5:379-395

---

A Appendix
A.1 Additional training details
We present here the learning rate and weight de- cay that were obtained for each model following hyperparameter search explained in Section 4.1.
Discourse Relation
Coherence Modeling
Sarcasm Detection
MAIN
Classification
PDTB
TED-MDB
WSJ
Wiki
Reviews
Headlines 0.01
Weight decay Learning rate
0.1
0.001
0.1
0.1
2e-5
4e-5
2e-5
le-5
4e-5
2e-5
Discourse Relation Classification
Coherence Modeling
Sarcasm Detection
RANDOM
PDTB
TED-MDB
WSJ
Wiki
Reviews
Headlines 0.1
Weight decay
0.001
0.1
0.1
0.1
0.001
Learning rate
le-5
2e-5
2e-5
le-5
2e-5
3e-5
Discourse Relation Classification
Coherence Modeling
Sarcasm Detection
PRETRAIN
PDTB
TED-MDB
WSJ
Wiki
Reviews
Headlines
Weight decay
0.001
0.01
0.001
0.1
0.1
0.01
Learning rate
4e-5
4e-5
3e-5
le-5
le-5
4e-5


Reviews:
[{"review_id": "MooDimgfkw", "reviewer": "Reviewer_PE3F", "summary": "", "questions": "a) While pre-finetuning on tasks related to discourse and pragmatics improves performance on ImpPres, does it also impact the model’s performance on MNLI? It would be helpful to include some analysis to see whether the performance improvement from pre-finetuning also comes at a cost.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer PE3F for their encouraging feedback and comments and for highlighting the many things that they liked about our paper. We thank the reviewer for their comment that the paper is well written and easy to follow and that the experiments are well-motivated and that we present extensive ablation studies. We appreciate the reviewer’s feedback that our work can inspire similar future work on improving transformer-based models’ performance on other hard tasks.\n\nBelow we'd like to respond to the main points: \n\n1. Thank you for bringing up the two references [1] and [2]. While both references offer valuable insights and present a wide array of experiments, our work is different in the sense that is focused on NLI and its intersection with pragmatics and to what extent NLI models exhibit some sort of pragmatic competence (specifically related to presupposition). In both references, none of the intermediate tasks focus on pragmatics so while the references support the general conclusion that a carefully chosen intermediate task can help performance on downstream tasks, we also shared 4 other references in our Related Work section that support the same general conclusion as this is not the main point we are trying to emphasize in the paper. Specifically, since we’re interested in the performance of NLI on presupposition, we are exploring this unique linguistic angle that ties various notions of discourse and pragmatics (via the 3 chosen tasks) to presupposition and, as you rightly point out, show empirically that leveraging this linguistic link can help the performance of NLI models on cases of presupposition. In the camera-ready version, we'd be happy to add the two suggested references [1] and [2] to the Related Work section and highlight their contribution and how our work is unique and different from these two references.\n\n2. Thank you for bringing up the question regarding test performance on MNLI. We did not include this analysis in our paper given the space limitation and because performance on MNLI is not central to the main point of the paper, namely, whether the linguistically motivated tasks can help with model performance on NLI cases involving presupposition. With extra space in the camera-ready version, we’d be happy to include these results and analyze any performance costs on MNLI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) The paper is well written and easy to follow\n\nb) The experiments are well-motivated with extensive ablation studies\n", "reasons_to_reject": "a) There are many previous works (for example, [1] [2] in missing references below) on pre-finetuning with intermediate tasks that are not discussed in this paper.\n\nb) In particular, one of the main claims in the paper that one single carefully chose task can be beneficial in pre-finetuning has already been studied and proven in previous work ([1] [2]), where they also experimented with a large number of different combinations of intermediate tasks and target tasks, making this paper less exciting.\n", "questions_to_authors": "a) While pre-finetuning on tasks related to discourse and pragmatics improves performance on ImpPres, does it also impact the model’s performance on MNLI? It would be helpful to include some analysis to see whether the performance improvement from pre-finetuning also comes at a cost.\n", "paper_topic_and_main_contributions": "This paper presents a study on whether pre-finetuning NLI models on intermediate tasks related to discourse and pragmatics would improve the models’ performance on NLI cases involving presupposition.\n\nThe pre-finetuning datasets are chosen from three tasks, including discourse relation classification, coherence modeling, and sarcasm detection. Experiments on the BERT model show that all pre-finetuning datasets lead to improved average performance on 9 subsets of ImpPres, an NLI datasets focused on presupposition and implicature.\n\nFurther ablation studies show that the improved performance is not merely a byproduct of additional training data, as the choice of the tasks also plays an important role in the performance boost. \n\nMain contributions:\n\na) Empirically established the connection between presupposition, discourse and pragmatics by showing the performance improvement from pre-finetuning.\n\nb) A case study that might inspire similar future work on improving transformer-based models’ performance on other hard tasks.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?. ACL 2020.\n\n[2] Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to Pre-Train on? Efficient Intermediate Task Selection. EMNLP 2021.\n"}}, {"review_id": "V3sr4hhYbq", "reviewer": "Reviewer_Gb86", "summary": "", "questions": "It is unclear to what extent the phenomenon targeted (presupposition) is one that is frequently occurring in practice when using NLI models, and so to what extent improvements on this subproblem are impactful.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Gb86 for their feedback and comments. We appreciate the reviewer's strong excitement towards our paper and appreciate that the reviewer believes that our work focuses on a foundational problem in evaluating generative text models.\n\nThank you for your comments regarding the scope and impact of the work. Below we’d like to respond to these comments.\n\nWhile we focus on presupposition in this work, we believe that our work, first and foremost, highlights an angle on NLI that is under-explored in the literature, specifically NLI involving pragmatics. Moreover, as you rightly point out, we also believe that NLI is a foundational benchmark for evaluating the progress of NLP models on understanding and our work targets a hard problem in the text understanding space, namely pragmatic understanding. Accordingly, we believe our research sheds light on a new and unexplored angle in the NLI space and has the potential to encourage new research on assessing and understanding capabilities of NLP models. This is especially timely with the renewed interest in language models and the recurring questions about the extent to which they exhibit understanding given their impressive performance on many tasks. We also believe that our work can encourage research that explores other kinds of pragmatic phenomena and their intersection with the NLU space and our experiments could inspire future research to showcase more rigor when evaluating NLP models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "NLI is a foundational component in evaluating generative text models and so any significant improvement in this area is very likely to have wide impact.", "reasons_to_reject": "The wealth of experimental data and really narrow scope of the paper risks to make it relevant to a very narrow audience.", "questions_to_authors": "It is unclear to what extent the phenomenon targeted (presupposition) is one that is frequently occurring in practice when using NLI models, and so to what extent improvements on this subproblem are impactful.", "paper_topic_and_main_contributions": "The paper targets a narrow phenomenon (presupposition) in the context of improving natural language inference (NLI) models.\n\nThe premise of the paper is that pre-finetuning models that will be later fine-tuned for NLI improve their performance. Moreover, if targeting a specific subproblem in NLI one may be able to improve model performance on teh target subproblem by pre-finetuning on an adequate task. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be good if the paper was accessible to the non-expert in NLI. A summary of results and their significance to the NLI field itself would be good.\n"}, "extra_text": {}}, {"review_id": "tfaoHI2Zvf", "reviewer": "Reviewer_joCE", "summary": "", "questions": "A) Why not experiment with additional architectures, e.g. RoBERTa which generally outperforms BERT?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer joCE for their encouraging feedback and positive comments and for mentioning the many things they liked about our paper. Specifically, we appreciate that the reviewer found that our paper presents a very good linguistics-focused background and explanation of presupposition, in a way that makes the reader understand the significance of presupposition for humans, as well as a way that underscores the importance of including it in NLP analysis. We thank the reviewer for their view that our paper presents an important finding regarding pre-finetuning on the chosen tasks. Finally, we appreciate the reviewer’s feedback that our analysis  presents an important practice given the increasing difficulty in evaluating model interpretability.\n\nThank you for your comment and suggestions regarding the label randomization experiment. We agree with all points raised, specifically that introducing noise is expected to lower the model performance and in the case where the performance remained stable, this would’ve shed light on the robustness of the model. Our motivation behind this experiment was for it to be more of a sanity check because sometimes models could behave in unexpected ways. We wanted to make sure that the performance would not actually improve because this would’ve necessitated further inquiry. The experiments you suggest are definitely insightful and we can add a comment that these are other tests that could be carried out.\n\nRegarding other architectures like RoBERTa, the main limitation for us is the computational resources. Given the number of datasets and setups and the resulting number of experiments, we had to limit our experiments to BERT.\n\nWe would like to thank the reviewer for the style and presentation improvement suggestions. We’d be happy to update these and incorporate them in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To increase models' understanding of language, it's important to experiment on elements of discourse that humans find easy, but models find difficult. In particular, it's important to investigate how we can make these elements useful to a model. This paper sheds light on one such element of discourse, namely presupposition. This paper shows how models can make use of pre-finetuning signal containing presupposition to solve target tasks that involve presupposition in some way. Additionally, the paper provides a very good linguistics-focused background and explanation of presupposition in the Related Works section, making the reader understand the significance of presupposition for humans, as well as underscoring the importance of including it in NLP analysis.\n\n- The paper makes the important finding that pre-finetuning on tasks that involve presupposition does improve performance on a target task that involves presupposition. This provides an encouraging foundation for future research.\n\n- The authors perform a model output analysis starting from Line 541, which provides a helpful glimpse into the models' exact behavior. I believe this is an important practice, since models are generally becoming less interpretable.", "reasons_to_reject": "- Line 355 (\"Randomizing the labels\"): The authors' intention to test whether performance gains are merely due to extra training data is logical, but to me, it seems like corrupting the data isn't a convincing way to do this. Corrupted data plays the role of noise, which generally tends to be harmful to performance. So one could argue that the low scores observed in the \"Random\" setting in Table 2 were due to label noise. Additionally, even if the performance had remained relatively the same, this would have said more about the model's robustness against noise than about whether or not the mere addition of more data originally helped it perform well. I think a better way to conduct the intended test would be to perform some form of over- and under-sampling on the data, such as duplication, and using that for pre-finetuning. This would ensure that the only thing that's being changed during the test is data quantity, and not label quality. Otherwise, I think a test should have been done to demonstrate that the impact of label noise was minimal on the \"Random\" setting.\n\n- Additionally, it's not stated whether the results in Tables 2, 3, and 4 are from single runs or averages of multiple runs. Therefore it could be the case that the method isn't robust.", "questions_to_authors": "A) Why not experiment with additional architectures, e.g. RoBERTa which generally outperforms BERT?", "paper_topic_and_main_contributions": "This paper investigates the method of pre-finetuning, which involves finetuning a model on an additional task before fine-tuning it on the target task. Specifically, the authors pre-finetune on 3 tasks (2 datasets each): discourse relation classification, coherence modelling, and sarcasm detection, all of which involve the linguistic phenomenon of presupposition. They then investigate whether this pre-finetuning stage is beneficial for the target task, namely the ImPres dataset, which also involves presupposition. \n\nThe paper makes the following contributions:\n\n- Continues research on an element of discourse (presupposition) that is easy for humans, but still challenging for models.\n- Sheds light on the benefits of pre-finetuning, by showing that it leads to performance gains on the target task given related pre-finetuning tasks.\n- Continues research on fine-tuning, which remains an important method despite the current focus on prompting.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- For the sample analysis in Section 6, it would have been helpful to have a small table showcasing some of the the responses of models with and without pre-finetuning.\n\n- In the caption of Table 2, I would state that the ImPres dataset was used as the test set as an extra reminder for the reader. Otherwise, the test set seemed to get lost in the other details of the paper.\n\n- Table 2, row \"Cleft Uniqueness\", column \"Vanilla\": the value is 11.10 for \"Main\", but 11.1 for \"Random\" and \"Pretrain\". For consistency, I'd change those to 11.10."}}]

Author rebuttal:

Meta-review: This work hypothesized that pre-finetuning NLI models with intermediate tasks related to discourse and pragmatics (namely coherence modeling, sarcasm detection, and discourse relation classification) would improve model performance on cases involving presupposition.

The paper provided empirical support for the authors' hypothesis, thus empirically establishing the connection between presupposition and these tasks. The reviewrs agreed that this was a useful result that could benefit future work. Overall, evaluation is solid with extensive ablation studies.

The rebuttal sufficiently addressed all of Reviewer joCE's concerns, particularly regarding the label randomization experiment and the use of the Vanilla setup as an informative baseline.

It's clear from Reviewer PE3F's response to the rebuttal that his/her concerns remained, specifically regarding the contribution of this paper in light of related work. I think the authors provided a reasonable response to PE3F's response and am personally satisified with their response. I view PE3F's concerns more as suggestions for improvements than serious flaws, and hope that the authors can take them into account when revising their paper.
Decision: Accept-Findings