GAPX: Generalized Autoregressive Paraphrase-Identification X
Authors: Yifei Zhou
Renyu Li
Hayden Housen
Ser-Nam Lim
Venue: NeurIPS 2022 Accept (2022)
Keywords: paraphrase identification
distribution shift
autoregressive transformer
out-of-distribution detection

Abstract: Paraphrase Identification is a fundamental task in Natural Language Processing. While much progress has been made in the field, the performance of many state-of- the-art models often suffer from distribution shift during inference time. We verify that a major source of this performance drop comes from biases introduced by negative examples. To overcome these biases, we propose in this paper to train two separate models, one that only utilizes the positive pairs and the other the negative pairs. This enables us the option of deciding how much to utilize the negative model, for which we introduce a perplexity based out-of-distribution metric that we show can effectively and automatically determine how much weight it should be given during inference. We support our findings with strong empirical results.

GAPX: Generalized Autoregressive Paraphrase-ldentifccttoo X
Yifei Zhou Cornell University
Renyu Li Cornell University r16260cornel1. edu
Hayden Housen Cornell University hth33@cornell.com edu
yz639@cornell.ddi
Ser-nam Lim Meta Al gernamlim@fb.c com
Abstract
Paraphrase Identification fundamental task in Natural Language Processing. While much progress has been made in the field, the performance of many state-of- the-art models often suffer from distribution shift during inference time. We verify that major source of this performance drop comes from biases introduced by negative examples. To overcome these biases, we propose in this paper train two separate models, one that only utilizes the positive pairs and the other the negative pairs. This enables us the option deciding how much to utilize the negative model, for which we introduce perplexity based out-of-distributio metric that we show can effectively and automatically determine how much weight should be given during inference. We support our findings with strong empirical results.
Introduction
Paraphrases are sentences phrases that convey the same meaning using different wording, and
fundamental to the understanding languages [7]. Paraphrase Identification well-studied
task of identifying if given pair of sentences has the same meaning [5]
47
56
57
and
has many important downstream applications such as machine translation
61
and
question-answering [5].
QOP
topic
beg-of-word
poor
topics
Emma Townshend is rogresented
The company then changed its company
What stool color change to yollow?
That amher getting annoying
DGA Associates.
was it.
David David Godwin is represented at DGA Associates Fimma Townshend
But changed the and there was nothing to deal with.
out as limle balls?
Why get amber nlents the
Figure I: Negative pairs in different datasets are mined differently in different datasets, and can lead to significant biases during training.
Recently, researchers have observed that neural network architectures trained different datasets could achieve state-of-ther art performances for the task of paraphrase identification [22 16 [50]. While these advances are encouraging for the research community, has however been observed
Our code is publicly available at:
paraphrase_ideeee identification
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that these models can be especially fragile in the face of distribution shift [45]. In other words, when a model trained on a source dataset D is tested on another dataset, D', collected and annotated independently, and with a distribution shift, the classification accuracy drops significantly [62].
This paper presents our findings and observations that negative pairs (i.e., non-paraphrase pairs) in the training set, as opposed to the positive pairs, do not generalize well to out-of-distribun pairs. Intuitively, negative pairs only represent a limited perspective of how the meanings of sentences can be different (and indeed it is practically infeasible to represent every possible perspectíve). We conjecture that negative pairs are so specific the dataset that they adversely encourage the model to learn biased representations. We show this observation in Figure I Quora Question Pair (QQP) eextaacts its negative pairs from similar topics. Paraphrase Adversarials from Word Scrambling (PAWS) Zhang et al. [62] generate negative pairs primarily from word swapping. World Machine Translation Metrics Task 2017 (WMT) g considers negative examples as poor machine translations. We therefore hypothesize that biases introduced by the different ways negative pairs are mined are major causes of the poor generalizability of paraphrase identification models.
test
Based on this observation, we would like to be able to control the reliance on negative pairs for out-of-distrbution prediction. In order achieve this, we propose to explicitly train two separate models for the positive and negative pairs (we will refer to them as the positive and negative models respectively), which will give us the option to choose when to use the negative model. is well known that just training on positive pairs alone can lead to degenerate solution [2] 59 42 [3], e.g., a constant function network would still produce perfect training loss. To prevent this, we propose a novel generative framework where we use an autoregressive transformer [39 49,, specifically BART 30. Given two sentences, we condition the prediction of the next token in the second sentence on the first sentence and the previous tokens. In Bayesian sense, this would mean that the next token predicted has higher probability of being a positive/negative pair to the first sentence for the positive and negative model respectively. This learning strategy has no degenerate solutions even when we are training the positive and negative models separately. We call our proposed approach GAP to stand for Generalized Autoregressive ignores the "interplay" between positive and negative pairs that would otherwise be learned if they utilized in training together. This is especially important when the test pairs are in-distribution. To overcome, we utilize an extra discriminative model, trained with both positive and negative pairs, to capture the interplay. We call this extension GAPX (pronounced "Gaps") to capture the eXtra discriminative model used.
Idettfiitinnnn....ttinn.... One potential pitfall of GAP is that it
For all practical purposes, the weights we placed the positive, negative and/or discriminative model in GAP and GAPX need to be determined automatically during inference. For in-distribution pairs, we desire to use them all, while for out-of-distributiin pairs, we hope to rely on the positive model much more heavily. This obviously leads to question of how to determine whether a given method ensembles the positive
pair is in or out of distribution 14 43 17 20). During testing, model, the negative model, and the discriminative model based on the degree of the similarity of the test pair to the training pairs, and found that this works well for our purpose. We measure this degree of similarity with probability cumulative density function (cdf) in terms of perplexity 255 and show that it is superior to other measures.
In summary, our contributions are as follow:
1. We report new research insights, supported by empirical results, that the negative pairs of dataset could potentially introduce biases that will prevent a paraphrase identification model from generalizing to
pairs.
2. To overcome, we propose novel autoregressive modeling approach to train both a positive and a negative model, and ensemble them automatically during inference. Further, we observe that the interplay between positive and negative pairs are important for in-distribution inference, for which we add a discriminative model. We then introduce a new perplexity based approach to determine whether given pair is out-of-distributton achieve auto ensembling.
3. We support our proposal with strong empirical results. Compared to state-of-the-art trans- formers in duitt--iiittibutton performance, our model achieves average of 1 11.5% and 8.4% improvement in terms of macro Fl and accuracy respectively over 7 different out-
htaps:/ggorraaat..ooocoooofmrsss Dataset-
Dww.aaatteeateeeeee-------e Release-Question-PPe

---

of-distribution scenarios. Our method is especially robust to paraphrase adversarials like PAWS, while keeping comparable performance for in-distribution prediction.
2 Related Works
2.1
Distribution Shift and Debiasing Models in NLP
The issue of dataset bias and model debiasing has been widely studied in a lot of field in NLP such as Natural Language Inference [23 3
and Question Answering [36 2 A). Notable work by [5 [0 15 (88 utilize ensembling to reduce models' reliance on dataset bias. These models share the same paradigm where they break down a given sample z into signal feature 2, and biased feature Tb, in the hope of preventing their model from relying on zb. which has been shown to the limiting factor preventing the model from generalizing to out-of-distribution samples [5]. Here, separate
model first either trained on T6 or on datasets with known biases [TO, 15 22 or acquired from
models known to have limited generalization capability. Then they train their main model with
regularization term that encourages the main model to produce predictions that deviate from that of the "biased model". However, this type of approach has shown limited success [45) in debiasing paraphrase identification models. In contrast, our method is based on our observation that negative pairs limit the generalization of paraphrase identification models.
2.2
Out-of-distribution Detection
Another line of work relevant to this paper is the task of detecting out- tittof--ditribution samples [4]
20 55) Researchers have proposed methods to detect anomaly samples by examining the softmax scores 32 or energy scores [33 produced by discriminative models, while others take
43 17
reconstruction error 38 In this paper, we introduce a novel perplexity based out-of out-of-distribution detection method that we -iitr-iiittiiutionn score is utilized to weigh the contributions from the positive and negative models: the higher the confidence that the sample out-of-distribution, the lesser the negative model's contribution.
more probabilistic approach estimate the probability density [14] 25 66 [
show empirically
work well for our purpose. Specifically, during inference,
2.3 Text Generation Metries
Finally, we would like to note the difference between our work and autoregressive methods that have been explored [58 [66 for evaluating text generation. Our work differs follows: 1) Paraphrase identification seeks to assign a label of paraphrase or not while text generation metrics seeks to assign to measure the similarity of sentences; 2) Current text generation metries either cannot be trained to fit to a specifc distribution [6]] [88] [3] or are limited to the i.i.d. setting 44 41 of the training distribution. In contrast, our method not only significantly improves out-of-distribution performances but is also competitive with state-of-the-art paraphrase identification methods for in-distribution predictions.
3 Methodology
We observe that negative pairs in paraphrase identification constitute the main source of bias. To overcome this, we propose the following training paradigms to learn a significantly less biased paraphrase identification model. We employ autoregressive conditional sentence generators with transformer architecture as the backbone of our model. Specifically, we train a positive and negative model to estimate the distribution of positive and negative pairs in dataset respectively. During testing, the two models are ensembled based on how likely the input pair is out of distribution. This section provides details on method.
3.1 Separation of Dependence on Positive and Negative Pairs
Let be the space all sentences, X (#1, 82) be the random variable representing a sample pair from S, and Y the random variable representing the labels, with Y = 1 indicating that 81 and 82 are paraphrases and otherwise when Y = O. We seek to separate the dependence between the distribution of positive and negative pairs, motivated by the observation of the presence of bias in the negative
3

---

Positive Model
Positive Distriution
Training
Negative Model
Negative
Dlstribution Model
Distribution
Valdation
Figure 2: An overview of the training procedure of our model GAP. GAPX ensembles GAP with another discriminative model.
pairs. To begin, we model the distribution of sentences by splitting the sentence $2 of length n into the autoregressive product of individual words, where w/ denotes the ith word in 82. By applying Bayesian Inference Law, we have:
(9) P(Y = 1/8,, 82) P(Y = [[ P( $1, Y (1:f-1))
(1)
I1= P(w [w2 1) ,1))
Subtracting the logarithm for Y and Y we get:
log(P log(P(Y = 1/81, 82)) - log( P(Y = 0|5i, 82))
(log P(Y = 1(81) - log P(Y = 0|s1)}
log P(w2 (6) |s1, Y w, (1:1-1))
log Y..Y-=.. = =0, ((
(log P(Y 1) - log loglog P(Y =0)
log (ww (6) 181, Y 1, w2 (1i-11)
log P(wz)) 181, Y 0, w/ (1))
(2)
In this way, break the probability inference into terms resulting in Eqn.
(1) (log P(Y
1) log P(Y= = 0)), which should just be a constant; (2) (E log P(w (0)
81, Y = 1, u 1::-1))
which depends only on the distribution of positive pairs; (3) (= log P(w $1 Y 0, w (1:- 1). )).
which depends only on the distribution of negative pairs. We define the score of confidence as follows:
82)1,82) = log ((w) (5) Y 1, 1
(1:1-1)
log P( (4)
(81, Y = O, m0 ())
(3)
Positive Modal
Negative Model
In the above, we are now left with two terms, the first representing the positive model and the second the negative model. If the resulting model can never be removed during inference, which we have observed to be a major source of bias. To avoid this, we propose train the first term and the second term separately, and then subsequently ensemble them based on the degree that given pair is out of distribution. We train each model on top of the pretrained autoregressive transformer described in [30] known BART. Given $1 and $2, we feed into the encoder as the condition, shift to the right by one-token, and feed shifted $2 to the decoder. While the decoder proceeds autoregressively, we record the nert-word probability distribution. We calculate the cross entropy between the next-word probability distribution and the target token in 82 to update the model parameters. Note that here the Bayesian formulation has been similarly raised in some of the previous work like Moore and Lewis [37] but to the best
train the two terms together, the effects of the negative pairs in
of our knowledge, we're the first to propose this Bayesian formulation to control the reliance on different components of the model.

---

1) and 3.2 Ensembling
To combine the positive and negative model, if we know priori D' is in the same distribution as D", we can directly substitute the prediction of the positive and negative model into Eqn. 3 We will refer to this as the In-distribution Predictor (IDP). If we have reason to believe that there significant distribution shift between D and D' (e.g., different sources of corpus and different dataset collection procedure), we observe empirically that we should only utilize the positive model and disregard the negative model due to the bias it introduces. We will refer to this as the Out-of-distribution Predictor (OODP).
3.2.1 Automatic Ensembling
However, most cases, we have little or no knowledge of the testing distribution, which case we need to automatically decide how important the negative model is by detecting how much test pair in the same distribution as the training set. We adopt weighted interpolation between a constant and the negative model in addition to the positive model as follow:
S(s1, s2) log P(s2|s1, Y 1) - (1 X()1, log P(s2|#1, Y 0) 82)1,
(4)
where A(51, is a weight parameter depending on 81 and 82, and is a constant that achieves regularization effect. See Appendix for ablations on how C' can be set. P(s2ls), P(s2|s1, Y = 0) are the same terms in Eqn To automatically assign A(31: 82) for different sentence pairs, measure
out-of-distribution score for (s1, 82) with regard to the training distribution. Specifically, we use the same set of training data, comprising both positive and negative pairs, from D', which we train another autoregressive model, which will refer to the distribution model. The distribution model is trained by feeding an empty string into the encoder and the concatenated 81 and 82 into the decoder, with the training goal of predicting the next token. We measure the perplexity of each sentence pair (81, 82) using the distribution model hased on the following formula, w² being the ith token of the concatenated (81, 82) of length
82) = P(w:lwala-1)
(5)
We then Weibull distribution the perplexity of a held-back of validation data, so that can better model the right-skewed property of the distribution. We derive the exponential parameter a, the shape parameter c, the location parameter loc, and the scale parameter scale. During testing, X(s1, 82) can now be determined as:
X(11, 52) cdf( s2), Weibull(a, loc, scale)).
For the final prediction, we predict the sentence pair to be paraphrase S(s1, 82) 2 and non- paraphrase otherwise. This forms what we referred to earlier as GAP (Generalized Autoregressive Paraphrase-(dentficaaton)
3.2.2 Capturing Interplay Between Positive and Negative Pairs
In practice, training positive and negative model separately disregards the interplay between the positive and negative pairs, which could important when the test pairs are in-distribution. To capture such interplay, we utilize both positive and negative pairs to train discriminative model for sequence classification. Specifically, we first define a thresholding function based on the value of
= 10
A 0.9
+())
else.
We then ensemble the discriminative model and GAP using the value of r(A):
1 r(X(51, (2)))
('1, M(1 = 1[s1, 2
(8)
5

---

where P(Y 1|s1, s2) can be estimated by any discriminative model, and M is sufficiently large constant. Note that this definition essentially the same trusting the discriminative model when we do not have statistical evidence that the pair is out- -it-r---istritttion (p-test 10%) while trusting the GAP model otherwise. For the final prediction, we predict the sentence pair be paraphrase S (2); 2 0 and non-paraphrase otherwise. This defines GAPX (Generalized Autoregressive Paraphrase Identfication X), for which we set M to sufficiently large (> 1000), so that when comparing the model confidence for different pairs, the score of the discriminative model will be prioritized.
4 Experiments
Our experiments are designed to (I) verify that the task of paraphrase identification suffers from biases in the datasets that is the main obstacle to generalization in this field of study, (2) test the accuracy of our perplexity based ditt----iitiitutionn detection method, and (3) test that balancing the utilization of the negative model can help outperform the state-of-ther -h-th---tt in the face of distribution shift, without losing in the mn-distribution scenarios.
4.1 Datasets
We compare method against the other state-of-the-art methods different combinations of the following datasets:
Quora Question Pair (QQP) consists of over 400,000 lines of potential question duplicate pairs. Since the original sampling method returns an imbalanced dataset, the authors attempt to balance with additional negative pairs collected from similar topics make them harder. Note that to scale QQP down to approximately the same size of PAWS and PIT (see below), we take the first 10k training pairs and 2k testing pairs from the train and test split by Wang et al. [52].
World Machine Translation Metries Task 2017 (WMT) g contains total 3793 manual ratings of machine translations from 7 languages to English. Each rating result contains a source sentence in the source language, a reference sentence English (ground truth translation), machine translated sentence in English, and manual rating of the quality of translations. We take the ground-truth reference sentence and the machine translated sentence as the sentence pair. Sentences with higher quality score (> 0) are labeled as paraphrases, while those with lower quality scores (S 0) are labeled otherwise. The resulting paraphase identification dataset is balanced. Note that this dataset is significantly smaller than other datasets, so we only use it as a test set.
54] contains 18762 sentence Paraphrase and Semantic Similarity in Twitter (PIT) [53
pairs automatically extracted from a similar distribution of topics as QQP. Annotators manually assigned integer scores from 0 to
each sentence pair, representing the degree of similarity between the sentence pair. To make it paraphrase identification dataset, we label sentence pairs with low scores (0, 1) as non-paraphrases and those with high scores (4,5) as paraphrases. The original dataset is unbalanced, so we randomly sample a maximum balanced subset of the dataset. The original test set processed in this way shrinks only 350, and is not comparable to the other datasets. Hence, we use the original development data of size 1896 as the test set while keeping original test of size 350 for development. The training set contains 5332 sentence pairs.
Paraphrase Adversarials from Word Scrambling (PAWS) 62 contains 49,401 sentence pairs, each of which is constructed from the same bag-of word to make the evaluation more challenging. Most of the negative pairs are generated by word swapping while positive pairs are supplemented by back translation. This dataset contains paraphrase pairs that are the adversarial counterparts of standard paraphrase datasets such as QQP and PIT.
With these datasets, we perform experiments where different models (Sec. 4.2 are trained on one dataset and evaluated on another in order to observe whether their performance hold the face of distribution shift.

---

4.2 Benchmarks
We benchmark the paraphrase identification task with these methods:
BOW [62] represents two input sentences with bag of words. The bag of words representa-
tion of each input sentence is passed through fully-connected network and cosine similarity between of the final layer is used to compute the classification output.
BiLSTM (28] passes each of the two input sentences through bidirectional LSTM network. The output state of the two sentences are then concatenated together and passed through fully-connected network to get the classification output.
BERT [66 is representative of the state ofe------heer transformer methods for text classification. We finetune the pretrained model "bert-base-uncasedd in standard way. We concatenate the sentence pair separated by [SEP] token and take the [CLS] token as aggregate rep- resentation for the sentence pair. The embedding for the [CLSS token is then fed into
output layer for classification.
BART [30 is the original transformer model that we build on by finetuning the pretrained We concatenate the sentence pair separated by 
 token, feeding it both into the encoder and the decoder. We use the 
 token at the end of the sentence pair for aggregate representation that it can attend to decoder states from the complete input. The embedding for 
 is then fed into an output layer for classification.
model
RoBERTa 34 shares the same transformer architecture with BERT, but uses a more robust pretraining strategy, and as result performs better than BERT in many NLP tasks 34 For our experiments, we employ RoBERTa in the same way as BERT.
IDP (In-distribution Predictor) is model for in-distribution prediction if we know a priori that the testing pairs come from the same distribution as the training pairs. combines the positive and negative models as given in Eqn. 3
OODP Duu-oo-ddistibution Predictor) is our model for cuttof-distribution prediction if we know a priori that the testing and training pairs are not in the same distribution. It only makes use of the positive model. We expect our OODP to have better generalizability than our IDP, because of its reduced reliance on negative examples.
GAP (Generalized Autoregressive Puraphrase-ddentifcation perplexity based out-of-distribution detector to automatically control the reliance on the negative model, using Eqn. 4 This setting is different from IDP and OODP in that do not have priori knowledge of the test distribution.
our method that utilizes the
GAPX (Generalized Autoregressive Paraphrase- Paraphrase-ldenttficaaa Identificatinn X) ensembles GAP with RoBERTa described above, because found RoBERTa to be the strongest baseline for in nn-distribution prediction. The intention is to capture via RoBERTa the interplay between positive and negative pairs. As depicted in Eqn. when do not have significant evidence that the given pair is out of distribution (p-test > 10%), we trust the prediction given by the discriminative model (RoBERTa). Otherwise, trust GAP.
Following the previous literature in sentence matching [22 16, we mainly use macro Fl score (F1) and accuracy score (ACC) to evaluate the models. Results based on Area-under-curve of the Receiver Operating Characteristic Curve (AUROC), a common metrics used to evaluate out-o--distribution metries, are also provided in Appendix
4.3 Measuring Distribution Shift
To situate our experiments properly. we note that different datasets does not equate to different distributions. It is thus important for us to be able measure the distribution gap between datasets, and shed light on the models that perform the best when transferring between datasets with high distribution gap. Metrics that is per sample based such as our perplexity measure are not suitable for measuring at dataset level. To this end, we take look at the Reverse Classification Accuracy* (RCA*) metric that has been proposed to predict the drop of model performance [8]] and model selection [19 64 Here, given Da and D'. we first train model M1 from the training set of D We then take certain amount of samples from the target distribution D' (1000 our experiments), and
use MI 1 to relabel them. The relabeled pairs are then utilized to train new model M2. We measure
7

---

the performance (in terms of AUC or ACC) of M2 on held-out test set from D As control, we train another model M3 following the same procedure except the relabeled data comes from D We denote the performance drop from M3 to M2 as the RCA* score indicative of the distribution gap.
To calibrate RCA*, for each dataset, shift from the dataset itself (which we expect to characterize an in-distribution RCA*), After 100 repetitions of measurements, we get probability distribution of RCA* scores for each distribution in itself. Calibration results are reported in Appendix. 2 All the distributions share a mean of around O and standard deviation around By a p-test of 10%, it is unlikely that datasets with RCA* * greater than 4.0 would belong to the same distribution. Based on this observation, most of the transfers between different datasets are likely cut-of-distributiin vice versa, which have a RCA* score of 2.8 and 3.4 respectively. This is probably because QQP and PIT are curated in a similar fashion, where they extract sentences with similar topics from social platforms, while all other datasets adopt different strategies to collect their data.
randomly selected 1000 held-out pairs and measure its
except for transfers from QQP to PIT and
Finally, important to note that although RCA* provides a good estimate of the distribution shift at the dataset level, it's utility does not easily extend to Eqn. 4 and B|as opposed to per sample out-of-distribution metries. RCA* * assumes availability of the entire test set, while in real world, we are much more likely to get a single or small batches of test pairs at a time, all of which could even be from different distributions.
4.4 Implementation details
In practice, for testing, it helps to average the cross entropy by the length of the sentence, and average the cross entropy of generating $2 from $1 and generating 81 from 82. To optimize conditional sentence generators, we use Adam optimizer with learning rate 2e-5, We adopt cross entropy loss for each word logit, All experiments are run on Nvidia 2080 Ti with 11 GB memory.
Comparisons of Macro F1 score of different models in face of varying degrees dietribution shifl (4]
[5]
[9]
BERTEP
BART MOSEHIA,
F1
CAP
COSREE SOFTMAX
GOCP
13"
MAHA ENERAY
22
20
10
22
20
20
30
Reverse Classifiction Accuracy" (%)
Reverse Classificaion Accuracy (%)
Reverse Clasailcation Accuracy
Figure 3: Comparing Macro F1 scores of different models at varying degrees of distribution shift.
Comparisons accuracy score of different models in face of verying degrees of distr buttio shift
[c]
FERPLEXIT EEXIT
GAPS
CORIKE
BERT
SOFTMAX
20
GAPK
BART
12
23
30
25
30
20
20
Reverse Classficction Acouracy
Reverse Acourasy
Reverse Classfication Accurasy"
Figure 4: Comparing ACC score of different models at varying degrees of distribution shift.
4.5 Main Results
Bias in Negative Pairs To understand whether negative pairs are major sources of bias, we plot the Macro Fl and ACC score against RCA* in Fig. (aa) and Fig. Ta). comparing the performances of OODP, IDP; and BART, all finetuned from the same pretrained checkpoint. The x-axis plotted
in ascending order of RCA* between pairs of datasets given in Table I and There are three pairs
8

---

Model
QQP
PIT
PIT
PAWS
PAWS
PAWS
QQP
WMT
WMT
PAWS
>> QQP
>> PIT (34.5)
WMT
>> PAWS
average
(5.5)
(11.4)
(32.4)
(33.2)
(35.6)
(43.7)
BOW
34,6/5.5
33.3/514
35.3/54.7
33.3/50.0
34//50.2
34.8/51.7
35.3/54.7
34.4/52.0
BiLSTM
34.4/51.5
50.7/51.1
48.6/48.7
36.8/50.4
43.3/50.6
34.9/51.2
37.1/54.7
40.8/51.2
BERT
67.4/67.7
50.0/57.7
31.2/45.5
33.8/622.
52.6/56.4
68.4/57.0
47.1/50.5
54.4/56.8
BERT+EP
66.5/66.5
50.8/57.5
31.2/45.5
51.8/56.4
$2.4/56.4
52.4/56.8
36.0/46.1
48.7/55.0
BART
68.3/68.5
61.6/63.7
31.4/45.5
62.5/63.5
61.7/61.6
64.4/65.5
43.2/50.0
56.2/59.8
RoBERTa
65.3/66.9
53.7/59.4
31.2/45.5
60.0/63.6
63.2/63.9
59.8/62.8
35.1/46.5
52.6/58.4
59.1/62.0
53.9/58.3
46.1/59.6
54.2754.3
54.0/53.8
50.5/52.0
52.6/49.2
$2.9/55.6
OODP
75.4/75.6
75.7/75.8
52.4/60.9
70.5/70.6
69.8/70.2
76.3/76.4
67 67.9/69.3
69.7/71.3
GAP 75.4/75.5
75.7/75.7
54.0/61.5
70.5/70.6
69.7/70.2
76.3/76.4
68.4/69.5
70.0/71.3
GAPX
75.5/75.5
74.4/74.5
55.1/55.5
70.8/70.8
69.7/70.2
76A/76.4
52.3/54.3
67.7/68.2
Table 1: Model performance on different out-of-distribtioon combinations of QQP, PAWS and PIT,
in terms of macro F1/accuracy (ACC). Parenthesized is the RCA* score for each combination of datasets.
Model
QQP
PAWS
PIT
QQP
PIT
>QQQ
SPAWS
->PIT
->PIT
-QQP
average
(0)
(0)
(0)
(2.8)
(3.4)
BOW
51.3757.8
48.8/56.8
33.3/50.0
41.7/49.7
33.3/50.0
40.7/52.9
BiLSTM
61.6/63.6
43.6/53.0
50.6/51.1
51.7/52.9
41.1/49.0
49.7/53.9
BERT
82.5/82.6
92.7/93.2
76.9/77.0
68.0/68.3
69.0/69.4
77.8/78.1
BERT+EP
81.6/81.7
89.7/89.7
75.3/74.9
58.6/62.7
67.4/63.6
74.5/74.5
BART
82.6/82.8
94.1/94.1
80.9/81.0
68.6/72.7
72.4/69.2
79.7/80.0
RoBERTa
84.4/84.5
93.5/93.6
81.0/81.1
66.5/68.8
73.0/74.9
79.7/80.6
OODP
65.3/73.2
67.9/77.1
41.5/65.2
62.2/65.5
65.2/71.2
60.4/70.4
IDP
79.0/79.1
88.2/88.5
77.4/77.7
65.2/66.9
68.3/69.0
75.6/76.2
GAP
68.8/71.0
85.1/85.2
56.6/76.5
62.2/65.0
71.3/71.7
68.8/73.
GAPX
84.4/84.5
92.7/92.7
79.3/79.3
62.3/63.4
72.0/72.4
78. 187./78.5
Table 2: Model performance for in-distribution performances QQP, PAWS, and PIT. in terms of
macro Fl/accuracy (ACC). Parenthesized is the RCA* & score for each combination datasets.
that have RCA* score (Table for which we average the performance in the plots. Both the Fl and ACC plots share a similar pattern. In the in-distribution region, where the RCA* score is less than 4% (Fig. D BART and IDP achieves similarly high performances 79.7% and 75.6% Fl on average respectively. The 4.1% gap in FI is potentially due to the fact that IDP trains negative model and positive model separately, neglecting the interaction between positive and negative pairs. Comparatively, the performance of OODP is significantly inferior to the other two, with only 60.4% average F1. This changes the cut-of-distribution region, where the RCA* score is now greater than 4%. OODP turns distribution shift, we observe that both BART and IDP are especially fragile and their performance drop significantly. When the RCA* greater than 20, both models' Fl drop to low as around 60%, which can be hardly useful in practice. In contrast, OODP maintains an advantage in Fl of as much 10-20% throughout the out-of-distribution region. Since the only difference between OODP and IDP is that OODP transfers only the positive model, it confirms our hypothesis that the negative model does not generalize as well as the positive model.
be the leading model over BART and IDP. With increasing degree of
Importance of the Interplay between Positive and Negative Pairs To understand the necessity of capturing the interplay for test pairs that are in distribution, we compare the performances of GAP, GAPX, and BART in Fig. a) and Fig. a). GAP only ensembles the positive and negative model trained separately, so does not contain any interplay information. On the other hand, both BART, trained with positive and negative pairs together, and GAPX (Sec. (322.2 capture interplay information. As shown in the plot, the performance of GAP in the in in-distribution region is not directly comparable to BART with a gap of 10.9% macro F1. contrast, GAPX's in-distribution result has much smaller margin of 1.6% macro F1 compared to BART, yet, by automatically weighing the contribution of the discriminative model, GAPX also closely matches the performance of GAP the dutoof-distributinn region with only 2.3% loss in macro Fl.

---

Effectiveness of Perplexity-bane Ensembling We also substitute PP(s1 S2) with other state-of-
the-art out-of-distribution metrics used for estimating the probability in Eqn. o oooor GAPX. Specifically,
we compare our perplexity metric with Maximum Softmax Probability (SOFTMAX) (24] 26 [
Energy Score (ENERGY) [3 Mahalanobis Distance (MAHA) [29]. and COSINE (65] See
Appendix for more details. The results are given in Fig. 4b) and Fig. Bb). F1 and ACC are all
similar in the in-distribution region. However, in the out-of-distribution region, SOFTMAX, MAHA,
and COSINE start perform poorly. ENERGY turned out to be the most robust but is still obviously
not matching our perplexity metric, often by a large margin.
Generalization We implemented one of the most popular methods for domain generalization in paraphrase identification, Expert Product [15] as a potential strong baseline for handling dataset bias. As described in Sec. 4.2] we train BERT classifier with only the first sentence 81 as the biased model for BERT+EP. We report the results in Fig. 4c and Fig. c))), together with BERT, BART and RoBERTa. addition, we also provide performance of traditional methods like BoW [62] and ESIM 12 in Table [ and Table BERT, BART, and RoBERTa all produce similar results on all combinations of datasets. We observe that their performances are consistently better than traditional methods like BoW and ESIM, showing that pretraining and finetuming can indeed improve the generalizability of classifiers for paraphrase identification. However, their performances in out-of- distribution setting are still far from their in in-distribution performances, with accuracy below 65% in most of the cases (around 20% drop). GAPX maintains an absolute margin of around 10% in terms of ACC and an absolute margin of 7-20% in terms of F1 in the out-of-distributie best transformer- based models in the out-of-distribution region is BART, with an average of 56.2% in F1 and 59.8% in ACC, while GAPX maintains average of 67.7% in F1 and 68. 68.2% in ACC, with an absolute gain of 11 11.5% in Fl and 8.4% in ACC. What is also encouraging is that GAPX's performance is close that of OODP, which is promising that do not need a priori information on the domain gap between the source and target. Lastly, BERT+EP fails to provide much gain, which we conjecture is due to the difficulty of "finding" the right bias model or features.
region. The
Conclusion
We have shown that negative samples introduce bias that prevent the generalization of paraphrase identification models. To overcome, present novel paradigm to train separate models for the distribution of positive and negative samples independently, and utilize a perplexity based out-of- distribution detection to ensemble them automatically. Experiments show that our method achieves an average 11.5% gain of Fl and 8.4% gain of ACC various different -ffto----ttiuutioon scenarios over other sate-of-the-an art methods.
5.1 Limitations
Our methodology is specifically designed for only "verification" problems, where the samples come in pairs. Scenarios that involve other types of bias will require non-trivial turn-key formulations to explicitly model the source of bias (much like the negative model).
Acknowledgements
This work is sponsored by Meta AI. We would also like to thank Prof. Yoav Artzi for helpful discussion in the early stage of this project.
References
[1] D. Abati, A. Porrello, S. Calderara, and R. Cucchiara. Latent space autoregression for novelty detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 481-490, 2019.
[2] A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971 4971-4980, 2018.
10

---

[3] H. Amirkhani, M. AzariJafari, Z. Pourjafari, S. Faridan-Jabromi, Z. Kouhkan, and A. Amirak. Farstail: A persian natural language inference dataset. arXiv preprint arXiv:2009.088220 2020.
[4] A. Anand, E. Belilovsky, K. Kastner, H. Larochelle, and A. C. Courville. Blindfold baselines for embodied QA. CoRR, abs/1811.0500 1.00011,,, 2018. URL
[5] H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh. Learning de-biased representations with biased representations. H. D. III and Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 528-539, PMLR, 13-18 Jul 2020. URL http:htts:///// htttps://ppoceeednnggg bahng20a... html
.mw.innn.mm........mm..mmm.m.mm. mlr press/v119/
[6] I.. Bergman and Y. Hoshen. Classification- anomaly detection for general data, ArXiv, abs/2005 02359,, 2020.
[7] R. Bhagat and E. Hovy. What Is Paraphrase? Computational Linguistics, 39(3):463-41) 472, 09 2013. ISSN 0891-2017. doi: 201162/COLLLa_00166 URL https://doi. org/ 10. 1162/ COLI a 00166
[8] O. Bojar, Y. Graham, and A. Kamran. Results of the wmt17 metrics shared task. In WMT, 2017.
[9] Bulusu, B. Kailkhura, B. Li, P. Varshney, and D. Song. Anomalous example detection in deep learning; A survey. IEEE Access, 8:132330-132347, 2020. doi: 10.1109/ACEESS22000 3010274.
[10] R. Cadène, C. Dancette, H. Ben-younes, M. Cord, and D. Parikh, Rubi: Reducing unimodal biases in visual question answering. CoRR, ahs/1906. 10169, 2019. httpp/ttp:/arrxiv.o. abs/1909 10169
[11] D. Cer, M. Diab, Agirre, Lopez-Gazpin and L. Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEvval20017), pages 1-14, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics. doi: 10. 111188611111112000
URL Utthttps::/ http::/:///aaaaahhllooooo/o orgee1/-gg/11111111 2001
[12] Q. Chen, X. Zhu, Z. Ling, S. Wei, and H. Jiang. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR, 0ts//609.06000 2016. URL lattp://arxiv
htg/abb/1609..oom
[13] T. Chen, S. Kornblith, M. Norouzi, and E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. BRLphtts:///axxi org/ Oba/2002.05709
[14] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection, 2019.
[15] C. Clark, M. Yatskar, and L. Zettlemoyer. Don't 't take the easy way out: Ensemble based methods for avoiding known dataset biases. CoRR, http//999.068333 2019. URL httpp:////ar org/ abs/ abs/1909. 006883)
[16] Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. Utt.btttp//arxiv org/abs/ /1tp//1110..000 04800
[17] T. DeVries and G. W. Taylor. Learning confidence for networks, 2018. URL htttttt//taxiiiii http:::://aaviv.r///a/aaase
detection in neural
04865
[18] H. Elsahar and M. Gallé. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference pages 2163-2173, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10. 10.18653/v 1/D19- 1222.
Natural Language Processing (EMNNL--IJCNLP),
URL https: http:/://aaaaaaaahaaaa www.looyy..rg/19-1120001
11

---

[19] W. Fan and I. Davidson. Reverse testing: an efficient framework to select amongst classifiers under sample selection bias. In KDD '06, 2006.
[20] S. Fort, J. Ren, and B. Lakshminarayanan. Exploring the limits of distiiibibtttinn detec- tion. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Procexsing Systems, volume 34, pages 7068-7081. Cur- Associates, Inc., 2021. http:htttss:::oeeeeeeeeeeeee wtw.eeedddinnnnnnnns...e. wttppsccpppppe///////// 39..:3888.177422234accc77/aaaaa3
pdf
[21] Grill, F. Strub, F. Altché, C. Tallec, H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning. CoRR, abs/2006.0772000 2020. httpss////aax wwwgg//gg//..oooo///o/
[22] H. He, S. Zha, and H. Wang. Unlearn dataset bias in natural language inference by fitting the residual. CoRR, abs/ 10763, 2019. URL pett://arrivy: wtg. atg/abs/1908 19111106881111333
[23] H. He, S. Zha, and H. Wang. Unlearn dataset bias in natural language inference by fitting the residual. arXiv preprint arXiv: 10763, 2019.
[24] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distriibttio examples neural networks. CoRR, abs/1610.02136, 2016. URL [ttp://///axxx .rgg abs/ 16110.0236
[25] D. Hendrycks, M. Mazeika, and T. G. Dietterich. Deep anomaly detection with outlier exposure. CoRR, abs/ (bbs/182..0466,, 2018. URL http::ttaxiii.oo/// abt/aab/...a///aaa/////aaaaattta
1812. 04606
[26] Y. Hsu, Y. Shen, H. Jin, and Z. Kira. Generalized ODIN: detecting out-of-distributie image without learning from out- -fi-riiiirtttttioon data. CoRR, abs/2002.11297, 2020. URL https 7/arxiv. org/ abs/ /00p..2002..22 11297
[27] Z. Kozareva and A. Montoyo. Paraphrase identification the basis of supervised machine learning techniques. In International conference on natural language processing (in Finland), pages 524-533. Springer, 2006.
[28] W. Lan and W. Xu. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. CoRR, abs/1 00/.043060333300 2018. URL http: //arxiv.../vv orgo htte//bs/18000 1806. 04330
[29] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting of samples and adversarial attacks, 2018. URL
03888
[30] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910. 13461, 2019. URL http:// arxiv arxiv.org/ abs/ 0tbb//999..... 19461
[31] B. Li, T. Liu, Wang, and L. Wang. Enhancing deep paraphrase identification via leveraging word aligmment information. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7884-7847, 2021. doi: 10.1109/ 0t..:339922..221994414
[32] S. Liang, Y. Li, and R. Srikant. Principled detection of out-of-distributtion examples in neural networks. CoRR, fass/7006.02690 2017. hol.//ttpp//arrivcrr//aa///////
[33] W. Liu, X. Wang, J. D. Owens, and Y. Li. Energy-based ootpsaasssdooootooooooirrrrrrrrr detection. CoRR, abs/2010.03759, 2020. URL httptttss///axiiit wwwp:/ogg//ab///e 03759
[34] Y. Liu, M. Out, N. Goyal, J. Du, Joshi, D. Chen, O. Levy, M. Lewis, I. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 1b2/1907.11692,
2019. BtLphttp://arxx
org/ abs//90//1 11692
[35] E. Marsi and E. Krahmer. Explorations in sentence fusion. 01 2005.
12

---

[36] S. Min, E. Wallace, S. Singh, M. Gardner, H. Hajishirzi, and L. Zettlemoyer. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4249-4244 -257, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.1 10.11653/v11/19-111 1416.. URL https Tacclanthology.o wwwpollooooooogggggeeeee wtg/p19-/10 1416
[37] R. C. Moore and W. Lewis. Intelligent selection of language model training data. In Proceedings of the ACI. 2010 Conference Short Papers, pages 220-224, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URI.https: haattpa://acllamthbloogoooo
[38] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL pttttppes//proocedingg wwwp:/iiinnnnnnnnnneeoem cctp:/ee/cccpppppe22/22 Paper pdf
[39] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[40] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, COMET: A neural framework for MT evaluation. CoRR, abs/2009.0900 2020. URLpttps:: pettps://arxivve org/ ab09 09025
[41] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie. COMET: A neural framework for MT evaluation, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702 Online, Nov. 2020. Association for Computational Linguistics. doi: 10. 1866://v/2000.emmmppmmooo 2020. emnlp-main. .13
URL httpst/tt////ctatttti
org/
[42] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. CoRR, abs/1908, 108.10088 10084, 2019. URL http:ttt://aaxii.rrrrrgr aaag/////// 1
[43] J. Ren, P.J. Liu, E. Fertig. J. Snoek, R. Poplin, M. Depristo. J. Dillon, and B. Lakshninarayanan. Likelihood ratios for out-of-distribtiion detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://///prroteeeeee
neurips Paper pdf
[44] T. Sellam, D. Das, and A. P. Parikh. BLEURT: learning robust metrics for text generation. CoRR, abs/2004. 04696, 2020. paattps://aarivvorg//aa
04696
[45] X. Shen and W. Lam. Towards domain-gennerlizzable paraphrase identification by avoiding the shorteut learning. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). pages 1318 - Held Online, Sept. 2021.
INCOMA Ltd. URL htttps:://aclaaatoollgg///// org 2021 ranlp main.ain............a.. 148
[46] B. Thompson and M. Post. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. CoRR, abs/2004. 14564, 2020. URL htt.ts::/taaii...rrrttr www.aarwwvv....raaaoo wrg.//s///////////// 14564]
[47] G. SoToa,,, T. Duque, O. Täckström, J. Uszkoreit, and D. Das. Neural paraphrase identification of questions with noisy pretraining. arXiv preprint arXiv:/704. 04565, 2017.
[48] A. Utama, N. Moosavi, and I. Gurevych. Towards debiasing NLU models from unknown biases. CoRR, abs/2009. 12303, 2020. URL htt.hhttt///xxiiii org/ tabb/2099... 12303
[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL httpp:////ar hrg/aba/1706. 1706603766226
[50] S. Wang, H. Fang, M. Khabsa, H. Mao, and H. Ma. Entailment as few-shot learner. CoRR, abs/2104. 102.11444.4400 14690, 2021. URL https:////a wwwpr//iiv.orgg/s///1/ee 14690
[51] Z. Wang, Mi, and A. Itycheriah. Sentence similarity learning by lexical decomposition and composition. CoRR, abs/1602.07019, 2016. utthttp://arxiv
13

---

[52] Z. Wang, W. Hamza, and R. Florian. Bilateral multi matching for natural language sentences. CoRR, abs/1702.0333 2017. http://:///rrt wttp:iiw..rg/bbb//////// abs/1702. 03814
org/
[53] W. Xu, A. Ritter, C. Callison-Burch, W. B. Dolan, and Y. Ji. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Compulational Linguistics, 435-448, 2014. doi: 10. a a a0004 BRL.https:/ /www.www.ooooo aclweb. org/ anthology/ Q14-1034
[54] W. Xu, C. Callison- Burch, and B. Dolan. SemEval-2011 Task 1: Paraphrase and semantic similarity in Twitter (PIT). In Proceedings of the 9th International Workshop Semantic Evaluation (SemEval 2015), pages 1-11, Association for Computational Linguistics, 2015. doi:
http://www.ccceeeboooor/tttttl// 0011853/v1//15-2000 URL wttp//////wttppp////aa/aeeee.e..
[55] Yang, K. Zhou, Y. Li, and Z. Liu. Generalized oatp://rrddddouoo--oo-dsssiiiiii detection: survey, CoRR, abs/2110.1 1334, 2021. URL https://arxxv.oorr//b//2221
[56] Y. Yang. Y. Zhang, Tar, and J. Baldridge. PAWS-X: cross-lingual adversarial dataset for paraphrase identification. CoRR, abs/1908. 11828, 2019. URL htttt://axxii.... org/ abs/
1908. 11828
[57] W. Yin and H. Schüitze Convolutional neural network for paraphrase identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 901-911, 2015.
[58] W. Yuan, G. Neubig, and P. Liu. Bartscore: Evaluating generated text text generation. CoRR, abs/2106.1 1520, 2021. URL. htt.:tttts:///tttttt oww.oogg/aa....o.o http//bb//0...... 2106 11520
[59] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-superriieed learning via redundancy reduction. CoRR, abs/2103.03230, 2021. URL
03230
[60] Zhai, Y. Cheng, W. Lu, and Z. Zhang. Deep structured energy based models for anomaly detection. CoRR, abs/1605.07717, 2016. URL http://///r org/ abs/ 1es//1660000777
[61] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09665, 2019. bettp://arxiv.orrg www.orgg/bbbbaaaoaae 09675)
[62] Y. Zhang, J. Baldridge, and L. He. PAWS: paraphrase adversaries from word scrambling. CoRR, abs/1904.01130, 2019. URL http://arxiv org abs 1904. 01130
[63] W. Zhao, M. Peyrard, F. Liu, Y. Gao, M. Meyer, and S. Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. CoRR, (0e/1999.0262,, 02622
2019. URL battp://arxits
wwtp://///aaa//aaaaaoaa httt:://arxi..orr//a///1990
[64] E. Zhong, W. Fan, Q. Yang, O. Verscheure, and J. Ren. Cross validation framework to choose amongst models and datasets for transfer learning. pages 547-562, 09 2010. ISBN 978-3-642- 15938-1. doi: 10.1007/978-341 6-6-2-111 1122-15931 15939-8 35.
[65] W. Zhou and M. Chen. Contrastive out-of-distribution detection for pretrained transformers. CoRR, abs/2104.08812 2021. /ttpp/tt:///rriiv.orr htrgg/ab/2104// 221.//1114.000 08812
[66] B. Zong, Q Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen. Deep autoencod- ing gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]. See our listed contributions in the introduction.
14

---

(b) Did you describe the limitations of your work? [Yes]. See the end of our conclusion,
Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This is traditional Natural Language Processing research topic. We do not any conceivable
negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]. See Sec. 3.1
(b) Did you include complete proofs of all theoretical results? [Yes]. See Section, 3.1
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes].
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. See Sec
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] The baseline models in NLP that we are comparing to do not report error bars in their paper.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]. See Sec. 4.4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]. See Appendix 4.
(b) Did you mention the license of the assets? [N/A] Those are all public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]. Those are all public datasets, allowing researching purpose. (e) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [N/A]. Those data does not contain personal information
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "mP82RPxlAYw", "reviewer": "Reviewer_KXFG", "summary": "**What is the task?**\nParaphrase identification\n\n**What has been done before?**\n* Many state-of- the-art models often suffer from distribution shift during inference time. Author(s) show a major source of this performance drop comes from biases introduced by negative examples.To overcome these biases, they propose in this paper to train two separate models, one that only utilizes the positive pairs and the other the negative pairs.\n\n* Authors have compared their work with different lines of work like distribution shift and debiasing models in NLP, out of distribution detection, text generation metrics etc. and are able to show the novelty of their work.\n\n\n**What are the main contributions of the paper?**\n* Reported a new research insight, supported by empirical results, that the negative pairs of a dataset could potentially introduce biases that will prevent a paraphrase identification model from generalizing to out-of-distribution pairs. \n\n* Proposed a novel autoregressive modeling approach to train both a positive and a negative model, and ensemble them automatically during inference. \n\n* Introduced a new perplexity based approach to determine whether a given pair is out-of-distribution to achieve auto ensembling. \n\n* SOTA results in out-of-distribution performance while keeping comparable performance for in-distribution prediction.\n\n\n\n**What are the main results? Are they significant?**\nAuthor(s) support their findings with strong empirical results using following experiments\n\n* (1) verify that the task of paraphrase identification suffers from biases in the datasets that is the main obstacle to generalization in this field of study, \n* (2) test the accuracy of proposed perplexity based out-of-distribution detection method, and \n* (3) test that balancing the utilization of the negative model can help outperform the state-of-the-art in the face of distribution shift.\n", "questions": "NA", "limitations": "Authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n\n* Reported new research insights well supported by empirical results for all the findings like \"bias in negative pairs\", \"importance of the interplay between positive and negative pairs\", \"effectiveness of perplexity-based ensembling\", \"generalization\" etc.\n* Good presentation - Paper is easy to understand\n* SOTA results in out-of-distribution performance while keeping comparable performance for in-distribution prediction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cp_PStZDSoz", "reviewer": "Reviewer_AaSh", "summary": "This paper addresses the problem of paraphrase identification. The improvement is motivated by an observation where distributions of negative examples exhibits serious corpus-specific bias and do not generalize well across different corpora. To solve this problem, the paper starts with a Bayesian formulation of the paraphrase identification problem (formula (1)-(3)), and proposed three different ways to remedy the corpus-level bias, including out-of-distribution predictor (OODP), automatic ensemble of the (generative) positive and negative model (GAP), and an extra ensemble with a discriminative model (GAPX).\n\nThe experiments cover several different corpus and distribution shift scenarios (measured by RCA). Overall, results show an significant macro F1/ACC improvement of OODP/GAP/GAPX model over the IDP model, as well as BERT/RoBERTa-based baselines, which shows the benefit of the paper's proposed anti-biasing solutions.", "questions": "- Please comment on weakness point 2. I could have misunderstood something.\n- It is not entirely clear to me why the distribution model (used to evaluated formula (5)) should be trained on the validation data -- isn't this trying to detect distribution shift from the training data?", "limitations": "Apart from the connection with Moore-Lewis filtering that the authors did not bring up, I also think the method could be further generalized and validated under cross-lingual scenarios -- for example, if $(s_1, s_2)$ are multilingual pairs and BART is substituted with mBART, the authors could use parallel data filtering task (https://www.statmt.org/wmt20/parallel-corpus-filtering.html) to further validate their findings.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The solution is very well-motivated by the authors' observation.\n- The paper is overall well-written -- most parts of the proposed solutions are presented in a clear and intuitive way.\n- The comparison is thorough, and shows clear improvement from the model across the board in out-of-distribution scenarios.\n\nWeaknesses:\n\n- The Bayesian formulation, while nicely laid out, is not a novel one. Specifically, Eqn. (3) has significant overlap with Moore-Lewis filtering (https://aclanthology.org/P10-2041.pdf). This connection is not pointed out in the paper.\n- While theoretically-interesting, GAP/GAPX models do not show a lot of empirical improvement compared to the very naïve (even ill-formed, because it completely dropped the $P(w_2^{(i)}\\mid s_1, Y=y, w_2^{(1:i-1)})$ term in the Bayesian formulation shown in (1)) OODP model, which is not clear to me why. Also, both GAP/GAPX models seem to have more components than OODP, so it's not clear whether GAP/GAPX is really worth it.\n\nSome minor comments/suggestions:\n- The paragraph of L34-51 is very long and verbose (especially since you are talking about a lot of modeling details without a formula). I would try to break it down to reflect the separation of different components (IDP -> OODP -> GAP/GAPX), as well as focusing more on providing a conciser summary of high-level intuitions.\n- L101: as follow -> as follows\n- L139-146: It would be clearer to have a separate subsection for IDP and OODP\n- L203: You are referring to Eqn. (4) instead of (3.2).\n- L242: \"cross entropy\"  -> what is this referring to? Eqn. (5)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Yr4fWThj8z", "reviewer": "Reviewer_mmQb", "summary": "This paper explores the distribution shift problem in paraphrase identification.  The authors first verify that the distribution shift problem is mainly caused by the bias of negative examples.  To address this problem, they train two separate models, a positive model and a negative model, and combine them together during inference where the weights are dynamically decided by the distribution similarity between inference pair and training pairs. Experiments show that the proposed approach achieves good transfer learning ability. ", "questions": "n/a", "limitations": "More comparison on datasets with in-pair data to show the generalization results over diverse tasks. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe motivation is interesting. The proposed method is motivated by their findings that negative examples do not generalize well to out-of-distribution data. To address this model, they propose to separate positive examples and negative examples and train two models. The combination weights is adjusted during inference. \n\n\nThe proposed model outperforms baselines with a large margin.  Table 1 shows significant performance improvements over baselines. \n\nWeaknesses:\n\nStrong baselines are missing. Multi-task learning is also an important solution to address the distribution shift problem. It would be better to add a baseline that combines all tasks together. On the other hand, distribution shift is a traditional and important problem. The comparisons with related literature is required to show the effectiveness of the proposed method. \n\nThe proposed method is specifically designed for paraphrase identification with in-pair data. It is unclear whether the proposed method can affect diverse NLP tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I4fa0I0t62F", "reviewer": "Reviewer_emd5", "summary": "This paper focuses on the distribution shift problem in paraphrase\nidentification task, and proposes several methods to better deal with\nthe bias brought by negative examples, including training two\nauto-regressive models exclusively and respectively on positive and\nnegative examples, combining them with an ordinary discriminative\nmodel, and determining their weights automatically on the basis of\naverage token-level perplexity.  Through a cross-dataset experiment,\nthe authors confirm that the proposed method is less affected by the\ndistribution shift of negative examples, while achieving a competitive\nclassification accuracy in the in-domain settings.  One advantage of\nthe proposed method is that it does not require us to have a knowledge\nof the degree of distribution gap between training and test data a\npriori.", "questions": "I have several questions regarding the proposed method.\n\n- Considering the diversity of negative examples, The distribution learned from a limited amount of negative examples would not be smooth enough.  Is there any justification to apply an auto-regressive model, a kind of generative model, rather than instance-based computation of the likelihood, such as nearest neighbors?\n\n- Two $P(s_{2}|s_{1},Y)$ in GAP (Eq.(4)) are not defined.  Are they respectively equal to the $\\sum\\log P()$ in IDP (Eq.(3))?\n\n- When computing the perplexity scores, did the authors concatenate given a pair of $s_{1}$ and $s_{2}$ without any special tokens or not?  If not, how can the model properly evaluate the adjacency of the last token of $s_{1}$ and the first token of $s_{2}$?  What is the advantage of concatenating it over separately scoring for $s_{1}$ and $s_{2}$?\n\n- The reason to choose the Weibull distribution and its parameters are unclear.\n\n- How large was $M$ in GAPX (Eq.(8))?  It is explained as \"a sufficiently large constant\" but this means that the second term (GAP) will be ignored.\n\n- There is $(\\frac{1}{2} - P())$ but isn't it $(P() - \\frac{1}{2})$?", "limitations": "Yes.  The proposed method exploits neural generative models but their usage is limited to labeling given pairs of sentences.  The trained models will not introduce any societal influence nor misleading conclusions.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are two noteworthy strengths.\n\n- It empirically demonstrates that the performance of existing methods\ncan dramatically worsen when the model is applied to the dataset that\nexhibits different distribution from the training data.\n\n- The proposed method automatically determines the weights of\ncomponent classifiers on the basis of perplexity adaptively to each\ntest example.  This should be much more emphasized, while all the\ncomponent models are not genuinely novel.\n\nSome descriptions are not precise enough.\n\n- As I acknowledge above, the perplexity-based automatic weighting is\nthe key of the proposed method.  However, the formulation of the\nweight lacks some information (see the questions below).\n\n- The presentation of the results has a room for improvement.  Figures\n3 and 4 are useless, because all the referenced information for the\ndiscussion in Section 4.4 are also seen in Tables 1 and 2.  Using line\ncharts is also inappropriate because the results for different\ndatasets are inherently incomparable.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept