Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper explores the distribution shift problem in paraphrase identification.  The authors first verify that the distribution shift problem is mainly caused by the bias of negative examples.  To address this problem, they train two separate models, a positive model and a negative model, and combine them together during inference where the weights are dynamically decided by the distribution similarity between inference pair and training pairs. Experiments show that the proposed approach achieves good transfer learning ability. 
strengths_and_weaknesses: Strengths:

The motivation is interesting. The proposed method is motivated by their findings that negative examples do not generalize well to out-of-distribution data. To address this model, they propose to separate positive examples and negative examples and train two models. The combination weights is adjusted during inference. 


The proposed model outperforms baselines with a large margin.  Table 1 shows significant performance improvements over baselines. 

Weaknesses:

Strong baselines are missing. Multi-task learning is also an important solution to address the distribution shift problem. It would be better to add a baseline that combines all tasks together. On the other hand, distribution shift is a traditional and important problem. The comparisons with related literature is required to show the effectiveness of the proposed method. 

The proposed method is specifically designed for paraphrase identification with in-pair data. It is unclear whether the proposed method can affect diverse NLP tasks. 
questions: n/a
limitations: More comparison on datasets with in-pair data to show the generalization results over diverse tasks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the problem of paraphrase identification. The improvement is motivated by an observation where distributions of negative examples exhibits serious corpus-specific bias and do not generalize well across different corpora. To solve this problem, the paper starts with a Bayesian formulation of the paraphrase identification problem (formula (1)-(3)), and proposed three different ways to remedy the corpus-level bias, including out-of-distribution predictor (OODP), automatic ensemble of the (generative) positive and negative model (GAP), and an extra ensemble with a discriminative model (GAPX).

The experiments cover several different corpus and distribution shift scenarios (measured by RCA). Overall, results show an significant macro F1/ACC improvement of OODP/GAP/GAPX model over the IDP model, as well as BERT/RoBERTa-based baselines, which shows the benefit of the paper's proposed anti-biasing solutions.
strengths_and_weaknesses: Strengths:

- The solution is very well-motivated by the authors' observation.
- The paper is overall well-written -- most parts of the proposed solutions are presented in a clear and intuitive way.
- The comparison is thorough, and shows clear improvement from the model across the board in out-of-distribution scenarios.

Weaknesses:

- The Bayesian formulation, while nicely laid out, is not a novel one. Specifically, Eqn. (3) has significant overlap with Moore-Lewis filtering (https://aclanthology.org/P10-2041.pdf). This connection is not pointed out in the paper.
- While theoretically-interesting, GAP/GAPX models do not show a lot of empirical improvement compared to the very naïve (even ill-formed, because it completely dropped the $P(w_2^{(i)}\mid s_1, Y=y, w_2^{(1:i-1)})$ term in the Bayesian formulation shown in (1)) OODP model, which is not clear to me why. Also, both GAP/GAPX models seem to have more components than OODP, so it's not clear whether GAP/GAPX is really worth it.

Some minor comments/suggestions:
- The paragraph of L34-51 is very long and verbose (especially since you are talking about a lot of modeling details without a formula). I would try to break it down to reflect the separation of different components (IDP -> OODP -> GAP/GAPX), as well as focusing more on providing a conciser summary of high-level intuitions.
- L101: as follow -> as follows
- L139-146: It would be clearer to have a separate subsection for IDP and OODP
- L203: You are referring to Eqn. (4) instead of (3.2).
- L242: "cross entropy"  -> what is this referring to? Eqn. (5)?
questions: - Please comment on weakness point 2. I could have misunderstood something.
- It is not entirely clear to me why the distribution model (used to evaluated formula (5)) should be trained on the validation data -- isn't this trying to detect distribution shift from the training data?
limitations: Apart from the connection with Moore-Lewis filtering that the authors did not bring up, I also think the method could be further generalized and validated under cross-lingual scenarios -- for example, if $(s_1, s_2)$ are multilingual pairs and BART is substituted with mBART, the authors could use parallel data filtering task (https://www.statmt.org/wmt20/parallel-corpus-filtering.html) to further validate their findings.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the distribution shift problem in paraphrase
identification task, and proposes several methods to better deal with
the bias brought by negative examples, including training two
auto-regressive models exclusively and respectively on positive and
negative examples, combining them with an ordinary discriminative
model, and determining their weights automatically on the basis of
average token-level perplexity.  Through a cross-dataset experiment,
the authors confirm that the proposed method is less affected by the
distribution shift of negative examples, while achieving a competitive
classification accuracy in the in-domain settings.  One advantage of
the proposed method is that it does not require us to have a knowledge
of the degree of distribution gap between training and test data a
priori.
strengths_and_weaknesses: There are two noteworthy strengths.

- It empirically demonstrates that the performance of existing methods
can dramatically worsen when the model is applied to the dataset that
exhibits different distribution from the training data.

- The proposed method automatically determines the weights of
component classifiers on the basis of perplexity adaptively to each
test example.  This should be much more emphasized, while all the
component models are not genuinely novel.

Some descriptions are not precise enough.

- As I acknowledge above, the perplexity-based automatic weighting is
the key of the proposed method.  However, the formulation of the
weight lacks some information (see the questions below).

- The presentation of the results has a room for improvement.  Figures
3 and 4 are useless, because all the referenced information for the
discussion in Section 4.4 are also seen in Tables 1 and 2.  Using line
charts is also inappropriate because the results for different
datasets are inherently incomparable.

questions: I have several questions regarding the proposed method.

- Considering the diversity of negative examples, The distribution learned from a limited amount of negative examples would not be smooth enough.  Is there any justification to apply an auto-regressive model, a kind of generative model, rather than instance-based computation of the likelihood, such as nearest neighbors?

- Two $P(s_{2}|s_{1},Y)$ in GAP (Eq.(4)) are not defined.  Are they respectively equal to the $\sum\log P()$ in IDP (Eq.(3))?

- When computing the perplexity scores, did the authors concatenate given a pair of $s_{1}$ and $s_{2}$ without any special tokens or not?  If not, how can the model properly evaluate the adjacency of the last token of $s_{1}$ and the first token of $s_{2}$?  What is the advantage of concatenating it over separately scoring for $s_{1}$ and $s_{2}$?

- The reason to choose the Weibull distribution and its parameters are unclear.

- How large was $M$ in GAPX (Eq.(8))?  It is explained as "a sufficiently large constant" but this means that the second term (GAP) will be ignored.

- There is $(\frac{1}{2} - P())$ but isn't it $(P() - \frac{1}{2})$?
limitations: Yes.  The proposed method exploits neural generative models but their usage is limited to labeling given pairs of sentences.  The trained models will not introduce any societal influence nor misleading conclusions.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: **What is the task?**
Paraphrase identification

**What has been done before?**
* Many state-of- the-art models often suffer from distribution shift during inference time. Author(s) show a major source of this performance drop comes from biases introduced by negative examples.To overcome these biases, they propose in this paper to train two separate models, one that only utilizes the positive pairs and the other the negative pairs.

* Authors have compared their work with different lines of work like distribution shift and debiasing models in NLP, out of distribution detection, text generation metrics etc. and are able to show the novelty of their work.


**What are the main contributions of the paper?**
* Reported a new research insight, supported by empirical results, that the negative pairs of a dataset could potentially introduce biases that will prevent a paraphrase identification model from generalizing to out-of-distribution pairs. 

* Proposed a novel autoregressive modeling approach to train both a positive and a negative model, and ensemble them automatically during inference. 

* Introduced a new perplexity based approach to determine whether a given pair is out-of-distribution to achieve auto ensembling. 

* SOTA results in out-of-distribution performance while keeping comparable performance for in-distribution prediction.



**What are the main results? Are they significant?**
Author(s) support their findings with strong empirical results using following experiments

* (1) verify that the task of paraphrase identification suffers from biases in the datasets that is the main obstacle to generalization in this field of study, 
* (2) test the accuracy of proposed perplexity based out-of-distribution detection method, and 
* (3) test that balancing the utilization of the negative model can help outperform the state-of-the-art in the face of distribution shift.

strengths_and_weaknesses: 
Strengths

* Reported new research insights well supported by empirical results for all the findings like "bias in negative pairs", "importance of the interplay between positive and negative pairs", "effectiveness of perplexity-based ensembling", "generalization" etc.
* Good presentation - Paper is easy to understand
* SOTA results in out-of-distribution performance while keeping comparable performance for in-distribution prediction.
questions: NA
limitations: Authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper tackles a discriminative problem by a generative model, where the generation probabilities can be twisted to adjust negative samples’ weights. 

Reviewers generally found the paper interesting. However, one concern is that the paper only considers the paraphrase-identification problem, which sounds narrow. It is expected that the approach may be generalized to different tasks.

confidence: Less certain
Decision: Accept