Noisy Pair Corrector for Dense Retrieval
Authors: Hang Zhang
Yeyun Gong
Xingwei He
Dayiheng Liu
Daya Guo
Jiancheng Lv
Jian Guo
Venue: EMNLP 2023 Findings (2023)
Keywords: Dense Retrieval
Noisy Pair

Abstract: Most dense retrieval models contain an implicit assumption: the training query-document pairs are exactly matched. Since it is expensive to annotate the corpus manually, training pairs in real-world applications are usually collected automatically, which inevitably introduces mismatched-pair noise. In this paper, we explore an interesting and challenging problem in dense retrieval, how to train an effective model with mismatched-pair noise. To solve this problem, we propose a novel approach called Noisy Pair Corrector (NPC), which consists of a detection module and a correction module. The detection module estimates noise pairs by calculating the perplexity between annotated positive and easy negative documents. The correction module utilizes an exponential moving average (EMA) model to provide a soft supervised signal, aiding in mitigating the effects of noise. We conduct experiments on text-retrieval benchmarks Natural Question and TriviaQA, code-search benchmarks StaQC and SO-DS. Experimental results show that NPC achieves excellent performance in handling both synthetic and realistic noise.

Noisy Pair Corrector for Dense Retrieval
Hang Zhang"', Yeyun Gong, Xingwei Hel?
Dayiheng Liu', Daya Guo', Jiancheng Lv Jian Guo
College of Computer Science, Sichuan University Engineering Research Center of Machine Learning and Industry Intelligence The University of Hong Kong" 3 Sun Yat-sen University 4 IDEA Research, China Sangzhang_scude foxmail.com, hexingweil 5 gmail.com
Abstract
Correctly Matched Pair
Most dense retrieval models contain im- plicit assumption: the training query-document pairs are exactly matched. Since is expen- sive to annotate the corpus manually, training pains in real-world applications are usually col- lected automatically, which inevitably intro- duces mismatched-pee noise. In this paper, we explore an interesting and challenging prob- lem dense retrieval. how train an effective model with mismatched-pait noise. To solve this problem, we propose a novel approach called Noisy Pair Corrector (NPC), which con- sists of a detection module and a correction module. The defection module estimates noise pairs by calculating the perplexity between notated positive and easy negative documents. The correction module utilizes an exponen- tial moving average (EMA) model to provide a soft supervised signal, aiding in mitigating the effects of noise. We conduct experiments text-retrieval benchmarks Natural Question and TriviaQA, code-search benchmarks StaQC and SO-DS. Experimental results show that NPC achieves excellent performance in han- dling both synthetic and realistic noise.
Query: Check variable lambda Code:
def is l__addaaaa__ Setidd_ffunctan Function(sbj): neturn isins Lance types, L and abj nane "clanbda>"
Mismatched Noise Pair
Query: Sending Dictionary using Sockets in Python
Code:
Import 1son
#data serialized
data date_string string json. (data) #data loaded data _oad__loded
(datt(
Two examples from StaQC training set. In the bottom example, the given code is mismatched with the query, since can not answer the query.
Figure I:
training data (Xiong et al., 2021) and samples nega- tive documents for each pair. Then the model learns to minimize the contrastive loss for obtaining a good representation ability (Zhang et al., 2022b; Qu al., 2021).
Recent studics on dense retrieval have achieved promising results with hard negative mining (Xiong et al., 2021), pretraining (Gao and Callan, 2021a), distillation (Yang and Seo, 2020), and adversarial training (Zhang et al., 2022a). All methods con- tain implicit assumption: each query is precisely aligned with the positive documents the train- ing set. In practical applications, this assumption becomes challenging to satisfy, particularly when the corpora is automatically collected from the in- ternet. In such scenarios, it is inevitable that the training data will contain mismatched pairs, incor- porating instances such as user mis-click noise in search engines low-quality reply noise in Q&A communities. As shown in Fig. 1, the examples are from StaQC benchmark (Yao al., 2018), which is automatically collected from Stack StarkOverflow. The document, i.c., code solution, can not answer the query but is incorrectly annotated as a positive doc-
1 Introduction
With the advancements in pre-trained language models (Devlin et al., 2019; Liu et al., 2019), dense retrieval has developed rapidly in recent years. is essential to many applications including search engine (Brickley et al., 2019), open-domain ques- tion answering (Karpukhin et al., 2020a; Zhang et al., 2021), and code intelligence (Guo et al., 2021). A typical dense retrieval model maps both queries and documents into low-dimensional vec- tor space and measures the relevance between them by the similarity between their respective represen- tations (Shen et al., 2014). During training, the model utilizes query-document pairs as labelled
"Work done during internship Research *Corresponding author

---

We reveal and extensively explore a long-neglected problem in dense retrieval, i.e., mismatched-par noise, which is ubiquitous in the real world. (2) We propose simple yet effective method for train- ing dense retrievers with mismatched-pan noise. (3) Extensive experiments on four datasets and comprehensive analyses verify the effectiveness of our method against symthetic and realistic noise. Code is available at https:://// /ttp:///gtthuii com/ hangzhang-nip//PPP
Correctly Matched
Mismatched Noise
Query
TP Doc
Doc
TN Doc
FN Doc
Push Apart
Pull Together
Figure 2: Effect of matched mismatched pair for training. Green objects refer annotated pairs, while pentagram and triangle are actually aligned pairs. In the left case, retrieval models are required to push the query with true-positive document (TP Doc) together and pull the query with true-negative documents (TN Doc) apart. the right case, the retrieval models are misled by the mismatched data pair, where the false-positive docu- ment (FP Doc) and the felse-negative document (FN Doc) are wrongly pulled and pushed, respectively.
Preliminary
Before describing our model in detail, we first in- troduce the basic elements of dense retrieval, in- cluding problem definition, model architecture, and model training.
Given query 9. and document collection DD, dense retrieval aims to find document d' rel- evant to from D. The training set consists a collection of query-document pairs, donated as C = [(11, ((01,d1 ) (9N, d where N is the data size. Typical dense retrieval models adopt dual encoder architecture to map queries and documents into dense representation space. Then the rele- vance score [qq, d) of query q and document d can be calculated with their dense representations:
ument. Such noisy pairs are widely present in au- tomatically constructed datasets, which ultimately impact the performance of dense retrievers.
To train robust dense retrievers, previous works have explored addressing various types of noise. For example, RocketQA (Qu et al., 2021) and AR2 (Zhang et al., 2022a) mitigate the false- negative noise with cross-encode, filter and distil- lation, respectively: coCondenser (Gao and Callan, 2021b) reduce the noise during fine-tuning with pre-training technique; RoDR (Chen et al., 2022) deal with query spelling noise with local ranking alignment. However, mismatched-pair noise (false positive problem) in dense retrieval has not been well studied. As shown in Fig. 2, mismatched-pur noise will mislead the retriever to update in the opposite direction.
So d) = =ii (E(g; 0), (dd:))
(1)
where E(:0)) denotes the encoder module param- eterized with 0, and sim is the similarity func- tion, e.g., euclidean distance, cosine distance. inner-product. Existing methods generally lever- age the approximate nearest neighbor technique (ANN) (Johnson et al., 2019) for efficient search. For training dense retrievers, conventional ap- proaches leverage contrastive learning techniques (Karpukhin et al., 2020a; Zhang et al., 2022b). Given training pair (g:, d, C, these methods sample m negative documents {d]1.-.. ..., dim) from large document collection D. The retriever's ob- jective is to minimize the contrastive loss, pushing the similarity of positive pairs higher than negative pairs. Previous work (Xiong et al., 2021) has veri- fied the effectiveness of the negative sampling strat- egy. Two commonly employed strategies are "In- Batch Negative" and "Hard Negative" (Karpukhin et al.. 2020a; Qu al., 2021).
Based on these observations, we propose Noisy Pair Corrector (NPC) framework to solve the false- positive problem. NPC consists of noise detection and correction modules. At each epoch, the detec- tion module estimates whether a query-document pair is mismatched by the perplexity between the annotated document and easy negative documents. Then the correction module provides a soft super- vised signal for both estimated noisy data and clean data via an exponential moving average (EMA) model. Both modules are plug-and-play. which means NPC is a general training paradigm that can be easily applied almost all retrieval models.
The above training paradigm assumes that the query-document pair (q: d'') in training set is correctly aligned. However, this assumption is dif- ficult to satisfy in real-world applications (Qu et al.,
The contributions of this paper are as follows: (1)

---

45 dizdi daust dia et dia
Noisy Dataset
11
d
42
d
Retriever
EMAA Updatt
leoss
Drop paky
Calculate
dg
diul
GMM
loou
Fit PPL Distribution
44
di
da.
Mini batch
Rutriever
Similanty Marrox
(a) Noise Detection
(b) Noise Correction
Figure 3: Overview of noise detection and noise correction. (a) Procedure of Noise Detection, At each epoch, first calculate the perplexity of all training query-document pairs using the retriever next fit the perplexity distribution with Gaussian Mixture Model to get the correctly matched probability of each pair; finally estimate the flag set {0<) by setting threshold. (b) Framework of Noise Correction. Given batch of data pairs, where di,, is the hard negative of and (93, dg is the estimated noisy pair, the retriever 0 and teacher 6* compute similarity matrices Sp and Sg- for all queries and documents, respectively. The retriever learns to minimize (1) Leons: negative likelihood probability true positive documents; (2) Leons: the KL divergence between Sg and the rectified soft label Sg- after normalization.
2021; Li et al., 2022; Wang et al., 2022). In prac- tice, most training data pairs are collected automati- cally without manual inspection, such as inevitably leading to the inclusion of some mismatched pairs.
this further in Appendix C. In practice, we adopt the "In-Batch Negative" strategy for efficiency.
After obtaining the perplexity of each pair, an automated method is necessary to differentiate be- tween the noise and the clean data. We note that there is bimodal effect between the distribution of clean samples and the distribution of noisy samples. An example
3 Method
We propose NPC framework to learn retrievers with mismatched-pair noise. As shown in Fig. 3, NPC consists of two parts: (a) the noise detection mod- ule described in Sec. 3.1, and (b) the noise cor- rection module as described in See. 3.2.
be seen in Figure 4(b). Motivated by this, we fit the perplexity distribution over all training pairs with a two-component Gaussian Mix- ture Model (GMM):
3.1 Noise Detection
K
p (PPL 0) Σ T (PPL k), k=1
(3)
The noise detection module is meant to detect mis- matched pairs in the training set. We hypothesize that: dense retrievers will first learn to distinguish correctly matched pairs from easy negatives, and then gradually overfit the mismatched pairs. There- fore, we determine whether training pair is mis- matched by the perplexity between the annotated document and easy negative documents.
where and é (PPL k) are the mixture coeffi- cient and the probability density of the k-th com- ponent, respectively. We optimize the GMM with the SopecttationMaaximzzale ster al., 1977).
algorithm (Demp-
Based on the above hypothesis, we treat training pairs with higher PPL as noise and those with lower PPL as clean data. So the estimated clean flag can be calculated as follows:
Specifically, given a retriever 0 equipped with preliminary retrieval capabilities, and an uncertain pair (gi, d,), we calculate the perplexity as follows:
0. T. (p(ks) PPL > A)
(4)
PPL(g..d.0) = log
folondd) + + foland
where 인: E {1, 0} denotes whether estimate the pair (q:, d,) be correctly matched not, k is the GMM component with the lower mean, À is the threshold. p(k: (PL) the posterior probability over the component K, which can in- tuitively understood as the correctly annotated con-
(2) where 7 is a hyper-parameter, d is the negative document randomly sampled from the document collection ID. Note that di is a randomly selected negative document, not hard negative. We discuss

---

fidence. We set À to 0.5 in all experiments. Note that before noise detection, the retriever should equip with preliminary retrieval capabilities. This can be achieved by initializing it with strong unsu- pervised retriever or by pre-training on the entire noise dataset.
Algorithm 1 Noisy Pair Corrector (NPC)
Require: Retriever 0; Noisy Training dataset C.
Warm up the retriever A.
Initial EMA model 0 with 0;
for i num__poch do
4: Calculate PPL of training pairs with random negatives using Eq.2:
3.2 Noise Correction
5: Fit PPL distribution with GMM;
Next, we will introduce how to reduce the impact of noise pairs after obtaining the estimated flag set {0r)e- One quick fix is to discard the noise data directly, which is sub-optimal since it wastes the query data noisy pairs. In this work, we adopt a self-ensemble teacher to provide rectified soft labels for noisy pairs. The teacher is an ex- ponential moving average (EMA) of the retriever, and the retriever is trained with a weight-averaged consistency target on noisy data.
6: Get the estimated flag set {ús} using Eq.4;
for i = 1 wuum_batch do Sample negatives with "In-Batch Nega- tive" "Hard Negative" strategy; Calculate rectified soft labels with EMA model 8";
10: Train by optimizing Eq.8;
II: Update EMA model 0 using Eq.5:
12: end for
13:
end for
Specifically, given retriever 0, the teacher 6* is updated with exponential moving average strategy as follows:
estimated clean pair, we calculate the contrastive loss and consistency loss. So the overall loss is formalized:
0f = = 1 (1-a))n
(5)
where is a momentum coefficient. Only the pa- rameters 0 are updated by back- kack-propagation For a query qi and the candidate document set Da. where Dg = {d=g]]= could consist of an- notated documents, hard negatives and in-batch negatives, we first get teacher's and retriever's sim- ilarity scores, respectively. Then, the retriever is expected to keep consistent with its smooth teacher 0r To achieve this goal, we update the retriever 0 by minimizing the KI. divergence between the student's distribution and the teacher's distribution.
L = y Leant + Lecms:
(8)
where is E {1,0) is estimated by the noise detec- tion module.
3.3 Overall Procedure
NPC is a general training framework that can be easily applied to most retrieval methods. Under the classical training process of dense retrieval, We first warmup the retriever with the typical contrastive learning method to provide it with basic retrieval abilities, and then add the noise detection module before training each epoch and the noise correction module during training. The detail is presented in Algorithm
To be concrete, the similarity scores between z and Dy are normalized into the following distribu- tions:
(foooddes)
Po(di (ig Dz.) = Σ effaladdis E (0,0"),
Experiments
(6)
Then, the consistency loss Lroms can be written as:
4.1 Datasets
To verify the effectiveness of NPC in robust dense retrieval, we conduct experiments four commonly-used benchmarks, including Natural Questions (Kwiatkowski
Leons = K KL(po(-lg: Da,,, Po- (-195; D).)), (7)
al., 2019), Trivia QA (Joshi al., 2017), StaQC (Yao et al., 2018) and SO-DS (Heyman and Van Cutsem, 2020).
where KL(-) is the KL divergence, paç. Dy,))
and pg- (-19;; D denote the conditional probabili-
ties of candidate documents Dg by the retriever and the teacher 8", respectively.
StaQC a large dataset that collects real query- code pairs from Stack Overflow". The dataset has
For the estimated noisy pair, the teacher corrects the supervised signal into a soft label. For the
htttps://sttccooverffl

---

StaQC
SO-DS
Methods
R@3
R@ 10
MRR
R@3
R@ 10
MRR
dM25 (Heyman and Van Cutsem, 2020)
8.0
13.3
7.5
23.8
32.3
21.6
NBOW (Heyman and Van Cutsem, 2020)
10.9
16.6
9.5
27.7
38.0
24.7
USE (Heyman and Van Cutsem, 2020)
12.8
20.3
11.7
33.3
48.5
30.4
CodeBERT (Feng et al., 2020)
23.4
23.1
GraphCodeBERT (Guo al.. 2021)
24.1
25.2
CodeRetriever (In-Batch Negative) (Li et al., 2022)
25.5
27.1
CodeRetriever (Hard Negative) (Li et al., 2022)
24.6
31.8
UniXcoder (In-Batch Negative) (Guo al., 2022)
29.98
47.47
28.04
31,90
51.21
28.29
UniXeoder (Hard Negative) (Guo et al., 2022)
31.18
48.38
28.63
33.42
53.37
29.97
NPC (In-Batch Negative)
33.07
50.35
30.39
35.58
54.54
30.96
NPC (Hard Negative)
34.38
52.20
31.36
38.00
56.51
32.49
Table 1:
Retrieval performance on StaQC and SO-DS, which are realistic-noiss datasets. The results of the first block are mainly borrowed from published papers (Heyman and Van Cutsem, 2020; al., 2022). results are provided, mark them "_"
been widely used on code summarization (Ped- damail et al., 2018) and code search (Heyman and Van Cutsem, 2020). SO-DS mines query-code pairs from the most upvoted Stack Overflow posts, mainly focuses the data science domain. Fol- lowing previous works (Heyman and Van Cutsem, 2020; Li et al., 2022), we resort to Recall of top- k (R@k) and Mean Reciprocal Rank (MRR) as the evaluation metric. StaQC and SO-DS are con- structed automatically without human annotation. Therefore, there are numerous mismatched pairs in training data.
StaQC and SO-DS, we adopt UniXcoder (Guo et al., 2022) as our backbone, which is the SoTA model for code representation. Following Guo et al. similar- (2022), we adopt the cosine distance ity function and set temperature À to 20. We up- date model parameters using the Adam optimizer and perform early stopping on the development set. The learning rate, batch size, warmup epoch, and training epoch are set to 2e-5, 256, 5, and 10, respectively.
the "Hard Negative" setting, we adopt the same strategy as Li ct al. (2022).
For NQ and TQ, we adopt BERT (Devlin et al., 2019) et al. (2020a), we adopt inner-produce larity function and set temperature À to The max sequence length 16 for query and 128 for pas- sage. The learning rate, batch size, warmup epoch, and training epoch are set to 2e-5, 512, 10, and 40, respectively. We adopt "BM25 Negative" and "Hard Negative" strategies as described in the DPR toolkit For fair comparison, DPR (Karpukhin al., 2020a) with the same hyper- parameters. All experiments are run 8 NVIDIA Tesla A100 GPUs. The implementation of NPC is based on Huggingface (Wolf et al., 2020).
Natural Questions (NQ) collects real queries from the Google search engine. Each question is paired with an answer span and golden pas- sages from the Wikipedia pages. Trivia QA (TQ) is a reading comprehension dataset authored by trivia enthusiasts. During the retrieval stage of both datasets, the objective is to identify posi- tive passages from large collection. Positive pairs in these datasets are assessed based striet rule, i.c., whether passages contain answers or not (Karpukhin et al., 2020a). Consequently, we consider these datasets to be of high quality. Thus, we leverage them for simulation experiments to quantitatively analyze the impact of varying propor- tions of noise. Drawing inspiration from the setting in the noisy classification task (Han et al., 2018), we simulate the mismatched-pair noise by ran- domly pairing queries with unrelated documents.
initial model. Following Karpukhin the simi-
implement
4.3 Results
Results on StaQC and SO-DS: Table shows the results the realistic-noisy datasets StaQC and SO-DS. Both datasets contain large number of real noise pairs. The first block shows the results of previous SoTA methods. BM25dess is a traditional
4.2 Implementation Details
NPC a general training paradigm that can be di- rectly applied to almost all retrieval models. For
Itttps: http:///ccccccaaacccooooeeeeea
DPR

---

Natural Questions
Trivia QA
Noisy
Methods
R@!
R@5
R@20
R@ 100
R @
R@5
R@20
R@ 100
BM25+
59.1
73.7
66.9
76.7
DPR*
78.4
85.4
79.4
85.0
DPR (BM25 Negative)
27.07
47.79
63.36
75.69
35.73
52.88
64.05
74.16
coCondenser (BM25 Negative)
29.12
51.02
67.45
77.93
39.41
56.72
67.34
76.04
Co-teaching (BM25 Negative)
26.02
52.48
63.46
76.11
28.65
53.01
64.99
74.05
DPR-C (BM25 Negative)
43.69
66.62
79.07
86.12
52.10
70.52
79.05
85.08
NPC (BM25 Negative)
45.22
68.42
79.76
86.56
52.34
70.22
79.10
84.86
20
DPR (Hard Negutive)
37.61
60.73
71.68
79.56
43.39
60.67
70.34
77.88
coCondenser (Hard Negative)
40.71
63.41
74.33
81.22
47.42
64.80
73.38
80.07
RocketQA (Hard Negutive)
43.32
64.25
74.96
81.42
49.90
65.72
74.04
80.39
Co-teaching (Hard Negative)
31.78
56.32
66.12
77.56
33.28
57.29
66.50
75.62
DPR-C (Hard Negative)
51.66
72.40
81,50
87.80
55.35
72.36
80.33
85.34
NPC (Hard Negative)
51.85
73.06
82.47
87.80
56.03
72.54
80.59
85.58
DPR (BM25 Negative)
16.12
33.88
49.70
63.38
20.09
34.63
47.42
61.04
coCondenser (BM25 Negative)
18.28
36.37
52.01
65.92
22.80
38.01
51.00
63.79
Co-teaching (BM25 Negative)
23.72
50.32
64.86
74.92
26.56
51.22
63.78
73.77
DPR-C (BM25 Negative)
41.29
65.21
78.48
85.70
49.61
68.81
78.00
84,23
NPC (BM25 Negative)
42.87
65.65
78.37
85.76
50.80
68.98
78.21
84,43
50
DPR (Hard Negative)
23.87
42.34
55.12
67.06
28.47
45.12
56.88
67.62
coCondenser (Hard Negative)
24.55
44.16
56.69
68.72
31.05
47.81
59.48
70.14
RocketQA (Hard Negative)
26.83
45.72
57.32
69.24
33.67
49.28
60.32
70.46
Co-teaching (Hard Negative)
30.12
55.94
65.81
76.90
31.85
55.37
65.29
75.02
DPR-C (Hard Negative)
48.87
70.52
81.44
87.17
53.07
70.36
79.02
84.69
NPC (Hard Negative)
48.81
70.60
81.17
87.20
53.09
70.27
79.31
84.96
Table 2: Retrieval performance Natural Questions and Trivia QA under the noise ratio of 20%, and 50%,
respectively. The results of BM25* and DPR* are borrowed from Karpukhin et al. (2020a). If the results are not provided, we mark them "."
sparse retriever based on the exact term matching of queries and code descriptions. NBOW is unsupervised retriever that leverages pre-trained word embedding of queries and code descriptions for retrieval. USE a simple dense retriever based on transformer. CodeBERT, GraphCodeBERT are pre-trained models for code understanding using large-scale code corpus. CodeRetriever is pre- trained model dedicated to code retrieval, which is pre-trained with unimodal and bimodal contrastive learning large-scale corpus. UniXcoder is also pretrained model that utilizes multi-modal data, including code, comment, and AST, for better code representation. The results are implemented by ourselves for a fair comparison with NPC. The bottom block shows the results of NPC using two negative sampling strategies.
cates that the mismatched noise greatly limits the performance of dense retrieval models, and NPC mitigate this negative effect. We also show some noisy examples detected by NPC in Appendix A.
Results on NQ and TQ: Table 2 shows the results on the synthetic-noisy datasets NQ and TQ under the noise ratio of 20%, and 50%. We compare NPC with BM25 (Yang et al., 2017) and DPR (Karpukhin et al., 2020a). BM25 is an un- supervised sparse retriever that is not affected by noisy data. DPR (Karpukhin al., 2020a) is a widely used method for training dense retrievers. coCondenser (Gao and Callan, 2021b) leverage pre-training to enhance models' robustness. Rock- etQA (Qu et al., 2021) adopts a cross-encoder to fil- ter false negatives in the "Hard Negative" strategy. Co-teaching (Han et al., 2018) uses the samples with small loss iteratively train two networks. which is widely used in the noisy label classifi- cation task. We implement baselines using two negative sampling strategies. Besides, we evaluate DPR on clean datasets by discarding the synthetic- noisy pairs, denoted by DPR-C. DPR-C a strong
From the results, we can see that our proposed NPC consistently performs better than the evalu- ated models across all metrics. Compared with the strong baseline UniXcoder which ignores the hismatched-paii problem, NPC achieves signif- icant improvement with both "in-bated" negative" and "hard negative" sampling strategies. indi-

---

Methods
NQ
StaQC
Setting
R
R@5
R@20
R@ 100
De
Co
HN
R@20
R@ 100
R@1
R@3
n=1
50.58
69.93
79.87
84.96
48.22
62.31
18.08
31.09
n=5
50.03
69.64
80.17
85.76
V
55.90
69.33
18.51
31.01
n=10
50.07
69.93
80.07
85.89
75.19
83.31
20.05
32.71
n=20
38.09
60.31
72.00
80.07
V
V
77.50
84.79
20.70
33.55
54.63
65.54
18.66
31.74
Table 4: Performance of NPC on NQ dev set with dif-
58.63
69.06
19.35
32.09
ferent warmup epoch number
77.59
85.03
20.93
33.55
V
80.07
85.89
21.93
34.51
Table Ablation studies on StaQC dev set and NQ dev set under noise ratio of 50%.
baseline that is not affected by mismatched pairs.
(a) Before warmup
After warmup
We can observe that (1) As the noise ratio in- creases, DPR, coCondenser, and RocketQA expe- rience a significant decrease in performance. At a noise rate of 50%, they perform worse than un- supervised BM25. (2) Despite Co-teaching hav- ing good noise resistance, its performance is still low. This indicates that methods for dealing with label noise in classification are effective for re- trieval. (3) NPC outperforms baselines by a large margin, with only a slight performance drop when the noise increases. Even comparing DPR-C, NPC still achieves competitive results.
(c) DPR
(d) NPC
Figure 4: Perplexity distribution of training pairs under different settings.
experiment, we adopt "Hard Negative" sampling strategy. We find that NPC achieves good results when the warmup epoch is relatively small (1- (1-10). However, when the warmup epoch is too large, the performance will degrade. We believe that a pro- longed warmup causes overfitting to noise samples.
4.4 Analysis
Ablations of Noise Detection and Noise Correc- tion: To get better insight into NPC, we con- duct ablation studies on the realistic-noisy dataset StaQC and the synthetic-noisy dataset NQ under the noise ratio of 50%. The results are shown in Table 3. "De" and "Co" refer to noise detection and noise correction, respectively. "HN" indicates whether to perform "Hard Negative" strategy. For both synthetic noise and realistic noise, we can see that the noise detection module brings significant gain, no matter which negative sampling strategy is used. Correction also enhances the robustness of the retriever since it provides rectified soft labels which can lead the model output to be smoother. The results show that combining the two obtains better performance compared with only using the detection module or correction module.
Impact Iterative Detection: In the training of NPC, we perform iterative noise detection ev- cry epoch. A straightforward approach is to detect the noise only once after warmup and fix the esti- mated flag set { To study the effectiveness of iterative detection, conducted an ablation study. The results shown in Table 5. We see that the model performance degrades after removing iterative detection.
Ablations of PPL: We distinguish noise pairs according to the perplexity between the annotated positive document and easy negatives. When cal- culating the perplexity, "Hard Negative" will cause trouble for detection. We construct ablation exper- iments to verify this, and the results are shown Table 5. We can see that the perplexity with "Hard Negative" results in performance degradation.
Impact of Warmup Epoch: According to the foregoing, NPC starts by warming up. In Table we pre-training the retriever on the noisy dataset for warming up, and show the performance of NPC with different various epoch numbers n. In this
Visualization of Perplexity Distribution: Fig. 4, we illustrate the perplexity distribution of

---

Setting
R@: I
R@5
R@20
R@ 100
the model produce an information-rich CLS rep- resentation (Gao and Callan, 2021a,b; Lu et al., 2021). As for effective fine-tuning strategies (He et al., 2022b), recent studies show that negative sampling techniques are critical to the performance of dense retrievers. DPR (Karpukhin et al., 2020b) adopts in-batch negatives and BM25 negatives; ANCE (Xiong et al., 2021), RocketQA (Qu et al., 2021), and AR2 (Zhang et al., 2022a) improve the hard negative sampling by iterative replacement, denoising, and adversarial framework, respectively. Several works distill knowledge from ranker to re- triever (Izacard and Grave, 2020; Yang and Seo, 2020; Ren et al.. 2021; Zeng et al.. 2022). Some studies incorporate lexical-aware sparse retrievers to convey lexical-related knowledge to dense re- trievers, thereby enhancing the dense retriever's ability recognize lexical matches (Shen et al., 2023; Zhang al., 2023).
NPC
50.07
69.93
80.07
85 85.89
-wlo iterative detection
47.29
68.39
78.79
85.38
-ppl with HN
42.81
65.06
75.22
83.09
Table 5: Ablation studies of iterative noise detection and perplexity variants
(a) w/o Hard Negative
(b) Hard Negative
Figure 5: Retrieval performance of DPR and NPC on NQ dev set under different noise ratios.
training pains before and after warmup, after train- ing with DPR, and after training with NPC. The experiment is on NQ under the noise ratio of 50%. We can see that the perplexity of most noisy pairs is larger than the clean pairs after warmup, which veri- fies our hypothesis in Sec. 3.1. Comparing Fig. 4(c) and Fig. 4(d), we find that the retriever trained with DPR will overfit the noise pairs. However, NPC en- ables the retriever to correctly distinguish clean and noisy pairs because it avoids the dominant effect of noise during network optimization.
Although the above methods have achieved promising results, they are highly dependent on correctly matched data, which is difficult satisfy in real scenes. The mismatched-pair noise problem has seldom been considered. Besides, some stud- ies utilize large-sized generative models (He et al., 2023) to guide retrievers, which achieve impres- sive performance without paired data (Sachan et al., 2022, 2021; Gao et al., 2022; He et al., 2022a). Al- though these models exhibit some robustness to noisy data, their success depends the availabil- ity of strong generative models. Moreover, their applicability will be limited in domains where gen- erative models do not perform well.
Analysis of Generalizability Fig. 5 shows the performance of DPR and NPC under the noise ratio ranging from 0% 80%. We can see that as the noise ratio increases, the performance degradation of DPR is much larger than that of NPC, which demonstrates the generalizability of NPC. Further- more, even though NPC is designed to deal with mismatched-por noise, it achieves competitive re- sults when used in a noise-free setting.
5.2 Denoising Techniques
One related task to our work Noisy Label. Nu- merous methods have been proposed to solve this problem, and most of them focus on the classifica- tion task (Han et al., 2020). Some works design robust loss functions to mitigate label noise (Ghosh et al., 2017; Ma et al., 2020). Another line of work aims to identify noise from the training set with the memorization effect of neural networks (Silva et al., 2022: Liang et al., 2022: Bai et al., 2021). These studies mainly focus on classification. NPC studies the mismatched noise problem in dense retrieval rather than the noise in category annotations, which is more complex to handle. Sev- eral pre-training approaches noticed the problem of mismatched noisy pairs. ALIGN (Jia et al., 2021) and CLIP (Radford et al., 2021) claim that utilizing
5
Related Work
5.1 Dense Retrieval
Dense retrieval has shown better performance than traditional sparse retrieval methods (Lee et al., 2019; Karpukhin et al., 2020a). The studies of dense retrieval can be divided into two categories, (1) unsupervised pre-training to get better initial- ization and (2) more effective fine-tuning la- beled data. In the first category, some researchers focus on how to generate contrastive pairs auto- matically from a large unsupervised corpus (Lee et al., 2019; Chang et al., 2019; Ma et al., 2022; Li et al., 2022). Another line of research enforces

---

large-scale image-text pairs can ignore the exis- tence of noise. E5 (Wang al., 2022) employs a consistency-based rule to filter the pre-training data. Although they slightly realized the existence of noisy pairs during pre-train, none of them give a specialized solution to solve it and extensively ex- plored the characteristics of noisy text pairs. Some recent works (Huang al., 2021; Han et al., 2023) study the noisy correspondence problem in cross- modal retrieval. Although the "mismatched- noisy" problem in cross-modal retrieval and text retrieval shares similarities, the specific settings and methods used in these two areas are notably distinct. it is challenging to directly apply these mooss- retrieval works to document and code retrieval. Our NPC is the first systematic work to explore mismatched-par pair noise in document/code retrieval.
National Science Foundation of China under Grant 61836006.
References
Yingbin Bai, Erkum Yang. Bo Han, Yanhua Yang. Jia- tong Li. Yinian Mao, Gang Niu, and Tongliang Liu. 2021. Understanding and improving early stopping for learning with noisy labels. NIPS.
Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google dataset search: Building a search engine for datasets in an open web ecosystem. In WWW.
Wei-Cheng Chang, X Yu Felix, Yin-Wen Wen Chang. Yim- ing Yang. and Sanjiv Kumar. 2019. Pre-training tasks for embedding-Bee national Conference on Learning Representations.
large-scale retrieval. In Inter-
Xuanang Chen, Jian Luo, Ben He, Le Sun, and Yingfei Sun. 2022. Towards robust dense retrieval via local ranking alignment. In Proceedings of the Thirty- First International Joint Conference on Artificial Intelli- gence, IJCAI, pages 1980-1986.
6 Conclusion
Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Sociery: Series B (Methodological), 39(1):1-22.
This paper explores a neglected problem in dense retrieval, i.e., mismatched-patt -pirr noise. To solve this problem, we propose a generalized Noisy Pair Corrector(NPC) framework, which iteratively de- tects noisy pairs per epoch based on the perplex- ity and then provides rectified soft labels via an EMA model. The experimental results and analysis demonstrate the effectiveness of NPC in effectively handling both synthetic and realistic mismatched- pair noise.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2079 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers).
Zhangyin Feng, Daya Guo, Duyu Tang. Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- bert: A pre-trained model for programming and nat- ural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020.
Limitations
This work mainly focuses on training the dense retrieval models with mismatched noise. There may be two possible limitations in our study.
1) Due to the limited computing infrastructure, we only verified the robustness performance of NPC based on the classical retriever training frame- work. We leave experiments to combine NPC with more effective retriever training methods such as distillation (Ren et al., 2021), AR2 (Zhang et al., 2022a), future work.
Luyu Gao and Jamie Callan. 2021a. your language model ready for dense representation fine-tuning? arXiv preprint arXiv:2104.02255
Luyu Gao and Jamie Callan. 2021b. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. arXiv preprint arXiv:2108.0553 055400.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. CoRR.
2) Mismatched-per noise may also exist in other tasks, such as recommender systems. We will con- sider extending NPC to more tasks.
Aritra Ghosh. Himanshu Kumar, and Shanti Sastry. 2017. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence.
Acknowledgement
This work supported by the Fundamental Re- search Funds for the Central Universities under Grant 10822041 12364 and the Key Program of the
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross- modal pre-training for code representation. In ACL.

---

Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang. Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundare- san, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodehert: Pre-training code representations with data flow. In International Conference on Learn- ing Representations.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. sclle similarity search with gpus. IEEE Transactions Big Data, 7(3):535-547.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In ACL.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. 2020. A survey of label-noise representa- tion learning: Past, present and future. arXiv preprint OrXiv:2011.04400
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a. Dense passage retrieval for open-domain question answering. arXiv preprint NXiv:2004.04900
Bo Han. Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NIPS.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020b. Dense passage retrieval for open-domain question answering. In EMNLP.
Haochen Han, Kaiyao Miao, Qinghua Zheng, and Min- nan Luo. 2023. Noisy correspondence learning with meta similarity correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 7517-7526.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein. Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova. Llion Jones. Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Xingwei He, Yeyun Gong, A Jin, Weizhen Qi. Hang Zhang. Jian Jiao, Bartuer Zhou, Biao Cheng, Siu Ming Yiu, Nan Duan, et al. 2022a. Metric-guided distillation: Distilling knowledge from the metric to ranker and retriever for generative commonsense rea- soning, arXiv preprint arXiv:2210.11 17708.
Kenton Lee, Ming-Wor Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Xingwei He, Yeyun Gong, A Jin, Hang Zhang, An- lei Dong, Jian Jiao, Siu Ming Yiu, Nan Duan, al. 2022b. Curriculum sampling for dense re- trieval with document expansion. arXiv preprint arXiv:2212.09114.
Xiaonan Li, Yeyun Gong. Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi. Daxin Jiang, Weizhu Chen, and Nan Duan. 2022. Coderetriever: Unimodal and bimodal contrastive learning. arXiv preprint arXiv:2207. 10866.
Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.1 16854.
Kevin Liang, Samrudhdhi B. Rangrej. Vladan Petrovic, and Tal Hassner. 2022. Few-shot learning with noisy labels. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9079- 9088.
Geert Heyman and Tom Van Cutsem, 2020. Neural code search revisited: Enhancing code snippet retricval through natural language intent. arXiv preprint arXiv:2008. 12193.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arxiv: 1907.1
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. 2021. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Process- ing Systems, 01-29400-294999
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain a strong siamese encoder for dense text retrieval using weak decoder. In Empirical Methods in Natural Language Processing.
Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question an- swering, arXiv preprint 204:22012.0484..
Chao Jia. Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and lasion-language representation learning with noisy text supervision. International Conference on Machine Learning.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Ro- mano, Sarah Erfami, and James Bailey. 2020. Nor- malized loss functions for deep learning with noisy labels. In ICMI.

---

Xinyu Ma. Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-train a discriminative text encoder for dense retrieval via contrastive span pre- diction. In SIGIR 22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-thhe the-art natural language processing. In EMNLP.
Jayavardhan Reddy Peddamail, Ziyu Yao, Zhen Wang. and Huan Sun. 2018. A comprehensive study of staqe for deep code summarization. In KDD.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. In International Conference on Learning Representa- lions.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In NAACL-HLT.
Peilin Yang. Hui Fang. and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML.
Sohee Yang and Minjoon Seo. 2020. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010. 10999.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang. and Ji-Rong Wen. 2021. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2825-2835.
Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. Stage: A systematically mined question- code dataset from stack overflow. In wwW.
Hansi Zeng, Hamed Zamani, and Vishwa Vinay. 2022. Curriculum learning for dense retrieval distillation. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, Madrid, Spain, July 11 15, 2022.
Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2022. Questions are all you need to train a dense passage retriever. CoRR.
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping. William L. Hamilton, and Bryan Catanzaro. 2021. End-to-end training of neural retrievers for open- question answering. In ACI/IJCNIP.
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li. Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Poolingformer: Long document modeling with pool- ing attention. In International Conference on Ma- chine Learning, pages 12437-12446. PMLR.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Kai Zhang, and Daxin Jiang. 2023. Unifier: A unified retriever for large-scale retrieval. In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, pages 4787-4799.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lx, Nan Duan, and Weizhu Chen. 2022a. Adversarial retriever-ranker for dense text retrieval. In Interna- tional Conference on Learning Representations.
Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. 2023. Led: Lexicon-enlightened dense retriever for large-scale retrieval. In Proceedings of the ACM Web Conference 2023, pages 9203-3213.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng. and Grégoire Mesnil. 2014. Learning semantic rep- resentations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pages 373-374.
Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2022b. Multi-view document repre- sentation learning for open-domain dense retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2022. Noise-robust learning from multiple unsupervised sources of inferred la- bels. In AAAI Conference on Artificial Intelligence.
Liang Wang, Nan Yang. Xiaolong Huang. Binx- ing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. ArXiv, abs/2211.03533.

---

A Qualitative Analysis
D Integration with stronger methods
We conducted comprchensive experiments that in- tegrated NPC into coCondenser and RocketQAv2. The subsequent experiments were conducted on the NQ dataset with a 50% noise ratio. We first combined NPC with coCondenser which is pre- trained model specialized for dense retrieval tasks.
Table lists some mismatched pairs detected by NPC in StaQC training set. We can see that these mismatched pairs are almost irrelevant and can be correctly detected by NPC. These examples are not well aligned, mainly due to the low-quality answers of the open community (cases 2 and 4), inappropriate data preprocessing in the collection phase (cases 2 and 3), and other reasons. It is well known that collecting and cleaning training data is expensive and complex work. Automatically con- structed datasets in real-world applications often contain such mismatched-pat noise. Our method can mitigate the impact caused by such noise dur- ing training.
The results are shown Table
R@ 1
R@5
R@20
R@ 100
BM25 Negative
coCondenser
18.28
36.37
52.01
65.92
coCondenser-C
44.75
68,91
80.89
87.31
coCondenser-NPC
47.31
70.38
81.58
87.38
Hard Negative
coCondenser
24.55
44,16
56.69
68.72
coCondenser-CC
49.31
71.99
82.41
88.38
coCooddnser+NNCC
50.66
72.42
82.64
88.31
Statistics of Datasets
Table 7: Retrieval performance on NQ after combining NPC with coCondenser.
The statistics of datasets are shown in Table 6.
It's evident that NPC significantly enhances the robustness of coCondenser against noise associ- ated with mismatched pairs. This observation un- derscores the compatibility between NPC and pre- trained dense retrievers.
Dataset
Train
Dev
Test
Corpus size
StaQC
203.7K
2.6K
2.7K
14.6K
SO-DS
12.1K
0.9K
1.1K
12.1K
NQ
79.2K
8,8K
3.6K
21 M
TQ
78.8K
8.8k
11.3K
2
Furthermore, we combined NPC with Rock- etQAv2 which adopted a cross-encoder as a teacher and dynamically distilled knowledge to the dense retriever. The results are shown in Table 8:
Table 6: The statistics of datasets. Corpus size means the size of document corpus for evaluation.
R@D 1
R@5
R@ 20
R@ 100
RocketQAv2
32.30
51.37
62.19
70.79
Discussion about Perplexity
RocketQAv2-C
52.63
73.51
83.21
88.71
Rocke1QAv2+NPC
52.59
73.83
83.32
88.69
We calculate the perplexity between the annotated document and easy negative documents during noise detection. We emphasize that the negative documents are randomly selected from the docu- ment collection ID: It is not suitable adopt "Hard Negative" sampling strategy when calculating the perplexity. Although hard negatives are important to train a strong dense retriever, they will cause trouble during noise detection. Specifically, it is ex- pected that the retriever confused only between false positive and negative documents and can con- fidently distinguish true positive and negative doc- uments. But if adopt "Hard Negative" when calculating the perplexity, the retriever will also be confused between true positive and hard negative documents, which will affect noise detection. We construct ablation experiments to verify this, and the results are shown in Table
Table 8: Retrieval performance NQ after combining NPC with RocketQAv2.
To combine NPC with RocketQAv2, we inte- grate the noise detection and the correction mod- ules in each training epoch of RocketQAv2. From the table, we can find that although RocketQAv2 powerful cross-encoder still limited by the noise of the training data and shows low performance, NPC can effectively har- monize with RocketQAv2 to mitigate the problems caused by mismatched pair noise. We will add these experiment results to the next version.
teacher, it is

---

Question
Code
Split words in nested list into letters
[list([[O]) for in mylist]
Dictionary python problem
getStartState()
Find the Common first name from Django Auth user Model
import operator
1st [1,2,4.6.3.8.0,,]]
for in lst[:]:
Find all text files not containing some text string
if n % 2 == O:
**> rstroem)
>> Ist
Table
Some noisy pairs detected by NPC in StaQC training set.


Reviews:
[{"review_id": "zSssCInhfZ", "reviewer": "Reviewer_8vRk", "summary": "", "questions": "A: Is there any direct evidence to support the assumption mentioned in W2, though it is intuitive? \n\nB: What is the difference between the noisy label problem in this study and the ones in [1-3]?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n> *Q1*:  Is there any direct evidence to support the assumption \"dense retrievers will first learn to distinguish correctly matched pairs from easy negatives, and then gradually overfit the mismatched pairs\".\n\nWe are grateful for your insightful question. In Figures 4 of our paper, we present evidence that addresses your concern. \n\nFigure 4.a and Figure 4.b illustrate the perplexity distributions for both correctly matched and mismatched query-document pairs before and after the warmup stage, respectively. We can find that Figure 4.a shows a narrow divergence between the perplexity distribution of correctly matched pairs and that of mismatched pairs, indicating the model can not distinguish them before training. After a few epochs, a remarkable shift becomes apparent. In Figure 4.b, the perplexity of correctly matched pairs experiences a rapid decrease, signifying the model's improving ability to distinguish them accurately. In contrast, the perplexity of mismatched pairs exhibits a contrasting trend. These empirical findings support our assumption *\"dense retrievers will first learn to distinguish correctly matched pairs from easy negatives\"*.\n\nFurthermore, Figure 4.c shows the perplexity distribution of query-document pairs when we continue training the retriever with the original contrastive learning objective. We can find that the perplexity of mismatched pairs is notably reduced compared to that in Figure 4.b. More remarkably, the perplexity of mismatched pairs is reduced to a range that is indistinguishable from correctly matched pairs. This phenomenon intuitively supports our assumption *\"then gradually overfit the mismatched pairs\"*.\n\n\n>*Q2*:  Similar ideas in cross-modal retrieval area[1-3].\n\nThanks for pointing out these missing cross-modal retrieval works, and we will incorporate discussions with them into \"Related Work\" section. Although the \"mismatched-pair noisy\" problem in cross-modal retrieval and text retrieval shares similarities, the specific settings and methods used in these two areas are notably distinct.\n\nPrimarily, the retrieval objective of dense text retrieval is unidirectional, i.e.,  query -> document, while that of cross-modal retrieval is bidirectional, i.e.,  text<->image. Dense text retrieval usually incorporates large document candidate sets for evaluation and hard negative mining, whereas cross-modal retrieval focuses on smaller candidate sets[1][2][3] (around 5k for the MSCOCO dataset). \n\nIn the area of dense text retrieval, the commonly used method is to employ contrastive loss combined with static global hard negatives mined from large document corpora (about 20 million). On the other hand, cross-modal retrieval commonly employs variations of the triplet loss[1][2][3], which relies solely on in-batch negatives and does not make use of global hard negatives. Moreover, in the context of dense text retrieval, there is a tendency to adopt cross-encoder rankers for distillation, whereas this practice is less common in cross-modal retrieval. \n\nConsidering these distinct methodological conventions, it is challenging to directly apply these cross-modal retrieval works to document and code retrieval. We would like to highlight that NPC is the first systematic work to explore mismatched-pair noise in document/code retrieval. NPC is suitable for a variety of existing dense text retrieval methods (i.e., DPR+Hard Negative, coCondenser, RocketQAv2), specific results can be found in our reply Reviewer#3 Q3. Our work also encompasses an in-depth analysis, yielding insightful findings. For instance, we recommend the use of random simple negative samples rather than hard negatives for noise detection. We highlight the effectiveness of incorporating correction modules to enhance model performance. \n\nWe believe that our research and findings on “mismatched-pair noise” are contributing to the development of the community in the direction of text/code retrieval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: The problem of mitigating the effects of the mismatched-pair noise is very interesting since it's very ubiquitous but hard to deal with.\n\nS2: The proposed method has good mathematical intuition. The design is straightforward and easy to follow.\n\nS3: The experimental results are pretty good compared to the generally comprehensive baselines.", "reasons_to_reject": "W1: Similar ideas have already been adopted in cross-modal retrieval tasks, like [1-3] listed in the missing reference, which hurts the novelty of the paper.\n\nW2: The assumption in section 3.1 : \"dense retrievers will first learn to distinguish correctly matched pairs from easy negatives, and then gradually overfit the mismatched pairs\" needs to be explained more thoroughly by some specific empirical analysis, since it serves as the basis of this study. \n\n", "questions_to_authors": "A: Is there any direct evidence to support the assumption mentioned in W2, though it is intuitive? \n\nB: What is the difference between the noisy label problem in this study and the ones in [1-3]?", "paper_topic_and_main_contributions": "The paper focuses on an important problem in dense retrieval: training an effective model in the presence of mismatched-pair noise. To address this challenge, the authors introduce a novel approach called Noisy Pair Corrector (NPC), which has proven effective in handling both synthetic and realistic noise. The authors validate this approach through experiments on various text-retrieval and code-search benchmarks, such as Natural Question, TriviaQA, StaQC, and SO-DS. The experimental results showcase that NPC achieves good performance, representing an advancement in the field of dense retrieval with noisy data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Learning with Noisy Correspondence for Cross-modal Matching. Proceeding of 35th Conference on Neural Information Processing Systems (NeurIPS 2021).\n\n[2] Learning Cross-Modal Retrieval with Noisy Labels. CVPR 2021.\n\n[3] Noisy Correspondence Learning with Meta Similarity Correction. CVPR 2023. "}}, {"review_id": "GIwLt8pfCI", "reviewer": "Reviewer_7QqM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n> *Q1*: Lack comparisons with other state-of-the-art models such as RocketQAv2 [1], LED [2], and Unifier [3].\n\n\nTo illustrate \"mismatched-pair noise\" problem more intuitively, we chose the most classical retriever training framework rather than other more complex training frameworks. We would like to highlight that NPC is orthogonal to these state-of-the-art methods. In fact, NPC can be easily applied to these models, helping them to effectively address noise challenges. \n\nThanks for pointing out the missing related models. The next version will add the discussions with these works. Due to the time limitation, we report the experiment results of RocketQAv2 on the NQ dataset under 20% and 50% noisy ratios. We follow the setting in RocketQAv2, wherein we jointly train a cross-encoder ranker and a dual-encoder retriever. During training, the ranker will dynamically distill knowledge to the dense retriever. To combine NPC with RocketQAv2, we integrate the noise detection and the correction modules in each training epoch of RocketQAv2.\n\n|   NQ Test(20% Noise)   |R@1  | R@5  | R@20 | R@100 |\n| ---- | ---- | ---- | ---- | ---- |\n| RocketQAv2 | 46.62 | 66.24 | 75.63 |82.79 |\n| RocketQAv2+NPC | 55.83| 75.79| 84.68 |89.25 |\n\n\n|   NQ Test(50% Noise)   |R@1  | R@5  | R@20 | R@100 |\n| ---- | ---- | ---- | ---- | ---- |\n| RocketQAv2 | 32.30 | 51.37 | 62.19 |70.79 |\n| RocketQAv2+NPC | 52.59| 73.83| 83.32 |88.69 |\n\nWe can see that RocketQAv2+NPC outperforms RocketQAv2. Although RocketQAv2 uses a powerful cross-encoder as a teacher, it is still limited by the noise of the training data and shows low performance. Our NPC can combine this advanced method to mitigate the problems caused by mismatched pair noise.\n\n\n\n> *Q2*: The model training used 8 NVIDIA Tesla A100 GPUs and 10 epochs, resulting in high resource consumption.\n\nThe training hyperparameters we adopted are following DPR. We only carry out warmup in the first 10 epochs, and carry out noise detection and correction in the last 30 epochs. Therefore, compared with the classical two-tower model training framework, there is no much higher training cost added. Each epoch takes about 40s on 8\\*A100. Note our experiments are adaptable and can be conducted with less demanding hardware resources, should the need arise. \n\n> *Q3*: Experimental results on the MSMARCO, TREC DL 2019, and TREC DL 2020 datasets.\n\nIn our initial version, we followed classic DPR to design experiments. As suggested, we have added the experiments on MSMARCO and TREC DL 2019 under a 20% noisy ratio. we implement the classical dense retriever training method, which uses bert-base-uncased mode to initialize our model, using contrast learning combined with one BM25 hard negative to train the dense retriever. The results are shown below:\n|      |MRR@10  | Recall@50  | Recall@1k |  NDCG@10 of TREC-DL-19 |NDCG@10 of TREC-DL-20|\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| DPR (BM25 Negative)  | 23.14     | 65.0     | 85.4     |56.0 | 55.1 |\n| DPR+NPC (BM25 Negative) | 31.93     | 80.6   | 96.4     | 66.5 | 65.6 |\n\nFrom the table, we can observe that our method NPC still outperforms DPR on the new datasets MSMARCO, TREC DL 2019, and TREC DL 2020. That indicates the effectiveness of NPC in dealing with mismatched pair noise.\n\n> *Q4*: Reproducibility.\n\nWe have already uploaded essential code components, and after the anonymous review period, we plan to make the dataset, preprocessing scripts, complete codebase, and model checkpoints publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a novel framework called Noisy Pair Corrector (NPC), which consists of a noise detection module and a noise correction module.\n- The writing and organization of the paper are well, making it easy to follow and understand.", "reasons_to_reject": "- The experimental results lack comparison with other state-of-the-art models such as RocketQAv2 [1], LED [2], and Unifier [3].\n- The model training used 8 NVIDIA Tesla A100 GPUs and 10 epochs, resulting in high resource consumption.\n- The paper lacks experimental results on the MSMARCO, TREC DL 2019, and TREC DL 2020 datasets, which could provide better validation of the model's performance.\n\n[1] RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. EMNLP 2021.\\\n[2] LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval. WWW 2023.\\\n[3] Unifier: A Unified Retriever for Large-Scale Retrieval. KDD 2023", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel framework called Noisy Pair Corrector (NPC), which consists of a noise detection module and a noise correction module. The noise detection module estimates the mismatched pairs in the training data by calculating the perplexity between the annotated positive document and easy negative documents. The noise correction module uses an exponential moving average model to provide rectified soft labels for both noisy and clean data.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. EMNLP 2021.\\\n[2] LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval. WWW 2023.\\\n[3] Unifier: A Unified Retriever for Large-Scale Retrieval. KDD 2023"}}, {"review_id": "NkVsZkjKN2", "reviewer": "Reviewer_YGVa", "summary": "", "questions": "Q1: What is a simple negative sample? The text does not state its definition. Is a negative sample a simple negative sample if it is not a hard negative sample?\n\nQ2: What is the exact expression of $L_{cont}$? I only found its textual description in Figure 3\n\nQ3: Are the results in the paper new SOTA? This part is too vague in its description", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and suggestions!\n\n> *Q1*: This paper may be the first study of the \"mismatched-pair noise\" problem in the field of document retrieval and code retrieval. Similar work exists in cross-modal areas.\n\nWe appreciate your observation, and indeed, NPC is the first comprehensive study of the \"mismatched-pair noise\" issue within the domains of document retrieval and code retrieval. Thanks for pointing out these missing cross-modal retrieval works, and we will incorporate discussions with them into the related work section. While the \"mismatched-pair noisy\" problem in cross-modal retrieval and text retrieval shares similarities, the specific settings and methods used in these two areas are notably distinct.\n\n\nPrimarily, the retrieval objective of dense text retrieval is unidirectional, i.e.,  query -> document, while that of cross-modal retrieval is bidirectional, i.e.,  text<->image. Dense text retrieval usually incorporates large document candidate sets for evaluation and hard negative mining, whereas cross-modal retrieval focuses on smaller candidate sets (around 5k for the MSCOCO dataset). \n\nIn the area of dense text retrieval, the commonly used method is to employ contrastive loss combined with static global hard negatives mined from large document corpora (about 20 million). On the other hand, cross-modal retrieval commonly employs variations of the triplet loss, which relies solely on in-batch negatives and does not make use of global hard negatives. Moreover, in the context of dense text retrieval, there is a tendency to adopt cross-encoder rankers for distillation, whereas this practice is less common in cross-modal retrieval. \n\nConsidering these distinct methodological conventions, it is challenging to directly apply these cross-modal retrieval works to document and code retrieval. We would like to highlight that NPC is the first systematic work to explore mismatched-pair noise in document/code retrieval. NPC is suitable for a variety of existing dense text retrieval methods (i.e., DPR+Hard Negative, coCondenser, RocketQAv2). Our work also encompasses an in-depth analysis, yielding insightful findings. For instance, we recommend the use of random simple negative samples for noise detection, and we highlight the effectiveness of incorporating correction modules to enhance model performance. \n\nWe believe that our research and findings on \"mismatched-pair noise\" are contributing to the development of the community in the direction of text/code retrieval.\n\n\n> *Q2*: NPC does not achieve the purpose of correction. The gap between the NPC results and the DPR-C results is weak, which seems the correction module does not work. \n\nWe understand your concern about the effectiveness of the correction module. However, we would like to clarify that the correction module does indeed yield positive results. The specific improvements can be found in Table 3 of our paper's ablation experiments. We can find that the correction module brings consistent improvements, including +2.5 on NQ R@20, +0.89 on NQR@100, +1.0 on StaQC R@1 and +0.96 on StaQC R@3.\n\nWe believe that *the weak gap between the NPC results and the DPR-C results* cannot be taken as evidence that *the correction module is ineffective*. The performance difference between NPC and DPR-C can be attributed to two main factors: (1) the negative impact of noise that NPC's noise detection module might have missed, and (2) the positive impact of the correction module. Although NPC's noise detection module is effective, it might not identify every instance of noise, resulting in a small fraction of noise remaining in the training data. Without the correction module, compared to DPR-C trained on clean data, the model's performance could degrade. The enhancement brought by the correction module helps bridge this gap, enabling the model to achieve performance similar to DPR-C. Hence, the correction module is indeed effective.\n\n> *Q3*:  Apply NPC to a variety of existing methods to highlight the plug-and-play nature.\n\nThank you for raising this point. We sincerely appreciate your suggestion to explore the plug-and-play nature of NPC. Following your suggestion, we conducted comprehensive experiments that integrated NPC into coCondenser and RocketQAv2. The subsequent experiments were conducted on the NQ dataset with a 50% noise ratio.\nWe first combined NPC with coCondenser which is a pre-trained model specialized for dense retrieval tasks. The results are shown below.\n\n|      |R@1  | R@5  | R@20 | R@100 |\n| ---- | ---- | ---- | ---- | ---- |\n| coCondenser (BM25 Negative) | 18.28 | 36.37 | 52.01 | 65.92 |\n| coCondenser-C (BM25 Negative) | 44.75     | 68.91 |80.89|87.31     |\n| coCondenser+NPC (BM25 Negative) | 47.31     | 70.38     | 81.58 |87.38 |\n\n|      |R@1  | R@5  | R@20 | R@100 |\n| ---- | ---- | ---- | ---- | ---- |\n| coCondenser (Hard Negative) | 24.55 |44.16 |56.69 |68.72 |\n| coCondenser-C (Hard Negative) | 49.31 | 71.99 |82.41|88.38 |\n| coCondenser+NPC (Hard Negative) | 50.66 |72.42|82.64|88.31  | \n\nFrom the above tables, it's evident that NPC significantly enhances the robustness of coCondenser against noise associated with mismatched pairs. This observation underscores the compatibility between NPC and pre-trained dense retrievers.\n\nFurthermore, we combined NPC with RocketQAv2 which adopted a cross-encoder as a teacher and dynamically distilled knowledge to the dense retriever. The results are shown below:\n\n|      |R@1  | R@5  | R@20 | R@100 |\n| ---- | ---- | ---- | ---- | ---- |\n| RocketQAv2 | 32.30     | 51.37     | 62.19     |70.79     |\n| RocketQAv2-C | 52.63     | 73.51     | 83.21     |88.71 |\n| RocketQAv2+NPC  | 52.59     | 73.83     | 83.32 |88.69 |\n\nTo combine NPC with RocketQAv2, we integrate the noise detection and the correction modules in each training epoch of RocketQAv2. From the table, we can find that although RocketQAv2 uses a powerful cross-encoder as a teacher, it is still limited by the noise of the training data and shows low performance. NPC can effectively harmonize with RocketQAv2 to mitigate the problems caused by mismatched pair noise. We will add these experiment results to the next version.\n\n\n\n\n> *Q4*:  The \"warmup\" and \"iterative detection\" are not mentioned before the experiment section.\n\nThe \"warmup\" refers to our initial training stage of the retriever with the traditional contrastive learning method, which provides it with basic retrieval abilities. The \"iterative detection\" means that we perform noisy detection at each epoch. We will provide additional explanations in Section 3.3 \"Overall Procedure\" to enhance clarity.\n\n> *Q5*: What is a simple negative sample?  Is a negative sample a simple negative sample if it is not a hard negative sample?\n\nYes. As explained in line 169 of our paper and further detailed in Appendix C, a \"simple\" negative document refers to a randomly selected negative document. We will make it clearer in the next version.\n\n> *Q6*: What is the exact expression of $L_{cont}$? \n\n$L_{cont} = - \\log p_\\theta(d_i^+|q_i,D_{q_{i}}) $\n\n$L_{cont}$ is the contrastive loss, where $d_i^+$ is the annotated positive document of question $q_i$, $D_{q_i}$ is the candidate collection, consisting of an annotated positive document and negative documents. \n\n> *Q7*: Are the results in the paper new SOTA? \n\nThe results presented in our paper do not claim to establish new state-of-the-art (SOTA). It's important to note that SOTA methods often leverage techniques such as distillation, data augmentation, and scaling up model sizes to achieve improved performance. Yet, these approaches do not directly align with the core problem we are addressing, making direct and equitable comparisons challenging. In our experiments, we adopt the classical contrastive learning framework as the baseline for a fair comparison. \n\nMoreover, as detailed in our response to Q2, we have compared the compatibility of NPC with more powerful methods (pre-trained retriever coCondenser, RocketQAv2 utilizing cross-encoder distillation), which underscores the effectiveness of our approach. We will add these experiment results to the next version and make the description clearer. \n\n**References**:\n\n[1] RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. EMNLP 2021.\n\n[2] coCondenser: Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. ACL 2022.\n\n[3] DPR: Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Too little research has been done on the problem of mismatched-pair noise, and this article is a keen probe into this pervasive and unresolved problem in real-world data.\n\n2. The experimental part of the article is well-detailed and shows the reader the effectiveness of the method in various aspects.\n\n3. The experimental results have been shown to be valid in a variety of task settings, demonstrating the applicability of this method in different domains", "reasons_to_reject": "1. I believe the first point of the contribution overstates the contribution of this paper. As far as I know, this paper may be the first study of the \"mismatched-pair noise\" problem in the field of document retrieval and code retrieval, but similar work already exists on other dense retrieval tasks, e.g., [1.2].\n2. The method proposed in this paper is called \"Noisy Pair Corrector\", but I think NPC does not achieve the purpose of correction. According to the results in Table 2, even though the proportion of noisy labeled pairs has come to 50%, the gap between the NPC results and the DPR-C results is weak, which seems to imply that the supervisory signals of Eq. 7 have almost no improvement on the results.\n3. The different sections of this paper are not organized closely enough. On the one hand, the \"warmup\" and \"iterative detection\" of the experiment section, which are described at great length, are not even mentioned before the experiment section, and I can't even fully understand what they mean. On the other hand, the paper highlights the plug-and-play nature of NPC in several places, so I was expecting to see the authors applying their NPC to a variety of existing methods in the experimental section, but the paper only experimented with one baseline on each dataset.\n\n[1] Learning with Noisy Correspondence for Cross-modal Matching. NeurIPS2021\n\n[2] Deep Evidential Learning with Noisy Correspondence for Cross-modal Retrieval. ACM MM2022", "questions_to_authors": "Q1: What is a simple negative sample? The text does not state its definition. Is a negative sample a simple negative sample if it is not a hard negative sample?\n\nQ2: What is the exact expression of $L_{cont}$? I only found its textual description in Figure 3\n\nQ3: Are the results in the paper new SOTA? This part is too vague in its description", "paper_topic_and_main_contributions": "This paper proposes a method called Noisy Pair Corrector for eliminating the effect of pseudo-positive examples on the fine-tuning step of dense retrieval fine-tuning. In principle, this method is simple enough to be theoretically applied to current dense retrieval models of multiple architectures. The experimental results show it has a large improvement in the data in a variety of areas.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See the \"Reasons To Reject\" part", "Typos_Grammar_Style_and_Presentation_Improvements": "I don't think perplexity can be defined casually since he already has a standard definition in the language modeling field"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies the noisy label issue in dense retrieval, which is neglected in the existing work. However, the technical novelty of the proposed method is limited since similar technique has been proposed in similar IR tasks. While the rebuttal has highlighted some differences between the proposed method and existing methods, these points are minor, making this work a more incremental one.
Decision: Accept-Findings