Rethinking Negative Pairs in Code Search
Authors: Haochen Li
Xin Zhou
Anh Tuan Luu
Chunyan Miao
Venue: EMNLP 2023 Main (2023)
Keywords: Code Search
Contrastive Learning

Abstract: Recently, contrastive learning has become a key component in fine-tuning code search models for software development efficiency and effectiveness. It pulls together positive code snippets while pushing negative samples away given search queries. Among contrastive learning, InfoNCE is the most widely used loss function due to its better performance. However, the following problems in negative samples of InfoNCE may deteriorate its representation learning: 1) The existence of false negative samples in large code corpora due to duplications. 2). The failure to explicitly differentiate between the potential relevance of negative samples. As an example, a bubble sorting algorithm example is less ``negative'' than a file saving function for the quick sorting algorithm query. In this paper, we tackle the above problems by proposing a simple yet effective Soft-InfoNCE loss that inserts weight terms into InfoNCE. In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE. Theoretically, we analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing a more precise mutual information estimation. We furthermore discuss the superiority of proposed loss functions with other design alternatives. Extensive experiments demonstrate the effectiveness of Soft-InfoNCE and weights estimation methods under state-of-the-art code search models on a large-scale public dataset consisting of six programming languages.

Rethinking Negative Pairs in Code Search
Haochen Li
Xin Zhou² Luu Anh Tuan' Chunyan Miao
SScoool of Computer Science and Engineering, Nanyang Technological University, Singapore 2 AAlibaba-NTU Singapore Joint Research Institute, Nanyang Technological University, Singapore {haochen003, xin. zhou, anhtuan. luu, ascymiao)entu. www.:ettuuuuuueeeeemmmem sg
Abstract
Recently, contrastive learning has become a key component in fine-tuning code search models for software development efficiency and effec- tiveness. pulls together positive code snippets while pushing negative samples away given search queries. Among contrastive learning, InfoNCE is the most widely used loss function due to its better performance. However, the following problems in negative samples of In- foNCE may deteriorate its representation learn- ing: 1) The existence of false negative samples large code corpora due to duplications. 2). The failure to explicitly differentiate between the potential relevance of negative samples. As an example. bubble sorting algorithm exam- ple is less "negative" than file saving func- tion for the quick sorting algorithm query. In this paper, tackle the above problems by proposing simple yet effective Loft-InfoNCE loss that inserts weight terms into InfoNCE. In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is special case of Soft-InfoNCE. Theoreti- cally, we analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing more pre- cise mutual information estimation. We fur- thermore discuss the superiority of proposed loss functions with other design alternatives. Extensive experiments demonstrate the effec- tiveness of Ioft-InfoNCE and weights estima- tion methods under state-of-the-art code search models on large-scale public dataset consist- ing of six programming languages. Source code is available at //tps://gtt c www.-aachhenniSfffIIIoooooo
O
Query Positive Code Negative Code
Figure 1: Contrastive learning pushes away negative
pairs in the representation space. Left: Existing works treat negative pairs equally. Right: Negative pairs should be pushed away according to their similarity with the query. A thicker arrow means that this sample is more negative than others.
and Pradel, 2022). To train fine-tune code search models, contrastive learning has become a key com- ponent in learning discriminative representations of queries and codes as it pushes apart negative query-code pairs and pulls together positive pairs (Shi et al., 2022a; Li et al., 2022a). InfoNCE (Van den Oord et al., 2018) is a representative choice of contrastive learning loss that considers the other in-batch samples as negative pairs given query (Huang et al., 2021).'
Although InfoNCE is effective in code search, we argue sub-optimal at discriminating code samples as it suffers from the following problems. First, the existence of false negatives in code bases. Lopes et al. (2017); Allamanis (2019) finds that code duplications are common in large code corpus, which means that many negative pairs are in fact false negatives. Training with false negative pairs may deteriorate code representation learning. Sec- ond, the setting of InfoNCE ignores the potential relevance negative codes (Li et al., 2022c). For example, for given query asking about quick sort-
1 Introduction
Code search is common activity in software devel- opment that can boost the productivity of software developers (Nie et al., 2016; Shuai al., 2020). Code search models can retrieve code fragments relevant to a given query from code bases (Grazia
'We conduct a survey to demonstrate the dominant adop- tion of InfoNCE, See Appendix A for details.

---

ing algorithms, among negative codes the bubble sorting algorithm expected to be retrieved before file-saving functions, but the current training proce- dure camnot model this relationship explicitly. The InfoNCE loss just treats all negative codes equally, as shown in Fig.1. In fact, the false negative can- cellation is a special case of modeling potential relevance, where the former only considers poten- tial relevance binary while the latter focus on describing continuous field. Although some methods (Huynh et al., 2022; Chen al., 2022; Li et al., 2022c) are proposed to solve these two prob- lems, they are all applied during model pre-training. In model fine-tuning stage. they still use InfoNCE loss. As a result, our aforementioned two problems are still largely unexplored in model fine-tuning stage.
bias in mutual information estimation. We also prove the superiority of Soft-InfoNCE loss over other choices of design and reach a conclusion that previous false negative cancel- lation works can be considered as a special case of our proposed methods by discussing the relation.
We apply Soft- Loft-InfNNCE loss on several code search models and evaluate them on the public CodeScarchnet dataset with six programming languages. Extensive experiment results ver- ify the validity of our theoretical analysis and the effectiveness our method.
Preliminaries
Code search aims at retrieving the most relevant code fragments for a given query. During train- ing, we take the comment of code as query and maximize the similarities between the query and its associated code. Meanwhile, we minimize the sim- ilarities between negative pairs that are generated by In-Batch Negative (Huang et al., 2021) strat- egy. Given code distribution C {a=) and a comment distribution Q = {u } where x; is a code fragment and y; is its corresponding query, K is the size of the dataset, a Siamese encoder CUQ - H is used to map codes and queries to shared representation space H. Thus, we ob- tain two representation sets, He {((x))) and Hg {)())))<
In this work, we first revisit the commonly used InfoNCE loss and explain why it cannot model po- tential relations among codes explicitly. Then we present Soft-InfoNCE to handle this problem, by simply inserting a weight term into the denominator of InfoNCE loss. We also propose three methods to estimate the weight terms and compare them empirically. To justify the effect of Soft-InfoNCE loss, theoretically analyze its properties with regard representation distribution and mutual in- formation estimation. Our analysis indicates that the inserted weight encourages negative pairs to ap- proximate a given distribution and reduces bias in the estimation of mutual information by leveraging importance sampling. Moreover, we prove that the proposed Soft-InfoNCE loss upper bounds other loss function designs like Binary Cross Entropy and weighted InfoNCE loss. We also relate existing false negative cancellation methods and our meth- ods. Finally, we demonstrate the effectiveness of the proposed Soft-InfoNCE loss by evaluating on several pre-training models across six large-scale datasets. Additional ublation studies also validate our theoretical analysis empirically.
We calculate the similaritics between query-code pairs by dot product or cosine distance. And we optimize the distribution of representations by con- trastive learning. Several loss functions are pro- posed for this objective (Mikolov et al., 2013; Weinberger and Saul, 2009; Hadsell al., 2006). Among them, InfoNCE loss (Van den Oord et al., 2018) is dominantly adopted by recent code search models due its better performance than others. We denote q E Hg and c; E H. as representa- tions of queries and codes, respectively, For given batch of data, we could generate I positive pair and N negative pairs for each query, where N is the batch size. The InfoNCE loss can be described
In summary, our contributions of this work are follows:
We propose a novel contrastive loss, Soft- InfoNCE, that models the potential relations among negative pairs explicitly by simply in- serting weight term to the vanilla InfoNCE loss.
We conduct theoretical analysis to show that Coft-InfoNCE loss can control the distribu- tion of learnt representations and reduce the
exp(g: c,) log cxp(x: ci) + Eje, exp(gi
L -

---

where (qi, (g) are positive pairs when = j and negative pairs otherwise. Here we adopt the dot product as the measurement of similarity.
between query di and code ej. The numerator in Eq.(4) puts smaller weights on less negative codes, and the denominator is a normalization factor make sure Σj Wij = N - 1, which is the same value as the vanilla InfoNCE loss. Now we can derive that the gradients of negative pairs are proportionally related to way:
3 Our Method
3.1 Revisiting InfoNCE Loss
We reformulate (1. to:
ac 2 @xp (0: cG) N Eÿ=1 wiy exp (4: cÿ)
Wij
(5)
L = -E [ ci]
N Σ exp(a cy) jzi
+ E
log explo: -4)
Note that wh = 1 in this equation when j. The vanilla InfoNCE loss is special case of Eq.(3), which sets all May as 1. Models may learn implicit relationships between a query and different nega- tive samples under the vanilla InfoNCE loss, but we argue that modeling this relationship explicitly by wi has positive influence on learning better representations.
(2)
As discovered by (Wang and Isola, 2020), the two terms correspond to two objectives of con- trastive learning. The first term can be expressed as the alignment of positive pairs that pulls posi- tive instances together. The second term enforces uniform distribution of negative pairs because pushes all negative pairs apart.
Then comes the estimation of similarity score simiy. The ideal solution is using human- annotated labels. However, there are existing datasets and it is challenging label all the pos- sible negative pairs since the number of them in- creases quadratically with the number of positive pairs (c.g., 100 positive pairs can generate 9900 negative pairs). Thus, we employ the following approaches to estimate the similarity scores simi and empirically analyze and compare them in Sec- tion 6.
However, we argue that negative pairs should not be distributed uniformly. In other words, unla- beled data may also share some similarities with the given query. Suppose we have a query "How im- plement a bubble sorting algorithm in Python?", al- though both a quick sorting algorithm and file sav- ing function are considered to be negative results, quick sorting is expected to be retrieved before file saving since it is more relevant to the query. More- over, Lopes et al. (2017); Allamanis (2019) find that code duplication is common in code corpora which means that there are many false negative examples during training. In conclusion, negative samples batch should not be treated equally, as illustrated in 1.
BM25. BM25 is enhanced version of TE- IDF, which matches certain terms in codes with the given query.
Sim. Unsupervised SimCSE (Gao al., 2021) a recently proposed method that has out- standing performance on sentence similarity tasks. We measure the similarity between a query and un- labeled code indirectly by measuring that between the query and code's positive query. The assump- tion behind this is that query and its positive code are exactly matched so that they are perfectly aligned in the representation space.
3.2 Soft-InfoNCE Loss
To address the aforementioned problems, we pro- pose Soft-InfoNCE loss by simply inserting weight term Wij into the original format, which can be described as:
Trained Model. Models that are trained with vanilla InfoNCE loss datasets may have certain capabilities to correctly predict the similarity.
exp(g, - ci)
L =
log
exp(y (.) + Eize Why exp(a c.)
3 a simij
Note that for the last two estimation methods, we load and freeze their pre-trained parameters during training. After calculating the similarity scores, we normalize the results with Softmax function with temperature t to satisfy Ei sim;j = 1.
Wij
(4)
8 N-1 Ejfts simyj"
where simy E [0, i simi = 1, and a, 3 are typer-parameters. simyy is the similarity score

---

4 Justification
Since the optimal value for exp(g c) is given by plcial the derivation can be described as follows: p(c)
In this section, we analyze the properties of Soft- InfoNCE loss and compare it with related works justify its effectiveness. Recall that the opti- mization objective of vanilla InfoNCE loss can be divided into two parts, alignment and uniformity. The insertion Wiy only has an influence on the second term. Thus, for simplicity, we analyze the second term in this section, which is:
picskgel p(ea)
C Eoogg
ple:las) p(a))
N
p(c))
1 p(c))
p(cylos)
log
(8)
p(cilg))
p(c+)
p(c;)
p(c)(a))
E log
(N 1)
E
Lunis =
log exp(4: ci)+
p(calge)
0j
GnCn p(ej)
(9)
(6)
N
p(ca) N p(c;)g))
Wij exp(a= c;))]
2 log
= - [gi, Ci)
log N,
4.1 Effect on Representation Distribution
where Cneg is the whole set of negative codes for the given query d:- A key step in the above derivation is the approximation from (8) (9)( In (q) we calculate the sum of Meclo:)) pley)
Intuitively, we control the distribution of nega- tive samples by setting different weights. Here we theoretically prove that Soft-InfoNCE loss upper bounds the KL divergence between the predicted similarity of negative pairs and simij-
in a batch estimate that of the whole negative set. As Van den Oord et al. (2018) mentioned, (8) becomes more accurate when N increases. By adding constant term log( N 1). would be clearer that InfoNCE loss builds a Monte-Carlo estimation sampling from a uniform distribution:
Theorem For a batch of query representations [- code representations {ci] and similar- ity scores S; = (sim;) Div we have:
1 Lunif 2 N(BN - a -
log (açclaa))
L log(NN -1)
j
p(ealss) plc.)
+ *KLL(S)Pa(c((4)) + const.w.r.tk Sp.,
-E log
plc.lg.) p(e))
p(cj
N
N
S4
play)
(7) where N is batch size, Po(cyla;) is predicted simi- larity between query q and code Oj by model O.
(10)
Incorporating wy in Eq.(10) reduces estimation bias since can be considered as using an impor- tance sampling strategy, resulting in more precise estimation. Specifically, we want to estimate the expectation of pic, 19:). according to real distribu- tion between the query qi and negative codes ej in the context of code search. However, the negative codes in batch are randomly sampled which fol- lows uniform distribution. To bridge this gap and get an unbiased estimator, importance sampling is adopted by inserting a weight term Wij- We denote the uniform distribution as p and the real distribu- tion as q. Thus, calculating expectations based on could be derived from p:
Proof of Theorem is presented in Appendix B.I. We can observe that addition to the original objective which minimizes the predicted similar- ity scores of negative pairs, the second term en- courages the similarity distribution to fit the given distribution Sj. When we set 3 and sim,, 1 it becomes the vanilla InfoNCE loss hence all Wij 1. This could also be used as an explanation for the uniformity objective of vanilla InfoNCE loss.
pie;]
4.2 Effect on Mutual Information Estimation
It has already been proved that optimizing InfoNCE loss improves the lower bounds mutual informa- tion for positive pair (Van den Oord et al., 2018) 8).
Enj^q p(cyla:)
p(c/14)) {(g)) p(c;) p(c))
Eejyp p(c;)

---

Proof of Theorem is presented in Appendix B.2. general, minimizing an upper bound leads to better performance compared with lower bounds. Weighted InfoNCE Loss. Another choice is using weighted InfoNCE loss. We take simig as weights, which can be described as:
p(cyle)) p(oj)
1
p(c
E
N
p(c,)
jfi
If we take both and B as 1 in (4), we have:
p(cylq:) ((c))
exp Ci) exp(q: 9y)
Lw - -E simyy log
p(cj) p(cz)
==
(13)
(N - 1)(1 simij)
p(cyla)
(11)
Note that we set the weight simy for the positive pair as
-2
p(o))
JFL
Proposition Soft-IffoNCE loss upper bounds
And we could find that ((e)) N-2 which is inversely proportional to sim.y. This property is line with our intuition that we should consider true negatives more when approximating the whole negative set. As we know, importance sampling provides more accurate estimates when the sam- pling distribution is closer to the real distribution. We hypothesize that for given query, the major- ity of codes share low similarity, which makes it suitable use the Softmax function to normalize simy.. We will empirically discuss this hypothesis in Section
simms
weighted InfoNCE loss:
C LW + log Po(cilqs)
N simyy log Po(eyla)- (14)
The equilibrium satisfies when all code fragments in the batch are actually false negatives.
The proof of Proposition pendix B.3. Without loss of generalization, our proof is deduced based on one query, however, it also holds for batch of queries. From the above proposition,
presented in Ap-
find that Soft-Inford Info-In loss Sofft upper bounds Weighted InfoNCE loss Lw- Thus, one would expect L be the superior loss function.
4.3 Relation with Other Loss Functions
In this part, we connect Soft-InfoNCE loss with other loss functions by analyzing them theoretically. Besides, in Section 6 we report their comparative evaluation results empirically.
KL Divergence Regularization. As proved Theorem 1, a KL. divergence term that measures the similarity between the given distribution S and predicted distribution Po is incorporated during the optimization implicitly. We try to explicitly add KL divergence regularization term to the vanilla InfoNCE loss and compare its performance with our proposed loss empirically.
Binary Cross Entropy Loss. We may con- sider simey as soft labels and hence use Binary
Cross-Entropy (BCE) loss to train the model. The exp(g.s)
probability in BCE is calculated by Σ explure;)" While InfoNCE loss upper bounds the KL di- vergence, BCE lower bounds it, which is described in Theorem
4.4 Relation with False Negative Cancellation
Theorem For batch of query representations N
Recently, several works focus on eliminating the effect of false negative samples by first detecting those samples and then removing them from the negative sets. Though these detection methods are different among tasks, their cancellation op- erations share the same principle. That is, false negatives removed from the denominator of the InfoNCE loss. It be considered special case of our proposed Soft-InfoNCE loss, which sets the weights Wig of false negatives while others remain 1. False negative cancellation meth- ods are effective in classification or unsupervised
N code representations {c and similar-
ity scores S: = {sim,) ) we have:
N
LBCE S
log(1 - Polcyla))
N
+ KL(S + const.w.r.l S
(12) where is batch size, Po(cylg)) is predicted simi- larity between query qi and code oj by model 6.

---

pre-training tasks since they only need to consider whether negative samples belong the same class or not. However, in the context of a similarity- based retrieval task, models are required to dis- criminate negative samples by continuous values. In Section 6, we report an empirical comparison between cancellation methods and ours.
calculation of Soft-InfoNCE loss, we provide im- plementation details in Appendix C.2. For BM25 estimation, we merely measure similarity based on in-batch data. For trained model estimation, we train the same model of each studied model follow- ing the training settings of the original paper on different programming languages separately until convergence. For SimCSE estimation, we first ini- tialize the model with the HuggingFace released parameters' and then train following the default setting of the original paper. Note that we col- lect all the natural language queries from different programming language training sets to boost per- formance on SimCSE unsupervised learning. The training epoch is set to 30 for all studied models. Experiments described in this paper are running with 3 random seeds 1234, 12345, and I 123456. All experiments meet p<0.01 of significance tests ex- cept for the results of GraphCodeBERT and UniX- Coder on the Go dataset. Experiments are con- ducted on a GeForce RTX A6000 GPU.
5 Experimental Setup
this section, we elaborate on the evaluated dataset, baselines, and our implementation details. Datasets. We use large-scale benchmark dataset CodeSearchNet (CSN) (Husain et al., 2019) evaluate the effectiveness of Soft- InfoNCE. loss which contains six programming lan- guages including Ruby, Python, Java, Javascript, PHP, and Go. The dataset is widely used in pre- vious studies (Feng et al., 2020; Guo et al., 2021, 2022) and the statistics are shown in Appendix C.1. For the training set, it contains positive-only query-code pairs while for the validation and test sets the model attempts to retrieve true code frag- ments from a fixed codebase. We follow (Guo et al., 2021) to filter out low-quality examples. The per- formance is measured by the widely adopted the Mean Reciprocal Rank (MRR) which is the aver- age of reciprocal ranks of the true code fragment for given query. It can be calculated as:
Results
In this section, first show the overall perfor- mance of three weight estimation approaches when applying Boft-InfoNCE loss. Then, we empirically compare our proposed method with other choices of loss functions and false negative cancellation methods. Finally, we conduct an ablation study to analyze the effect of papper-paramtters..
Q
MRR-
E
(15)
Q 1=1
Rank;
Overall Results. The results in Table 1 reveal that baseline models equipped with Sofo-Inf loss can gain an overall 1%-2% performance im- provement in MRR over that of InfoNCE loss across six programming languages. The consis- tent improvements observed in all three estima- tion approaches demonstrate the effectiveness of our proposed Soft-InfoNCE loss in code search. Time efficiency comparison is also performed in Appendix D.I.
where Rank; is the rank of the true code fragment for the i-th given query Q. Baselines. We apply Soft-InfoNCE loss on sev- eral code search models: CodeBERT is a bi-modal pre-trained model pre-trained on mask language modeling and replaced token detection (Feng et al., 2020). Note that in this work refer Code the siamese network architecture described in the original paper. GRA incorporates the structure information of codes and further de- velops two structure-based pre- pre-training tasks: node alignment and data flow edge prediction (Guo et al., 2021). UniXCoder unifies understanding and gen- eration pre-training tasks to enhance code repre- sentation leveraging cross-model contents like Ab- stract Syntax Trees (Guo et al., 2022).
Comparison among Estimation Approaches. As shown in Table 1, trained-model estimation sig- nificantly outperforms the other two methods on CodeBERT, while all estimation approaches im- prove on the other models similar extent. Among them, SimCSE is the most robust one con- cerning model types.
Implementation Details. For all the settings re- lated to model architectures, we follow the original paper. For poper--arameter settings that affect the
htttps: http///ggggggcceee.e//// htwpugiingace.cc/senttaaaeo ttr.senteneetrrafffomeeee msmarco- cirrrroddistibbrn distilbert-dott

---

Model
Loss
Estimator
Ruby
Python
Java
JavaScript
PHP
Go
InfoNCE
0.648
0.636
0.663
0.594
0.615
0.878
CodeBERT
BM25
0.660
0.664
0.682
0.600
0,621
0.882
Soft-InfoNCE
Trained Model
0.682
0.675
0.682
0.615
0.631
0.898
SimCSE
0.666
0.668
0.670
0.607
0.623
0.887
InfoNCE
0.705
0.690
0.691
0.647
0.648
0.896
GraphCodeBERT
BM25
0.730
0.697
0.698
0.652
0.655
0.892
Soft-InfoNCE
Trained Model
0.719
0.692
0.692
0.653
0.648
0.889
SimCSE
0.721
0.700
0.702
0.656
0.657
0.894
InfoNCE
0.740
0.720
0.726
0.684
0.676
0.915
UniXCoder
BM25
0.753
0.728
0.733
0.693
0.684
0.916
Soft-InfoNCE
Trained Model
0.753
0.728
0.731
0.694
0.682
0.915
SimCSE
0.753
0.726
0.731
0.699
0.684
0.913
Table I: Results of different weight estimation approaches under MRR.
there are contradictory predictions when we use InfoNCE-tuned code search models to predict Wig- This phenomenon indicates that although existing code search models could find true code snippets well, they cannot recognize potential relevance among negative codes, which is the core motivation for proposing Soft-InfoNCE. Kindly note that in this paper we do investigate estimation methods fully but mainly focus on the effectiveness of Soft- InfoNCE, as discussed in the 'Limitations' Section.
80-
Simose
Co
60-
GruphCoelcdtrt
Undiiaader
Weight
Sinilerity
40
0.8
20
0.4
0.0
2000
4000 6000 Sample
8000 10000
Sample
Figure 2: Similarity estimations by different approaches for a random batch of samples in CSN-Python.
Comparison with Other Loss Functions. Ta- ble shows the overall performance of different loss functions. In Table we also give the de- tailed results for each programming language. For the calculation of these loss functions, we follow the definition described in Sec.4.3. Note that for KL regularization we set the weights of the origi- nal loss and regularization term as 1.3 and 0.7 to fairly compare with Soft-InfoNCE loss, and we use SimCSE estimation for all experiments. We can see significant drops of MRR compared with Soft-InfoNCE loss, which agreement with our theoretical analysis. We think those three loss func- tions somewhat improve the mutual information of negative pairs. Take weighted InfoNCE as an example. Though weights for negative pairs are only at around 0.03, optimizing negative pairs still lower bounds the mutual information with stant log N according Eqn.9. It makes feature vectors distribute closer and hence hard distin- guish each other, which makes the performance even worse than InfoNCE. The analysis also works for BCE and KL regularization, which try to im-
In 22g.2, we take random batch of samples from CSN-Python to analyze the difference among estimation approaches. From the left figure, can find that predicted similarity scores roughly follow softmax distribution, per our hypothesis. As for the right one, we can see that compared with other estimation methods, BM25 generates simi- lar weights for the majority negative samples. This is because BM25 calculates similarities only based on keyword matching and for most negative codes there is no keyword overlapping all. While for neural model based methods, they can capture latent semantic similarities. Considering the esti- mated weights of samples from trained models in the right panel of Fig. 2, we find that some weights that are predicted by the three models are similar while others are not, or even contradictory with each other.
To better understand the differences among these estimation methods, we perform case studies for the eight samples in the right panel of Fig. 2 in Appendix D.2. From the case study, we find that

---

Loss
CodeBERT GraphCodeBERR UniXCoder
0.599
2.00
0:12,4:23
0-13,5-0) 5-007
BCE
0.641
0.679
0.686
1775
p-1A E-0.5
0.600
pels. 5-0.
W-InfoNCE
0.639
0,674
0.714
0.697
15
pel pw0.4
KL Reg. 0.648
0.688
0.716
0.696
1.00
Ours
0.687
0.722
0.751
0.695
0.25
Table Overall performance of different loss function designs on six programming languages under MRR. "W- InfoNCE" stands for weighted InfoNCE loss and "KL Reg." stands for KL regularization.
0.693
12
1.3
1.4
5
0
Sample ld
Figure 3: MRR and estimated weights under different and [ settings on GraphCodeBERR over CSN- -St-Python.
Methods
MRR
Methods
MRR
original InfoNCE loss. The left part of Fig.3 shows that the performance of Soft-InfoNCE is relatively stable with different settings and a = 1.3, B = 0.7 reaches its best performance. Besides, since a and 3 are incorporated into the calculation of weights, they also have effects weights estimation. As shown in the right part of Fig.3, the increasing of makes the weights more distinct.
0.685
0.7
0.689
Dynamic
Top-K 3 0.681
0.5
0.688
Threshold
5
0.680
0.3
0.688
Table 3:
Performance of false negative cancellation methods applied to GraphCodeBERT on CSN-P ISN-Python.
prove the similarity between a query and negative codes as well. Therefore, we argue negative pairs should not be placed at the numerator of InfoNCE.
7 Related Works
Code Search Models. There are mainly threc stages in the development of code search mod- els. Traditional information retrieval techniques match keywords between queries and code frag- ments (Hill et al., 2011; :; Yang and Huang, 2017: Satter and Sakib, 2016; Lv et al., 2015; Van Nguyen et al., 2017). Since natural language and program- ming language have different syntax rules, they often suffer from vocabulary mismatch problems (MeMillan et al., 2011). Then, with the popularity of neural networks, several methods are proposed to better capture the semantics of both queries and codes (Gu et al., 2021; Cambronero et al., 2019; Gu et al., 2018; Husain et al., 2019). Generally, they are encoded by neural encoders into shared representation space. Recently, transformer-based pre-trained models significantly outperformed pre- vious methods. CodeBERT (Feng et al., 2020) is pre- teaitraind via masked language modeling and re- placed token detection. GraphCodeBERT (Guo et al., 2021) leverages data flow as additional infor- mation to model the relationship among variables. UniXCoder (Guo et al., 2022) is a model that can support understanding and generation tasks at the same time. This allows it to further boost perfor- mance by using the pre-training tasks (e.g. unidi- rectional language modeling, denoising objective) both. In this work, we mainly consider pre-trained models due to their better performance.
Comparison with False Negative Cancellation Methods. We apply two types of false negative cancellation methods and evaluate their perfor- mance, as shown in Table 3. The first type involves removing top-K similar negatives. We observe that performance decreases when more negative sam- ples are removed because sometimes there are no false negative samples in the batch. Hence, re- moving top-k negatives directly may accidentally remove hard negative samples. The other option the dynamic threshold. takes negative samples that have similarities greater than certain ratios of the positive sample as false negatives. Besides hav- ing the same drawbacks as the top-k method, it is also hard to determine appropriate ratio. Thus, there slight drops when upplying the dynamic threshold. While using Soft-InfoNCE can achieve a MRR of 0.700 which is better than the result in Table 3, we argue that setting a weight on negatives would less risky than removing them directly.
Effect of a and B. a and 3 control the weights of two terms in Theorem I, one for minimizing pre- dicted similarities and the other for KL divergence, which contradicts each other some extent. To bal- ance training, we performed empirical experiments to guide the setting of these two hyper-parameters, as shown in Fig.3. Note that we follow at0 1 to make Soft- InfoNCE fall on the same scale as the
False Negatives in Unsupervised Contrastive

---

Learning, The false negative problem in unsu- pervised contrastive learning has been studied by some researchers. Since in contrastive learning we automatically consider other in-batch examples as negatives, we may sample false negatives dur- ing training, which results in discarding semantic information and slow convergence. Several ap- proaches are proposed to detect those false neg- atives. The key insight behind these detection methods is that false negative examples are sim- ilar to positive ones. Huynh et al. (2022) calculates the similarity between multiple views of images to distinguish false negatives and true hard nega- tives. Chen et al. (2022) leverages the property that similar instances are closer in the representa- tion space to incrementally detect false negatives in vision tasks. Zhou et al. (2022) uses another external model to measure the similarity of two sentence representations and selects pairs whose similarity scores are higher than the threshold as negative pairs. code representation pre-training, Li et al. (2022c) handles it in an iteratively adver- sarial manner. However, the false negative problem in the fine-tuming of code search is not investigated yet, and it also suffers due to code duplication in code corpora mentioned by Lopes et al. (2017); Allamanis (2019). The aoove-mentioned works can be seen as special case of the proposed Soft- InfoNCE loss.
Limitations
There mainly three limitations of this work. First, we propose and evaluate three methods estimate similarity scores simyj by using BM25, trained models and SimCSE. Since we mainly fo- cus on the justification of Soft-InfoNCE loss itself in this work, the estimation methods are not fully in- vestigated. Which method is better? Are there any other more efficient and precise estimation meth- ods? We leave this our future work. Second, in this work, we aim to show and justify the effec- tiveness of Soft-InfoNCE over InfoNCE. For the efficiency of the two, since there is an additional step to estimate the weight of each negative pair, the training time-cost of Soft-InfoNCE) greater than InfoNCE. We compare them in Table 8 and leave improving the efficiency of Soft-InfoNCE our future work. Third, as we claimed in the paper. this work focuses the order of negative codes for given query. However, the widely adopted benchmarks only consist of binary labels, positive and negative. As a result, we could only demon- strate its effectiveness by indirectly showing that it can help models learn better representations.
References
Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN Interna- tional Symposium on New Ideas, New Puradigms, and Reflections on Programming and Software, pages 143-
8 Conclusion
In this work, we revisit the commonly used In- foNCE loss in code search and analyze its draw- back during fine-tuning. By simply inserting weight terms, we propose Soft-InfoNCE to model the potential relation of negative codes explicitly. We further theoretically analyze its effect on rep- resentation distribution, mutual information esti- mation, and superiority over other loss functions and false negative cancellation methods. We evalu- ate our Soft-InfoNCE loss on several datasets and models. Experiment results demonstrate the effec- tiveness of our approach, and justify the theoretical analysis.
Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2021. Self- supervised contrastive learning for code retrieval and summarization via semantic-preservvee tions. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 511-521.
transforma-
Jose Cambronero, Hongyu Li, Scohyun Kim, Koushik Sen, and Satish Chandra. 2019. When deep learning met code search. In Proceedings the 2019 27th ACM Joint Meeting on European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering, pages 964-974.
Acknowledgement
Tsai-Shien Chen, Wei-Chih Hung, Hung-Ye Yu Tseng, Shao-Yi Chien, and Ming-Hsuan Yang. 2022. cremental false negative detection for contrastive learning. In The Tenth International Conference Learning Representations.
We thank the anonymous reviewers for their helpful comments and suggestions. This research is sup- ported by Alibaba-NTU Singapore Joint Research Institute (JRD), Nanyang Technological University, Singapore.
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin,

---

Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- bert: A pre-trained model for programming and nat- ural languages. In Findings of the Association for Computational Linguistics, pages 1536-1547.
Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, and Maryam Khademi. 2022. Boost- ing contrastive self-supervised learning with false negative cancellation. In IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 986-996.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang. and Lei Li. 2020. On the sentence em- beddings from pre-traine trained language models. arXiv preprint arXiv:201 1.05864.
Luca Di Grazia and Michael Pradel. 2022. Code search: A survey of techniques for finding code. ACM Com- puting Surveys (CSUR).
Haochen Li, Chunyan Miao, Cyril Leung, Yanxian Huang, Yuan Huang. Hongyu Zhang, and Yanlin Wang. 2022a. Exploring representation-level aug- mentation for code search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4924-4936.
Jian Gu, Zimin Chen, and Martin Monperrus. 2021. Multimodal representation for neural code search. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 483 494.
Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. 2022b. Coderetriever: Unimodal and bimodal contrastive learning. arXiv preprint arXiv:2201 220666600..0
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In 2018 TEEF/ACM 40th Interna- tional Conference on Software Engineering, pages 333-944.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross- modal pre- pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 7212- 7225.
Xiaonan Li, Daya Guo, Yeyun Gong. Yun Lin, Ye- long Shen, Xipeng Qiu, Daxin Jiang. Weizhu Chen, and Nan Duan. 2022c. Soft-labeled contrastive pre- training for funetion-level code representation. arXiv preprint wwxiv2210.00997.
Cristina V. Lopes, Petr Maj, Pedro Martins, Vaib- hav Saini, Di Yang, Jakub Zitny Hitesh Sajnani, and Jan Vitek. 2017. Déjävu: map of code du- plicates on github. Proc. ACM Program. Lang.. (O0PSLLA):84: 1-884:22::88422
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng Duyu Tang, Shujie Liu, Long Zhou, Nan Duan. Alexey Svy- atkovskiy. Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundare- san, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations.
Fei Lv, Hongyu Zhang, Hlan-guang Lou, Shaowei Wang. Dongmei Zhang, and Jianjun Zhao. 2015. Code- how: Effective code search based api understand- ing and extended boolean model (e). In 2015 30th IEEF/ACM International Conference on Automated Software Engineering, pages 260-270.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition, volume 2, pages 1735-1742.
Collin McMillan, Mark Grechanik. Denys Poshyvanyk, Qing Xie, and Chen Fu. 2011. Portfolio: finding relevant functions and their usage. In Proceedings of the 33rd International Conference on Saftware Engineering, pages 111-120.
Emily Hill. Lori Pollock, and K Vijay-Shnnker. 2011 1. Improving source code search with natural language phrasal representations method signatures. In 2071 26th IEEE/ACM International Conference on Auto- mated Software Engineering, pages 524-527.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Cosqa: 20, 000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1thh International Joint Confer- ence on Natural Language Processing, pages 5690- 5700.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022. Text and code embeddings by contrastive pre- training. arXiv preprint arXiv:2201.10005
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidr. 2019. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: 2itv:1900 1909.09436
Liming Nie He Jiang, Zhilei Ren, Zeyi Sun, and Xi- aochen Li. 2016. Query expansion based on crowd knowledge for code search. IEEE Transactions on Services Computing, 95):771-773.

---

Abdus Satter and Kazi Sakib. 2016. A search log mining based query expansion technique to improve effec- tiveness in code search. In 2016 19th International Conference on Computer and Information Technol- ogy. pages 586-591.
Kilian Q Weinberger and Lawrenee Saul. 2009. Dis- tance metric learning for large margin nearest neigh- bor classification. Journal of machine learning re- search, 10(2).
Yangrui Yang and Qing Huang. 2017. lecs: Intent- enforced code search via extended boolean model. Journal of Intelligent Fuzzy Systems, 33(4)22565- 2576.
Ensheng Shi, Wenchao Gub, Yanlin Wang, Lun Du, Hongyu Zhang. Shi Han, Dongmei Zhang. and Hong- bin Sun. 2022a. Enhancing semantic code search with multimodal contrastive learning and soft data augmentation. arXiv preprint arXiv:2204.0329
Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. 2022. Dehiased contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pages 6120- 6130.
Yucen Shi, Ying Yin, Zhengkui Wang, David Lo. Tao Zhang, Xin Xia, Yuhai Zhao, and Bowen Xu. 2022b.
How to better utilize code graphs in semantic code search? In Proceedings of the 30th ACM Joint Eu- ropean Software Engineering Conference and Sym- posium on the Foundations of Software Engineering. pages 722-733.
A Survey on the adoption of InfoNCE loss
Zejian Shi, Yun Xiong, Xiaolong Zhang. Yao Zhang. Shanshan Li, and Yangyong Zhu. 2022c. Cross- modal contrastive learning for code search. In 2022 IEEE International Conference on Software Mainte- nance and Evolution (ICSME), pages 94-105.
To show the dominant adoption of InfoNCE, we conduct survey on recent code search studies. We collect the papers top-tier conferences from 2020 and then filter out researches that focus model training, as shown in Table 4. As we can see, among 16 papers there are only papers that use other loss functions for training.
Jianhang Shuai, Ling Xu, Chao Liu, Meng Yan, Xin Xia, and Yan Lei. 2020. Improving code search with co-attentive representation learning. Proceedings of the 28th International Conference on Program Comprehension, pages 196-207.
Model
InfoNCE
Weisong Sun, Chunrong Fang, Yuchen Chen, Guan- hong Tao, Tingxu Han, and Quanjun Zhang. 2022. Code search based on aonee code translation. arXiv preprint arXh::2202.0802.
RoBERTa (code) (Feng et al., 2020)
CodeBERT (Feng et al.. 2020)
GraphCodeBERT (Guo et al., 2021)
UniXCoder (Guo et al., 2022)
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv e-prints, pages 180i
SyncoBERT (Wang et al., 2021)
SCodeR (Li et al., 2022c)
CoCoSoDa (Shi al., 2022a)
Thanh Van Nguyen, Anh Tuan Nguyen, Hung Dang Phan, Trong Duc Nguyen, and Tien N Nguyen. 2017. Combining word2vec with revised vector space model for better code retrieval. In 2017 IEEE/ACM 39th International Conference on Soft- ware Engineering Companion, pages 183-185.
CodeRetriever (Li et al.. 2022b)
RACS (Li al., 2022a)
CrossCS (Shi et al., 2022c)
Code-MVP (Wang et al., 2022)
cpt-code (Neelakantan et al., 2022)
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-99999 99399.
Corder (Bui et al., 2021)
TranCS (Sun et al., 2022)
CoCLR (Huang et al., 2021)
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li. Hao Wu, Jin Liu, and Xin Jiang. 2021. Syncobert: syntax-guided trastive pre-training for code representation. arXiv preprint arXiv:2108.0455 04555.
CQIL (Gu et al., 2021)
GSMM (Shi et al., 2022b)
multi-modal con-
Table 4: The adoption InfoNCE loss on the training of recent code search models.
Xin Wang. Yasheng Wang, Yao Wan, Jiawei Wang. Pingyi Zhou, Li Li, Hao Wu, and Jin Liu. 2022. Code-mvp: Learning to represent source code from multiple views with contrastive pre-training. arXiy preprint arXi: 02029.
"Specifically, venues ranked A or B under the section of Artificial Intelligence and Software Engineering in the CCF ranking: httpf://ccf... atom, atmm.im/.

---

B Proof
greater than O. By subtracting a constant term N E Log(NV-1 and applying Jensen's 's Inequal- ity, we get Eq.(18). Then, we subtract (q. from Zi=1 Σjjt B a-simas log 2n exp(g: BN-a-1 cj). According previous studies (Li et al., 2020; Zhou al., 2022), wansformer-basee models learn an anisotropic embedding space which means that nearly any dot product of two representations is greater than O. We also find similar phenomena in the context of code search models. Thus, the in- equality in Eq.(19) holds in most cases. To further satisfy the condition, we clamp the dot product re- sults that are less than 0 to be O. Finally, by turning the subtraction outside the logarithms into division inside it, we get the probability Po(cylas (q:) that refers to the similarity of q: and cj. Therefore, we reach the conclusion Theorem 1. And we could find that the constant term is:
B.1
Proof of Theorem 1
I
N
log exp(e: Gi)
1=1
N
Wij exp(e : g]]l
(16)
N
N
log
Σ
a+ simii
N
3
Ejji simig
1=1
N
expla . 13)
(17)
N
3 a= simy
log
3
ji simig
1=1
N
N
N
N
exp(a c4)
log(N~ 1)
1
N
simi log simij
N(8N a -
9 a simai
log
BN- a -1 exp(9: .g)
which is related to S: simm LN
N 1=1
B.2
Proof of Theorem 2
3 a: simij
log exp(a (4)
The proof only discusses the effect of BCE loss for negative pairs. For the optimization of Positive pairs, BCE loss is equal to Soft-InfoNCE loss.
BN a
jz
(18)
N
N
3 a: simij
log exp(p: Gj)
LBCE
BN=
1=1 jHi
1
N
N
exp(9: Cj) Ejit exp(% (3)
Σ
simy log
N N
3 - a sims
N
2
log
exp(1 Cj)
N
3N-
E
Jti
+ (1 - simiz) log (1 explgi cj)
(19)
Ex expl(: cj)
3 a simii
N
N
log Poçclla)
BN- a
log(1 - Po(cylai))
1=1
N2
1=1 j=1
N
N (B - asim.j)
1
exp(g: - cj)
N(8N a-11
simij log
1=1
exp(a c,)+ + Ejfts exp(e Gj) (21)
log Po(clla))
(20)
N
N
N
NV
1
Σ
1
Σ
log Po(cyla))
log(1 - Polcylat)
N(3N a- 1)
1=1
+ +KKL(S
Σ simig log simmy
+ simi log(Po(s)123)
(22)
577
We drive from (16 to (17) by ignor- ing the result of positive pair that is always
log(1 [ Po(cy(4))
N2

---

Model
Ruby
Python
Java
JavaScript
PHP
Go
Overall
CodeBERT
-BCE
0.603
0.615
0.639
0.555
0.572
0.863
0.641
-Weighted InfoNCE
0.608
0.609
0.634
0.553
0.566
0.861
0.639
-KL. Regularization
0.615
0.623
0.643
0.557
0.586
0.861
0.648
-Soft-InfoNCE
0.666
0.668
0.670
0.607
0.623
0.887
0.687
GraphCodeBERT
-BCE
0.705
0.648
0.669
0.571
0.597
0.883
0.679
-Weighted InfoNCE
0.695
0.633
0.655
0.608
0.588
0.865
0.674
KKL Regularization
0.691
0.658
0.677
0.620
0.611
0.869
0.688
-Soft-InfNCCE
0.721
0.700
0.702
0.656
0.657
0.894
0.722
UniXCoder
-BCE
0.721
0.666
0.669
0.571
0.608
0.883
0.686
-Weighted InfoNCE
0.729
0.689
0.697
0.649
0.623
0.894
0.714
-KL Regularization
0.729
0.692
0.699
0.659
0.637
0.882
0.716
-Soft-InfONCE 0.753
0.726
0.731
0.699
0.684
0.913
0.751
Table 5: Results of different loss functions on six programming languages under MRR.
N
N
log exp(ai . Gj)
+ (L((S)P((())) + sim,y log simaj
Σ
simyy
Σ/= exp(x cq)
jfi
By removing the - exp(a: eg) in the denomina- tor of the second term in (21() the inequality to (22) holds. Therefore, we reach the conclusion in Theorem 2. And we could find that the constant term is:
Wij log bogexp((; cy)
(24)
jfi
N simij log exp(x= cq)
1=1
N ΣΣ
log
exp(x . (ÿ)
simn, log simij
No
j=1
j
(N 1)(1 simis)
log cogexp(> 0j)
which related to S = { (simig IN 1735
N -
ifi
B.3 Proof of Proposition 1
N
N
simys log cxp(x.p...cc cj)
N
i4i
Lw LSoft
N
+ 2 2og
exp(q: cj) - log exp(x= ci)
exp(g: gj)
simij log
Σラ=1 explex 0z)
N
N
simug log Po(cgla)
exp(9: (i)
+ log
N-2
j= Wij exp(p: Gj)
j+i
log Po (o(cilg:)
exp(g: Cj)
simij log
E exp(p. cj)
llog
exp(q: G.) + Σ wij cxp(a:c))
We derive (2. from E2. by first ignoring exp(q qj) within the logarithm and then applying Jensen's 's Inequality. Therefore, we get:
jzi
+ log exp(p: c;)
(23)

---

Loss
Time per batch
InfoNCE
0.75s
Lsoft 2 Lw log Po(ci(4s) N simij
Soft-InfoNCE
log Po(eyla))
-BM25
0.77s
N -2
-SimCSE
0.81s -Trained Model
0.98s
The equilibrium satisfies when code fragments in the batch are actually positives and the model perfectly predicts them.
Table 8:
Training time efficiency comparison of In-
foNCE and Soft-InfoNCE.
Experiment Settings
D Detailed Experimental Results
D.1 Efficiency Comparison between Soft-InfoNCE and InfoNCE
C.1 Dataset Statisties
The dataset statistics of CodeSearchNet are shown in Table 6.
To compare the time efficiency, we take CodeBERT as an example, and calculate time cost per batch based on the average value of 30 epochs, using the same batch size for training. The results shown in Table 8 reveal that the implementation of Soft- InfoNCE has negligible increase in time over- head while improving performance. As discussed in Limitations, we aim show and justify the ef- fectiveness Soft-InfoNCE over InfoNCE in this work and we leave the efficiency improvement Soft-InfoNCE as our future work.
Language
Training
Validation
Test
Codebase
Ruby 24,927
1,400
1,261
4,360
Python
251,820
13,914
14,918
43,827
Java 164,923
5,183
10,955
40,347
JavaScript
58,025
3,885
3,291
13,981
PHP 241,241
12,982
14,014
52,660
Go
167,288
7,325
8,122
28,120
Table 6: CodeSearchNet dataset statistics.
D.2 Case Study
We think that it would be beneficial to see more detailed analysis. Thus, we analyze the eight exam- ples in the right part of Fig.2 in detail. We first list out all eight examples. The natural language de- scription is written in the caption above each code snippet.
C.2 paaae--prrameter Settings
The settings of and / are shown in Table 7. The settings of and B in BM25 are different because we calculate BM25 scores only based on in-batch data, which results in similar scores for negative pairs. Thus, to better distinguish negative pairs, we set higher a compared with the other two estima- tion methods. And t is used to tune the distribution of negative similarity scores sim;j similar to soft- max distribution, which is in accordance with our hypothesis. Note that all the calculated weights whj are clamped be greater than 0.1.
The positive example is the first one and the left seven examples are negative ones.
Listing I: Extracts encoded genotype data from binary formatted file.
def extract extract_ genotypes for b. bytes:
genogggesstssssll...... bytes):
for in range 4)
((b>>(1-2))
hwtpeeapppeeeee...goooooeeeeeeee genotypes.. appeedpss.append(sse)
raturn ge (8: iindutinn_dddnt]
Listing 2: Gets the SasL.ggiclllneeroomeett API client
Estimation Methods
sas sas_logica) hes._logccaa__neeeeeeeecccc www.reennectaaaeeeee....
-_ 528
BM25
1.5
0.5
1.0
sas _og..oocccl...terrccmeee
bas.ogicalinneercomettss.. connection)
SimCSE
1.3
0.7
0.1
self sog_ogee interconnects
Trained Model
1.3
0.7
5.0
Listing 3: Read the specified number of bytes from the stream.
Table a,, B and of different estimation methods.

---

transitive relationships are not wented, so remove then
def read, btaad_bttssee (self, length) -> bytes:
value self-strean. stream. (tre (length) return value
for 5, réf supjectt_bbeccssseee broader rewave sxas. broaderrransstiive, o))
5, rdf. subjectt
harrowarTrans subject_cbiects
Listing 4: Copy a full directory structure.
rdf. ref.remove ([s, 5K0S. 3a0S.narrowerrramsiitie Q))
def
copy, Sopy_tree((s (src, dst, symlinks-felse. ign names os. listdir (src) If not os. path. exists
infer. ctt.skosoooccoceet(rrllll
0s. makedirs((st) errars for name in nanes:
Listing 7: Retrieves all running tunnels for a specific
if nane in ignore:
com:inue
srenane pol-pathhjj_nsste name)
dstnane es.path.joinn join(dst, nane)
try:
symlinks
os.path. (srcaae))
nethod client. request(method, endpoint)
'/rest/vi /C//unnels format
linkto = inadllnk as synlink (linkto, dstname) elif as. as.hath. path. copy_ (ddir(srcname) dstnome,
return
synlinks. ignure)
alse:
Listing 8: power down the OpenThread
file (srenane, dstnane) except (Ioerror, orrar)) as exc:
errors. errors..ppend (srrennn,,,,,,,nnn dstname, str( exc)))
def poverDown(sa)
print 86 call powerDown
selff port
except clerrar exe: errors. (xtend(exx..errrs)
if self._. AuttossoocaateeTteaaao
'setprep
false [8]
errors:
1= 'ail" "
CTError
tine (0.ep If self -eendComann
'reset"
[0] 'Fall":
Fsooeroonnn
Listing A memoizing recursive Fibonacci to exercise RPCs.
False
return
Fulse
def mmmiiiiin_.fbonccciii
if n <n raise ndb. Return(n)
As we could see, the positive query could be summarized as extracting data from a binary file. For SimCSE and BM25, the estimated weights are calculated based on the natural language descrip- tions of positive codes and negative codes. The second negative example contains tokens like "read from" and "bytes", which makes it perform like reading data from a file as well. Thus, BM25 and SimCSE consider as the most similar negative sample hence predicting a small weight. However, the natural language description is deceptive. The code of the second negative example in fact reads bytes not based on file directory but given number. On the contrary, since weights that are pre- dicted by trained models are calculated by the posi- tive query and negative codes, real potential false negative samples are captured like the third nega- tive code. We also find that there are cases where BM25 and SimCSE perform better than trained models when the code snippets are too long and complicated which makes it hard for trained mod- els to capture the main purpose of the code.
key ndb. (Fy(FobonacciNen (n))
naño yteld kay
httpcbb_shhuudccccheeeeaaaa ca
if neno is not None:
assert meno arg
logging. info neme
n, neme.
value)
Return (memo- value)
logging. inf.iinfo nero Fibonacci perotzing--ibooacee
Rd"
yield fihonacccc
bibonaccieennkeyytee logging.info( value)
ad Ad", valuesans)
yiald nemo raise ndb. Raturn(nns)
Listing 6: Enrich the SKOS relations according to SKOS semantics, including subproperties of broader and symmetric related properties.
def anrich_relationssrdr anrich_nappings.. use_narrowers pnre_transiiiver first enrich napping relationships because they affect regular ones)
if mapiihgnapping:::
skos_symmetrae symmetric_napun (roeings infer wwt.sskess.. infer_skos skos_ use use_narrower)
2. then enrich regular relationships
related reos_r)lsted(rdf)
# broaderGeneric -> broader Inverse narroverGenerte
far in rdf rdf.e broaderGeneric)) raf add(s. 3X0S.broade, o))
D.3 Comparison with Other Loss Functions
broaderPartitive breader inverse perrowerPartitive
For s, in rdf subject_.ojjects broaderPartit raf. add((s, SKOS. broader, 6))
Table 5 shows detailed results each program- ming language. Table 2 calculates the aver- age value of different programming languages demonstrate the overall performance.
skos hierarchicallrdf. use_narrower)
# transitive closure: broaderTrensittvve and narroverTren wrroowrrraanstiiy
if use tre_sransi:iv:: infer. skos. tros.trrnsitiveeeee use_narrower)
else:


Reviews:
[{"review_id": "kLoPWqaH9P", "reviewer": "Reviewer_Ls9b", "summary": "", "questions": "A The paper focuses on code search tasks. Could you discuss the potential challenges or considerations when applying Soft-InfoNCE to other NLP tasks?\n\nB The paper does not discuss the computational efficiency of Soft-InfoNCE. Could you provide information on the computational cost of the proposed method compared to other methods, especially considering the additional computations for weight assignment?\n\nC Could you provide more details on the choice of hyperparameters β and γ? It is mentioned that \"performance of Soft-InfoNCE is relatively stable with different settings and α = 1.3, β = 0.7 reaches its best performance\". Could you give reasons for that?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your reviews! We are very encouraged that you found our paper could stimulate further research.\n\n> Q1. The paper focuses on code search tasks. Could you discuss the potential challenges or considerations when applying Soft-InfoNCE to other NLP tasks?\n\nIn this paper, we focus on the code search task because there are published papers that discuss the existence of false negative samples and their negative influence on representation learning, as written in Lines 56-60. Thus, we aim to solve the problem of false negative samples in code search.\n\nIntuitively, other NLP tasks may also suffer from the same problem. For example, in passage retrieval, texts that describe Shiba are expected to be retrieved before texts of Rabbit for a given query asking about Corgi because both Corgi and Shiba are dogs. When applying Soft-InfoNCE to passage retrieval, we need to first make sure that the existence of false negatives definitely deteriorates the representation learning. Then, the potential relevance between queries and passages could be captured to a certain extent, although not necessarily extremely accurately. We leave the analysis of the false negative problem and application of Soft-InfoNCE in passage retrieval as our future work.\n\n> Q2. The paper does not discuss the computational efficiency of Soft-InfoNCE. Could you provide information on the computational cost of the proposed method compared to other methods, especially considering the additional computations for weight assignment?\n\nWe discuss the computational efficiency of Soft-InfoNCE in Appendix D.1. We record the average time cost per batch during training in Table 8. However, as discussed in ‘Limitations’, in this paper we aim to show and justify the effectiveness of Soft-InfoNCE and we leave the efficiency improvement of it as our future work.\n\n> Q3. Could you provide more details on the choice of hyperparameters β and γ? It is mentioned that \"performance of Soft-InfoNCE is relatively stable with different settings and α = 1.3, β = 0.7 reaches its best performance\". Could you give reasons for that?\n\nAs shown in the left part of Fig.3, we try different α and β choices and find that α=1.3, β=0.7 achieves the best performance. From the right part of Fig.3, we could find that increasing α (decreasing β correspondingly) could make the weights more distinct, which we think could help models better distinguish those negative samples. However, setting α too high also brings risk. That is, the weights of some negative samples are too small. Although those small-weight negatives are less negative compared with other negative samples, models still need to distinguish them with the positive code snippet.\n\nWe may also explain from another perspective. As proven in Theorem 1, α controls the weights of the KL divergence term that fits predicted negative similarity to the given similarity scores $S_i$ while β for lowering the predicted negative similarity. Those two terms are contradictory and α=1.3, β=0.7 achieves a balance between the two.\nWe will add this discussion in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The paper presents a novel theoretical perspective on the role of negative pairs in contrastive learning. This not only challenges the existing assumptions in the field but also provides new insights that can stimulate further research.  \n\n2 The proposed Soft-InfoNCE method is also a significant contribution. By assigning different weights to negative pairs based on their hardness levels, the method addresses a key challenge in contrastive learning. The approach is novel and could inspire similar methodologies in other areas of NLP. \n\n3 The authors further provide a comprehensive empirical evaluation of their proposed method. The experiments are well-designed and conducted on a publicly available, multi-language code search dataset, strengthening the validity of the results.", "reasons_to_reject": "1 While the paper provides valuable insights for contrastive learning in code search tasks, it does not thoroughly explore the implications of their proposed method for other NLP tasks. This somewhat limits the generalizability of the results.\n\n2 The paper does not discuss the computational efficiency of the proposed method. As the Soft-InfoNCE method involves additional computations for weight assignment, it would be important to understand the trade-off between improved performance and increased computational cost.\n\n3 While the authors present the results of their experiments, they do not provide an in-depth analysis of these results. More detailed analysis, including a discussion of cases where the proposed method performs exceptionally well or poorly, could have added depth to the paper.", "questions_to_authors": "A The paper focuses on code search tasks. Could you discuss the potential challenges or considerations when applying Soft-InfoNCE to other NLP tasks?\n\nB The paper does not discuss the computational efficiency of Soft-InfoNCE. Could you provide information on the computational cost of the proposed method compared to other methods, especially considering the additional computations for weight assignment?\n\nC Could you provide more details on the choice of hyperparameters β and γ? It is mentioned that \"performance of Soft-InfoNCE is relatively stable with different settings and α = 1.3, β = 0.7 reaches its best performance\". Could you give reasons for that?", "paper_topic_and_main_contributions": "This paper discusses contrastive learning, focusing on the role of negative pairs and their varying contributions to the learning process in the context of code search tasks. The paper challenges the conventional contrastive learning assumption that treats all negative pairs equally, highlighting the potential for improved performance when considering the \"hardness\" level of different negative pairs.\nThe paper provides an in-depth theoretical understanding of the role of negative pairs in contrastive learning. It introduces Soft-InfoNCE, a new method that assigns weights to negative pairs based on their hardness levels. The paper validates the effectiveness of Soft-InfoNCE through empirical evaluations on code search tasks. The authors discuss the potential for their method to be applied to other contrastive learning tasks and propose future research directions. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w9OMmAxaVA", "reviewer": "Reviewer_swyC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. We are sincerely grateful for your feedback and are glad you appreciated that our work could significantly advance the field and inspire further research in this area.\n\n> Q1. The paper could provide more thorough comparisons with other potential methods for addressing the identified issues with the InfoNCE loss function.\n\nTo the best of our knowledge, our work is the first work to address the false negative problem with InfoNCE in fine-tuning code search models. We compare Soft-InfoNCE with Weighted InfoNCE, which is another choice of design, both theoretically and empirically in Proposition 1 and Table 2. As we could see, Soft-InfoNCE upper bounds Weighted InfoNCE and outperforms it on a dataset that consists of six programming languages.\n\nThere are also some other ways to address the false negative problem, although not using InfoNCE loss function. We also discuss and compare Soft-InfoNCE with BCE loss and KL divergence regularization and show its superiority in Section 4.3 and Table 2.\n\nAt the same time, there are two works addressing the false negative problem during pre-training. However, those approaches could not be applied during fine-tuning since they require large amounts of data. Reviewer ftqB also mentioned this point. Please see our response to Reviewer ftqB (Response to Q2) for a detailed discussion of those two works if you are interested.\n\n> Q2. It would be beneficial to see a more detailed analysis of the results, including a discussion of any limitations or potential drawbacks of the Soft-InfoNCE loss function.\n\nAs discussed in the ‘Limitations’ Section, Soft-InfoNCE introduces additional computational cost due to the weight estimation. Besides, it relies on the precision of weight estimation. In this paper, we evaluate three weight estimation methods, and all of them achieve better performance. However, since we aim to show and justify the effectiveness of Soft-InfoNCE in this paper, the potential choices of more precise and efficient estimation methods remain uninvestigated. We leave them as our future work.\n\nWe agree that it would be beneficial to see a more detailed analysis. Thus, we analyze the eight examples in the right part of Fig.2 in detail. We first list out all the eight examples:\n\nThe positive example is:\n* Extracts encoded genotype data from binary formatted file.\n\nAnd the seven negative examples are:\n* Gets the SasLogicalInterconnects API client.\n* Read the specified number of bytes from the stream.\n* Copy a full directory structure.\n* A memoizing recursive Fibonacci to exercise RPCs.\n* Enrich the SKOS relations according to SKOS semantics, including subproperties of broader and symmetric related properties. \n* Retrieves all running tunnels for a specific user.\n* power down the OpenThreadWpan\n\nNote that due to the limited space of the rebuttal, we do not show all of the code snippets here, but we will show complete code snippets in the appendix of our next version.\n\nAs we could see, the positive query could be summarized as extracting data from a binary file. For SimCSE and BM25, the estimated weights are calculated based on the natural language descriptions of positive codes and negative codes. The second negative example contains tokens like ‘read from’ and ‘bytes’, which makes it perform like reading data from a file as well. Thus, BM25 and SimCSE consider it as the most similar negative sample hence predicting a small weight. However, the natural language description is deceptive. The code of the second negative example is:\n\n```\ndef read_bytes(self, length) -> bytes:\n        value = self.stream.read(length)\n        return value\n```\n\nThis method reads bytes not based on a file directory but on a given number. \n\nOn the contrary, since weights that are predicted by trained models are calculated by the positive query and negative codes, real potential false negative samples are captured. The code of the third negative example is:\n\n```\ndef copy_tree(src, dst, symlinks=False, ignore=[]):\n    …\n    if symlinks and os.path.islink(srcname):\n        linkto = os.readlink(srcname)\n        os.symlink(linkto, dstname)\n    …\n```\n\nHowever, there are also cases that BM25 and SimCSE perform better than trained models when the code snippets are too long and complicated which makes it hard for trained models to capture the main purpose of the code. \n\nWe think this case study will give readers a better understanding of estimation methods. We will add this case study to our next version. However, kindly note that in this paper we do not investigate estimation methods fully but mainly focus on the effectiveness of Soft-InfoNCE, as discussed in the ‘Limitations’ Section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's main strength lies in its novel approach to improving contrastive learning for code search models. The Soft-InfoNCE loss function is an innovative solution that addresses significant issues with existing methods. The authors' thorough theoretical analysis and extensive empirical evidence provide strong support for the effectiveness of their proposed method. If presented at the conference, this paper could significantly advance the field of contrastive learning for code search models and inspire further research and development in this area.", "reasons_to_reject": "While the paper presents a novel and theoretically grounded approach, it does have some weaknesses. The paper could provide more thorough comparisons with other potential methods for addressing the identified issues with the InfoNCE loss function. Additionally, while the authors provide extensive empirical evidence supporting their proposed method, it would be beneficial to see a more detailed analysis of the results, including a discussion of any limitations or potential drawbacks of the Soft-InfoNCE loss function. \n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of improving contrastive learning for code search models. It identifies significant issues with the widely used InfoNCE loss function, namely the existence of false negative samples and the inability to differentiate the potential relevance of negative samples. The paper's main contribution is the proposal of a novel Soft-InfoNCE loss function, which includes weight terms to address the identified problems. The authors suggest three methods for estimating the weights of negative pairs and demonstrate theoretically and empirically that their proposed Soft-InfoNCE loss function improves the control of the distribution of learned code representations and provides a more precise mutual information estimation.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7uU9l1Dxba", "reviewer": "Reviewer_ftqB", "summary": "", "questions": "* (major) Please mention distinct challenges in finetuning stage, so that the existing approaches are not easily applied. \n\n* Regarding the code duplications in line 56 and 183, it would be more helpful if an analysis on the finetuning corpus used in this study (CodeSearchNet) is supported, rather than only adding citations. \n\n* (minor) I think there are some works with opposite intuitions: adversarial training like [2] and minimizing the effect of false negatives [1]. From the perspective of adversarial training, the performance improvement might be come from learning more (lexical) bias. It could be interesting to evaluate the proposed method on AdvTest, and see whether the method incurs such bias. \n\n* In line 488, what is the previous statement?\n\n\n[2] Li, X., Guo, D., Gong, Y., Lin, Y., Shen, Y., Qiu, X., Jiang, D., Chen, W., & Duan, N. (2022). Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 118–129). Association for Computational Linguistics.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for the time volunteered to review our work. We are glad you appreciate that our proposed approach is simple and effective.\n\n> Q1. Typos Grammar Style And Presentation Improvements\n\nWe thank you for your careful consideration of typos and potential presentation improvements. We will revise them in our next version according to your suggestion.\n\n> Q2. (major) Please mention distinct challenges in finetuning stage, so that the existing approaches are not easily applied.\n\nThere are two works in code search that also address the false negative problem during pre-training, CodeRetriever [1] and SCodeR [2]. However, they can not be easily applied to fine-tuning because they require large amounts of data. As we know, during fine-tuning, we often can not have nearly the same size corpus as pre-training. We will discuss this in detail then.\n\n* CodeRetriever: It proposes a data mining approach to mine potential positive pairs from the pre-training corpus.\n\n* SCodeR: It introduces a discriminator and trains the encoder in an adversarial training style to minimize the effect of false negative samples.\n\nTo support our statement, we randomly sampled 20000 pairs from the CodeSearchNet Python and Java datasets (around 10% of the original dataset), respectively. And compare our methods with CodeRetriever and SCodeR. We still take the MRR as evaluation metric.\n\n| Methods | Python | Java |\n| ---- | ---- | ---- |\n| Default (InfoNCE) | 0.320 | 0.355|\n| CodeRetriever | 0.322 | 0.354 |\n| SCodeR | 0.298| 0.337 |\n| **Ours (Soft-InfoNCE)** | **0.334** | **0.363** |\n\nAs we can see in the Table, our approach could still improve the performance while CodeRetriever and SCodeR are even outperformed by merely training with InfoNCE. We find that there are only 17 and 23 pairs mined from the Python and Java datasets by CodeRetriever, which has little influence on improving the performance. For SCodeR, we think that the small amount of data can not support the training of the discriminator hence deteriorating adversarial learning. As a result, the performance even drops.\n\n**In conclusion, the distinct challenge for fine-tuning is the small corpus size, which makes pre-training methods not easily applied during fine-tuning.**\n\nWe agree with you that it is important to mention this distinct challenge to show the necessity of addressing the false negative problem during fine-tuning. We will add this discussion in our next version.\n\n[1] Li X, Gong Y, Shen Y, et al. CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 2898-2910.\n\n[2] Li X, Guo D, Gong Y, et al. Soft-Labeled Contrastive Pre-training for Function-level Code Representation[J]. arXiv preprint arXiv:2210.09597, 2022.\n\n> Q3. I feel this work is incremental unless there are distinct challenges that occur in finetuning stage, but not in pre-training.\n\nWhile we have addressed the distinct challenges during fine-tuning in our response to Q2, we do not consider our work to be merely incremental. **Besides proposing Soft-InfoNCE for addressing the problem, another important contribution of our work is the theoretical justification.** We conduct theoretical analysis to show that Soft-InfoNCE loss can control the distribution of learnt representations and reduce the variance in mutual information estimation. We also prove the superiority of Soft-InfoNCE loss over other choices of design.\n\n> Q4. In Section 4.2, though line 301-309 is the key part, it does not theoratically explain why the incorporating $w_{ij}$ results in more precise estimation. (Section 4 in [1] might be helpful for better presentation.)\n\n**As mentioned in Lines 297-299, InfoNCE loss builds a Monte-Carlo estimation sampling from a uniform distribution. And incorporating $w_{ij}$ can be considered as using the importance sampling strategy in Monte-Carlo estimation. The idea behind the importance sampling is to reduce the statistical uncertainty of Monte-Carlo estimation by drawing samples from a given distribution instead of a uniform distribution, which makes it focus on the most important region of the space [3, 4].**\n\nIn the context of our paper, we want to estimate the expectation of $\\frac{p(c_j|q_i)}{p(c_j)}$ according to a real distribution between $q_i$ and negative codes $c_j$. However, the negative codes in a batch are randomly sampled which follows a uniform distribution. To bridge this gap and get an unbiased estimator, importance sampling is adopted by inserting a weight term $w_{ij}$.\nWe denote the uniform distribution as $p$ and the real distribution as $q$. Thus, calculating expectations based on $q$ could be formulated as:\n\n$$\n\\mathbf{E_{c_j \\sim q}} \\left[ \\frac{p(c_j|q_i)}{p(c_j)} \\right] =\\mathbf{E_{c_j \\sim p}}  \\left[ \\frac{p(c_j|q_i)}{p(c_j)} \\cdot \\frac{q(c_j)}{p(c_j)} \\right]\n$$\n\nHere, the uniform distribution $p$ is $\\frac{1}{N-1}$, and the constructed real distribution is $\\frac{1-sim_{ij}}{N-2}$. Thus, $\\frac{q(c_j)}{p(c_j)} = w_{ij}$ when we take both $\\alpha$ and $\\beta$ as 1 according to Eq.4. In Lines 305-309, we discuss the validity of constructed real distribution $q$ and we empirically verify it in the left part of Fig.2. As we assumed, the real distribution is inversely proportional to $sim_{ij}$ and follow a long-tailed distribution.\n\n**Theoretical analysis in Section 4 of your mentioned paper [5] is a special case of our study.** In that paper, the authors theoretically analyze the bias only based on true negative samples and false negative samples. However, in the context of code search, it is not a discrete case. Some negative samples are more negative while some are less negative, which is a continuous case. In Table 3 of our paper, we also compare our Soft-InfoNCE with merely dividing negative samples into true and false ones. Our Soft-InfoNCE outperforms the discrete case.\n\nWe agree with you that since we skipped some steps in the theoretical analysis, this part is a bit unclear. We will add this detailed derivation in our next version.\n\n[3] Arouna, Bouhari (2004). \"Adaptative Monte Carlo Method, A Variance Reduction Technique\". Monte Carlo Methods and Their Applications.\n\n[4] Monte Carlo Methods – Lecture Notes, N. Shimkin, Spring 2015\n\n[5] Chuang, Ching-Yao, et al. \"Debiased contrastive learning.\" Advances in neural information processing systems 33 (2020): 8765-8775.\n\n> Q5. In the current presentation, the possibility of mispredicting $w_{ij}$ is not addressed, which may cause the sampling distribution far from the real distribution.\n\nAs we discussed in the ‘Limitations’ Section, the estimation methods are not fully investigated because in this paper we mainly focus on the justification of Soft-InfoNCE loss itself. \n\nYes, we admit that there is a possibility of mispredicting $w_{ij}$. As we can see, in the right part of Fig.2, for the same batch of sample, different estimation methods may even predict contradictory weights. There must be correct estimation and wrong estimation. **However, as shown in Table 1, overall Soft-InfoNCE could reach better performance, which indicates that Soft-InfoNCE is robust to mispredicted $w_{ij}$.** The detailed analysis of the Fig.2 can be found in our response to Q7.\nWe leave investigations of estimation methods as our future work.\n\n> Q6. In line 488, what is the previous statement? And for line 488-494, is that mean the scores from models are bad because of the ignorance of potential similairty in neg codes, thus other methods who have the contradictory scores (e.g. SimCSE) are better?\n\nWe find that there are contradictory predictions when we use InfoNCE-tuned code search models to predict $w_{ij}$. This phenomenon indicates that although existing code search models could find true code snippets well, they cannot recognize potential relevance among negative codes. This is the core motivation for proposing Soft-InfoNCE, which we give an example in Lines 177-182.\nSorry for not making this part clear, we will rewrite these lines in our next version.\n\n> Q7.  Regarding the code duplications in line 56 and 183, it would be more helpful if an analysis on the finetuning corpus used in this study (CodeSearchNet) is supported, rather than only adding citations. Authors may also need to show the code samples (especially those recieved contradictory scores; e.g., sample 2 of the x-axis in Fig 2's right figure)\n\nYes, we agree with you that adding case studies will help readers better understand the problems we address, and the successful and failed cases of our proposed approach. **Reviewer swyC also mentions this point. Please see our response to Reviewer swyC (Response to Q2) for a detailed case study if you are interested.** We will add this case study in our next version.\n\n> Q8. (minor) I think there are some works with opposite intuitions: adversarial training like [2] and minimizing the effect of false negatives [1].\n\nThe adversarial training method SCodeR is the paper we discuss in the response to Q2. It requires large amounts of data which often can not be satisfied during fine-tuning.\n\nThe other paper is discussed in our response to Q4. This paper could be considered as a special case of our proposed method. And our Soft-InfoNCE outperforms it as shown in Table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The approach is simple but effective", "reasons_to_reject": "* (major) In line 73-80, authors argue that, despite there exist similar approaches during pre-training, this work is novel as it conducts during finetuning. However, I feel this work is incremental unless there are distinct challenges that occur in finetuning stage, but not in pre-training. \n\n\n* In Section 4.2, though line 301-309 is the key part, it does not theoratically explain why the incorporating $w_{ij}$ results in more precise estimation. (Section 4 in [1] might be helpful for better presentation.) In the current presentation, the possibility of mispredicting $w_{ij}$ is not addressed, which may cause the sampling distribution far from the real distribution. Also, line 279-299 may fit to lie in Preliminaries, and can be summarized more. \n\n* Figures should be better presented: Figure 1 is not informative, and for Figure 2, it was hard to understand the meaning of y-axis in the left figure (samples sorted by the similarity scores?).  And for line 488-494, is that mean the scores from models are bad because of the ignorance of potential similairty in neg codes, thus other methods who have the contradictory scores (e.g. SimCSE) are better? Authors may need to show the code samples (especially those recieved contradictory scores; e.g., sample 2 of the x-axis in Fig 2's right figure) \n\n\n[1] Chuang, Ching-Yao, et al. \"Debiased contrastive learning.\" Advances in neural information processing systems 33 (2020): 8765-8775.", "questions_to_authors": "* (major) Please mention distinct challenges in finetuning stage, so that the existing approaches are not easily applied. \n\n* Regarding the code duplications in line 56 and 183, it would be more helpful if an analysis on the finetuning corpus used in this study (CodeSearchNet) is supported, rather than only adding citations. \n\n* (minor) I think there are some works with opposite intuitions: adversarial training like [2] and minimizing the effect of false negatives [1]. From the perspective of adversarial training, the performance improvement might be come from learning more (lexical) bias. It could be interesting to evaluate the proposed method on AdvTest, and see whether the method incurs such bias. \n\n* In line 488, what is the previous statement?\n\n\n[2] Li, X., Guo, D., Gong, Y., Lin, Y., Shen, Y., Qiu, X., Jiang, D., Chen, W., & Duan, N. (2022). Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 118–129). Association for Computational Linguistics.", "paper_topic_and_main_contributions": "This paper introduces Soft-InfoNCE, a weighting mechanism in the contrastive loss (InfoNCE). The key hypothesis of this work is to reduce the negative impact of false negatives by giving low weight scores to (estimated) false negative pairs. The weighting scores are obtained by 1) lexical matching (BM25), 2) unsupervised learning (SimCSE), or 3) supervised learning (a trained model with InfoNCE). The experimental results in the CodeSearchNet benchmark, with 6 different languages, generally show the effectiveness of Soft-InfoNCE over InfoNCE.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Figure 1 is too ambiguous. Please checkout Figure 1 in [1]. \n\n* Please update Figure 2, considering the item 3 in `Reasons to Reject'. \n\n* line 56, 183, 295: Please use \\citet for textual citation. \n\n* Section 4.2 should be better presented, especially from line 301 to 309. Section 4 in [1] might be helpful for revision. \n\n* line 296: Eq.(8) (Not important, but it would be good to align notations: Fig.X Eq.(X) Sec.X or Figure X Eq (X) Section X)\n\n* line 651: Soft-InfoNCE\n\n* line 550: $\\dfrac{\\alpha+\\beta}{2}=1$? (as $\\alpha = 1.3$, $\\beta = 0.7$)"}}]

Author rebuttal:

Meta-review: All the reviewers have agreed that it is interesting to see such work in code search that tackles with false negatives.

The reviewers have raised a few questions, and during the rebuttal period, the authors have largely addressed these questions.

I suggest the authors further follow the suggestions of the reviewers to make this paper more promising, e.g., applications to other NLP fields.
Decision: Accept-Main