ContraGen: Effective Contrastive Learning For Causal Language Model
Authors: Nihal Jain
Dejiao Zhang
Wasi Uddin Ahmad
Zijian Wang
Feng Nan
Xiaopeng Li
Ming Tan
Ramesh Nallapati
Baishakhi Ray
Parminder Bhatia
Xiaofei Ma
Bing Xiang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural Language Processing
Contrastive Learning
Causal Language Model
Natural Language Generation and Discrimination
Code Generation and Discrimination

Abstract: Despite exciting progress in large-scale language generation, the expressiveness of its representations is severely limited by the \textit{anisotropy} issue where the hidden representations are distributed into a narrow cone in the vector space. To address this issue, we present ContraGen, a novel contrastive learning framework to improve the representation with better uniformity and discrimination at both sequence-level and token-level. We assess ContraGen on a wide range of downstream tasks in natural and programming languages. We show that ContraGen can effectively enhance both uniformity and discrimination of the representations and lead to the desired improvement on various language understanding tasks where discriminative representations are crucial for attaining good performance. Specifically, we attain $45.9\%$ relative improvement on the Semantic Textual Similarity tasks and $33.5\%$ on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of the representations, ContraGen also boosts the source code generation capability with $9\%$ relative improvement on execution accuracy on HumanEval benchmark.

Under review as a conference paper at ICLR 2023
CONTRAGEN: EFFECTIVE CONTRASTIVE LEARNING FOR CAUSAL LANGUAGE MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
Despite exciting progress in large-scale causal language models, the expressive- ness of its representations is largely limited by the anisotropy issue where the hid- den representations are distributed into narrow cone in the vector space. To re- solve this problem, we present CONTRAGEN, a novel contrastive learning frame- work at both token-level and sequence-level. We assess CONTRAGEN range of downstream tasks and show that CONTRAGEN can effectively enhance both isotropy and discrimination of the representations. This leads the desired improvement on various language understanding tasks, which helps bridge the gap with the encoder-only models and makes causal language models more suited for tasks beyond language generation. Specifically, we attain 44% relative im- provement on the Semantic Textual Similarity tasks and 34% on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of the representa- tions, CONTRAGEN also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval benchmark.
a wide
INTRODUCTION
Causal Language Models (CLM) have seen remarkable success in language generation, both in natural language (Radford et al., 2018; 2019; Brown et al., 2020) and programming language (Chen al., 2021: Nijkamp al. 2022). However, one limitation at its core is the anisotropy issue, where the hidden representations, especially those output from the final layers of transformers, are squeezed into a tiny cone in the representation space. As observed by Ethayarajh (2019), the average cosine similarity of any two words from two randomly sampled sequences is almost at one when evaluating at the last hidden layer of GPT-2 (Radford et al., 2019). Such anisotropic often causes large performance gap (see Appendix D.I) with the encoder-based models on the discriminative tasks and hence limits the wide usage of CLM beyond language generation.
Two main reasons have been posited for causing the degenerated representations. Dong et al. (2021) theoretically prove that the key ingredient of language models - self-attention (Vaswani et al., 2017) possesses strong inductive bias towards the representation collapse. Another line of work argues that the autoregressive next token prediction objective is the main cause for the non-isotropic rep- resentations. We thereby face conundrum. On the one hand, both self-attention and next token prediction are the driving force behind the widespread successes of CLM on language generation (Lewis al., 2020; Radford al., 2019; Chen al., 2021; Chowdhery al., 2022). On the other hand, as shown in Section 4 and Appendix D.I, the resulting anisotropic representations cause infe- rior performance on language understanding tasks, e.g. Semantic Textual Similarity (Agirre al., 2012: 2013; 2014; Marelli et al., 2014; Agirre et al., 2015; 2016; Cer et al., 2017) and Code Search (Lu et al., 2021; Huang et al., 2021; Guo et al., 2022).
We tackle this challenge by leveraging contrastive learning (Chen et al., 2020; He et al., 2020; Gao al., 2021). Intuitively, by separating each instance apart from each other, contrastive learning can produce more uniformly distributed representations. Previous studies (Wang & Isola, 2020; Wang & Liu, 2021) have shown that uniformity is important for yielding discriminative representa- tions. We thereby present CONTRAGEN, novel contrastive learning framework at both token-level (CONTRAGEN- Tok) and sequence-level (CoNTrAGEN-SEOQ. Our goals are two-fold. First, an
We will release our code and checkpoints after the final decisions of acceptance out.

---

Under review as a conference paper at ICLR 2023
Representation Discriminaton
Model
Discrimination
Generation
STS-AVG
Pass I Rank
0.4
CLM
27.83
13.42
15.48
0.3
SimCTG
36.19
13.26
15.24
0.2
CLM
CONTRAGEN- -EQ
37.04
12.96
15.24
ContraCer-Tak
CONTRAGEN-
42.03
13.64
16.99
0.1
ContraGienSSe
CONTRAGEN 45.32
14.63
17.07
ContraCien
Figure 1: CONTRAGEN and its components enhance (Left) the representation discrimination ap- proximated as one minus the ratio of inter-similarity or uniformity (the cosine similarity between tokens from randomly sampled sequences) and intra-similarity (the cosine similarity between a se- quence and its constituent tokens)- - high discrimination score is better; and (Right) the performance of discrimination (STS) and generation (HumanEval) tasks (details in Section 4).
ideal CLM should be able to better leverage the representation space by dispersing apart semanti- cally different tokens or sequences. Second, we aim to attain separable features by mapping tokens or sequences under similar contexts distinct but comparatively closer locations in the vector space.
We conduct extensive experiments and analyses to assess the effectiveness of our proposed frame- work with particular focus address the following - (1) Does CONTRAGEs improve the discrimi- nation ability of representations learned by CLM? (2) Do the enhanced representations lead to better performance on language generation tasks? (3) Is the joint contrastive learning both token-level and sequence-level necessary and how do they benefit from each other? (4) How does the impact of CONTRAGEN vary across language domains?
RELATED WORK
Anisotropic Representation of Language Models Despite the remarkable success achieved by large scale language models (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2019; Raffel al., 2020; Lewis al., 2020), they suffer from the anisotropy issue where the representations are distributed into a tiny cone in the vector space (Gao et al., 2019; Ethayarajh, 2019; Li et al., 2020; Wang et al., 2020). In particular, Ethayarajh (2019) shows that the degeneration is much severer on CLM, where the average cosine similarity between two words sampled from randomly selected sequences is almost one when evaluating at the output from the last hidden layer of GPT-2 (Radford et al., 2019). Inspired by the previous findings (Arora et al., 2017: Mu & Viswanath, 2018), several efforts have focused on regularizing the representations to isotropic distribution either via postprocessing (Su et al., 2021; Li et al., 2020) directly optimizing variant regularization terms during training (Gao et al., 2019; Wang et al., 2020). Although promising improvement has been observed, performance is still inadequate.
Contrastive Learning An alternative comes to light as the popular contrastive learning techniques Chen et al. (2020); He et al. (2020) have remarkable successes in Natural Language Processing (NLP). A large amount of research has focused on sentence representation learning for the encoder- only model, with the main differences lying in how the augmentations are generated (Fang & Xie, 2020; Giorgi et al., 2021; ; Wu et al., 2020; Meng et al., 2021; Yan et al., 2021; Kim et al., 2021; Gao al.. 2021; Zhang al.. 2022). Recently there is emerging interest in developing effective con- trastive learning approach for text generation models. However, most existing work mainly focus on the encoder-decoder structure Dong et al. (2019); Raffel et al. (2020); Lewis et al. (2020) by con- trasting the suboptimal model generation, obtained via diverse sampling (An et al., 2022) adding perturbations on the embedding space (Lee et al., 2021), against the ground truth. On the other hand, it is not intuitive and even elusive to develop effective contrastive learning strategy for the decoder- only models. A recent work (Su et al., 2022) proposes, SimCTG, a token-level contrastive learning approach that aims to separate each token apart from others within the same sequence by prefixed distance. However, we find that our temperature- based token-level contrastive learning approach, CONTRAGEN TOK, consistently outperforms SimCTG across different tasks. We conjecture that
2

---

Under review as a conference paper at ICLR 2023
the fixed margin based contrastive learning allows less flexibility for the token-level representation separation within each sequence, especially considering that the semantic relevance among tokens can vary actoss contexts (sequences).
Code Generation and Beyond Language modeling for source code is an actively growing area of research. Various model architectures have been explored recently, including the encoder-only (Feng et al., 2020; Guo et al., 3021), encoder-decoder (Ahmad et al., 2021; Wang et al., 2021; Li et al., 2022), and decoder-only models (Chen et al., 2021; Nijkamp al., 2022: Chowdhery al., 2022). Among them, the decoder-only models found effective on the code generation front. However, as shown in Section 4.3.2 and Appendix D.I, they suffer from the less satisfied performance on the discriminative tasks (Lu et al., 2021: Huang et al., 2021; Guo et al., 2022). It motivates to improve the decoder-only models on the discriminative tasks so as to extend its main usage beyond language generation. Furthermore, code is fundamentally different from natural language in that it is more structured, which helps validate the generalizability of our approach beyond plain text.
CONTRASTIVE LEARNING FOR LANGUAGE GENERATION
Before present the main model, first call out the following notations that we will use through- out this section. Let x = [x1, X2, X|x| denote a sequence with variable length |x), egg., a text document or a code snippet, and h = [h1 h2. ... x]] be its representation output by the last hid- den layer of the decoder. For randomly sampled batch
{x with N sequences, we use x and by to denote the jth token and its representations in the jth sequence, respectively, Let h2, hh* denote the representation pair of sequence x and h h correspond the representations of the i-th token. Such representation pairs are referred as the positive pairs contrastive learning, which often obtained via data augmentation.
3.1 LANGUAGE GENERATION MODELING
Language generation modeling is usually formulated sequence distribution estimation over a set of examples, B = [x+] j=1-- For tractable estimation, common practice is to factorize the joint distribution of each sequence into the product of conditional token prediction probabilities. The language generation model is then trained via maximum likelihood estimation as LCLM
> |xe| i=ll log (x{))) where x²; = "x.,..,x___... x denotes the subsequence before x! refers to the sequence length. In this paper, mainly focus the decoder-only Causal Language Model (CLM), though we believe our strategy can be applied to most language models.
and x
Despite promising progress in language generation, CLM suffers from the anisotropy issue, which limits the wide usage of it beyond language generation. Ideally, to better leverage the representation space, aim to disperse apart the representations of tokens sequences from different contexts while simultaneously pulling close those from the similar contexts for better discrimination. To this end, present a novel contrastive learning framework both token-level and sequence-level.
3.2 TOKEN-LEVEL CONTRASTIVE LEARNING
As aforementioned, h h." pair of representations for x/. the i-th token in the j-th sequence. Let I, = (1.2...., | denote the indices of tokens in xj. Further let 7 denote the temperature hyper-parameter and o denotes the cosine similarity, i.e., a = a" b//la|]]||lbbl] Then for teken-level contrastive learning, minimize the following.
N log exp(h? o h² /T) EiET,AS exp(h? o h///)) exp(h) o h²" /T)
exp(h? h2 /T)
CCONTRAGEN-TON
1=1
exp(h: o h///T)
log
exp(h)* o hh/T) EiETYy
exp(hh" b exp(h"" o /T)
The above objective tries to separate the positive representation pairs of each token at distinct loca- tions apart from the others within the same sequence. Intuitively, pushing representations of different
3

---

Under review as a conference paper at ICLR 2023
tokens away from each other should indeed cause them be more uniformly distributed. On the other hand, as suggested in Section 4, the token-level contrastive learning leads to the grouping of semantically similar sequences, which conjecture is due to implicit clustering effect of con- trastive learning (Wang & Liu, 2021; Zhang et al., 2021) I) on tokens and so as the sequences consist of semantically similar tokens.
3.3 SEQUENCE-LEVEL CONTRASTIVE LEARNING
Let LB (1,2..... ( 2+ ...... N+} denote indices of all 2N sequence-level representa- tions for batch B. The sequence-level contrastive loss then follows as
exp(h' o hhTT7 /T)
CCONTRAGEN-STO
Σ
log
exp(h) ////T)+++++ ENEIEVII exp(h o h² h*/T)
==1
exp(h" ///TT)
+ log exp(h/* o o//T)) + ENEINAS exp(h* o hh*/7)
The sequence-leved contrastive loss aims to separate each sequence apart by separating its positive representation pairs from those of randomly sampled sequences (in-batch negatives). The token- level and sequence-level can complement each other for two main reasons. First, the sequence-level discrimination can help cross-sequence token-level separation. Second, as semantically irrelevant sequences still share some similar tokens, better token- level separation can help the sequence- level discrimination be more driven by the semantically irrelevant tokens.
3.4 CONTRAGEN
In summary, CONTRAGEN optimizes both the token-level and sequence-level contrastive learning objectives addition to the standard causal language modeling objective as follows
LcOnTRAGEn LCLM + LCONTRAGEN-TOK + LCOSTRAGEN-SEO
Unless otherwise specified, we weight each loss in CONTRAGEN equally and set the temperature T 0.05 for both contrastive losses. Although better performance can achieved by hyperparam- eter optimization, boost the zero-shot transfer learning performance on both discrimination and generation tasks. We therefore leave hyperparameter optimization in supervised setting for future work.
mainly investigate how CONTRAGEN improves the representation quality and
Positive pair of representations We consider the simple yet effective dropout- based augmenta- tion (Gao al., 2021). where the representation pairs are obtained by forwarding each sequence twice. Unlike the existing findings that the dropout-based augmentation can boost the contrastive learning performance when (continually) training a language model, we find that the trends can be different when evaluating on the discrimination tasks and the generation tasks. We discuss with detailed ablation study both Section 4.4 and Appendix D.2.
EXPERIMENTS
We design experiments in both natural language and programming language tasks address the four main questions listed in the Introduction section.
4.1 DATA AND MODELS
Data Models For text, we continue train GPT-2 (Radford et al., 2019) on WikiText-103: collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia (Merity et al., 2017). For code, we continue train CodeGen 350M monolingual (Nijkamp al., 2022) collected permissively licensed Python code from GitHub. Please refer to Appendix A. for the training details. We consider the following objectives for continual training both models.
CLM. The standard left to right autoregression objective for training causal language models, which is also the objective used for pretraining both GPT-2 and CodeGen.
4

---

Under review as a conference paper at ICLR 2023
SimCTG (Su et al., 2022). A margin- baagin token-level contrastive learning framework aims to separate tokens at each distinct location within a sequence apart from each other based on a predefined margin value.
CONTRAGEN-TON & CONTRAGEN-SEQ, In addition to our full model CONTRAGEN, also evaluate its components CONTRAGEN- ToK and CONTRAGEN SEQ defined in Section 3.4.
4.2
EVALUATE ON NATURAL LANGUAGE
We first evaluate our model on discrimination and generation tasks in natural language.
Model
STS12
STSI3
STS14
STSI5
STS16
SICK-R
STS-B B
Avg.
GPT2
25,84
28.90
26.20
34.74
35.70
42.72
26.27
31.48
CLM
27.14
20.34
18.73
37.56
27.40
35.70
27.97
27.83
SimCTG
30.32
37.10
31.99
39.68
42.73
46.26
25.27
36.19
CONTRAGEN-TE 37.28
37.63
31.33
54.78
50.16
48.10
34.95
42.03
CONTRAGEN-SEQ 29.66
39.89
34.50
43.20
41.99
44.52
25.51
37.04
CONTRAGEN
37.54
45.23
36.41
56.74
50.30
51.52
39.49
45.32
Table I: Spearman rank correlation between the cosine similarity of sentence representation pairs and the ground truth similarity scores.
4.2.1 SEMANTIC TEXTUAL SIMILARITY
We first evaluate semantic textual similarity (STS), the most commonly used benchmark for evalu- ating the semantic discrimination capability of representations. STS consists of seven tasks, namely STS 201 2-2016 (Agirre et al., 2012; 2013; 2014; 2015; 2016), the STS Benchmark (Cer et al., 2017), and the HICK-Relatednes (Marelli et al., 2014). Human annotators provide fine-grained similarity score ranging from () to 5 for each sequence pair in STS. Same as Reimers & Gurevych (2019), for sequence pairs in each dataset, we report the overall Spearman's 's correlation between the cosine similarities of representations and the providd similarity scores in Table 1.
Contrastive Learning Leads to Discriminative Representations Table shows both GPT-2 and the one continually trained with CLM perform poorly STS, which is consequence of the poorly discriminative representations where the cosine similarities between both semantically similar and dissimilar pairs almost at (see Figure 4 in Appendix B. Also, continuing to train GPT-2 with CLM on WikiTeex- 103 worsens performance. We hypothesize this is because WikiText- is much smaller than the original dataset used for GPT-2 pretraining and hence degenerates the trans- fer learning performance on STS. In contrast, both CONTRAGEN and SimCTG largely outperforms GPT-2, yet still, CONTRAGEN attains 25% relative improvement over SimCTG. Moreover, with the token-level contrastive learning only, CONTRAGEN- ToK still outperforms SimCTG on almost all STS benchmarks and the trend remains the same even without the dropout-based augmentation (see Appendix D.3). Therefore, we posit our temperature- bility in encouraging the semantic-dependent separations among tokens, while requiring a prefixed separation margin between tokens within the same sequence (context) as SimCTG does, is not ideal.
contrastive learning allows more flexi-
Sequence-level vs. Token-level Contrastive Learning Table also indicates that CONTRAGEN- ToK and CONTRAGEN -EQ complement each other, CONTRAGEN consistently performs better than them on all STS tasks. Note that, CONTRAGEN-SEE performs worse than CONTRAGEN- It is surprising, especially since STS mainly assesses the sequence-level representation quality. We investigate this by dividing the sequence pairs into two groups semantically similar pairs with suman-annotale similarity scores no less than 0.7 and dissimilar pairs with human scores larger than 0.3.² We plot the rank of the model inferred similarity scores against the human similarity in Figure 2 (left). As
see, CONTR GEN-SEQ struggles in ranking semantically dis- similar sequence pairs higher and similar pairs lower. This suggests that the token-level contrastive loss is essential for making the sequence-level representations robust spurious patterns of tokens
FFor all experiments in this section, we set the margin = 0.5 as recommended in Su et al. (2022). For better illistration, scaled the human annotated scores from [0, 5] [0,1] in Figure 2.
5

---

Under review as a conference paper at ICLR 2023
Dassimilar Sequence Fairs
Similar Secuence Pairs
1200
Similar Sequence Pair
Model
Rank]
Simiarity fark (beccending)
Sann-o conte soan-bc
ST: a woman is stabbing
Ground Truth
1200
Joan
potato with fork S2: a woman is punctu- ring potato with a fork
CONTRAGEN-SOE
501
1100
CONTRAGEN- GEN-TOK
272
5.0
CONTRAGEN
251
1000
430
Dissimilar Sequence Pair Model
Rankt
300
S1: a man is opening a bux and laking out puper S2: woman peeling potato
Ground Truth
1310
CONTRAGEN-
120
CONTRAGEN-TRE
1054
CONTRAGIN
1181
Ground Truth Similanty
Ground Truth Similanity
essential for making the sequence-level representations robust to spurious patterns of words phrases (results reported on STS-B).
Figure 2:
Ground Truth Text
Generated Text
Model
PPL. ISim ESim(D) Disc(D))
MAUVE
ISim ESim(S) Disc(S)) ESim(D)
Disc(D)1
GPT-2 47.50 0.96
0.96
0.00
0.893
0.95
0.95
0.00
0.95
0.00
CLM 22.48 0.95
0.95
0.00
0.945
0.95
0.95
0.00
0,94
0.01
SimCTG 22.51 0.40
0.19
0.44
0,952
0.40
0.35
0.11
0.19
0.54
CONTRAGEN-TONE 22.99
0.35
0.15
0.56
0.953
0.35
0.29
0.12
0.16
0.49
CONTRAGEN-SAE SEo 22.60
0.29
0.08
0.88
0.933
0.29
0.22
0.23
0.06
0.83
"............................... CONTRAGEN 23.01 0.33
0.11
0.67
0.947
0.33
0.25
0.18
0.10
0.62
Table 2: Perplexity and representation quality evaluated on the Wikitext- 103 test set. Disc(D) or Esim(D) is computed among two randomly sampled sequences. Disc(S) is computed between the ground truth text and the generated one associated with the same prompt. A lower Disc(S) can indicate better coherence with the ground truth. All metric reported over the entire test set.
or phrases, e.g., ranking semantically similar sequences with different synonyms low and ranking dissimilar sequences high even in presence of the same phrase (see Figure 2 (right)).
4.2.2 TEXT GENERATION
Next, we assess the open-ended language generation capability, where each model is required to generate text continuations given the prefixes from the WikiText-103 test set. Following Su et al. (2022), we set the lengths of prefix and continuation to 32 and 128, respectively. We use nucleus sampling (Holtzman et al., 2020) with top-p = 0.95. We report the model perplexity (PPL) in Table 2. PPL evaluates the model prediction confidence on the ground truth continuations in our case. The lower the PPL value, the higher the prediction probabilities assigned to the human-written text. Additionally, we propose the following for evaluating representation quality.
Intra-Similarity (ISim) is the average cosine similarity between the representations of a sequence and its constituent tokens. this paper, the sequence representation is obtained via mean pooling-
Inter-Similarity (ESim) is the average cosine similarity between the representations of quence and each token of another sequence. Regarding contextual representations, it is desired that tokens under Similar contexts (sequences) are pulled closer and hence attaining higher ESim(S) score, while tokens from Different contexts are separated apart with lower ESim(D) score.
Discrimination (Dise) defined as Disc = ESim ISim where Disc(S) and Dise(D) associate with ESim(S) and ESim(D) defined above, respectively. Thereby, lower discrimination score (Disc(S)) between each ground truth and generation is desired for better semantic coherence, while higher discrimination scores (Disc(D)) desired for generations of randomly sampled prompts for more diverse generations and better isotropic representations.
Contrastive Learning Trades Off between Model Perplexity and Representation Quality As reported in Table 2, despite slight increase in PPL. contrastive learning improves MAUVE (Pillutla et al., 2021) and the representations with better isotropy and discrimination. Furthermore, CONTRA- GEN attains better semantic coherence between the ground truth and generation indicated by the

---

Under review as a conference paper at ICLR 2023
Ruby
Python
Java
Model
Avg.
Ruby
Python
Java
Ruby
Python
Java
Ruby
Python
Java
CodeGen
16.18
5.90
0.52
2.66
18.11
0.36
1.61
1.65
10.16
6.35
CLM
16.36
6.67
0.80
3.07
15.72
0.46
1.41
2.11
10.25
6.32
SimCTG
17.66
7.19
1.94
7.63
18.31
1.78
1.63
2.32
10.83
7.70
CONTRAGEN- -OO 18.02
7.84
2.51
8.76
20.46
2,48
1.91
2.58
11.43
8.44
CONTRAGEN-SBE 16.76
5.45
1.06
7.40
16.74
1.41
1.55
2.25
10.23
6.98
CONTRAGEN 17.90
7.78
2.56
9.05
19.74
2.64
1.90
2.50
11.32
8.38
Table 3: MAP score (%) of zero-shot code search task. The language names mentioned in the top two rows indicate the languages queries and candidates are written in.
lower Disc(S) score and more diverse generations for randomly sampled prompts as indicated by the higher Disc(D) value. We stress that the zero valued Disc(S) score attained by GPT-2 and CLM is indeed a natural consequence of the inferior representations, since the discrimination score be- tween semantically irrelevant sequences also zero. Also, the slight increase on PPL is probably expected considering PPL is better aligned with the standard CLM objective. Thereby, contrastive learning can be interpreted as regularization that trades-off between PPI. and the desired represen- tation properties. We hypothesize CONTRAGEN can improve PPL with new decoding strategy that better leverages the enhanced representations.
4.3 EVALUATE ON PROGR AMMING LANGUAGE
In this section, we study the effectiveness of our proposed contrastive learning framework on pro- gramming language applications code search, code completion, and code re-ranking. Different from Section 4.2, CodeGen pretrained without using dropout from the very beginning. We follow the same setting for all models in this subsection so to study the effectiveness of CONTRAGEN without the dropout-based augmentation. We also further investigate how dropout would affect the decoder-only models when evaluated on the downstream tasks in Section 4.4 and Appendix D.2.
4.3.1 CODE SEARCH
Code search is the task of retrieving relevant code fragments given a code fragment as a query. We perform in-language (query and relevant code are in the same language) and cross-language (query and relevant code are in different languages) code search. We provide an example in Figure 5 (Appendix C. In this study, we experiment in the zero-shot setting- use the models described in Section 4.1 to generate dense representations of code and perform nearest neighbor search to retrieve relevant code fragments. We use publicly available implementations of Guo al. (2022)
Contrastive Learning Yields Discriminative Code Representations For code-to-code search task, Guo et al. (2022) used problem solutions in Ruby/Python/Java languages from CodeNet (Puri et al., 2021). They propose to use each program as a query and retrieve all programs that solve the same problem. We present detailed statistics of the dataset Table 5 (Appendix C. We set the maximum sequence length 512 and cosine similarity between two mean vectors of the last hidden states as relevance scores. We then sort the candidates by their scores to calculate the Mean Average Precision (MAP) score. We present the results for the code search tasks in Table 3.6
We observe CONTRAGEN- Tok and CONTRAGEN frameworks improve CodeGen trained with CLM by 33.5% (absolute 2. 12) and 32.6% (absolute 2.06) on average, respectively. We also point out that the performance gap between CONTRAGEN-TONN and SimCTG are apples-to-apples com- parison since the bropout-based augmentation is not used in either cases. As aforementioned, the consistent better performance of CONTRAGEN-11 ToK suggests the superiority of our temperature based contrastive learning objective. On the other hand, CONTRAGEN-Sic improves over the
"ttpps: / //gttubb... com/mi crosoft/ /Coee//ooeee /tteee/teeeeeeeeeeee weteee//eaaatte /wwst////eeeeeeeeee /wweee//niiiocooo /ww/dwtteeeenmmmee tasks $ WW also performed experiments with maximum length 1024 but didn't 't observe any significant difference. WW present comparison with encoder-only and encnder-deeoder models Table 7 in the Appendix.
7

---

Under review as a conference paper at ICLR 2023
CLM
SimCTG
ContraGen-
SDa ContraGen
ContraGen
MAP score
0.10
0.55
8.00
0.0.0
0.25
9.0
2.0
0.50
9.0
1..
1000
9.0
0.08
1000
0.20
9.9
20.00
0.00000
0.5)
[0.5, 0.75)
[0.75, 1.0)
(a) Performance breakdown based on edit similarities (x-axis).
MAP score
0.12
0.225
0.030
0.200
Poo 1000
0:000000
0.025
k.a.a.10
0.175
Inroin 00000
0.10
1..000
Poo 1000
[0. 64)
[64, 256)
[256, 512)
(b) Performance breakdown based on length differences (x-axis).
Figure 3: Code search performances based on (a) and (b) between the query code fragments (in Python) and their relevant code fragments (in Python). We observe that both cases, CONTRAGEN- ToK outperforms CLM, SimCTG, and CONTRAGEN-SEQ,
CLM baseline by 10.4% only. Code search results indicate that CONTRAGEN-SHEE performs poorly compared CONTRAGEN- language evaluation. We conjecture that CONTRAGEN- Tok generates better discriminative repre- sentations for code sequences since the finer-grained understanding of the code tokens is crucial to understanding the code sequences' functionality (semantics). To verify, we check if non-semantic factors impact model performances in the following section.
the performance gap is larger than what observed in natural
Token-level Contrastive Learning Effective for Code Understanding We break down the code search performance based on edit similarities and length differences between query code and their relevant code fragments. While edit similarity indicates how much queries and their relevant code overlap, the length difference indicates whether models effectively capture relevance between two code fragments if they are similar in length or differ significantly. We present the results for Python language in Figure 3 (for all the languages, see Figures show that CONTRAGGEN-TR edit similarities and length differences. Therefore, we can conclude that sequence overlap or length not the reasons for improvements in CONTRAGEN- TOK.. Presumably, finer-graing under- standing code tokens makes CONTRAGEN- ToK more effective for code representations.
in Appendix C.3). The results outperforms CLM, SimCTG, and CONTRAGEN-SS irrespective of
4.3.2 CODE COMPLETION AND RE-RANKING
Given a sequence tokens composed of natural language, function signature, and ouput- examples (as whole, call them prompt), the goal of the code completion task is to complete the function. To evaluate the functional correctness of complete code, we use existing benchmarks that include unit tests. If the generated code successfully passes the unit tests, successful execution. We compute pass @ k for k S following Chen et al. (2021). In addition, we compare the models in the code re-ranking task given sampled code using a code completion model, the goal of code re-ranking is to find an ordering of the code. We use the mean log probability of the sampled code to order them (Chen et al., 2021). For code re-ranking evaluation, we report ranked pass @ (Inala et al., 2022). We present an example in Figure 6 (Appendix C.1) to illustrate the code completion and re-ranking tasks. We detail the evaluation metrics Appendix C.2.
refer to this as
Contrastive Learning Improves Source Code Generation Chen et al. (2021) introduced Hu- manEval, collection of 164 handwritten programming problems and their respective unit tests. Each problem in this dataset is presented using function signature and supporting docstring, and the task is to complete the body of the function, such that the complete function can pass unit tests. In all our experiments, We sample n = 10 completions per problem with sampling temperature 0.2. Table 4 presents the evaluation results the HymanEyal benchmark.
nucleus sampling (Holtzman et al., 2020) with top p 0.95.
8

---

Under review as a conference paper at ICLR 2023
We see that CONTRAGEN-TON and CONTRAGEN- SANTRAGEN-SEQURR perform compa- rably to CLM and SimCTG, while CONTRAGEN outperforms them significantly (by 9% and 10.3% in terms of pass@1 accuracy).
Pass@k
Ranked Pass@k
Model
k=1
k=5
k=1
k=5
CodeGen
12.65
16.89
13.42 (+0.77)
17.07 (10.18)
CLM
13.42
18.08
15.38 (+1.96)
18.29 (+0.21)
Similarly, in the code re-ranking task, CONTRAGEN outperforms CLM and SimCTG baselines by 11% and 12% in terms of ranked pass@1 accuracy, respectively. While CONTRAGEN-SFQ under- performs in code completion, boosts code re-ranking significantly. We hypothesize the improvement due to the contrastive learning objective aligns with the mean log probability- re-ranking choice.
SimCTG
13.26
17.29
15.24 (+1.98)
18.29 (+1.00)
CONTRAGEN-TOR 12.96
17.01
15.24
(+2.96)
17.68 (+0.6)).
CONTRAGEN-SEE 13.64
15.85
16.99
(43.35)
16.46 (40.61)
CONTRAGEN 14.63
18.83
17.07 (+2.4) 18.90 (+0.07)
Table 4: Evaluation results on the HumanEval benchmark. The numbers in the subscript indicates the difference be- tween ranked pass@k k and pass@k. While CONTRAGEN- Tok and CONTRAGEN-SER perform competitively to the baselines, CONTRAGEN significantly outperforms them.
4.4 DISCUSSION
Impact of Dropout
Dropout-based augmentation (Gao et al., 2021) has proven to be effective for contrastive learning on language models, which often leads significant improvement when evalu- ating on discriminative tasks. We observe the same trend on both GPT-2 and CodeGen (see Tables 8b & 8a in Appendix D.2). However, we observed the opposite when evaluating on language gener- ation, no matter when training with CLM only or together with contrastive learning (see Appendix D.2). Dropout has been one of the key ingredients for training large models, further investigation on the proper ways to use and evaluate it is indeed required. On the bright side, even without dropout, Section 4.3 shows CONTRAGEN still consistently yield considerable improvement.
Bridge the Gap between Causal and Bidirectional Attention Models on Discriminative Tasks In comparison with the causal (left-to-right) attention mechanism of the decoder-only models, the bidirectional attention mechanism allows to better leverage the context of the sequence and hence yield better representations for discrimination tasks. Take the encoder-only model as an example, as Table 6 in Appendix shows, both GERT-Base (Devlin et al., 2019) and RoBERTa Base (Liu et al., 2019) outperform GPT-2 by at least 60% relative performance on STS. Despite the performance gap between CodeGen and the encoder-only or encoder-decoder models decreases in Table 7, it is still significant considering that both the model size and pretraining data size used by CodeGin are much larger. Such large performance gap severely limits the decoder-only models being used in many discriminative tasks. On the bright side, contrastive learning shows the promise to bridge the gap. As seen in Table 6, on STS, CONTRAGEN reduce the relative performance gap from 67.24% (absolute 21.12%) to 16.17% (absolute 7.33%) regarding BERT-Base and from 84.62% (absolute 26.64%) to 28.24% (absolute 12.8%) regarding RoBERTa-Bas that CONTRAGEN outperforms encoder-decoder models and perform comparably to encoder-only model GrRRTCodeBERT. Please refer to Appendix D.I for more detailed discussions.
Similarly, Table 7 shows
CONCLUSION
In this paper, we present CONTRAGEN, an effective contrastive learning framework to resolve the representation degeneration issue of causal language models trained with the standard autoregres- sion objective. We assess the effectiveness of the proposed strategy a variety of downstream tasks in both natural language domain and programming language domain, where we attain sig- nificant improvement on both discrimination tasks and language generation tasks. Moreover, we conduct in-depth analyses on our proposed token-level and sequence-level contrastive losses. Al- though only explored the decoder-only causal language models, our proposed contrastive learning framework can serve as a drop-in term for encoder decoder,,,,,, encoder-only, or prefixLM. We leave the explorations future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computa- tional Semantics Volume 1: Proceedings the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguisties. URL https: /acl antho wanthology. ogy orgg 512-1-11100 121011.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agiree, and Weiwei Guo. *SEM 2013 shared task:
Semantic textual similarity. In Second Joint Conference on Lexical and Computational Se- mantics ( (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pp. 32 -3,4,, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics. URL ht ps: / /aa/laaannttt /acllannntlloyyyy/o org/ 1-044.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Rada Mihalcea, German Rigau, and Janyce Wiehe. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Eval- uation (SemEval 2014), pp. 81-91, Dublin, Ireland, August 2014. Association for Computational URL https: httpa/aaannnooooooyooooooooooooa
Linguistics. doi: 10.3115/v 1/V11114-
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Iñigo Oazzio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), 5), pp. 252-263, Denver, Colorado, June 2015. Association for Computational Lin- guistics. doi: 10.18653/v1/S15.1 2045. URL ht / aclanthology org/s15-2044
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, Ger- man Rigau, and Janyce Wiebe. 2e06val-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemFval-20016,, pp. 497-511, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18633/1/SS16- 1081. URL https:://///c. httpp:/://aaaaathlloooooo org/ 21///16 6-1166111 1081.
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Unified pre- pre-training for pro- gram understanding and generation. Proceedings of the 2021 Conference of the N American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 2655-2668, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653vvlll URL ht : /atpaaaaannnnooccccacaca org/2021 nnaccl sesacc-mainn maa..aannn...mm 211.
Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. Cont: Contrastive neural text generation. arXiv preprint 2tt::2225.144990 2022.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat bascline for sentence embeddings. In International Conference on Learning Representations, 2017. URL https: //openreview. net./ forum? 1ddsyk0005xxx.....
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- neurips purrpss... pdf.
ciates, Inc., 2020. URL https: https:ttt/cceeeeeeeeeeee http:://c/cccccceceeeeeee 1457c0d6bfcb4s5 Att.c/fffo..111111aaaaaaaeaeeee.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-201 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of
10

---

Under review as a conference paper at ICLR 2023
the 1 11th International Workshop on Semantic Evaluation (SemEval-2017) (0ts)), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: (0.18653/11/117-20011 URL http:s/s//ccaantooooo...ooo.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL https: //ppoceedings.not mmr....s.......... mlr. press/ srress/viis /t11//v11///// chen20j html.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm:
Scaling language modeling with pathways. arXiv preprini urXiv:2204.0231 2022.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wer Wen Li, Scott Yih, Yoon Kim, and James Glass. DifiCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, pp. 4207-4218, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022 naacl 1-11-mm ma iin.in.... 311.
main.311. URL ht http///caannthccoco org/2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-111 1423. URL https: // //aclantholoyy wwwg/ogggg-........o 9-1223999111...
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: /ttp:///////ceeeeeeeeeeee newrieeiip.... seuurips... http//ppprr2211110 Paaeaaaaaaaaaaaaaapaae.ae
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 2793-2803. PMLR, 18-24 Jul 2021. URL https:
/ //proceedings www.r..ress//m.. / /13.//1//00 w13l//oog2lllvv dong2 html.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/11/D19. 1006. URL https: /acl antho ogy. org/Do 90066.
Hongchao Fang and Pengtao Xie. Cert: Contrastive self-supervised learning for language under- standing. arXiv preprint arXiv:2005. 12766 2020.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for pro- gramming and natural languages. In Findings of the Association for Computational Linguis- fics: EMNLP 2020, pp. 1536-1547, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 0006653//12202.... h0200.0..nninggsemmmp...... org/2020 f niddi naddingseentt emnn 139.
URL https: /aclant
11

---

Under review as a conference paper at ICLR 2023
Jun Gao, Di He, Xu Tan, Tao Qin. Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019. URL https: : http://ppppweeeeeeeeeeeee htt/://tntteeoeeee///ee/e
forum? wiw.mmmiiiiaa S=SXEYo )Rqt..
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1/255///12022..m//m maalpp-mai.5522.. URL org/ 2021. emnlp-ma 1...mmiin..5.... 552.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learning for unsupervised textual representations. Proceedings of the 59th Annual Meeting of the As- sociation for Compulational Linguistics and the 1 1Ith International Joint Conference Natu- ral Language Processing (Volume 1: Long Papers), pp. 879-895, Online, August 2021. As- sociation for Computational Linguistics. doi: 10. Lac-ooaaa--o URL https: // //aclanthology org/ acl-lonccloonnn 72.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yin, Daxin Jiang, et al. Graphcodehert: Pre-training code representations with data flow. In International Conference on Learning Representations, 2021. URL https: pttttt:://pperee/eee /openrevee.. net/ fest/fose htte/fooumm?cc 7id= 1tte//ooonndd-lllleee3333 4332
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7212-7225, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 499. URL https: : //a/ccaanntt wwwlanthllogyyoo.ooooooooooollle org / 22222 coccllloogg 499.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: / //openreview net http://ommmmmmdryree
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. CoSQA: 20,000+ web queries for code search and question answering, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference
Natural Language Processing (Volume 1: Long Papers), pp- 5690-5700, Online, August 2021. Association for Computational Linguistics. doi: 18.15/3//v/ 2021 lonl.acl-long..o. 442. URL https: http////aaaaaatc/eeeeeee
org/2021111.. acl-long. 442.
Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. In Advances in Neural Information Processing Systems, 2022.
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-guided contrastive learning for BERT sentence representations. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume Online, August 2021. Association for Computational Linguis-
1: Long Papers), pp. tics. doi: 10. 10.18865//1/2021..aa 197. URL https: http///aaaaaattoool.oaoooo org/ org/2021. 2021. acl-long.197.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of Association for Computational Linguistics (Volume Dublin, Ireland, May 2022. Association for Computational
1: Long Papers), pp. Linguistics. doi: 10.18653/v 1/253//110022.cllll 122022..cll-ooo..577.. 2.22.. 222.cclloog 577.
URL https: / //////aaanttt /aclanthooooggyyyoooo /rr/
12

---

Under review as a conference paper at ICLR 2023
Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. Contrastive learning with adversarial perturbations for conditional text generation. In International Conference on Learning Representations, 2021. URL htttps://operreviewt
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-seqquee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguisties. doi: 10.186.80 18853/v1/2020.odl/ maa.:aan.mmmmmm0000 703. URL https::h////mm https///aaaaanthoooco
orr.oo/////ooooo 2020. acl-mainn 703.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 9119-9130, Online, November URL
2020. Association for Computational Linguistics. doi: 10. http://0..0../1mp1ee https: aataacaannnthtoo hat/a/aaaannttoooooooooeoe org/ 2020.00202 mam.am0m0m0mmmmmmmmmmmmmmmmmm emnlp-ma in....nn....... 733.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-loeel code generation with alphacode. arXiv preprint arXiv:2203.078144 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:/9077 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: /tt///pperr/e/// ew. net/ forum? id= Bkg6RicqY7.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco. Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021. URL https: / http://pppeeeeeeeeeeeeeeeeeeeeee net// 2axum??dd=1EE40xxUUU
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), 14), pp. 223, Reykjavik, Iceland, May 2014. European Language Resources Associ- ation (ELRA), URL http://tt///. www. 1rec conf wwwoooog//oooooooooooo http://eeeinnn//eeee2ee / 363 163__Paper.por .dd.
Yu Meng, Chenyan Xiong, Payal Bajaj, saurabh tiwary, Paul Bennett, Jiawei Han, and XIA SONG. Coco-Im: Correcting and contrasting sequences for language model pretraining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 23102-23114. Curran Asso- ciates, Inc., 2021. URL ht http::::pppppp..ceiggs..ooo newrips. cc hetpiissccppappee////eee/ee pttpp//ppppp////eeeee pdf.
c2c2a0451 1a0.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference Learning Representattonss 2017. URL https:
/ //openrevien wtt.:/..eeeeeeeeeeeeeeeeeeeeeeee forum?? d=By jt 72dxe.
Jiqqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word
representations. International Conference on Learning Representations, 2018. URL https:
/oooeeeeiies ew. sat/fooumm' ditpdid-uuuuaa d=HkuGJ3kCB.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13477. 13474, 2022.
13

---

Under review as a conference paper at ICLR 2023
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- gence frontiers. Advances in Neural Information Processing Systems, 34:4816-4828 2021.
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolo- tov, Julian T Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning diversity of coding tasks. In Vanschoren and S. Yeung (eds.), Proceedings of the Neural Infor- mation Processing Systems Track Datasets and Benchmarks, volume 1, 2021. URL https: /att::/ddaaaetttee wwwpreteeeeeeemmma-e-rerreeeddde www.:/ignnnneuun.noooommmm.m...e neurips http//pp..ccccppppre///1 paper/202// pdf..
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http:
// //jmlr. http//ppprss/2211/
074. html.
Jeff Rasley, Samyam Rajbhandari, Olatuji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. In Pro- Knowledge Discovery and Data Mining, KDD '20, pp. 3505-3506, New York, NY, USA, 2020. Association for Com- URL https:
ceedings of the 26th ACM SIGKDD International Conference
puting Machinery. ISBN 978 9781450379984. doi: 10. 10.1445/33448663406700.
[/ddi.org/// 10. 3145/33944
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http: http///axivvvrrr abs/ 1ots/1904 1908 10084.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semanties and faster retrieval. arXiv preprint arX1v:2103.15316, 2021.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. contrastive framework for neural text generation. In Advances in Neural Information Processing Systems, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: http:s:///ppceeeeegsssse neurips hwwppeeeeeeiiin..ee.ie.oeooop... cttp///ccc..////////////111 cc/paper/201//Ele. 3f5ee243547dem wtt.eeeeeee11311eaaacaaaaeae Paper. Paper.pdf.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 2495-2504, 2021.
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representations, 2020. URL https: /opp/opeeeviieeeeeeeeeee/ee net / forum? Biriiiiyy-yxxxteee...
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference Machine Learning. volume 119 of Proceed- ings of Machine Learning Research, pp. 9929-9939. PMLR, 13-18 Jul 2020. URL https:
wrw.ss/vllll / //proceed igs ml www.m:......p......mm..m.m //sss/vlll //wgg/2wnkkk html.
14

---

Under review as a conference paper at ICLR 2023
Yue Wang. Weishi Wang. Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-awan unified pre-trained encoder-decoder models for code understanding and generation. Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 8696-8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational 2ttp:////2221..mpmmmaaaaal 2021. emnlp main. 685.
Linguistics. doi:
URL /aap::///calaa/ahco/caooo ogy org/
Xing Wu, Chaochen Gao. Liangjun Zang. Jizhong Han, Zhongyuan Wang. and Songlin Hu. Es- imcse: Enhanced sample building method for contrastive learning unsupervised sentence em- bedding. In Proceedings of COLING 2022, the 29th International Conference Computational Linguistics, 2022.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint (rXXi:20/2./5466 2020.
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A trastive framework for self- sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 506555067777 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 acc-oo...3.-- URL httph:tpp///ccc...tm http://ccaaantltlgooeo org/ 2021 aalalclllnnns 393.
Zhilin Yang, Zihang Dai. Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Н. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http://petoogs..neee/e/e neurips /tcep/pae//21//illee/ Papee3eeeeeeeecoeeepae.ee..
Dejiao Zhang, Shang- Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. Pairwise supervised contrastive learning of sentence representations. Proceedings of the 2021 Conference
Empirical Methods in Natural Language Processing, pp. 5786-5798, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/1120011.mmml) www.e/emainnnnnni55. m6in.467. URL https: http:::s///aaaatthhloooooo 2021. www.p-main........mmemmmmmmmaawm 466.
org/
Dejiao Zhang, Wei Xiao, Henghui Zhu, Xiaofei Ma, and Andrew Amoll. Virtual augmenta- tion supported contrastive learning of sentence representations. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 876-876, Dublin, Ireland, May 2022. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2022 finddiiinnns acl.70. URL https:
//aclanthology.coo www.hhowwo..o....oo.eoooeo 2022. fone2..finnde findings-ach acl. 70.
15

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS
A.1 SETUP DETAILS
Training Data For text, we use WikiText-103, collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia (Merity et al., 2017). For code, we collected permissively licensed Python code from GitHub. Following Chen et al. (2021); Ni- jkamp et al. (2022), we perform filtering and deduplication and further remove data that contains a significant use of non-English languages or is not parsable, resulting dataset of 101GB code.
Model We use GPT-2 (Radford et al., 2019) and CodeGen 350M monolingual (Nijkamp et al., 2022) for all experiments on natural language (text) and programming language (code), respectively. We set the batch size to 512 and continue to train GPT-2 on WikiText-103 and CodeGen on the GitHub data for 12 and 2 epochs, respectively. We trained both models using a max sequence length of 512 tokens and 1024 for WikiText-103 and Code data, respectively. We set the learning rate to 2e-5, warm-up steps as 500 with linear annealing after peak learning rate, weight decay of 0.1, the temperature of 0.05 (when using contrastive losses), and gradient clipping of 1.0. We use Adam optimizer (Loshchilov & Hutter, 2019) with 31 = 0.9, 33 = 0.999, and e = 10" following (Nijkamp et al., 2022). Our training pipeline is based on PyTorch Lightning" and use DeepSpeed (Rasley et al., 2020) for training optimization.
Processing Code Training Data Our preprocessing strategy for code datasets used for training is designed to ensure that we optimize for data utilization while retaining the syntactic structure of programming language sequences. We also eliminate duplicate sequences since this benefits training large language models (Lee et al., 2022). Specifically, we break long sequences into chunked sequences of smaller lengths to retain most parts of the original program. Further, we maintain syntactic structure in the chunks by ensuring that each chunk ends with a "\n' char- acter. Each chunk obtained this way contains most max chars per seq characters where max_chars.per.ser = max_to max_tokens www.tokenssper_eeee
* chars-per_tok = 1024. We also perform dedu- plication using character-based exact matches between chunked sequences over the entire dataset. This step helps eliminate duplcaduplicates that might be present after the chunking stage.
In our experiments,
we fix chars.per.tok = 3.2 and mex.ttoees.per.see.
B MORE ON NATURAL LANGUAGE EVALUATION
B.1 REPRESENTATION QUALITY EVALUATION ON STS
In this section, we dive deep into the model performance on the STS benchmarks. For each sequence pair in STS, a fine- aineggraindd similarity score ranging from 0 to 5 is provided, with high similarity score indicating semantically similar pairs and low similarity scores suggesting semantically dis- similar or irrelevant pairs. For better illustration, we scale the human-annotated similarity scores to [o, 1] to align with the model- model-predicted cosine similarity scores. This does not affect the evaluation as the spearman correlation reported in Section 4.2 is rank-based correlation metric.
CLM yields poorly discriminative representations We report the model predicted similarity scores of sequence pairs in Figure 4a. A good model is expected to yield representations that at- tain higher similar scores between similar sequence pairs and lower similarity values for dissimilar sequences. A large gap between the similarity scores of similar and dissimilar pairs is desired. How- ever, as seen in the left column of Figure 4, the similarity scores attained by the model trained with the standard CLM only objective are almost one for both similar sequence pairs and dissimilar pairs. This again suggests that the representations yielded by CLM are squeezed into tiny cone in the representation space rather than being scattered apart to leverage the vector space's 's capacity better. Despite the resulting similarity ranks are not entirely flattened, as shown in the right column in Figure 4b, CLM struggles in ranking similar sequences lower and dissimilar sequences higher as consequence of its poorly discriminative representations such that low similarity scores can be assigned to the semantically similar and high similarity scores to dissimilar sequence pairs.
Phttppp http:////www w/wwwwptttwowwwolll/o/ooooo ng.
16

---

Under review as a conference paper at ICLR 2023
0.95
0.90
2250
Smilarg
2000
Infered
0.80
1750
0.75
1500
0.70
1250
ContraGon
1000
0.6
0.8
1.0
500-1000 1000-1500
150C-2000 2000-2500 2500-3000 3000-3500 3500-4000 Ground Tuut Bank
Ground Truth Similarity
(a) STS14: (I.eft) Predicted cosine similarity human annotated ground truth. Right Similarity ranking according to model predicted similarity scores vs Human similarity based ranking.
1.00
2250
2,000
0.90
3.700
0.85
Infered
0.75
1000
0.65
750
0.0
0.2
0.4
0.6
0.5
1.0
0-500
500-1000
1000-1500
1500-2000 2000-2500 2500-3000 3000-3500
Ground Truth Similaityy
Ground Tuth Rank
STS15: (Left) Predicted cosine similarity vs. human annotated ground truth. Right Similarity ranking according to the model predicted similarity scores Human similarity based ranking.
Figure 4: CLM versus Contrastive Learning in Similarity Prediction and Ranking. We report the results on two STS benchmarks: (1) STSI4 where CLM performs the worst compared its perfor- mance on the other STS tasks; and STS15 where CLM attains the best performance when comparing with its own. For the purpose of illustration, we scale the human annotated similarity scores from [0, 5) to [0, 1). A good language model is expected to predict discriminative similarity scores such that the resulting ranking results are as close as the ranks provided by human as possible.
CONTRAGEN leads representations with better discrimination and contextulization In comparison, Figure 4a also validates that contrastive learning effectively yields more discriminative representations with comparatively larger similarity gap between the similar pairs and dissimilar pairs. Thereby, the similarity ranking results of the sequence pairs are more aligned with those obtained according to similarity scores provided by human. Figure 4b shows that CONTRAGEN consistently outperforms the other models, with the ranking results better matching the ground truth.
C MORE ON PROGRAMMING LANGUAGE EVALUATION
C.I TASKS EXAMPLES AND STATISTICS
In Figure 5, present example of query code fragment in Python and relevant code fragments in Python and Java, respectively. While langagge code-to-code search refers to retrieving relevant code fragments in the same language, cross- lanss-ang code-to-code search refers to retrieving code fragments in a different language. We present the statistics of the code search dataset in Table 5. To demonstrate the code completion task, illustrate example in Figure 6.
C.2 EVALUATION METRICS
Mean Average Precision (MAP) For set of queries, it indicates the mean of the average preci- sion scores for each query.
MAP De AveP(g)
Q
where Q is the number of queries.
17

---

Under review as a conference paper at ICLR 2023
Ruby
Python
Java
Total problems
1,708
2072
3142
Total #solution
11,744
15,594 7.5
23,530
Avg. #solution problem
6.9
7.5
Avg. length solution
160.4
214.4
894.9
Stdev. of length solution (problem-wie w
112.80
113.79
813.0
Solutions with length 512 Solutions with length 1024
409
1,200
10,023
278
4,766
Avg. edit similarity
(.48(+0.13)
(.52()
(.49(xa.13)
Table 5: Statistics of code-to-code search task dataset created from CodeNet (Puri et al., 2021). We truncate the code if its length exceeds maximum sequence length which 512.
Pass@k Given a problem (code prompt as as shown in Figure 6), pass@k indicates functional correctness of model generated code samples. A problem is considered solved if any sample passes the unit tests. Following Chen et al. (2021). we generate n 2 k samples per problem (in this paper, we use n 10 and & {1, 5)), count the number of correct samples e that pass unit tests, and calculate the unbiased estimator of pass @ k as:
pass
Problems
Ranked @ass@k Unlike Pass@k, where we randomly chose out of n samples, in ranked pass@1 we chose the top-k samples based on model-provided scores and then compute pass@k.
C.3 DETAILED CODE SEARCH RESULTS
We provide comparison between encoder-only (Feng et al., 2020; Guo et al., 2021), encoder-decoder (Ahmad et al., 2021; Wang al., 2021), and decoder-only models (main focus of this work) on the zero-shot code-to-code search task in Table 7. We see that CONTRAGEN-TOV and CONTRAGEN outperforms the encoder-only model CodeBERT and both the encoder-decoder models. is im- portant to note that the comparison across these models is not size, the scale of pretraining, and language settings. This comparison's 's purpose is show the promise of decoder-only models being used in discrininative tasks like code search.
apple as these models differ
We further break down the code search performances based on edit similarities and length differ- ences between query code and their relevant code fragments. We present the results in Figure and 8. observe a similar performance trend in three languages, although cross-lingual search performance still needs improve. Nonetheless, the objective of this performance analysis is to show that sequence overlap or length are not the reasons for improvements in CONTRA ToK. Instead, finer-grained understanding of code tokens due the token-level contrastive learning makes CONTRAGEN- more effective for code representations.
D MORE ANALYSIS AND DISCUSSIONS
D.1 CONTRASTIVE LEARNING BRIDGE THE GAP BETWEEN CAUSAL AND BIDIRECTIONAL. ATTENTION MODELS ON DISCRIMINATIVE TASKS
Compared to the causal (left-to-right) attention mechanism of the decoder-only models, the bidi- rectional attention mechanism in both encoder-only and encoder-decoder models allows to better leverage the context of the sequence. It hence leads to better representations when evaluated discrimination tasks.
Taking the encoder-only models example. As shown in Table 6, average, BERT-Base (Devlin et al., 2019) and Roberta-Band Baseeta-Base (Liu et al., 2019) outperform GPT-2 with 67.24% (abso- lute 21.12%) and 84.62% (absolute 26.64%) relative improvement on STS, respectively. Although
18

---

Under review as a conference paper at ICLR 2023
the performance gap between CodeGen and the BERT models trained programming languages (CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2021)) decreases or even diminishes when evaluated on the Code-Search tasks, the performance gap is still significant because both the model size and pretraining data in CodeGen are much larger than those by the encoder-only models in Table 7. Similar trends and encoder-decoder models on both natural language (Lewis et al., 2020; Raffel et al., 2020) and programming language (Ahmad al., 2021; Wang et al., 2021).
observed regarding the performance gap between the decoder-only
Model STS12
STS13
STS14
STS15
STS16
SICK-R
STS-B
Avg.
Encoder-only Models
BERT-Base 30.92
59.96
47.72
60.35
63.72
58.25
47.36
52.65
RoBERTa-Bass 53.95
47.42
55.87
64.73
63.55
62.94
58.40
58.12
Encoder- Models
BART-Basc 34.46
52.49
44.50
62.51
61.99
57.72
52.30
52.28
T5-Basc
37.78
56.81
49.37
65.50
64.65
60.11
57.52
55.96
Decoder-only Models
GPT2
25.84
28.90
26.20
34.74
35.70
42.72
26.27
31.48
CLM
27.14
20.34
18.73
37.56
27.40
35.70
27.97
27.83
SimCTG
30.32
37.10
31.99
39.68
42.73
46.26
25.27
36.19
CONTRAGEN Tok
37.28
37.63
31.33
54.78
50.16
48.10
34.95
42.03
CONTRAGEN-SER 29.66
39.89
34.50
43.20
41.99
44.52
25.51
37.04
CONTRAGEN 37.54
45.23
36.41
56.74
50.30
51.52
39.49
45.32
Table 6: Spearman rank correlation between the cosine similarity of sentence representation pairs and the ground truth similarity scores.
Ruby Python
Python Python
Java Python
Model
Avg.
Ruby
Java
Ruby
Java
Ruby
Encoder-only Models
CodeBERT
13.55
3.18
0.71
3.12
14.39
0.96
0.55
0.42
7.62
4.94
GraRhCodeBERT
17.01
9.29
6.38
5.01
19.34
6.92
1.77
3.50
13.31
9.17
Encoder-Decoder Decoder Models
Encoder-
PLBART
18
18.60
10.76
1.90
8.27
19.55
1.98
1.47
1.27
10.41
8.25
C5deT55base
18.22
10.02
1.81
8.74
17.83
1.58
1.13
0.81
10.18
7.81
Decoder-onk Models
CodeGen
16.18
5.90
0.52
2.66
18.11
0.36
1.61
1.65
10.16
6.35
CLM
16.36
6.67
0.80
3.07
15.72
0.46
1.41
2.11
10.25
6.32
SimCTG
17.66
7.19
1.94
7.63
18.31
1.78
1.63
2.32
10.83
7.70
CONTRAGEN-TEL
18.02
7.84
2.51
8.76
20.46
2.48
1.91
2.58
11.43
8.44
CONTRAGEN SEQ
16.76
5.45
1.06
7.40
16.74
1.41
1.55
2.25
10.23
6.98
CONTRAGEN
17.90
7.78
2.56
9.05
19.74
2.64
1.90
2.50
11.32
8.38
Table 7: MAP score (%) of zhoo code-to-code co-dee search task. The language names mentioned in the top two rows indicate the languages queries and candidates are written in.
The large performance gap severely limits decoder-only models used many discriminative tasks. To this end, contrastive learning shows the promise to largely bridge the gap. As seen in Table 6, on STS. CONTRAGEN reduce the relative performance gap from 67.24% (absolute 21.12%) to 16.17% (absolute 7.33%) regarding BERT-Base, and from 84.62% (absolute 26.64%) to 28.24% (absolute 12.8%). Similarly, Table shows that CONTRAGEN outperfors encoder-decoder models and perform comparably encoder-only model GraphCodeBERT.
19

---

Under review as a conference paper at ICLR 2023
Ruby Python
Python
Java
Model
Avg.
Ruby
Java
Ruby
Python
Java
Ruby
Python
Java
CLM +Drapout
18.04
6.47
1.21
5.52
18.70
1.18
1.62
2.35
11.26
7.37
CLMM Dropout
16.36
6.67
0.8
3.07
15.72
0.46
1.41
2.11
10.25
6.32
CONTRAGEN Drspout
20.09
8.84
3.66
9.25
22.39
3.13
1.93
3,06
12.02
9.37
CONTRAGEN Dropout
17.98
7.57
2.51
8.94
20.25
2.55
1.92
2.57
11.34
8.40
(a) MAP score (%) of zero-shot code-to-code scarch.
Model
STS12
STS13
STS14
STS15
STS16
SICK-R
STS-B
Avg.
CL.M Dropout
27.14
20.34
18.73
37.56
27.40
35.70
27.97
27.83
CLM Dropout
25.60
15.23
13.95
31.64
28.13
34,96
26.15
25.09
CONTRAGEN
Dropoout
37.54
45.23
36.41
56.74
50.30
51.52
39,49
45.32
CONTRAGEN
Dropout
38.22
40.15
33.57
53.16
45.35
47.47
36.10
42.00
(b) Spearman rank correlations between the cosine similarity of sentence representation pairs and the ground truth similarity scores for STS benchmarks.
Table Discriminative task performances with (-Dropou) and without (-Dropou) Dropout augmenta- tion applied to CLM and CONTRAGEN. We apply Dropout (0.1) to all the layers of the models.
D.2 DROPOUT FOR CONTRASTIVE LEARNING WITH DECODER-ONEE MODEL
Gao et al. (2021) showed that dropout-based augmentation is an effective strategy for unsupervised contrastive learning and followup works (Chuang et al., 2022; Wu et al., 2022) endorse the effective- ness. This motivates us to study dropout-basee augmentation our proposed contrastive learning framework. We present the results discriminative and generation tasks in Table 8 and 9, respec- tively. From results, it is evident that the adoption of dropout-based augmentation improves the discrimination task performances, which corroborates the findings of Gao et al. (2021).
On the other hand, dropout-bused augmentation hurts the generation task performances. While for code completion, anticipated that dropout- augmentation would hurt the performances since we use CodeGen (Nijkamp et al., 2022) in this work which does not use dropout during pre- surprise, we observe drop in perplexity due the dropout-basee augmen-
training. However, tation that does not go with our anticipation (unlike CodeGen, GPT-2 is pretrained with dropout). Therefore, leave this future exploration dive deeper into the reasoning behind this finding.
Pass @k
Ranked Pass
Model
k=1
k=5
k=1
k=5
CLM Dropouut
13.19
15.92
13.41 (40.22)
16.46 (+0.54)
CLM Dropout
13.42
18.08
15.38 (41.6)
18.29 (40.21)
CONTRAGEN +Dropout
13.19
15.92
13.41 (40.22
16.46 (+3.05)
CONTRAGEN
-Dropout
13.41
17.89
15.24 (+1.83)
18.90 (+1.01)
(a) Evaluation results on HumanEval benchmark. The numbers the subscript indicates the difference between ranked pass@k and pass @k accuracy.
CLM
Dropout
CLM Drpouut
CONTRAGEN -rropout
CONTRAGEN +roopout
Perplexity
21.86
22.48
22.07
23.01
(b) Perplexity of continually trained GPT-2 on the test set of WikiText-
Table Generation task performances with (4Dropour) and without (-Dropout) Dropout augmentation applied to CLM and CONTRAGEN. We apply Dropout (0.1) to all the layers of the models.
20

---

Under review as a conference paper at ICLR 2023
D.3 CONTRAGEN CONSISTENTLY OUTPERFORM SIMCTG
To better understand the performance gap between CONTRAGEN and SimCTG (Su et al., 2022), we the following ablations GPT-2 and report the evaluations STS.
In Table 10, we report the results of (1) running CONTRAGEN w/o dropout-based data augmenta- tion and compare it with the original SimCTG model; and (2) augmenting SimCTG with both the sequence-level contrastive loss and dropout-based augmentation and compare it with our proposed CONTRAGEN model. As we can see, CONTRAGEN consistently outperforms SimCTG. Figure 10 together with our results reported in Section 4.3 where we disabled the dropout-based augmentation for CONTRAGEN and its variations, but still observed consistent better performance than SimCTG on both discrimination and generation tasks, conclude that CONTRAGEN is better than SimCTG across domains and settings.
Model
STS12
STS13
STS14
STSI5
STSI6
SICK-R
STS-B
Avg.
SimCTG
30.32
37.10
31.99
39.68
42.73
46.26
25.27
36.19
CONTRAGEN -rropou
38.22
40.15
33,57
53.16
45.35
47.47
36.10
42.00
SimCTG Lavy Dropout
38.70
43.60
36.29
50.01
45.19
48.25
33.36
42.20
CONTRAGEN Dropout
37.54
45.23
36.41
56.74
50.30
51.52
39.49
45.32
Table 10: CONTRAGEN consistently outperform SimCTG (Su et al., 2022) even without dropout based data augmentation (first rows); or augmenting SimCTG with dropout and sequence-level contrastive loss defined in Equation 1.
21

---

Under review as a conference paper at ICLR 2023
Query Program in Python
Import math
2
b, (innt, input ( caplit ()
h$33=D w43-=0;
4
print (0)
y-ymmox min (hi ana-y
8
For hi in range (1,h1:
9
X-max ths (h-hi) ((*+1)//2), (b-hi)
nemin thiew, mwy (h-h1) ((w11)/)))))) (w) (w//2)
11
ans-min N-m)
wi range (1,
Xemox (te +h, (M-W)) ((h+1)(1+1/))) ( (h//2))
nemin wl, (M-W)) (1h11) //2), (M-w1)+ (h//2))
ans-min (ans,
print (ans)
Relevant Program in Python
(F,
[K//3*w, (B-H//3)-(K//2),, 18-8//3) (N-W//2)1
p2 [cetl (F/31 +N, (8-ce1) (8/3)) 18-ost (8/3)) (X-W/ //))
(p1)
5 22 max (p2) min
6
883 else
7
recurn (in, S2,
9.
from math import ceil F, map input () (psp1tt()))
print (solve H), solve (1)
Relevant Program Java
import java.util.4;
2
public class Main public
vold (Stringt arga)
4 Scaaner so - new Scanner (8ystem. in17 long sc.nextint
6
long h nc.nextint
7 if (x83==0 II h83-=0)
System. p0t.printin
9
elae
culiem.out..em www.nt.....nnnnn.o............n. (Math hi, thlve w)));
11
static long solve (long M, long h)
long min = Long. Maoag.MMX_LB. VALJR;
(intint im1/i<hri4t] long long b, c. 0, 0;
12Iw32--0) b w/2+ (h-i); =: min Math.min (min, Math.max math e)) MM)-Mhh..min Math.min (b, e)));
20
else 1f1(hh1142=0)1
23
w> (hh-1)2)))
25
min Math.min (min, Xath. Math.max (a, e) -Mteh-teaaaatt... Math.min e)))s
27
elset
28
w- (-hh(h1)) /2);
1/2+12;
30
nin Math.min (nin, Math Math.max et) Math.minta, Mathamin c)1);
w/24 (h-i)
32
(M/2+11
Mach.min Math. Math.max el) mMach Math.min e)));
37
Figure 5: An example of a query and relevant documents code-to-code search task, where both query and candidates complete programs that solve programming problem. In this search task, each program used as a query and retrieve all programs (from a collection of 11,744/55.54/223530 programs in Ruby, Python, and Java, respectively) that solve the same problem.
22

---

Under review as a conference paper at ICLR 2023
Prompt (function signature and docstring)
typing import
2
def wwl_coloseeemeetiie (numbers: List [flost], threshold: Elest) bool:
5
check if in given list of numbers, are any two numbers closer to each other than given threshold.
7
>>> has_clese_elemants ([1.0, o, 2.0, 3.01, 0.5)
9
>>> has_cllose_elemntts ([1.0, 2,8, 3.0, 4,0, 5.0, 2,0], 0.3)
Izue
Unit tests
1
def check (candddata))
2
asaert candidate![1.G] 2.8, 3.3, 4.3, 5.3, 2.2), 0.3) True
candidate:[1.0] 2.0, 3.9, 4.0, 5.0, 2.21, 0.05) -- Palse
4
asaert cancidate! 2.0, 5.8, 4.9, 5.31, 0.95) True
candidate
2.0,
5.3,
5.01,
6 assert candidate: [1.,,, 2.0,
3.0, 4.0, 5.0, 2.0], 0.2) == True
assert candidate [1.1, 2.2,
3.1, 4.1, 5.1], 1.0)
8 asaert cancidate [1.1, 2.2, 3.1, 4.1, 5.11, 0.5) == False
check caw.s_ooosse___eettee_eeeeee
Completion (passed; mean logp: -0.1146)
for in range (len (numbers) 1):
for j in zange (1 1, len (nunbers)):
if abs (numoara [i] numbera(])) threshold:
return True
Completion 2 (failed; mean logp: -0.1211)
For in range (len (numbers) 1):
for j in ronge (1 1, len (sumbers)) abs return True
[1] numbera(:]) threshold:
recurn
Completion 3 (failed; mean logp: -0.1598)
in range (len (numbers) 1):
iE als (numbers [i] numbers(i 11) threshold: return
recurn False
Completion 4 (failed; mean logp: logp: -0.1607)
For in range (len (numbers) 1):
(numbers(1) numbers(1 11) threshold:
return Falsa
return Tzue
Completion 5 (failed; mean logp: -0.3904) return (numbers)
(aumbers) len (numbezs)) threshold
Figure 6: An example of prompt from HumanEval and five unique code completions generated by the CONTRAGEN model. We rank them based on the mean logp scores. Considering these completions and one of the passes unit tests, pass@1 is 0.2 while ranked pass 1 is 1.0.
23

---

Under review as a conference paper at ICLR 2023
CLM
SimCTG
Contra Gen-Tok ContraGen-Tok
hadd ContraGen-Seq
ContraGen
Ruby
0.100
0.27
0.71
6.6
6.9
19.6
0.095
0,26
6.00
9.9
0.70
0.00
0.090
6.6.p000
0.25
1.9.000
0.69
1,0.0
0.24
0.0 1000
0.68
0.00 POASE
[0.
[0.5, 0.75)
[0.75,
Python
MAP score
0.10
0.26
0.0.
0.09
0.24
139
100000--0
0.22
0.0.0
2006
6.0 40.00
1.0 1000
[0, 0.5)
[0.5, 0.75)
[0.75, 1.0)
Java
0.052
0.54
0.050
6.0
000
0.17
190
0.52
0.0.0
0.16
1.2.0000
0.51
1.9.0
1000
[0. 0.5)
[0.5,0.75) 0.75)
[0.75, 1.0)
Edit Similarity
Figure 7: Code-to-code search performance (MAP score) breakdown for different models based on the edit similarities ([O, IJ) between the query code fragments and the relevant code fragments. Higher and lower edit similarity indicates the search task is trivial or difficult, respectively.
Z
CLM
SimCTG
Contra Gen-TaG
bec
ContraGen-Seq
ContraGen
Ruby
0.028
6oc
0.120
0.026
0.19
626
00
0.115
0.024
200000-
6al 5000
0.0
2000
0.18
0.0 2005
22.0000
0.022
0.00
[0,
[64, 256)
[256, 512)
Python
0.24
MAP score
0.12
0.22
0.0
0.030
0.11
0.20
2600000
0.18
0.0 200C
0.10
10..000
8. 1004
0.025
10000
000
PO0 2004
5. 5000
2 1000
[0,
[64, 256)
2256, 512)
Java
0.0060
0.080
0.0058
100c
63.0
0.00 2000
0.0056
9900000
000
0.135
0.070
0.0
0.0
(000
0.130
0.0052
[0, 64)
(56)
[256, 512)
Length Difference
Figure 8: Code-to-code search performance (MAP score) breakdown based on (absolute) length differences between the query code fragments and their relevant code fragments.
24


Reviews:
[{"review_id": "pbdXLE2tvkS", "reviewer": "Reviewer_PcBG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates adding contrastive learning objectives to LM to combat the anisotropy issue, under the assumption that increasing representation discrimination would lead to better LM. It shows a positive impact in similarity based tasks and mixed/unclear impact on LM quality.\n", "summary_of_the_review": "The experiment design can be improved and some findings cast doubt on the modeling assumptions.\n", "strength_and_weaknesses": "Strength\nIt introduces contrastive learning objectives into LM and shows it benefits similarity based tasks.\n\nWeakness\nThe experiment result cast some doubt on the experiment design and modeling assumptions.\n\n- Each new objective shows mixed results in code experiment compared to CLM, and only when combining both objectives does it outperform CLM. It casts doubt on the assumption that reducing anisotropy improves LM quality.\n- Natural language experiment (sec 4.2) should include some type of automatic metric (e.g. MAUVE) to evaluate the generation quality.\n- Given the performance gap between GPT-style and BERT-style models on similarity-based tasks, shouldn’t the paper include experiments with BERT-style models?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "feGKblVJPk", "reviewer": "Reviewer_Jhu9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "One weakness of current language models, especially decoder-only ones, is in lack of expressiveness of representations due to the CLM objective it is retrained on, which poses an issue when used for tasks such as STS or retrieval. The authors suggest augmenting this loss with contrastive loss at both a sequence and token level to mitigate this. Additionally, they suggest three metrics that measures representation quality, and show that their training regime improves upon the CLM only baseline for both code search and STS, while also improving upon source code generation.", "summary_of_the_review": "To the reader who is somewhat familiar with broader work in PLMs but not specifically in representation learning or code generation/search, this seems like a solid paper with a well-outlined motivation, methodology that makes sense, and results that back up their claims. This work is a step towards making decoder-only models more suited for all tasks (including ones that rely more on representations such as information retrieval), which, if the methodology is novel, would be worth putting in front of researchers.", "strength_and_weaknesses": "This work is a step towards making decoder-only models more suited for all tasks. This paper clearly outlines the motivation for improving model representations, defines metrics to do so, and contributes a training regime to improve representation learning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XvOI9e3yue", "reviewer": "Reviewer_9pG3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address the problem of the anisotropy issue (i.e., hidden representations of transformer models are squeezed into a tiny cone), this paper proposes to use contrastive learning for language generation (henceforth, ContraGen). The model was evaluated on both text generation and code generation. It was also assessed on tasks like semantic textual similarity and code to code search.", "summary_of_the_review": "This paper could be a good paper about language modelling, but it is nearly nothing about language generation. Since this paper does something that is inconsistent with what it claims, I cannot recommend an acceptance. ", "strength_and_weaknesses": "Contrastive learning is quite a natural idea for targeting the anisotropy issue. The author has applied the model to various NLP tasks to assess its validity. \n\nThis paper could be a good paper about language modelling, but what makes me uncomfortable is that this paper repeatedly says it is about language generation. Nevertheless, except for the code generation experiment that appears at the very end of this paper, this paper does nothing related to language generation. This results in many errors:\n1. Semantic textual similarity and code search are not downstream tasks of language generation\n2. How the PPL was used in 4.2.2 is actually for evaluating language models\n3. More justification is needed for why ISim, ESim and Disc are evaluating generation quality", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DBMui454HP", "reviewer": "Reviewer_Rmep", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The current research piece introduces a dual contrastive learning objective for training GPT-like (or Causal language ) models (CLM). The main motivation comes from the finding shown in previous work (Ethayarajh, 2019) where the representations in upper layers of CLMs  show a particular clustering (the authors called it anisotropy) that limits the capacity of such representations. Authors also hypothesize that text generation might benefit if such anisotropy is reduced.\nThe Model, ContraGen, is trained using two contrastive objectives, one token based and one whole sequence based. Two models using each individual strategy are also included for comparison purposes. ContraGen-Tok and ContraGen-Seq\nThe authors put special emphasis on comparing this approach with SimCTG, a similar approach to ContraGen-Tok with the addition of a temperature in the contrastive loss.\nExperimentation is carried out in many aspects, particularly on tasks where semantic discrimation is needed: for natural language, semantic textual similarity (STS) and text generation, and for code, Code Search and Code Completion.\nBenchmarks on STS and Code Search show a clear advantage of the current approach against baseline (GPT/CodeGEN), the model furter trained with CLM, and SimCTG.\nOn the Generation tasks, the current approach cannot surpass CLM in terms of perplexity on natural language but improves over Pass rate on code tasks\n", "summary_of_the_review": "Judging the paper from its conclusion “we present ContraGen, an effective contrastive learning framework for language generation  ..to resolve the representation degeneration issue” I think the authors did a good job in solving the latter aspect but I fail to bring enough evidence to prove the former. ", "strength_and_weaknesses": "The paper is quite ambitious but is really easy to read, clear and concise. \nA vast analysis is included in the Appendix that helps the overall understanding of the approach.\nThe improvements in terms of Semantic similarity and Code search are significant.\n\nOn the weaknesses the two main issues (discussed below) are the justification of the motivation and the results that support it.\nAuthors argue that the anisotropy is the source of limited performance on generation models but reducing that aspect didn’t bring improvement in terms of PPL for natural language, and a limited improvement in terms of code generation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B-RgLvH25y", "reviewer": "Reviewer_6wZm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new contrastive learning algorithm to deal with the anisotropy issues of large-scale language model representation. The authors present three variants of their algorithm: CONTRAGEN, CONTRAGEN-Tok, and CONTRAGEN-SEQ, where each variant either includes token loss, sequence loss, or both.", "summary_of_the_review": "The paper is well-written and straightforward to follow. The ideas in this paper are novel, and the authors empirically justify their claims. However, there are some concerns around the experiment setup, as a list in the weakness part. Because of these concerns, my initial score is borderline accept.", "strength_and_weaknesses": "Strength:\n- The contrastive loss is very intuitive and builds on prior work.\n- The experimental setup and very thorough, and most decisions are justified. \n- Experiments are run across different domains (natural language and programming language) and models (GPT2 and CodeGen) to show the generality of the proposed algorithm.\n- Ablations studies are strategically done to address concerns regarding surprising results (e.g., CONTRAGEN-SEQ performing worse than CONTRAGEN-Tok)\n\nWeakness:\n- Most semantic textual similarity models are Bert-based models. Although this paper focuses on CLM, comparing this algorithm to algorithms that use those models seems more relevant.\n- Text generation results lack evaluation metrics: BLEU, BLEURT\tBERTScore, Distinct1, Distinct2, rep-2, rep-3, rep-4, diversity, MUAVE,  etc. It is unclear how the proposed metrics compare with the standard generation metrics.\n- Natural Language results are missing a baseline algorithm, Unlikelihood training. The Natural language section introduces new metrics, but it is unclear how non-contrastive ideas perform with respect to the new metrics.\n- Lack of justification of new metrics for evaluating generation quality.\n- The motivation of the paper is not entirely empirically supported.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject