Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates adding contrastive learning objectives to LM to combat the anisotropy issue, under the assumption that increasing representation discrimination would lead to better LM. It shows a positive impact in similarity based tasks and mixed/unclear impact on LM quality.

strength_and_weaknesses: Strength
It introduces contrastive learning objectives into LM and shows it benefits similarity based tasks.

Weakness
The experiment result cast some doubt on the experiment design and modeling assumptions.

- Each new objective shows mixed results in code experiment compared to CLM, and only when combining both objectives does it outperform CLM. It casts doubt on the assumption that reducing anisotropy improves LM quality.
- Natural language experiment (sec 4.2) should include some type of automatic metric (e.g. MAUVE) to evaluate the generation quality.
- Given the performance gap between GPT-style and BERT-style models on similarity-based tasks, shouldn’t the paper include experiments with BERT-style models?

clarity,_quality,_novelty_and_reproducibility: This paper is well written and it could be reproduced with some effort.

- What is the x-axis of fig-1? Why not use a bar chart if the x-axis imply a different model?

summary_of_the_review: The experiment design can be improved and some findings cast doubt on the modeling assumptions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new contrastive learning algorithm to deal with the anisotropy issues of large-scale language model representation. The authors present three variants of their algorithm: CONTRAGEN, CONTRAGEN-Tok, and CONTRAGEN-SEQ, where each variant either includes token loss, sequence loss, or both.
strength_and_weaknesses: Strength:
- The contrastive loss is very intuitive and builds on prior work.
- The experimental setup and very thorough, and most decisions are justified. 
- Experiments are run across different domains (natural language and programming language) and models (GPT2 and CodeGen) to show the generality of the proposed algorithm.
- Ablations studies are strategically done to address concerns regarding surprising results (e.g., CONTRAGEN-SEQ performing worse than CONTRAGEN-Tok)

Weakness:
- Most semantic textual similarity models are Bert-based models. Although this paper focuses on CLM, comparing this algorithm to algorithms that use those models seems more relevant.
- Text generation results lack evaluation metrics: BLEU, BLEURT	BERTScore, Distinct1, Distinct2, rep-2, rep-3, rep-4, diversity, MUAVE,  etc. It is unclear how the proposed metrics compare with the standard generation metrics.
- Natural Language results are missing a baseline algorithm, Unlikelihood training. The Natural language section introduces new metrics, but it is unclear how non-contrastive ideas perform with respect to the new metrics.
- Lack of justification of new metrics for evaluating generation quality.
- The motivation of the paper is not entirely empirically supported.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written overall, and I understood most of the claims and results. The quality is the paper is very good. The authors do not include their code, so verifying the method's reproducibility is hard.
summary_of_the_review: The paper is well-written and straightforward to follow. The ideas in this paper are novel, and the authors empirically justify their claims. However, there are some concerns around the experiment setup, as a list in the weakness part. Because of these concerns, my initial score is borderline accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: One weakness of current language models, especially decoder-only ones, is in lack of expressiveness of representations due to the CLM objective it is retrained on, which poses an issue when used for tasks such as STS or retrieval. The authors suggest augmenting this loss with contrastive loss at both a sequence and token level to mitigate this. Additionally, they suggest three metrics that measures representation quality, and show that their training regime improves upon the CLM only baseline for both code search and STS, while also improving upon source code generation.
strength_and_weaknesses: This work is a step towards making decoder-only models more suited for all tasks. This paper clearly outlines the motivation for improving model representations, defines metrics to do so, and contributes a training regime to improve representation learning. 
clarity,_quality,_novelty_and_reproducibility: I am not very familiar with representation learning or code generation/search research workstreams, thus cannot speak to the novelty of the paper. It seems very clear and of high quality due to the ablations that the authors made to explore some anomalies in results (i.e. ContraGen-Seq doing worse than ContraGen-Tok in STS-B). 
summary_of_the_review: To the reader who is somewhat familiar with broader work in PLMs but not specifically in representation learning or code generation/search, this seems like a solid paper with a well-outlined motivation, methodology that makes sense, and results that back up their claims. This work is a step towards making decoder-only models more suited for all tasks (including ones that rely more on representations such as information retrieval), which, if the methodology is novel, would be worth putting in front of researchers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: To address the problem of the anisotropy issue (i.e., hidden representations of transformer models are squeezed into a tiny cone), this paper proposes to use contrastive learning for language generation (henceforth, ContraGen). The model was evaluated on both text generation and code generation. It was also assessed on tasks like semantic textual similarity and code to code search.
strength_and_weaknesses: Contrastive learning is quite a natural idea for targeting the anisotropy issue. The author has applied the model to various NLP tasks to assess its validity. 

This paper could be a good paper about language modelling, but what makes me uncomfortable is that this paper repeatedly says it is about language generation. Nevertheless, except for the code generation experiment that appears at the very end of this paper, this paper does nothing related to language generation. This results in many errors:
1. Semantic textual similarity and code search are not downstream tasks of language generation
2. How the PPL was used in 4.2.2 is actually for evaluating language models
3. More justification is needed for why ISim, ESim and Disc are evaluating generation quality
clarity,_quality,_novelty_and_reproducibility: The paper is novel but the clarity needs further improvement.

More specifically, the model description section is a bit hard to follow. For example, what does h^{j+} mean?
summary_of_the_review: This paper could be a good paper about language modelling, but it is nearly nothing about language generation. Since this paper does something that is inconsistent with what it claims, I cannot recommend an acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The current research piece introduces a dual contrastive learning objective for training GPT-like (or Causal language ) models (CLM). The main motivation comes from the finding shown in previous work (Ethayarajh, 2019) where the representations in upper layers of CLMs  show a particular clustering (the authors called it anisotropy) that limits the capacity of such representations. Authors also hypothesize that text generation might benefit if such anisotropy is reduced.
The Model, ContraGen, is trained using two contrastive objectives, one token based and one whole sequence based. Two models using each individual strategy are also included for comparison purposes. ContraGen-Tok and ContraGen-Seq
The authors put special emphasis on comparing this approach with SimCTG, a similar approach to ContraGen-Tok with the addition of a temperature in the contrastive loss.
Experimentation is carried out in many aspects, particularly on tasks where semantic discrimation is needed: for natural language, semantic textual similarity (STS) and text generation, and for code, Code Search and Code Completion.
Benchmarks on STS and Code Search show a clear advantage of the current approach against baseline (GPT/CodeGEN), the model furter trained with CLM, and SimCTG.
On the Generation tasks, the current approach cannot surpass CLM in terms of perplexity on natural language but improves over Pass rate on code tasks

strength_and_weaknesses: The paper is quite ambitious but is really easy to read, clear and concise. 
A vast analysis is included in the Appendix that helps the overall understanding of the approach.
The improvements in terms of Semantic similarity and Code search are significant.

On the weaknesses the two main issues (discussed below) are the justification of the motivation and the results that support it.
Authors argue that the anisotropy is the source of limited performance on generation models but reducing that aspect didn’t bring improvement in terms of PPL for natural language, and a limited improvement in terms of code generation.

clarity,_quality,_novelty_and_reproducibility: Some comments to review 

_Abstract The claim ”the expressiveness of its representations is severely limited by the anisotropy issue where the hidden representations are distributed into a narrow cone in the vector space” stated in the abstract should be addressed later in the paper 

_ Introduction. The claim “the resulting anisotropic representations fail to provide the desired semantic discrimination that is crucial for language understanding tasks, e.g., Semantic Textual Similarity” is arguable. The best existing models for semantic text similarity are currently all based on attention-based models such as transformers.  

_ Related Work: “.. the anisotropy issue where the representations are [..] Such degeneration is undesired, especially considering expressiveness capacity
one would expect on modern language models ”. Ok from the first comment. But stating that the degeneration is not desired and it might hurt expressiveness it’s not the same as stating that the degeneration “severely” limits it.

_ Related Work: “However, they suffermost on the discriminative tasks such as code search or clone detection (Lu et al., 2021; Huang et al., 2021; Guo et al., 2022) due to the anisotropy issue (Ethayarajh, 2019)”. First of all, none of the papers analyze the limitations of their performance due to the anisotropy issue. Second, Lu et al., 2021 (aka, CodeXGlue paper) only included Bert-based and GPT-based models for baselines purposes. CodeGPT was not even tested on clone detection nor code search. Third, Guo et al., 2022 presented an encoder- decoder model which surpassed other previous encoder only models. 

_ Section 4.3.1 Code Search:  Why the authors choose CodeNet challenge and not CodeXGlue (Lu et al., 2021). More models could have been used to locate where the performance of the ContraGen model.

summary_of_the_review: Judging the paper from its conclusion “we present ContraGen, an effective contrastive learning framework for language generation  ..to resolve the representation degeneration issue” I think the authors did a good job in solving the latter aspect but I fail to bring enough evidence to prove the former.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The similarity between pairs of tokens or sentences induced by the last layer of a language model concentrates in a small range of values (anisotropy). This paper proposes to combine classical language modeling loss with contrastive loss at the sentence and token level. This allows a model covering a wider range of similarity values. Experiments show that this strategy preserves the generative ability of the LM while performing better on similarity and search tasks over English text and computer code.
 
Strength:
- simple modification to classical neural net training, easy to implement and use.
- clarity, motivations, good description and analysis of the experiments.
 
Weaknesses:
- many tasks (e.g. classification, retrieval, parsing, tagging) relying on the similarity induced by language model can resort to extra data for fine tuning and it is not clear that anisotropy would be an issue in this case.
- the retrieval literature is ignored although it commonly trains language model with sequence level contrastive loss, often in combination with (pre) training with a likelihood objective (e.g. K. Guu et al REALM 2020, G. Izacard et al Unsupervised Dense Information Retrieval with Contrastive Learning 2021, G. Izacard et al Atlas 2022).

justification_for_why_not_higher_score: - the paper does not refer to retrieval augmented language models which blend generative and contrastive loss already.
- the paper does not show that addressing anisotropy might be fixable by fine tuning.
justification_for_why_not_lower_score: - the paper is clear, the approach is simple and easy to incorporate.
Decision: Reject