Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning

Authors: Peichao Lai, Zhengfeng Zhang, Wentao Zhang, Fangcheng Fu, Bin CUI
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Contrastive Learning, Unsupervised Sentence Representation Learning, Few-shot Learning, NLP

Abstract:
Recently, using large language models (LLMs) for data augmentation has led to considerable improvements in unsupervised sentence embedding models. However, existing methods encounter two primary challenges: limited data diversity and high data noise. Current approaches often neglect fine-grained knowledge, such as entities and quantities, leading to insufficient diversity.  Additionally, unsupervised data frequently lacks discriminative information, and the generated synthetic samples may introduce noise. In this paper, we propose a pipeline-based data augmentation method via LLMs and introduce the Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (GCSE) model to enhance unsupervised sentence embeddings. To tackle the issue of low data diversity, our pipeline utilizes knowledge graphs (KGs) to extract entities and quantities, enabling LLMs to generate more diverse, knowledge-enriched samples. To address high data noise, the GCSE model uses a Gaussian-decayed function to limit the impact of false hard negative samples, enhancing the model's discriminative capability. Experimental results show that our approach achieves state-of-the-art performance in semantic textual similarity (STS) tasks, using fewer data samples and smaller LLMs, demonstrating its efficiency and robustness across various models.

Decision: reject
Reviewer scores: 5, 6, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1729434831065

thread_type:
official_review

<thread_object>
creator: reviewer_zenj

<content>
summary:
This paper proposes a pipeline-based data augmentation method via LLMs and introduces the Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (GCSE) model to enhance unsupervised sentence embeddings. The data augmentation first constructs a knowledge graph (KG) with the extracted data, guides LLM to generate more diverse positive samples, and filter noisy data. Experimental results show its effectiveness.

strengths:
1. The paper is well-written.
2. The data augmentation is interesting.

weaknesses:
1. The main results are based on BERT and Roberta. However, LLMs are effective in sentence embedding [1,2]. Please use LLMs for evaluation. Using only ChatGLM-3 for evaluation is not convincing. Therefore, the experimental evaluation in this paper should be redone.

2. Analysis of \alpha and \beta.

3. Two missing references. [3] is about data noise and [4] is about data augmentation.

[1] BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings, NAACL 2024

[2] AoE: Angle-optimized Embeddings for Semantic Textual Similarity, ACL 2024

[3] Debiased Contrastive Learning of Unsupervised Sentence Representations, ACL 2022

[4] Improving Text Embeddings with Large Language Models, ACL 2024

questions:
Why the Gaussian-decayed function G() in the denominator? It seems that the Gaussian-decayed function has no connection with the denominator. Can we just add the Gaussian-decayed function to the contrastive learning loss?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730014275672

thread_type:
official_review

<thread_object>
creator: reviewer_wcpf

<content>
summary:
This paper proposes a data augmentation method related to the STS task, aiming to prompt LLMs for entity extraction, knowledge graph construction, and data generation. Additionally, it introduces a Gaussian-decayed function to align hard negatives between a training encoder and a frozen encoder.

strengths:
The proposed pipeline, which leverages prompt engineering, entity extraction, and knowledge graph construction, introduces a novel paradigm for data augmentation in STS tasks. The paper is well written and easy to follow.

weaknesses:
1、The motivation is not clearly clarified. why was a Gaussian-decayed function chosen? 
2、The experimental improvement is marginal.  

missing references;
[1]  Zhou K, Zhang B, Zhao W X, et al. Debiased contrastive learning of unsupervised sentence representations[J]. arXiv preprint arXiv:2205.00656, 2022.
[2] Wu X, Gao C, Su Y, et al. Smoothed contrastive learning for unsupervised sentence embedding[J]. arXiv preprint arXiv:2109.04321, 2021.

questions:
1、Why was a Gaussian-decayed function chosen? 
2、Why train a frozen encoder first, followed by training a new encoder with LLM-generated data? What's the benefit?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730088366502

thread_type:
official_review

<thread_object>
creator: reviewer_pz9x

<content>
summary:
This paper proposes a STS task-related data augmentation method, prompting LLM for entity extraction, KG construction and data generation. It also introduces a Gaussian-decayed function to align hard negatives between a training encoder and a frozen encoder.

strengths:
1. The paper is well-written. The figures are neat and clear to read.
2. The pipeline of utilizing prompt engineering, entity extraction and KG construction serves as a new paradigm of data augmentation for STS tasks.

weaknesses:
1. Analysis of false positive (FP) and negative (FN) in Figure 1 is unconvincing. See Question 1 for details.
2. The improvements (0.28 with BERT-base, 0.23 with BERT-large, and 0.46 with RoBERTa-large) seem very marginal.
3. The motivation of choosing Gaussian-decayed function is unclear. No theoretical proves on the modification with Gaussian-decayed function.

questions:
1. In figure 1, why choosing 0.5 as a hard limit to distinguish Positives & Negatives? Can we choose other limits, like >0.8 is Positives, < 0 is Negatives, and 0~0.8 is ambiguous? Theoretically,  most embedding models like SimCSE are trained with a list-wise temperature-scaled contrastive loss, which is basically optimized via Cross-Entropy. The distribution of query-positive cosine similarity scores will be pushed greatly towards 1.0, and query-negative scores towards -1.0. If SimCSE is trained with some optimization object like a point-wise MSE loss, choosing 0.5 will be more rational. Thus, I suggest you to set a hard limit Top-k window for the prediction of Positives & Negatives. For example, the prediction with largest similarity is positive, and lowest is negative.
2. Can you do statistic significant tests on your results?
3. Where is the evaluation model E′ in Figure 4?
4. Why choosing Gaussian-decayed function? Does it have some special mathematical traits? Can we just choose a MSE loss / Cross-Entropy loss / KL loss to align hard negatives between the training encoder and frozen encoder?
5. Why do you want to align the hard negatives between the training encoder and frozen encoder?
6. Why do you train a frozen encoder first, then train a new encoder with the LLM-generated data? Can we just merge these steps to one stage? 
7. Line 311: How would the “spatial distribution of negatives” change? How will the gradients update?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730390789057

thread_type:
official_review

<thread_object>
creator: reviewer_hrdn

<content>
summary:
To address the challenges of unsupervised sentence embedding models in terms of data diversity and noise, this paper first proposes a pipeline-based data augmentation method using large language models. This method utilizes knowledge graphs to extract entity and quantity information, generating more diverse samples. Next, the paper introduces a Gaussian Contrastive Sentence Embedding (GCSE) model assisted by a Gaussian decay function to reduce the impact of false negative samples and enhance the model's discriminative ability. Finally, the paper demonstrates that this approach achieves SOTA performance on semantic textual similarity  tasks, while exhibiting efficiency and robustness with fewer data samples and smaller LLMs.

strengths:
1. By integrating knowledge graphs and LLMs, the paper proposes a novel data augmentation approach that effectively enhances sample diversity and embedding quality.
2. The introduction of the Gaussian decay function significantly mitigates the impact of false negative samples, bolstering the model's robustness.
3. While reducing data requirements and model size, the method presented in this paper achieves state-of-the-art performance across multiple benchmarks, demonstrating its efficiency and generalization capability.

weaknesses:
1. The core motivation of this paper's approach is data diversity and data noise. Although higher benchmark performance has been achieved, there is a lack of significant quantitative analysis on how the proposed methods or modules enhance data diversity and reduce data noise. For example, could some quantitative metrics be used to intuitively demonstrate the improvements in data diversity and reductions in data noise brought about by the proposed method or individual modules?

questions:
1. Is it possible to visually demonstrate the improvement in data diversity and reduction in data noise brought about by the proposed method through some quantitative metrics (same as Weakness) ?

2. Ablation analysis of the hyperparameters α and β in equations 10 and 11.

3. The impact of different LLMs on performance, and whether replacing with a more powerful LLM could raise the method's upper limit.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731066400887

thread_type:
official_review

<thread_object>
creator: reviewer_n2jx

<content>
summary:
This paper investigates the use of large language models for data augmentation to generate unsupervised sentence embeddings. To overcome two limitations of existing methods: limited data diversity and high data noise, the authors propose using knowledge graphs to extract entities and quantities to address low data diversity. Additionally, they suggest using a Gaussian-decayed function to limit the impact of false hard negative samples in the Contrastive Sentence Embedding model to tackle the issue of high data noise. Experimental results on semantic textual similarity (STS) and reranking tasks indicate that the proposed method offers certain improvements over existing large language model-based approaches.

strengths:
The authors have provided a thorough analysis and summary of the shortcomings of existing methods.

This paper is easy to follow, and the technical approach is sound.

weaknesses:
Although introducing the Gaussian-decayed function into SimCSE is straightforward, its contribution does not seem very significant.

From the results in Tables 2 and 3, the improvement of GCSE over SynCSE appears to be relatively minor in many cases. I suggest conducting multiple experiments and reporting the results of statistical significance tests.

questions:
If SynCSE and GCSE use the same LLM, would the improvement be greater? Could you provide these results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111415960

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
Thanks very much for taking the time to review our manuscript, and we really appreciate all the comments and suggestions.
In response to the two major concerns regarding the motivation behind the Gaussian-decayed function and whether the model performance can achieve the upper limit with more powerful LLMs:
(1) We have expanded on the motivation of the Gaussian-decayed function in the Introduction to provide further clarification. Additionally, we have included a detailed analysis in Appendix G, where we visualize and demonstrate how the Gaussian-decayed function helps in mitigating the noise from negative samples.
(2) Regarding model performance, we extended our experiments using the same setup and evaluated the performance of our model with samples synthesized by GLM4-9B and ChatGPT. The results of these experiments have been added to Tables 2 and 7. Our model outperforms previous state-of-the-art methods in average scores for semantic textual similarity (STS) tasks, achieving improvements of:
	1.05% with BERT-base
	1.62% with BERT-large
	0.49% with RoBERTa-base
	1.50% with RoBERTa-large
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a pipeline-based data augmentation method utilizing LLMs combined with a novel Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (model to enhance unsupervised sentence embeddings. The proposed approach integrates knowledge graphs for data diversity and employs a Gaussian decay function to mitigate data noise, resulting in good performance on semantic textual similarity tasks. 

However, as pointed out by the reviewers, there are several concerns, including the marginal improvements over existing methods, the unclear theoretical justification for the Gaussian-decayed function, and the lack of significant quantitative analysis demonstrating how the proposed methods enhance data diversity or reduce noise. Even though the authors tried to address these issues during the rebuttal, the reviewers were not fully satisfied with the answers, particularly regarding the practical significance and theoretical underpinnings of the improvements reported.