Harnessing Dimension-level Contrastive Learning and Information Compensation Mechanism for Sentence Embedding Enhancement
Authors: Kang He, Yuzhe Ding, Bobo Li, Haining Wang, Fei Li, Chong Teng, Donghong Ji


Reviewer paper summaries: This paper  design a novel sentence embedding enhancement method, MSSE, consisting of an information compensation mechanism (ICM) and a dimensional-level contrastive learning mechanism (DCM). The ICM is achieved by an AutoEncoder, while the DCM is achieved by dimension-wise contrastive learning, i.e., BarlowTwins.
Summary of strengths: 1. The experimental results of the proposed method surpass the baselines.
Summary of weaknesses: 1. Overlap with existing studies on sentence embeddings. The proposed Dimension-wise contrastive learning is already proposed and discussed in [1][2][3]. ([1] tackles the rank bottleneck problems for [2], also the upper bound as well.) All of those works are identical or strongly correlated with the method the author proposed, however, I see no citation or discussion on them at all. Besides, the AutoEncoder structure is not interesting as well. 

2. Unfair comparison. The data augmentation used here is roundtrip translation, which is not an unsupervised setting at all (conventional unsupervised settings in sentence embeddings are only dropout augmentation). However, authors still compare their proposed method with unsupervised methods such as DistllCSE, etc. Utilizing supervised data such as roundtrip translation, paraphrasing, etc. would significantly affect the embedding model's performance, leading to erroneous conclusions for embedding studies. I strongly disagree with such a practice of claiming the proposed method as an unsupervised method while using roundtrip translation. If it is purely unsupervised, how could you train a roundtrip translation NMT system? Seriously? Why don’t authors use GPT4 for data augmentation and claim it as unsupervised as well? All NLP researchers are responsible for doing rigorous research.

3. Missing some important baselines. Such as SimCSE++[1], SCE[3], CT-BERT[4], and ConSERT-BERT [5].

**Refs**

[1] Jiahao Xu, Wei Shao, Lihui Chen, and Lemao Liu. 2023. SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12028–12040, Singapore. Association for Computational Linguistics.

[2] Zbontar, Jure, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. "Barlow twins: Self-supervised learning via redundancy reduction." In *International conference on machine learning*, pp. 12310-12320. PMLR, 2021.

[3] Tassilo Klein and Moin Nabi. 2022. [SCD: Self-Contrastive Decorrelation of Sentence Embeddings](https://aclanthology.org/2022.acl-short.44). In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 394–400, Dublin, Ireland. Association for Computational Linguistics.

[4]  Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations.

[5] Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601–1610, Online. Association for Computational Linguistics.
Comments, suggestions and typos: See weakness
Overall assessment scores: 1.5