SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts
Authors: Marc Felix Brinner, Sina Zarrieß
Accepted at: acl25


Reviewer paper summaries: This work investigates the problem of scientific abstract representation learning. Instead of using external signals such as citation network, the author prioritize the semantic information from the scientific content itself and proposed a simple-yet-effective method called SemCSE. The proposed SemCSE is built on a contrastive loss leveraging the power of LLM and advantages the SOTA due to its fully unsupervised style. Extensive experiments on open benchmark and proposed dataset validate the effectiveness.

This paper introduces SemCSE, an unsupervised method for generating semantic embeddings of scientific abstracts. The approach avoids noisy citation-based supervision, instead training a model to align summaries of the same abstract while separating those of unrelated abstracts. Evaluations on a novel semantic benchmark and the SciRepEval benchmark show SemCSE outperforms citation-based models.

This paper presents SemCSE: an algorithm for fintuning sentence embeddings on scientific abstracts that leverages LLMs to provide multiple sentence summaries of the abstracts, and then learns to align the embeddings of abstracts with their LLM-generated summaries using a contrastive loss.

The paper includes an evaluation section that shows some uplift on a baseline dataset.
Summary of strengths: a) The proposed SemCSE is a fully unsupervised method.

b) A new dataset sampled from open-sourced benchmark SciRepEval is proposed.

Eliminates dependency on labeled data or citation networks, enhancing adaptability to new domains.

Uses LLM-generated summaries to capture core content, avoiding citation noise.

Outperforms citation-based models on semantic tasks and matches larger models (e.g., NvEmbed-V2) with fewer parameters.

- The paper is well-written and easy to read and follow.

- The method is very simple conceptually and easy to implement and verify.

- The three new tasks suggested are useful proxies for the embedding of scientific articles, and I can see them being used in more work on scientific document understanding.
Summary of weaknesses: a) The writing needs to be improved, it's hard to understand the challenges and the contributions of this paper e.g., how the proposed method is different from existing methods, what would be the challenges that existing methods can not solve in sci-doc representation learning.

b) The paper mentioned four limitations of using citation network (most are about potential noises) while a bunch of graph-based methods such as GraphSage [1] have been proven effective. The authors may want to add some justifications or experimental findings to prove the four claims for example, how SemCSE mitigate those problems, would it be benefit if de-noisy methods are used for SemCSE to include citation networks.

c) Lack the details of proposed dataset e.g., qaulity, human evaluation etc.

d) Unbalanced positive-negative pairs in training.

**Reference**

[1]  Robust optimization as data augmentation for large-scale graphs.

How does the choice of LLM (e.g., Llama-3/3.1-8B vs 70B vs Phi vs. GPT-4) impact summary quality and embedding performance?

How does SemCSE handle interdisciplinary papers where summaries might diverge significantly?

Were certain prompts more effective for summary generation? How were the final prompts selected?

Why use Euclidean distance instead of cosine similarity for contrastive loss? Did you experiment with alternatives?

- While the simplicity of the method is a strength, it is also lacking in novelty: many previous works have explored similar approches for finetuning embedding models with llm summaries.

- The average rank of the positives as evaluation metric: is a very noisy metric because one badly ranked positive could result in meaningless result , I would have appreciated to see more ranking measures like MAP and NDCG.

- The negative random sampling for evaluation can result in a weak baseline with easy negatives. Usually for contrastive learning papers we would want to see "hard" negatives and "easy/soft" negatives. The method for choosing such hard negatvies is not well explained or not included.

- The results table is underwhelming, while it shows good improvment over baseline models like "SciDeBERTa", it was only the best model in a single case with small ~1% difference. In all other cases it was beaten by other models, sometimes with a large margin.

- I can't find any reason why the scientific articles were exclusively chosen for this work. The same method could have been tried on other domains where the llms summaries could be better (i.e. news). Furthermore, depending on how deep is the scientific article/topic, llm's knowledge might be limited, resulting in bad summaries.
Comments, suggestions and typos: Please also refer to the Weaknesses.

a) In line 14, "a models'" -> "a model's".

b) In line 209-213, would the negative samples be much more than positive ones?

Investigate cases where SemCSE underperforms citation-based models (e.g., SPECTER2).

Report training/inference latency and memory usage.

- line 214-215: "here m is ...", while m is not included in the equation.
Overall assessment scores: 2

3

2.5
Metareview score: 3.5
Metareview: The paper augments the existing paper abstracts with multiple aspect-based summaries generated by LLMs to improve the semantic encoder for scientific papers. Instead of assuming the abstracts of two papers that cite each other are similar, this work assumes the generated summaries from the same paper are similar and uses a simple contrastive learning loss to train its semantic encoder. It could be viewed as an approach to distilling the good relevancy estimation ability from LLM, which is a common strategy in the IR community to train a dense retriever, so it is not surprising that the resulting model is better than the existing state-of-the-art encoder.

Reviewers suggest several directions that can further strengthen the papers. For example, some reviewers encourage the authors to try other tasks. Some reviewers encourage the authors to try more sophisticated contrastive learning approaches. Nevertheless, I agree with the author that the existing experiments have already been sufficient to support the main claims of this paper. I studied scientific encoders before, so I know that achieving significant improvements in these benchmarks is not easy even though the absolute performance difference looks small. The proposed method is simple, is sufficiently different from existing work, and brings about state-of-the-art results. Thus, I believe the paper is sound and should at least be accepted as a finding paper.
Summary of reasons to publish: This proposed method achieves new state-of-the-art results in a practical NLP problem. The proposed method can potentially be used to improve some practical paper recommendation tasks such as reviewer assignments in ACL-ARR.