ArturoBE21/simcse-bert-base-snli-sup

SimCSE (supervised) trained from bert-base-uncased on SNLI

A sentence embedding model that replicates the SimCSE training method (Gao, Yao and Chen, 2021) for Universidad Politécnica de Yucatán project. It maps sentences to 768-dimensional vectors that you compare with cosine similarity.

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("ArturoBE21/simcse-bert-base-snli-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])

Training data

33,351 (premise, entailment hypothesis) pairs from a 100k SNLI subset; contradiction hypotheses as hard negatives for about 28% of the pairs. A specific SNLI subset was used, which is much smaller than the data in the paper.

Recipe

• Start: google-bert/bert-base-uncased, all weights trained
• Objective: contrastive loss with entailment pairs as positives and contradictions as hard negatives
• Pooling: [CLS] with an MLP head during training; the MLP is kept at inference
• Temperature 0.05, dropout 0.1, learning rate 3e-05, batch size 128, epochs 3, max length 32
• Seed 44, precision float16, hardware Tesla T4
• Checkpoint chosen by best Spearman on the STS-B dev set, evaluated every 50 steps (best at step 100)

Results (STS-B, Spearman x 100, cosine similarity, no regressor)

Split	Spearman
dev	82.29
test	78.26

Alignment 0.169 and uniformity -3.091 on STS-B dev (lower is better).

For reference, the paper reports 86.2 dev and 84.25 test for its supervised BERT-base model, trained on different data. The reasons for the difference are analyzed in the project report.

Limitations

• Trained on SNLI, which is made of short English image captions, so it works best on simple everyday sentences.
• Only evaluated on STS-B. Do not assume it is good for other domains, long documents or other languages.
• Sentences are truncated at 128 tokens.
• It inherits the biases of BERT and of the crowd-written SNLI sentences.
• Results are from the seed with the best STS-B dev score among 3 seeds (42, 43, 44). Scores vary across seeds; all runs are in training_logs/.