echo-sbert-domain

A sentence embedding model for short, messy, real-world app-store reviews.

Built for Echo, which turns 100,000 Swiggy Google Play reviews into tracked themes. Trained in two stages:

1. Sentence-BERT reproduction. distilroberta-base on 300,000 SNLI + MultiNLI pairs, siamese, mean pooling, classifier on (u, v, |u-v|) — written by hand in raw PyTorch, reproducing Reimers & Gurevych (2019).
2. Domain adaptation. Continued with MultipleNegativesRankingLoss on 53,061 pairs mined from the reviews themselves, where TF-IDF and the stage-1 encoder independently agree, plus SimCSE dropout self-pairs.

Results

benchmark	score
STS average (7 datasets) after stage 1	72.17
STS average after stage 2	74.54
Review retrieval, Precision@10	61.15
Review retrieval, + cross-encoder rerank	75.77
Theme assignment, blind hand-audit	82.4%

Stage 2 improved generic STS by +2.37 while adapting to the domain, which was predicted to degrade and did not. Note that 74.54 is not "beating the paper's 74.21": that number comes from NLI training alone, this adds a second stage.

Limitations, stated plainly

• It does not bridge Hinglish. "khana thanda tha" against "the food was cold" scores 0.066, versus 0.049 for a genuinely unrelated pair. Romanised Hindi reviews cluster by language rather than by subject.
• On its own it loses to TF-IDF for review retrieval (61.15 vs 65.00). It only wins in front of a cross-encoder reranker.
• Part of the retrieval gain may be circular — mined pairs required TF-IDF to agree, so the model may have partly learned to imitate it.
• Retrieval numbers rest on 26 hand-judged queries, one judge.

Use

from transformers import AutoModel, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("aynaval2003/echo-sbert-domain")
model = AutoModel.from_pretrained("aynaval2003/echo-sbert-domain").eval()

def embed(sentences):
    x = tok(sentences, padding=True, truncation=True, max_length=128,
            return_tensors="pt")
    with torch.no_grad():
        h = model(**x).last_hidden_state
    mask = x["attention_mask"].unsqueeze(-1).float()      # mean pooling,
    v = (h * mask).sum(1) / mask.sum(1).clamp(min=1e-9)   # ignoring padding
    return torch.nn.functional.normalize(v, dim=1)

Mean pooling, and it matters — this model was trained with it. CLS pooling scores 5.1 points lower in the ablation.