FlipAware_all_MiniLM_L6_v2_smoke

sentence-transformers/all-MiniLM-L6-v2 fine tuned to tell apart sentences that look alike but mean different things, while keeping its normal search quality. It is a drop in replacement for the base model: same architecture, same vector size, same prompts.

Dataset: FlipAware_Triples_smoke | Leaderboard: FlipAware Leaderboard | Logs: FlipAware Runs | Code: GitHub

Results

Trap accuracy is the share of triples where the true paraphrase scores closer to the anchor than the look alike trap. Tuned numbers are mean ± std over 1 seed(s). Test sentences never appear in training.

Test	Rows	Base	Tuned
Trap test (same generator, held out sentences)	15	66.7	66.7 ± 0.0
Cross generator test (written by a different model family)	38	81.6	81.6 ± 0.0
Passage retrieval traps (pairwise)	5	80.0	80.0 ± 0.0
Passage retrieval traps (recall@1)	5	80.0	80.0 ± 0.0
NevIR negation retrieval (pairwise)	300	6.3	7.3 ± 0.0
NanoBEIR nDCG@10 (general retrieval)		51.2	47.8 ± 0.0

Verdict: GOOD. Trap accuracy +0.0 points, retrieval nDCG@10 -3.4 points, cross generator +0.0 points.

Confidence

Paired bootstrap over test triples, best seed against the base model.

Test	Gain	95% interval	Chance it is not better
Trap test	+0.0	+0.0 to +0.0	100.0%
Cross generator test	+0.0	+0.0 to +0.0	100.0%

Accuracy by trap type

Trap type	Base	Tuned
entity swap	50.0	50.0 ± 0.0
hedge	100.0	100.0 ± 0.0
negation	100.0	100.0 ± 0.0
numbers	100.0	100.0 ± 0.0
role swap	33.3	33.3 ± 0.0
scope	0.0	0.0 ± 0.0
time order	50.0	50.0 ± 0.0

Shorter and quantized vectors

Vector	Base	Tuned
384 dims float	66.7	66.7 ± 0.0
256 dims float	66.7	66.7 ± 0.0
128 dims float	66.7	66.7 ± 0.0
64 dims float	66.7	66.7 ± 0.0
int8	66.7	66.7 ± 0.0
binary (1 bit per dim)	60.0	66.7 ± 0.0

Ablation

Variant	Trap	Cross generator	Passage	NanoBEIR nDCG@10
full guard	66.7	81.6	80.0	47.8
full noguard	66.7	81.6	80.0	47.6
mined guard	66.7	81.6	80.0	47.8

mined guard trains only on the triples this base model got wrong, plus a sample of easy ones.

How much data it takes

Training triples	Trap	Cross generator	NanoBEIR nDCG@10
0	66.7	n/a	51.2
50	66.7	81.6	47.9
116	66.7	81.6	47.8
120	66.7	81.6	47.5

Blending tuned and base weights

Weights blended as (1 - alpha) * base + alpha * tuned, with no extra training.

alpha	Trap	NanoBEIR nDCG@10
0	66.7	51.2
0.25	66.7	48.5
0.5	66.7	48.4
0.75	66.7	48.1
1	66.7	47.8

Best blend inside the retrieval budget: alpha 0.

Where embedders fail

Anchor length	Rows	Base	Tuned
8 to 12 words	6	83.3	83.3
13 to 17 words	6	66.7	66.7
18 to 22 words	3	33.3	33.3

Words changed	Rows	Base	Tuned
1 to 2	11	72.7	72.7
3 to 4	2	50.0	50.0
5 or more	1	100.0	100.0

Compared with other open embedders

Full table: FlipAware Leaderboard

Training curves

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("AmareshHebbar/FlipAware_all_MiniLM_L6_v2_smoke")

a, b = model.encode(["The drug is safe for children.", "The drug is not safe for children."])
print(model.similarity(a, b))

queries = model.encode(["is the drug safe for kids"])
docs = model.encode(["The drug is not safe for children under 12."])
print(model.similarity(queries, docs))

Shorter vectors: pass truncate_dim=256 (or any trained size) when loading.

Vectors from this model are not compatible with vectors from the base model. Re embed your corpus after switching.

Training

• Data: FlipAware_Triples_smoke, anchor, paraphrase, and look alike trap across 8 trap types, plus Natural Questions retrieval pairs so search skill does not decay.
• Loss: CachedMultipleNegativesRankingLoss inside MatryoshkaLoss, with the trap as a hard negative.
• Guard: CosineSimilarityLoss pulling similarity on general sentence pairs back to the base model's own scores.
• Matryoshka dims: 384, 256, 128, 64
• Epochs 1, batch 32 (mini batch 16), learning rate 2e-05, max length 128, precision fp16, LoRA no
• Hardware: Tesla T4, 0.05 GPU hours for all runs of this base model
• Training curves: the Training metrics tab and assets/curve.png.
• Every log and metric: FlipAware Runs, live in TensorBoard and MLflow.
• Code: https://github.com/amareshhebbar/flipaware

Prior work

NevIR (Weller et al., 2024) showed that neural retrievers fail when two documents differ only by negation. This model trains a fix across eight trap types, checks it on human written and cross generator test sets, and reports general retrieval next to it, so any regression is visible.

Limitations

• English only.
• Training triples are written by Qwen/Qwen3-4B-Instruct-2507, which belongs to the same family as the Qwen3 embedders. The cross generator and human test sets exist to check for that bias.
• Eight trap types; other meaning flips are not covered.
• NanoBEIR is a small sample of BEIR, used as a quick check for general retrieval.