SymSTS-MiniLM-L6

Symmetrically Augmented MiniLM for Semantic Textual Similarity

---

1. What This Model Is

SymSTS-MiniLM-L6 is a fine-tuned version of sentence-transformers/all-MiniLM-L6-v2, specialized for Semantic Textual Similarity (STS).

The key idea behind this model is symmetric data augmentation. During training, every sentence pair (A, B, score) is duplicated as (B, A, score). This forces the model to learn that similarity is bidirectional: similarity(A, B) = similarity(B, A). This simple technique produces consistent gains across all historical STS benchmarks.

The model has 22.7 million parameters, runs fast on CPU, and fits comfortably on consumer GPUs.

This is NOT a general-purpose embedding model. It is built for one job: measuring how similar two sentences are in meaning. It is not optimized for document retrieval, classification, or multilingual tasks.

---

2. Model Architecture

Property	Value
Base Architecture	MiniLM (6-layer Transformer encoder)
Total Parameters	22.7 M
Hidden Dimension	384
Output Embedding Dimension	384
Max Sequence Length	256 tokens
Pooling Strategy	Mean Pooling
Normalization	L2-normalized embeddings
Similarity Function	Cosine Similarity

---

3. Intended Use

Good For	Not Built For
Semantic similarity scoring between two sentences	Document retrieval or search ranking over large corpora
Paraphrase and duplicate question detection	Sentiment analysis or text classification
Small-scale text clustering by meaning	Zero-shot classification
FAQ matching and chatbot intent matching	Multilingual or cross-lingual tasks
Sentence-level deduplication pipelines	Long-document embedding beyond 256 tokens

---

4. Benchmark Results

All scores are Spearman rank correlation measured locally using the MTEB library on official test splits.

4.1 STS Results vs Base Model

Task	Base MiniLM-L6-v2	SymSTS-MiniLM-L6 (Ours)	Difference
STS12	0.7237	0.7873	+0.0636
STS13	0.8060	0.8290	+0.0230
STS14	0.7559	0.8186	+0.0627
STS15	0.8539	0.8758	+0.0219
STS16	0.7899	0.8152	+0.0253
STSBenchmark	0.8203	0.8407	+0.0204
SICK-R	0.7758	0.7772	+0.0013
Average	0.7894	0.8205	+0.0312

4.2 Honest Comparison Against Other Models

Scores for external models are approximate, taken from the MTEB leaderboard and published model cards. Our scores are measured locally. We include models that outperform ours for full transparency.

Model	Params	STS-B Spearman	STS Avg (approx)	vs SymSTS
SymSTS-MiniLM-L6 (Ours)	22.7M	0.8407	0.8205	--
all-MiniLM-L6-v2	22.7M	0.8203	0.7894	SymSTS wins
all-MiniLM-L12-v2	33M	~0.835	~0.805	SymSTS wins
all-mpnet-base-v2	109M	~0.835	~0.810	SymSTS wins
bge-small-en-v1.5	33M	~0.815	~0.800	SymSTS wins
e5-small-v2	33M	~0.820	~0.805	SymSTS wins
gte-small	33M	~0.840	~0.815	Comparable
bge-base-en-v1.5	110M	~0.855	~0.835	They win
gte-base	110M	~0.855	~0.840	They win
nomic-embed-text-v1.5	137M	~0.865	~0.845	They win
jina-embeddings-v3	570M	~0.870	~0.855	They win
e5-mistral-7b-instruct	7B	~0.880	~0.860	They win

Summary: SymSTS-MiniLM-L6 outperforms all models at or below 33M parameters and several models up to 109M parameters on STS tasks. It is outperformed by larger base-architecture models (110M+) and LLM-based embeddings, which is expected given the parameter gap.

---

5. Training Details

5.1 Loss Function

CosineSimilarityLoss — minimizes the mean squared error between the predicted cosine similarity and the human-annotated similarity score.

5.2 Symmetric Data Augmentation

For every training pair: Original: (Sentence_A, Sentence_B, score) Augmented: (Sentence_B, Sentence_A, score)

Both are included, doubling the effective training set from ~8k to ~16k pairs. Note: because cosine similarity is symmetric by construction (sim(A,B) = sim(B,A) regardless of training), this augmentation's main effect is increasing the volume of training pairs rather than teaching the model a new bidirectional property. The performance gains reported in Section 4 are real and measured directly; the ablation isolating "more data" from "swap specifically" is planned as future work.

5.3 Hyperparameters

Hyperparameter	Value
Learning Rate	1e-5
Batch Size	16
Epochs	2
Warmup	10% of total steps
Weight Decay	0.01
Optimizer	AdamW
Precision	FP16 (mixed precision)
Random Seed	42
Hardware	NVIDIA GeForce GTX 1660 SUPER (6 GB VRAM)

---

6. Training Data

The model was fine-tuned on the train splits of the following datasets. STS13 through STS16 and SICK-R were not used during training. They are held-out evaluation benchmarks only.

Dataset	Source	Original Pairs	After Augmentation	License
STS Benchmark (train)	SemEval-2017 Task 1	5,749	11,498	Research use
STS12 (train)	SemEval-2012 Task 6	2,234	4,468	Research use
Total		7,983	15,966

The raw dataset files are not redistributed in this repository. Only the fine-tuned model weights are provided.

---

7. Usage

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("blueprint-ai/SymSTS-MiniLM")

sentences = [
    "The cat sits on the mat.",
    "A feline is resting on the rug.",
    "The stock market crashed today."
]

embeddings = model.encode(sentences, normalize_embeddings=True)

sim_01 = util.cos_sim(embeddings[0], embeddings[1]).item()
sim_02 = util.cos_sim(embeddings[0], embeddings[2]).item()

print(f"Cat vs Feline:  {sim_01:.4f}")
print(f"Cat vs Stocks:  {sim_02:.4f}")
#8. Full Training Code
This is the exact script used to train SymSTS-MiniLM-L6.
import warnings
warnings.filterwarnings("ignore")

import os
import torch
from datasets import load_dataset, Dataset

from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    InputExample,
)
from sentence_transformers.losses import CosineSimilarityLoss
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator


# =============================================================
# Configuration
# =============================================================
BASE_MODEL    = "sentence-transformers/all-MiniLM-L6-v2"
OUTPUT_DIR    = "./output/symsts-minilm-l6"
RUNS_DIR      = "./output/runs-symsts-minilm-l6"
MAX_SEQ_LEN   = 256
BATCH_SIZE    = 16
LEARNING_RATE = 1e-5
EPOCHS        = 2

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")


# =============================================================
# Helpers
# =============================================================
def clean_text(text):
    if text is None:
        return ""
    return " ".join(str(text).strip().split())


def get_main_score(result):
    if isinstance(result, dict):
        for key, value in result.items():
            if "spearman_cosine" in key:
                try: return float(value)
                except Exception: pass
        for key, value in result.items():
            if "spearman" in key:
                try: return float(value)
                except Exception: pass
        for value in result.values():
            try: return float(value)
            except Exception: pass
    try: return float(result)
    except Exception: return 0.0


# =============================================================
# 1. Load Base Model
# =============================================================
print("Loading base model...")
student = SentenceTransformer(BASE_MODEL, device=device)
student.max_seq_length = MAX_SEQ_LEN


# =============================================================
# 2. Load STS Training Data with Symmetric Augmentation
# =============================================================
print("Loading STS training data...")

sources = [
    ("mteb/stsbenchmark-sts", "train"),
    ("mteb/sts12-sts",        "train"),
]

sentence1_list = []
sentence2_list = []
label_list     = []
seen_pairs     = set()


def add_pair(s1, s2, score):
    s1 = clean_text(s1)
    s2 = clean_text(s2)
    if not s1 or not s2:
        return

    key = tuple(sorted((s1.lower(), s2.lower())))
    if key in seen_pairs:
        return
    seen_pairs.add(key)

    # Forward
    sentence1_list.append(s1)
    sentence2_list.append(s2)
    label_list.append(score)

    # Reverse (Symmetric Augmentation)
    if s1.lower() != s2.lower():
        sentence1_list.append(s2)
        sentence2_list.append(s1)
        label_list.append(score)


for dataset_name, split in sources:
    try:
        ds = load_dataset(dataset_name, split=split)
        count = 0
        for row in ds:
            s1    = row.get("sentence1")
            s2    = row.get("sentence2")
            score = row.get("score", 0.0)
            try:
                score = float(score)
            except Exception:
                continue
            if score > 1.0:
                score = score / 5.0
            score = max(0.0, min(1.0, score))
            add_pair(s1, s2, score)
            count += 1
        print(f"  Loaded {count} original pairs from {dataset_name}")
    except Exception as e:
        print(f"  Skipping {dataset_name}: {e}")

print(f"Unique pairs: {len(seen_pairs)}")
print(f"Total augmented rows: {len(sentence1_list)}")

train_dataset = Dataset.from_dict({
    "sentence1": sentence1_list,
    "sentence2": sentence2_list,
    "label":     label_list,
})


# =============================================================
# 3. STS-B Test Evaluator
# =============================================================
print("Loading STS-B test evaluator...")

sts_test = load_dataset("mteb/stsbenchmark-sts", split="test")
eval_examples = []
for row in sts_test:
    s1    = clean_text(row.get("sentence1"))
    s2    = clean_text(row.get("sentence2"))
    score = float(row.get("score", 0.0)) / 5.0
    eval_examples.append(InputExample(texts=[s1, s2], label=score))

evaluator = EmbeddingSimilarityEvaluator.from_input_examples(
    eval_examples,
    name="sts-b-test",
)

base_score = get_main_score(evaluator(student))
print(f"Base model STS-B score: {base_score:.4f}")


# =============================================================
# 4. Train
# =============================================================
print("Training SymSTS-MiniLM-L6...")

train_loss = CosineSimilarityLoss(model=student)

training_args = SentenceTransformerTrainingArguments(
    output_dir                  = RUNS_DIR,
    num_train_epochs            = EPOCHS,
    per_device_train_batch_size = BATCH_SIZE,
    per_device_eval_batch_size  = BATCH_SIZE,
    learning_rate               = LEARNING_RATE,
    warmup_steps                = 0.1,
    weight_decay                = 0.01,
    fp16                        = torch.cuda.is_available(),
    bf16                        = False,
    logging_steps               = 50,
    save_strategy               = "no",
    eval_strategy               = "epoch",
    dataloader_num_workers      = 0,
    report_to                   = "none",
    remove_unused_columns       = False,
    seed                        = 42,
)

trainer = SentenceTransformerTrainer(
    model         = student,
    args          = training_args,
    train_dataset = train_dataset,
    loss          = train_loss,
    evaluator     = evaluator,
)

trainer.train()

os.makedirs(OUTPUT_DIR, exist_ok=True)
try:
    trainer.save_model(OUTPUT_DIR)
except Exception:
    student.save_pretrained(OUTPUT_DIR)

print(f"Model saved to: {OUTPUT_DIR}")


# =============================================================
# 5. Final Evaluation
# =============================================================
final_score = get_main_score(evaluator(student))

print("=" * 50)
print(f"Base model STS-B:   {base_score:.4f}")
print(f"SymSTS STS-B:       {final_score:.4f}")
print(f"Improvement:        {final_score - base_score:+.4f}")
print("=" * 50)

#9. Evaluation Code
This is the script used to produce the benchmark tables above.
import warnings
warnings.filterwarnings("ignore")

import os, json, glob, mteb
from sentence_transformers import SentenceTransformer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

TASKS = [
    "STS12", "STS13", "STS14", "STS15",
    "STS16", "STSBenchmark", "SICK-R",
]

def get_scores(model_path, out_dir):
    model      = SentenceTransformer(model_path)
    tasks      = mteb.get_tasks(tasks=TASKS)
    evaluation = mteb.MTEB(tasks=tasks)
    evaluation.run(model, output_folder=out_dir, verbosity=0)

    scores = {}
    for task_name in TASKS:
        files = glob.glob(f"{out_dir}/**/*{task_name}*.json", recursive=True)
        if files:
            with open(files[0]) as f:
                d = json.load(f)
                if "scores" in d and "test" in d["scores"]:
                    scores[task_name] = d["scores"]["test"][0].get("main_score", 0.0)
    return scores

base = get_scores("sentence-transformers/all-MiniLM-L6-v2", "./mteb_base")
ours = get_scores("./output/symsts-minilm-l6",              "./mteb_ours")

print(f"{'Task':<16} {'Base':>8} {'Ours':>8} {'Diff':>8}")
print("-" * 44)
for t in TASKS:
    b = base.get(t, 0)
    o = ours.get(t, 0)
    print(f"{t:<16} {b:>8.4f} {o:>8.4f} {o-b:>+8.4f}")

10. Limitations

Trained on a small curated dataset (~16,000 augmented pairs). May not generalize well to domains far from the STS benchmark distribution (news, forums, headlines, image captions). Symmetric augmentation assumes similarity is perfectly symmetric. This is generally true for semantic similarity but may not hold for all retrieval scenarios. Inherits biases from the base MiniLM architecture and SemEval training data, which is predominantly English, web-sourced text. Should not be used as the sole decision-making system in high-stakes applications without human oversight.

11. License

This fine-tuned model is released under the Apache 2.0 License, consistent with the license of the base model sentence-transformers/all-MiniLM-L6-v2. The training datasets (STS Benchmark, STS12) are released for research and evaluation purposes by their respective authors. The raw dataset files are not redistributed here.

12. Credits and Citations

Base Model: Nils Reimers and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of EMNLP 2019. Model Card STS Benchmark: Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia. "SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation." Proceedings of SemEval-2017. STS12: Eneko Agirre, Daniel Cer, Mona Diab, Inigo Lopez-Gazpio, Lucia Specia. "SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity." Proceedings of *SEM 2012. Evaluation Framework: MTEB: Massive Text Embedding Benchmark

13. Contact

For questions, bug reports, or collaboration inquiries: Email: blueprintai.help1@gmail.com

_{Trained on consumer hardware. No datacenter required.}