mine-embed-v1

Version 1: first experimental test release, private. This is MineAI Technology's first embedding model, trained from scratch. It is published to document an early experiment, not as a competitive or production-ready model. On a 28-task MTEB subset it scores well below standard small embedding models (see Evaluation).

mine-embed-v1 is a 57.8M-parameter English text embedding model. It maps a sentence to a 512-dimensional unit vector; cosine similarity between vectors measures how close two texts are in meaning. It uses a custom encoder architecture (not a standard transformers model) and the GPT-2 byte-pair tokenizer.

1. Quick start

The repository contains a custom model.py, so the model is loaded through its Embedder class, not AutoModel or SentenceTransformer.

import sys
from huggingface_hub import snapshot_download          # private repo: log in with an HF token first

folder = snapshot_download("MineAITechnology/mine-embed-v1")
sys.path.insert(0, folder)
from model import Embedder

e = Embedder(folder)                                    # loads config.json + model.safetensors, GPT-2 tokenizer
vecs = e.encode(["A man is playing a guitar.", "Someone strums an instrument.", "Stocks fell today."])
print(vecs.shape)                                       # torch.Size([3, 512]), already L2-normalised
print(vecs @ vecs.T)                                    # cosine similarities

Requirements: torch, transformers (only for the tokenizer), safetensors, huggingface_hub. Embedder.encode(texts, max_len=128, bs=64) truncates to 128 tokens by default. No query/passage prefixes are used.

2. Model summary

Parameters	57,781,248
Architecture	Encoder-only transformer: 10 layers, hidden 512, 8 heads, feed-forward 2048, pre-norm, GELU, learned position embeddings
Output	512-dimensional, mean pooling over non-padding tokens + L2 normalisation
Tokenizer	GPT-2 BPE, vocabulary 50,258 (id 50256 = end-of-text/pad, id 50257 = mask)
Max positions	1,024 (contrastive training used 128 tokens)
Language	English (GPT-2 BPE is weak on other languages)
Weights	model.safetensors, fp32, about 231 MB
Initialisation	Random: trained from scratch

3. Training

Two stages. The full data description is in the dataset card: MineAITechnology/mine-embed-v1 (dataset).

Stage 1: masked language modelling (MLM).

• 15% dynamic masking (80% mask, 10% random token, 10% unchanged), sequence length 512, 128 sequences per step.
• AdamW, peak learning rate 4e-4 with cosine decay to 1e-5, weight decay 0.01, fp16 on 2x Tesla T4.
• 22,118 steps (5.0 hours), about 1.8 passes over a corpus of roughly 895M tokens.
• The corpus is a mixture of code, multilingual text, math, security/alignment text and corporate-fact text. Large parts are synthetic or template-generated, so the amount of distinct natural text is far below the nominal token count, and general English prose is only a small share.

Stage 2: contrastive fine-tuning.

• In-batch and hard-negative InfoNCE on 471,400 triplets, mostly AllNLI with Quora duplicate questions and a small amount of private domain-specific data, 128-token inputs.

4. Intended use

Suitable for experiments and baselines: sentence-similarity scoring on short English texts, studying small-scale embedding training, and as a reference point for later versions. It is not recommended for retrieval, clustering, or any production search system (see the limitations).

5. Evaluation

Protocol

MTEB English subset of 28 tasks: 9 STS, 4 classification, 3 clustering, 3 pair classification, 3 reranking and 6 retrieval tasks. Every model was run with the same settings: max length 128, batch size 64, fp32, one Kaggle T4, mteb 1.x. Scores are each task's main metric times 100. Competitors were loaded with their own prompts/prefixes. QuoraRetrieval is excluded from averages because this model was fine-tuned on Quora.

Overall results

Model	Params (M)	Dim	Avg (28)	STS	Classif.	Cluster	Pair	Rerank	Retrieval
bge-base-en-v1.5	109.5	768	66.25	78.51	66.69	39.68	86.56	67.77	49.94
bge-small-en-v1.5	33.4	384	64.37	77.32	64.89	37.88	84.92	66.84	46.35
gte-small	33.4	384	63.95	76.66	62.72	39.15	83.55	66.21	47.17
all-mpnet-base-v2	109.5	768	62.66	75.19	60.06	39.43	83.04	68.84	43.92
all-MiniLM-L12-v2	33.4	384	61.60	74.56	59.79	37.56	82.46	67.58	41.99
e5-small-v2	33.4	384	61.58	75.33	61.74	33.16	85.43	61.94	42.97
all-MiniLM-L6-v2	22.7	384	61.14	73.67	59.28	37.08	82.37	67.12	42.04
GloVe average (baseline)	120.0	300	45.77	55.47	55.11	21.39	72.48	48.76	22.35
mine-embed-v1	57.8	512	37.78	59.63	43.84	11.19	54.06	40.76	4.64

mine-embed-v1 ranks last. It scores about 23 points below all-MiniLM-L6-v2 and 8 points below the GloVe baseline.

Per-task scores vs all-MiniLM-L6-v2 and GloVe

Task	mine-embed-v1	all-MiniLM-L6-v2	GloVe average
STS (avg)	59.6	73.7	55.5
STSBenchmark	68.6	82.0	61.5
STS12	62.6	72.4	54.6
STS13	60.6	80.6	69.2
STS14	58.6	75.6	60.8
STS15	73.6	85.4	72.3
STS16	68.6	79.0	65.3
STS17	17.3	28.8	15.1
SICK-R	73.4	77.6	55.4
BIOSSES	53.5	81.6	44.9
Classification (avg)	43.8	59.3	55.1
Banking77Classification	47.5	80.0	67.8
EmotionClassification	24.6	40.8	37.6
TweetSentimentExtractionClassification	44.3	54.0	49.0
ToxicConversationsClassification	58.9	62.2	66.1
Clustering (avg)	11.2	37.1	21.4
TwentyNewsgroupsClustering	13.5	46.5	24.9
BiorxivClusteringS2S	5.4	32.9	18.7
MedrxivClusteringS2S	14.6	31.9	20.5
Pair classification (avg)	54.1	82.4	72.5
SprintDuplicateQuestions	49.8	94.6	87.0
TwitterSemEval2015	43.5	67.9	53.1
TwitterURLCorpus	68.8	84.7	77.3
Reranking (avg)	40.8	67.1	48.8
AskUbuntuDupQuestions	45.6	63.5	49.6
StackOverflowDupQuestions	31.4	50.8	34.1
SciDocsRR	45.3	87.1	62.6
Retrieval (avg)	4.6	42.0	22.3
SciFact	8.1	62.4	29.6
NFCorpus	2.2	31.5	13.9
FiQA2018	0.8	36.6	10.1
ArguAna	10.6	48.8	36.3
SCIDOCS	1.3	21.7	8.0
TRECCOVID	4.8	51.3	36.2

QuoraRetrieval (not in the averages, trained-on data): 74.1.

What it does well and badly

• Best area: sentence similarity. SICK-R 73.4 (within about 4 points of all-MiniLM-L6-v2), STS15 73.6, STSBenchmark 68.6, and it beats the GloVe baseline on 7 of 9 STS tasks. This matches its AllNLI-style training data.
• Weak areas: retrieval (4.6), clustering (11.2) and duplicate-question detection (SprintDuplicateQuestions 49.8). The contrastive stage used a small, narrow dataset, so the embeddings separate fine differences between similar sentences but do not group texts by topic.

Training-time evaluation

• STS-B test Spearman: 0.686 (the MLM-only model before contrastive tuning scored 0.357).
• Held-out triplet accuracy (in-distribution, not a measure of general quality): AllNLI 0.977 (n=2,000), Quora 0.969 (n=1,589), overall 0.974 (n=3,606); MLM-only baseline: 0.585, 0.844 and 0.700. Three tiny extra domains (corporate, math/code, security: 6, 9 and 2 examples) are too small to report.

Inference-time variants (12-task subset)

Changing the maximum length, shrinking the embedding with truncation or PCA, centering, or removing top principal components moved the average by between -2.0 and +0.15 points. None closes the gap to the standard models. Details are in the benchmark report.

Efficiency (one Tesla T4, 128-token inputs, batch 64, fp32)

389 sentences/s on GPU, 18.8 sentences/s on 2 CPU threads, peak GPU memory 456 MB including weights.

Embedding geometry check

The average cosine similarity between random, mostly unrelated sentences is 0.047 (all-MiniLM-L6-v2: 0.059), so the embeddings are not collapsed into a narrow cone.

6. Limitations

• Far below standard small embedding models on general benchmarks (see the table).
• Retrieval, clustering and duplicate detection are poor. Do not use it for search or topic grouping.
• Narrow training data. Contrastive data is dominated by AllNLI and Quora; the MLM corpus is code/math/security and synthetic-heavy with little natural English prose.
• English only, and the GPT-2 tokenizer is inefficient for non-English text.
• Custom architecture: needs model.py; it does not load with AutoModel or SentenceTransformer.
• Short inputs: contrastive training used 128 tokens.
• License: mixed sources, see below. Not cleared for commercial use.

7. License and data terms

The weights are derived from data under several different licenses and terms, so the model is released under license: other and kept private until a license review is finished. Not legal advice.

• AllNLI = SNLI (CC BY-SA 4.0) + MultiNLI (per-genre terms). Share-alike obligations may apply.
• Quora Question Pairs: governed by Quora's own terms; commercial use unclear.
• MLM corpus (code, web, multilingual, math, security): each source has its own license; code in particular is often copyleft or attribution-required.
• Private domain-specific and synthetic data: owned by the author.

8. Files in this repository

File	Purpose
model.safetensors	Weights (231 MB)
config.json	Architecture settings (encoder_config)
model.py	Model definition and the Embedder class
eval_report.json	Training-time evaluation (triplet accuracy, STS-B)
README.md	This card

9. Citation

@misc{mine_embed_v1_2026,
  title        = {mine-embed-v1: a 57.8M-parameter English text embedding model trained from scratch},
  author       = {{MineAI Technology}},
  year         = {2026},
  howpublished = {Hugging Face model card}
}