EchoesML/bge-m3-georgian-literary

BGE-M3 Georgian — Fine-tuned Embedding Model

A fine-tuned version of BAAI/bge-m3 for dense retrieval over Georgian literary text. Trained on 14,340 triplets derived from 9 classical Georgian authors (2,390 chunks, sourced from ka.wikisource.org).

---

Benchmark Results

Evaluated on a held-out test set of 7,170 queries against 2,390 passages — all unseen during training.

Model	Recall@8	nDCG@8	MRR
Qwen3-Embedding-0.6B base	0.2169	0.1434	0.1202
Qwen3-Embedding-0.6B fine-tuned	0.2685	0.1822	0.1550
BGE-M3 base	0.5170	0.3817	0.3401
BGE-M3 fine-tuned v1 (5 epochs)	0.5442	0.3992	0.3548
BGE-M3 fine-tuned v2 best checkpoint	0.7248	0.5621	0.5114
BGE-M3 fine-tuned v2 averaged (this model)	0.7278	0.5643	0.5133

Fine-tuning improved Recall@8 from 0.517 → 0.7278 (+40.8% relative gain over the base model).

---

Intriguing Questions

The full methodology is documented in two companion files. But here are some questions that might pull you in:

→ Why does the averaged checkpoint outperform the best single checkpoint — almost every time? Each epoch's model lands in a slightly different region of the loss landscape. Individual checkpoints sit in sharp minima — well-fitted to their specific batch order, but brittle. The averaged model lands in the center of the region explored by the last three epochs: a flatter, wider basin that generalises better to queries the model has never seen. No additional training. Free improvement. [Full explanation with loss landscape intuition → training.md]

→ Why did I abandon the most popular chunking strategy (sliding window with overlap) — and get better results with a simpler one? Sliding window overlap is widespread because it helps boundary sentences appear in multiple chunks, raising their retrieval odds. But it comes at a cost: your index contains duplicated content, reconstruction score drops below 1.0, and you lose the ability to map any retrieved chunk back to its exact position in the source. The sentence-boundary nearest-neighbor strategy used here hits a 1.0 reconstruction score on all 28 source files, keeps every chunk within tight size bounds (500–1200 chars), and never embeds the same character twice. [Full algorithm with edge cases → chunking_strategy.md]

→ How did a smaller model with worse tokenization beat a larger model with a purpose-built tokenizer? Qwen3-Embedding-0.6B (596M parameters) tokenizes Georgian whole-word — it sees ვეფხისტყაოსანი as a single token, not fragmented subwords. BGE-M3 (568M parameters, XLM-RoBERTa based) chops the same word into pieces. By every surface measure, Qwen3 should win on Georgian. It didn't — not even close (Recall@8: 0.2685 vs 0.7278). The reason is pretraining: BGE-M3 was pretrained on 1.2 billion retrieval pairs across 100+ languages. Its representations are already calibrated for semantic similarity at the vector level. Qwen3 Embedding's superior tokenization raises the ceiling of what fine-tuning can achieve, but the floor it started from was so much lower that three epochs of LoRA couldn't close the gap. Tokenizer quality is not the same as retrieval quality. [Full breakdown with training configs → training.md]

Model Details

Property	Value
Base model	BAAI/bge-m3
Architecture	XLM-RoBERTa (encoder-only)
Parameters	568M
Pooling	CLS (position 0)
Fine-tuning method	Full fine-tuning (all 568M params)
Final model	Checkpoint average (epochs 6, 7, 8)
Training framework	FlagEmbedding native encoder trainer
Max sequence length	512 tokens
Embedding dimension	1024

---

Training Data

Property	Value
Corpus	9 Georgian classical authors, public domain
Source	ka.wikisource.org
Chunks	2,390 (sentence-boundary chunking, target 800 chars)
Query generation	Gemini 2.0 Flash — 3 natural Georgian queries per chunk
Training triplets	14,340 (query, positive passage, 7 hard negatives)
Hard negative mining	Top-8 from BGE-M3 base, excluding ground truth
Training epochs	8 (final model = avg of epochs 6, 7, 8)
Batch size	8 (effective 16 with gradient accumulation)
Learning rate	2e-5
Temperature	0.02

---

Companion Documentation

• training.md — Full training methodology: data pipeline, hard negative mining, why the custom training loop caused catastrophic forgetting, checkpoint averaging explained with loss landscape geometry.
• chunking_strategy.md — The chunking algorithm in full: sentence-boundary search, reconstruction scoring, why overlap was rejected, contextual chunking with author/title metadata, and the script annotated line by line.

---

License

Model weights: Apache 2.0 (inherited from BAAI/bge-m3).
Training corpus: public domain (ka.wikisource.org).