AtmicQuoterv2

Fine-tuned from SriRamanaAtmic/AtmicQuoterv1 (itself a fine-tune of BAAI/bge-small-en-v1.5) on a new expert-vetted Q&A set.

Training data: mined from expert_pass.csv (expert-vetted {Question, Response} pairs). For each question, the expert Response was used as an anchor and searched against the 2,875-passage Sri Ramana Maharshi citation corpus using a weighted, per-query min-max-normalized blend of three retrieval signals:

• AtmicQuoterv1 cosine similarity — weight 0.25
• AtmicEmbeddingv3 cosine similarity — weight 0.45
• BM25 (lexical) — weight 0.30

The top 4 distinct-source-family matches per question were kept as positives (4 separate training rows, one positive each), sharing a pool of 10 mined hard negatives that excludes all 4 positives' source families. 345 questions -> 1,380 rows -> 1,103 train / 276 val (query-level split, 0 leakage).

Benchmark: closed-pool citation retrieval (276 val queries, 2,857-passage pool)

Full results, all four models scored on the same 276-row val set / 2,857-passage closed pool (dense-only, each model's own embedding space):

metric	baseline (bge-small)	bge-m3	v1	v2
accuracy@1	0.0217	0.0181	0.0254	0.0471
recall@3	0.0471	0.0580	0.0652	0.0978
recall@5	0.0725	0.0942	0.0833	0.1268
recall@10	0.1232	0.1341	0.1413	0.2138
mrr@3	0.0326	0.0344	0.0429	0.0688
mrr@10	0.0447	0.0480	0.0549	0.0860
ndcg@3	0.0363	0.0404	0.0486	0.0763
ndcg@10	0.0627	0.0682	0.0749	0.1152
map@100	0.0520	0.0550	0.0641	0.0972

v2 wins outright on every metric, clearly ahead of bge-m3 despite bge-m3 being a much larger general-purpose multilingual model — v2 (33M params, domain fine-tuned) beats it by roughly 60-90% relative on ranking metrics (mrr@3, ndcg@3). v1 also modestly beats both untrained baselines, and bge-m3 beats stock bge-small on most metrics except accuracy@1/recall@1 — a bigger general model helps somewhat out-of-the-box, but domain fine-tuning (v1->v2) matters far more than model scale for this task.

Benchmark: production-shaped pipeline (dense top-20 -> monoBERT rerank -> top-4)

This mimics how the model is actually meant to be served, rather than raw closed-pool ranking: for each of 69 unique held-out questions (each with 4 mined valid citations), the quoter model dense-retrieves the top 20 candidates from the full 2,857-passage corpus (pure query-mode, the standard BGE-instruction retrieval), castorini/monobert-large-msmarco reranks those 20, and the top 4 are what would be shown to a user.

metric	base (bge-small)	v1	v2
stage1_recall@20	0.1884	0.2029	0.2790
precision@4	0.0688	0.0725	0.1123
hit_rate@4 (>=1 of 4 found)	0.2319	0.2609	0.3333

stage1_recall@20 — of a question's 4 true citations, the fraction that even made the top-20 retrieval pool (the ceiling the reranker can't exceed). precision@4 — of the 4 citations served, the fraction that are correct (equivalently recall@4, since 4 are served against exactly 4 true citations per question). hit_rate@4 — the fraction of questions where at least one of the 4 served citations is correct.

In production, the app will show at least one correct citation for about 33.3% of queries with AtmicQuoterv2 — a 44% relative lift over stock bge-small (23.2%) and 28% over AtmicQuoterv1 (26.1%). The gain traces back to stage 1: v2's retriever gets more of the true citations into the top-20 pool in the first place (27.9% vs. 18.8%/20.3%), which the reranker then has more to work with — retrieval recall is the ceiling here, not the reranker, and v2 raises that ceiling the most of the three.

Absolute scores are low in every configuration tested — this is a genuinely hard closed-pool task (natural questions against ~2,857 short, often mutually confusable quote fragments) — but AtmicQuoterv2 consistently raises the retrieval ceiling the most, which is what any downstream reranking or serving strategy is bounded by.

---

SentenceTransformer based on SriRamanaAtmic/AtmicQuoterv1

This is a sentence-transformers model finetuned from SriRamanaAtmic/AtmicQuoterv1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: SriRamanaAtmic/AtmicQuoterv1
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Represent this sentence for searching relevant passages: Can we surrender to solve our problems?',
]
documents = [
    'Leave everything to God, your burden will cease and He will take on your burden. He knows what to do',
    'Self-surrender is the only way to Peace',
    'There is One who governs the world and it is His task to look after the world. He who has given life to the world knows how to look after it also. He bears the burden of this world, not you',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4747, 0.5341, 0.3339]])

Evaluation

Metrics

Information Retrieval

• Dataset: atmic-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0471
cosine_accuracy@3	0.0978
cosine_accuracy@5	0.1268
cosine_accuracy@10	0.2138
cosine_precision@1	0.0471
cosine_precision@3	0.0326
cosine_precision@5	0.0254
cosine_precision@10	0.0214
cosine_recall@1	0.0471
cosine_recall@3	0.0978
cosine_recall@5	0.1268
cosine_recall@10	0.2138
cosine_ndcg@3	0.0763
cosine_ndcg@10	0.1152
cosine_mrr@3	0.0688
cosine_mrr@10	0.086
cosine_map@100	0.0972

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,103 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, negative_9, and negative_10
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10
  type	string	string	string	string	string	string	string	string	string	string	string	string
  modality	text	text	text	text	text	text	text	text	text	text	text	text
  details	min: 18 tokens mean: 26.23 tokens max: 52 tokens	min: 11 tokens mean: 41.99 tokens max: 128 tokens	min: 13 tokens mean: 38.46 tokens max: 81 tokens	min: 10 tokens mean: 39.62 tokens max: 128 tokens	min: 12 tokens mean: 35.92 tokens max: 95 tokens	min: 13 tokens mean: 43.42 tokens max: 93 tokens	min: 11 tokens mean: 40.35 tokens max: 112 tokens	min: 16 tokens mean: 46.46 tokens max: 128 tokens	min: 8 tokens mean: 47.81 tokens max: 128 tokens	min: 17 tokens mean: 43.46 tokens max: 128 tokens	min: 13 tokens mean: 38.77 tokens max: 128 tokens	min: 10 tokens mean: 43.88 tokens max: 128 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10
  Represent this sentence for searching relevant passages: What is the danger of engaging in too much karma?	The fruit of action passes. But action leaves behind Seed of further action Leading to an endless ocean of action; Not at all to moksha.	Both are trying only to take the ego back to the source from which it sprang and make it merge there	What action remains to be done by that great Yogi whose mind has been extinguished and who rests in his own true and transcendent state of Being?	In the sheer presence of the Lord / Himself free from all trace of thought, / Jivas set out on numerous paths / Of action, work away, and wearied / Turn inward and return to freedom	"Bhagavan saw a branch of a tree, cut it and spent one hour shaping it into a nice walking stick. At about that time, an elderly shepherd who had no walking stick came that way walking slowly and with difficulty. Bhagavan gave him the stick which he just made and said 'Action is over so also desireless action'."	The screen is always there and is never affected by the action of the pictures	He who's contented with his lot, from jealousy is free; balanced in affluence and mishap; not bound by action he.	What is done with peaceful and pure mind is righteous action; whatever is done with the mind agitated and from desire is wrong action	The goings on in the world do not / Affect the sun; the properties / Of earth, water, fire and air touch not / The infinite ether. Even so, / Men’s actions do not reach or move / The mind-transcending Lord supreme	His every action, His every movement was the eternal Upadesa — the Golden Silence	Hearken! It stands as an insentient hill. Its action is mysterious, past human understanding. From the age of innocence it had shone in my mind that Arunachala was something of surpassing grandeur...
  Represent this sentence for searching relevant passages: What is the danger of engaging in too much karma?	Action yields fruit, For so the Lord ordains it. How can action be the Lord? It is insentient.	Both are trying only to take the ego back to the source from which it sprang and make it merge there	What action remains to be done by that great Yogi whose mind has been extinguished and who rests in his own true and transcendent state of Being?	In the sheer presence of the Lord / Himself free from all trace of thought, / Jivas set out on numerous paths / Of action, work away, and wearied / Turn inward and return to freedom	"Bhagavan saw a branch of a tree, cut it and spent one hour shaping it into a nice walking stick. At about that time, an elderly shepherd who had no walking stick came that way walking slowly and with difficulty. Bhagavan gave him the stick which he just made and said 'Action is over so also desireless action'."	The screen is always there and is never affected by the action of the pictures	He who's contented with his lot, from jealousy is free; balanced in affluence and mishap; not bound by action he.	What is done with peaceful and pure mind is righteous action; whatever is done with the mind agitated and from desire is wrong action	The goings on in the world do not / Affect the sun; the properties / Of earth, water, fire and air touch not / The infinite ether. Even so, / Men’s actions do not reach or move / The mind-transcending Lord supreme	His every action, His every movement was the eternal Upadesa — the Golden Silence	Hearken! It stands as an insentient hill. Its action is mysterious, past human understanding. From the age of innocence it had shone in my mind that Arunachala was something of surpassing grandeur...
  Represent this sentence for searching relevant passages: What is the danger of engaging in too much karma?	The life of action need not be renounced	Both are trying only to take the ego back to the source from which it sprang and make it merge there	What action remains to be done by that great Yogi whose mind has been extinguished and who rests in his own true and transcendent state of Being?	In the sheer presence of the Lord / Himself free from all trace of thought, / Jivas set out on numerous paths / Of action, work away, and wearied / Turn inward and return to freedom	"Bhagavan saw a branch of a tree, cut it and spent one hour shaping it into a nice walking stick. At about that time, an elderly shepherd who had no walking stick came that way walking slowly and with difficulty. Bhagavan gave him the stick which he just made and said 'Action is over so also desireless action'."	The screen is always there and is never affected by the action of the pictures	He who's contented with his lot, from jealousy is free; balanced in affluence and mishap; not bound by action he.	What is done with peaceful and pure mind is righteous action; whatever is done with the mind agitated and from desire is wrong action	The goings on in the world do not / Affect the sun; the properties / Of earth, water, fire and air touch not / The infinite ether. Even so, / Men’s actions do not reach or move / The mind-transcending Lord supreme	His every action, His every movement was the eternal Upadesa — the Golden Silence	Hearken! It stands as an insentient hill. Its action is mysterious, past human understanding. From the age of innocence it had shone in my mind that Arunachala was something of surpassing grandeur...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 6.0
• learning_rate: 1e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 6.0
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	atmic-val_cosine_ndcg@10
0.7246	50	4.6409	-
1.0	69	-	0.0872
1.4493	100	4.2389	-
2.0	138	-	0.0956
2.1739	150	3.8763	-
2.8986	200	3.6176	-
3.0	207	-	0.1023
3.6232	250	3.4962	-
4.0	276	-	0.1078
4.3478	300	3.4141	-
5.0	345	-	0.1137
5.0725	350	3.2106	-
5.7971	400	3.1744	-
6.0	414	-	0.1152
-1	-1	-	0.1152

• The bold row denotes the saved checkpoint.

Training Time

• Training: 27.0 minutes
• Evaluation: 46.1 seconds
• Total: 27.8 minutes

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}