Aman0026/ArogyaAI-BioBERT-BioLORD

SentenceTransformer based on FremyCompany/BioLORD-2023-M

This is a sentence-transformers model finetuned from FremyCompany/BioLORD-2023-M. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FremyCompany/BioLORD-2023-M
• Maximum Sequence Length: 64 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Leukemia',
    'test mein blood cancer bataya hai, safed khoon (WBC) count abnormal hai',
    'मला मूळव्याध झाला आहे आणि बसायला त्रास होतोय',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7930, 0.4414],
#         [0.7930, 1.0000, 0.1904],
#         [0.4414, 0.1904, 1.0000]], dtype=torch.bfloat16)

Evaluation

📊 RAG Retrieval Benchmark (M.Tech Thesis)

Rather than just raw cosine similarity against isolated labels, the model is evaluated in a Retrieval-Augmented Generation (RAG) pipeline against a FAISS vector index of 73 diseases. The benchmark tests complex symptom combinations (Hindi, Hinglish, Marathi, Tamil, Gujarati) against standard clinical profiles.

Metric	Local Laptop CPU	Description
Precision @ 1	97.3%	The correct clinical disease was the absolute #1 retrieved profile.
Precision @ 3	100.0%	The correct disease was within the top 3 retrieved profiles.
Avg Retrieval Latency	225.4 ms	Fast sub-second local retrieval via FAISS.

📈 Confusion Matrix (Multi-language Retrieval)

Below is the confusion matrix generated for raw phrase-to-class mapping without RAG profiles (showing the semantic baseline for raw symptom matching):

======================================================
                 CONFUSION MATRIX                     
======================================================
True / Predicted | Fever    | Diabetes | Heart At | Asthma   | Hyperten
-----------------------------------------------------------------------
Fever            | 1        | 0        | 1        | 1        | 0       
Diabetes         | 0        | 1        | 1        | 1        | 0       
Heart Attack     | 0        | 0        | 2        | 0        | 1       
Asthma           | 0        | 0        | 2        | 1        | 0       
Hypertension     | 0        | 0        | 1        | 0        | 2       
======================================================
Overall Top-1 Retrieval Accuracy: 46.7% (7/15)
======================================================

Note: Directly matching patient-described symptoms to bare English disease names is a characteristically hard task (getting 46.7% accuracy), but storing symptom lists in a FAISS RAG index increases overall diagnostic recall to a perfect 100.0% Top-3 accuracy.

Training Details

Training Dataset

Unnamed Dataset

• Size: 8,879 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1
  type	string	string
  modality	text	text
  details	min: 4 tokens mean: 17.61 tokens max: 59 tokens	min: 4 tokens mean: 10.0 tokens max: 34 tokens

• Samples:

  sentence_0	sentence_1
  বুকের বাঁ দিকে হঠাৎ খুব ভারী লাগছে।	Myocardial Infarction
  chokher chani operation hoyeche	Cataract surgery
  चावल के पानी जैसे पतले दस्त आ रहे हैं	Cholera

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 30
• per_device_eval_batch_size: 256
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• per_device_train_batch_size: 256
• num_train_epochs: 30
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 256
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	spearman_cosine
1.0	35	0.2122
2.0	70	0.3213
3.0	105	0.3320
4.0	140	0.3404
5.0	175	0.3568
5.7143	200	0.3697
6.0	210	0.3790
7.0	245	0.3877
8.0	280	0.3958
9.0	315	0.4021

Training Time

• Training: 1.7 minutes
• Evaluation: 0.2 seconds
• Total: 1.7 minutes

Framework Versions

• Python: 3.11.15
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.5.1+cu121
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}