MinhPhuc0804/me5-docling-checkthat-task1-v1

SentenceTransformer based on intfloat/multilingual-e5-large-instruct

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large-instruct. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-large-instruct
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MinhPhuc0804/me5-docling-checkthat-task1-v1")
# Run inference
sentences = [
    "query: @user You're more prone to get COVID-19 with more of the jabs. They're aiming to cut down Earth's population.",
    'passage: doses. Results Among 51017 employees, COVID-19 occurred in 4424 (8.7%) during the study.\n\ntitle: Effectiveness of the Coronavirus Disease 2019 (COVID-19) Bivalent Vaccine\nIn multivariable analysis, the bivalent vaccinated state was associated with lower risk of COVID-19 during the BA.4/5 dominant (HR, .71; 95% C.I., .63-.79) and the BQ dominant (HR, .80; 95% C.I., .69-.94) phases, but decreased risk was not found during the XBB dominant phase (HR, .96; 95% C.I., .82-.1.12). Estimated vaccine effectiveness (VE) was 29% (95% C.I., 21%-37%), 20% (95% C.I., 6%-31%), and 4% (95% C.I., -12%-18%), during the BA.4/5, BQ, and XBB dominant phases, respectively. Risk of COVID-19 also increased with time since most recent prior COVID-19 episode and with the number of vaccine doses previously received.',
    'passage: d the bacterial counts from the same surgeon, a significant increase was noted in the 2-hours group.\n\ntitle: Surgical masks as source of bacterial contamination during operative procedures\nMoreover, the bacterial counts were significantly higher among the surgeons than the OR. Additionally, the bacterial count of the external surface of the second mask was significantly higher than that of the first one. The source of bacterial contamination in SMs was the body surface of the surgeons rather than the OR environment. Moreover, we recommend that surgeons should change the mask after each operation, especially those beyond 2 hours. Double-layered SMs or those with excellent filtration function may also be a better alternative. This study provides strong evidence for the identification that SMs as source of bacterial contamination during operative procedures, which should be a cause for alarm and attention in the prevention of surgical site infection in clinical practice.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6161, 0.0673],
#         [0.6161, 1.0000, 0.0795],
#         [0.0673, 0.0795, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: 10-percent-dev-split
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5621
cosine_accuracy@3	0.7699
cosine_accuracy@5	0.827
cosine_accuracy@10	0.8784
cosine_precision@1	0.5621
cosine_precision@3	0.2566
cosine_precision@5	0.1654
cosine_precision@10	0.0878
cosine_recall@1	0.5621
cosine_recall@3	0.7699
cosine_recall@5	0.827
cosine_recall@10	0.8784
cosine_ndcg@10	0.7248
cosine_mrr@10	0.675
cosine_map@100	0.6791

Training Details

Training Dataset

Unnamed Dataset

• Size: 17,319 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 19 tokens mean: 60.83 tokens max: 106 tokens	min: 26 tokens mean: 184.65 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  query: Financial ties between heads of powerful US professional health societies and the corporate sector: cross‑sectional study | The BMJ	passage: title: Financial ties between leaders of influential US professional medical associations and industry: cross sectional study
  abstract: To investigate the nature and extent of financial relationships between leaders of influential professional medical associations in the United States and pharmaceutical and device companies.Cross sectional study.Professional associations for the 10 costliest disease areas in the US according to the US Agency for Healthcare Research and Quality. Financial data for association leadership, 2017-19, were obtained from the Open Payments database.328 leaders, such as board members, of 10 professional medical associations: American College of Cardiology, Orthopaedic Trauma Association, American Psychiatric Association, Endocrine Society, American College of Rheumatology, American Society of Clinical Oncology, American Thoracic Society, North American Spine Society, Infectious Diseases Society of America, and American College of Physicians.Proportion ...
  query: Récente recherche du Mexique sur l'ivermectine associée aux #azithromycine, #montelukast & aspirine sur 768 patients. La mortalité était diminuée de 81%, et 74% de baisse des hospitalisations. Un rétablissement 3,4 fois plus rapide pour les #Covid19. En prépublication	passage: title: Effectiveness of a multidrug therapy consisting of Ivermectin, Azithromycin, Montelukast, and Acetylsalicylic acid to prevent hospitalization and death among ambulatory COVID-19 cases in Tlaxcala, Mexico abstract: There is an urgent need for effective treatments to prevent or attenuate lung and systemic inflammation, endotheliitis, and thrombosis related to COVID-19. This study aimed to assess the effectiveness of a multidrug-therapy consisting of Ivermectin, Azithromycin, Montelukast, and Acetylsalicylic acid ("TNR4" therapy) to prevent hospitalization and death among ambulatory COVID-19 cases in Tlaxcala, Mexico.
  query: 🧠💥 Suite à un #AVC, on peut constater une réponse inflammatoire persistante qui favorise alors une altération cognitive. La mise d'une molécule à des #souris 🐁 a autorisé de restaurer le métabolisme lipidique dans le #cerveau et diminuer ce risque	passage: in the striatum and thalamus and c-Fos immunoreactivity in hippocampal regions.

  title: Repeated Administration of 2-Hydroxypropyl-β-Cyclodextrin (HPβCD) Attenuates the Chronic Inflammatory Response to Experimental Stroke Additionally, HPβCD improved recovery through the protection of hippocampal-dependent spatial working memory and reduction of impulsivity. These results indicate that systemic HPβCD treatment following stroke attenuates chronic inflammation and secondary neurodegeneration and prevents poststroke cognitive decline. SIGNIFICANCE STATEMENT Dementia is a common and debilitating sequela of stroke. Currently, there are no available treatments for poststroke dementia. Our study shows that lipid metabolism is disrupted in chronic stroke infarcts, which causes an accumulation of uncleared lipid debris and correlates with a chronic inflammatory response. To our knowledge, these substantial changes in lipid homeostasis have not been previously recognized or investigated... |
• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 10
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	10-percent-dev-split_cosine_ndcg@10
0.9225	500	0.6227	-
1.0	542	-	0.7037
1.8450	1000	0.2537	-
2.0	1084	-	0.7074
2.7675	1500	0.1664	-
3.0	1626	-	0.7144
3.6900	2000	0.1109	-
4.0	2168	-	0.7159
4.6125	2500	0.0779	-
5.0	2710	-	0.7230
5.5351	3000	0.0664	-
6.0	3252	-	0.7161
6.4576	3500	0.0563	-
7.0	3794	-	0.7142
7.3801	4000	0.0478	-
8.0	4336	-	0.7198
8.3026	4500	0.0379	-
9.0	4878	-	0.7243
9.2251	5000	0.0379	-
10.0	5420	-	0.7248

Training Time

• Training: 1.1 hours

Framework Versions

• Python: 3.12.6
• Sentence Transformers: 5.4.1
• Transformers: 4.56.0
• PyTorch: 2.8.0+cu129
• Accelerate: 1.10.1
• Datasets: 4.8.4
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}