aliRafik/embeddinggemma_300m_finetuned_16bit

SentenceTransformer

This model was finetuned with Unsloth.

based on unsloth/embeddinggemma-300m

This is a sentence-transformers model finetuned from unsloth/embeddinggemma-300m on the generator dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: unsloth/embeddinggemma-300m
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • generator

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'PeftModelForFeatureExtraction'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "How do statin drugs potentially reduce the risk of Alzheimer's disease?\n",
]
documents = [
    'Reducing cholesterol with drugs in the statin family appears to alleviate this risk dramatically (35, 36) (see "Greasing Aging\'s Downward Slide" ¶ ¶ ). Cholesterol congregates in so-called lipid rafts that float around in the cellular membrane. APP, β-secretase, and γ-secretase are attracted to these rafts, and as Brian Austen of St. George\'s Hospital Medical School in Tooting, U.K., says, they\'re "marooned together" where they can interact and brew up β amyloid. APP appears to exist in two separate pools; those proteins floating on the lipid rafts are clipped by β-secretase and can spawn β amyloid; APP floating free of the rafts is more likely to be cut by α-secretase and yield harmless proteins (37) . At the Stockholm meeting in 2002, Konrad Beyreuther of Heidelberg University in Germany reported preliminary results from a small clinical trial of statins in people with mild AD and normal cholesterol concentrations. Patients taking the drug appeared to decline more slowly than those taking a placebo, he reported.\n\n Other epidemiological studies have uncovered promising hints that estrogen replacement therapy might protect women from Alzheimer\'s. One followed almost 2000 women in Cache County, Utah-the United States\' longest-lived county. Those who had been on hormone replacement therapy (in this study, most women were treated with estrogen only) in the past were less likely to develop AD than were those who had never been on HRT (38) . Current use didn\'t seem to help, leading researchers to conclude that there might be a "critical time" during which estrogen can forestall AD (39) . But the purported benefits of HRT didn\'t hold up in a massive, case-controlled, randomized study. The Women\'s Health Initiative (WHI) showed that estrogen plus progestin increased rather than decreased the risk that postmenopausal women would develop dementia or mild cognitive impairment (see "Weathering the HRT Storm" ## ) (40) . Another arm of the WHI trial examined whether estrogen (without progestin) alleviates the clinical symptoms of women already suffering from mild or moderate AD; it turned up no evidence of such a benefit (41). However, an earlier postmortem neuroanatomical study from the WHI showed that at a cellular level, at least, estrogen did seem to exert a positive effect. Women with early signs of AD who had taken estrogen lost fewer neurons in vulnerable brain regions than did women who had taken placebos (42) .\n\n Other ongoing clinical prevention trials will assess the protective effects, if any, of antioxidants such as vitamin E and ginkgo biloba. One trial of vitamin E in people with established AD showed that the treatment delayed their entry into nursing homes, a major milestone in the progression of the disease (43) . The AD-prevention hypothesis dearest to the hearts of many researchers goes by several different names: the cognitive reserve hypothesis, the cortical reserve hypothesis, and the "use it or lose it" hypothesis. According to the theory, stimulating the brain, say, by regularly reading research or introductory review articles, decreases the risk of developing Alzheimer\'s. Although the mechanism for such a protective factor is a bit murky-scientists talk about building extra neural connections that somehow compensate for early AD neuropathology-a few studies have backed up initial observations that people with more years of education are less likely to develop Alzheimer\'s. For instance, those with AD were less likely to have engaged in intellectual activities in early and middle adulthood than their nondemented peers (44) .\n\n \n\n \n\n Some of the most intriguing evidence for the cognitive reserve hypothesis comes from a longitudinal study of nuns in convents (see "Sister Knows Best" *** ). These women are an epidemiologist\'s dream: They live in similar rooms, eat the same kinds of foods, and have similar chores and hobbies, thus reducing the number of confounding variables that normally plague population studies. Researchers analyzed writing samples gathered when the young women entered their convents. Low grammatical and cognitive complexity correlated with likelihood of AD (45); in contrast, one particularly sharp woman lived to age 101 with all her wits, even though her brain was studded with plaques and tangles at autopsy (46) , providing more anecdotal evidence for the cognitive reserve hypothesis.\n\n Alzheimer\'s disease used to be one of the grimmer areas of research, but these days, AD researchers are remarkably optimistic. New transgenic and knockout mouse lines offer "ways to get into the nervous system of living animals," says Donald Price. The potential for helping millions of people live fuller, longer lives contributes to the new sense of urgency and enthusiasm. But there\'s also the specter of an aging population; a person\'s risk of AD doubles every decade.',
    'Furthermore, we showed that heterozygous carriers of the ADAMTS13-related mutations who, thus, have reduced ADAMTS13 bioactivity have a normal phenotype.\n\n The plasma of the patients with congenital TTP did not present the ADAMTS13 band. This may be due to altered synthesis, secretion or antigenicity, or due to increased breakdown of the protease in plasma. The fact that two antibodies directed to two different domains in ADAMTS13 were unable to detect the protease band makes altered antigenicity less likely to be the cause for the lack of the ADAMTS13 band in these patients. Previous studies have shown impaired secretion of the 4143insA (patients 1, 2, 4, and 5) [35, 42, 44] and P353L (patient 3) [42] mutants from cells, thus, indicating that the protease may accumulate intracellularly, at least in some patients with congenital TTP. This may be due to a missing cell sorting signal, as in the case of 4143insA [44] or due to conformational changes in the protein, impairing its secretion. Similar findings regarding two other ADAMTS13 mutations, V88M and G239V, have been recently reported [34] . A total lack of ADAMTS13 activity in the plasma is thought to be incompatible with life [27] , thus, the patients may have very low amounts of ADAMTS13 activity in their plasma, which we were unable to detect with this method. The patients with acquired TTP presented with a normal ADAMTS13 band, which is consistent with the fact that ADAMTS13 protease genotype and expression are normal, but their activity is lower, due auto-antibodies [11, 14, 41, 45, 52] . The finding that the immunoadsorption of immunoglobulins from the plasma of the patients with acquired TTP also led to the removal of the ADAMTS13 antigen from their samples indicates that the protease is complexed with the auto-antibodies in the circulation of these patients, and that this occurs even during clinical remission.\n\n The majority of the ADAMTS13 assays available today detect ADAMTS13 activity levels in plasma (Table 2) . This is performed either by detecting the VWF products resulting from ADAMTS13 cleavage (assays [1] [2] [3] [4] [5] or by measuring the residual VWF activity (assays 6-7). The VWF substrate utilized in these assays may be high molecular weight VWF (plasma-derived or recombinant; assays 1-4) or VWF domains or short synthetic peptides (assay 5). Two multicenter studies evaluating methods 1, 3-5, and 6-7 showed that all assays were able to detect severely ADAMTS13-deficient plasma samples and indicated that methods 1, 6, and 7 were the most consistent and reliable methods [48, 49] . A recent smaller study evaluating the FRETS-VWF73 method (assay 5) showed that this is a reliable assay which provides results in good accordance with other methods [26] .\n\n Few recent studies have shown that ADAMTS13 antigen was detectable in plasma by ELISA (assay 8 in Table 2 ). ELISA assays are a valuable complement to the ADAMTS13 activity assays, and offer a fast analysis of the ADAMTS13 antigen levels in plasma samples using antibodies directed to ADAMTS13. These assays are able to distinguish ADAMTS13-deficient TTP plasma from normal plasma samples. Other methods are capable of detecting anti-ADAMTS13 auto-antibodies ( Table 2 , assays 8-9). Mutational analysis can be carried out by polymerase chain reaction (PCR) in order to detect mutations in the ADAMTS13 gene (assay 11 in Table 2 ).\n\n In the present study, we developed a qualitative, semiquantitative assay capable of detecting ADAMTS13 antigen and anti-ADAMTS13 auto-antibodies in plasma and, IRMA: immunoradiometric assay; FRET: fluorescence resonance energy transfer; FRETS-VWF73: a 73-amino-acid-long synthetic peptide which provides a minimal substrate for ADAMTS13 [24] that has been made fluorogenic; PCR: polymerase chain reaction thus, distinguishing between TTP and normal plasma, as well as distinguishing between congenital and acquired TTP. All patients with congenital TTP, regardless of their ADAMTS13 mutations, presented with undetectable levels of ADAMTS13 antigen, whereas patients with acquired TTP presented a normal phenotype. Heterozygous carriers of ADAMTS13 mutations also revealed a normal ADAMTS13 antigen band. Although not a quantitative method like the above-mentioned ELISA assays, the present assay offers the advantage of not only demonstrating the presence or absence of ADAMTS13 antigen, but also the molecular size of ADAMTS13 present in plasma samples. Furthermore, this assay was able to show the presence of anti-ADAMTS13 antibodies indirectly by performing immunoadsorption of plasma immunoglobulins prior to immunoblotting. In conclusion, this assay is able to distinguish between normal and TTP plasma, and also between congenital and acquired TTP. It is easy to perform and can be used at any hospital laboratory routinely using SDS-PAGE electrophoresis and immunoblotting, and offers a complement to existing ADAMTS13 methods in the diagnosis of TTP.',
    'Diabetic-cardiomyopathy-related complications are the major cause of morbidity and mortality in diabetes. Diabetes not only affects the heart causing diabetic-cardiomyopathy independent of coronary artery disease, hypertension, and hyperlipidemia but also exacerbate the cardiac injury after myocardial infarction or IR. Association of cardiac fibrosis, hypertrophy, and cardiomyocyte apoptosis has been discussed in literature. Diabetic mouse models have been used in cardiac research to elucidate the underlying cellular and molecular mechanism of fibrosis and hypertrophy [103] . Hyperglycemia of the left ventricular wall in diabetes stimulates cardiac remodeling and disrupts contractile function, thereby impairing the heart function by declining the systolic and diastolic function of the heart [104] . The mutation in diabetic (db/db) mice results in expression of a non-functional truncated long-form of the leptin receptor leading to hypothalamic resistance to leptin. This results in voracious appetite, marked obesity and overt diabetes at a young age in mice. Obesity and diabetes at young age (4-6 months) results in cardiac interstitial fibrosis in db/db mice and has been documented using both histological and biochemical techniques. Activation and expansion of the resident cardiac fibroblast in diabetes may result in cardiac fibrosis [105] . Further the cardiac fibroblast-dependent matrix accumulation in type II diabetes has been reported using the genetically induced mouse model of type II diabetes (db/db murine model). Increased expression of collagen type I, TIMP2 and TGF-b, and phenotypic changes in diabetic db/db fibroblast isolated from diabetic mouse was associated with collagen accumulation, myocardial stiffness and diastolic dysfunction [106] .\n\n Cardiomyopathies are a heterogeneous group of diseases affecting the heart muscle and frequently lead to progressive heart failure-related disability or cardiovascular death. Since genetics have been shown to play an increasingly important role in the onset of cardiomyopathies [107] , mouse models have emerged as the most commonly used animal model for cardiomyopathy research, based on the availability of genetic information and technologies to manipulate the mouse genome [108, 109] . Hypertrophic cardiomyopathy (HCM) is a disease of the cardiac muscle that occurs due to mutations in genes encoding for the cardiac sarcomere. To date, approximately 1400 mutations have been identified as responsible for HCM pathology. About 70 % of these mutations are in the sarcomere genes encoding cardiac b-myosin heavy chain (MYH7) and cardiac myosin-binding protein C (MYBPC3). Other sarcomeric genes, such as regulatory myosin light chain (MYL2), cardiac troponin T (TNNT2), cardiac troponin I (TNNI3), and actin (ACTC) account for the other cases of HCM [19] . The phenotypic expression of HCM includes massive hypertrophy primarily involving the ventricular septum. The mechanisms of HCM are complex and include dynamic LV outflow tract obstruction due to bulging of the thickened basal anterior septum beneath the aortic valve, mitral regurgitation, diastolic dysfunction, myocardial ischemia, and cardiac arrhythmias that may lead to sudden death [8] . The microscopic hallmarks are myocyte hypertrophy and disarray together with expansion of the interstitial fibrosis. Myocyte disarray is characterized by architectural disorganization of the myocardium, in which adjacent hypertrophied myocytes are aligned perpendicularly or obliquely to each other around central cores of collagen in either a pinwheel configuration or herringbone pattern (Fig. 6) .\n\n A number of myosin heavy chain transgenic animal models of HCM have been created in an attempt to understand the pathogenesis of HCM phenotypes typically seen in humans. A transgenic mouse model with a point mutation of the Arg 403 -Gln in a-myosin heavy chain shows similarities with familial HCM in humans [110] . A similar phenotype has been seen in transgenic mice lacking the light chain-binding domain of b-myosin heavy chain [111] . These models will help in understanding the pathophysiology of HCM. Overexpression of the H-ras gene targeted to the ventricle with MLC2v promoter causes ventricular hypertrophy with myofiber disarray, which causes left ventricular outflow tract obstruction leading to diastolic dysfunction [112] .',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.6547, -0.0359, -0.0823]])

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8752
cosine_accuracy@3	0.9603
cosine_accuracy@5	0.9769
cosine_accuracy@10	0.9886
cosine_precision@1	0.8752
cosine_precision@3	0.3201
cosine_precision@5	0.1954
cosine_precision@10	0.0989
cosine_recall@1	0.8752
cosine_recall@3	0.9603
cosine_recall@5	0.9769
cosine_recall@10	0.9886
cosine_ndcg@10	0.937
cosine_mrr@10	0.9199
cosine_map@100	0.9204

Training Details

Training Dataset

generator

• Dataset: generator
• Size: 100,000 training samples
• Columns: question and passage_text
• Approximate statistics based on the first 100 samples:

  	question	passage_text
  type	string	string
  modality	text	text
  details	min: 10 tokens mean: 21.83 tokens max: 60 tokens	min: 529 tokens mean: 942.59 tokens max: 1024 tokens

• Samples:

  question	passage_text
  What factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction?	I n 1966, Ross and Somerville 1 reported the first use of an aortic homograft to establish right ventricle-to-pulmonary artery continuity in a patient with tetralogy of Fallot and pulmonary atresia. Since that time, pulmonary position homografts have been used in a variety of right-sided congenital heart lesions. Actuarial 5-year homograft survivals for cryopreserved homografts are reported to range between 55% and 94%, with the shortest durability noted in patients less than 2 years of age. 4 Pulmonary position homografts also are used to replace pulmonary autografts explanted to repair left-sided outflow disease (the Ross operation). Several factors may be likely to favor increased pulmonary conduit durability in Ross patients compared with those with right ventricular outflow tract obstruction, including later age at operation (allowing for larger homografts), more normal pulmonary artery architecture, absence of severe right ventricular hypertrophy, and more natural positioning of ...
  How does MCAM expression in hMSC affect the growth and maintenance of hematopoietic progenitors?	After culture in a 3-dimensional hydrogel-based matrix, which constitutes hypoxic conditions, MCAM expression is lost. Concordantly, Tormin et al. demonstrated that MCAM is down-regulated under hypoxic conditions. 10 Furthermore, it was shown by others and our group that oxygen tension causes selective modification of hematopoietic cell and mesenchymal stromal cell interactions in co-culture systems as well as influence HSPC metabolism. [44] [45] [46] Thus, the observed differences between Sharma et al. and our data in HSPC supporting capacity of hMSC are likely due to the different culture conditions used. Further studies are required to clarify the influence of hypoxia in our model system. Altogether these findings provide further evidence for the importance of MCAM in supporting HSPC. Furthermore, previous reports have shown that MCAM is down-regulated in MSC after several passages as well as during aging and differentiation. 19, 47 Interestingly, MCAM overexpression in hMSC enhance...
  What is the relationship between Fanconi anemia and breast and ovarian cancer susceptibility genes?	( 31 ) , of which 5% -10 % may be caused by genetic factors ( 32 ) , up to half a million of these patients may be at risk of secondary hereditary neoplasms. The historic observation of twofold to fi vefold increased risks of cancers of the ovary, thyroid, and connective tissue after breast cancer ( 33 ) presaged the later syndromic association of these tumors with inherited mutations of BRCA1, BRCA2, PTEN, and p53 ( 16 ) . By far the largest cumulative risk of a secondary cancer in BRCA mutation carriers is associated with cancer in the contralateral breast, which may reach a risk of 29.5% at 10 years ( 34 ) . The Breast Cancer Linkage Consortium ( 35 , 36 ) also documented threefold to fi vefold increased risks of subsequent cancers of prostate, pancreas, gallbladder, stomach, skin (melanoma), and uterus in BRCA2 mutation carriers and twofold increased risks of prostate and pancreas cancer in BRCA1 mutation carriers; these results are based largely on self-reported family history inf...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

generator

• Dataset: generator
• Size: 10,000 evaluation samples
• Columns: question and passage_text
• Approximate statistics based on the first 100 samples:

  	question	passage_text
  type	string	string
  modality	text	text
  details	min: 12 tokens mean: 22.39 tokens max: 56 tokens	min: 525 tokens mean: 947.01 tokens max: 1024 tokens

• Samples:

  question	passage_text
  What are some hereditary cancer syndromes that can result in various forms of cancer?	Hereditary Cancer Syndromes, including Hereditary Breast and Ovarian Cancer (HBOC) and Lynch Syndrome (LS), can result in various forms of cancer due to germline mutations in cancer predisposition genes. While the major contributory genes for these syndromes have been identified and well-studied (BRCA1/ BRCA2 for HBOC and MSH2/MSH6/MLH1/PMS2/ EPCAM for LS), there remains a large percentage of associated cancer cases that are negative for germline mutations in these genes, including 80% of women with a personal or family history of breast cancer who are negative for BRCA1/2 mutations [1] . Similarly, between 30 and 50% of families fulfill stringent criteria for LS and test negative for germline mismatch repair gene mutations [2] . Adding complexity to these disorders is the significant overlap in the spectrum of cancers observed between various hereditary cancer syndromes, including many cancer susceptibility syndromes. Some that contribute to elevated breast cancer risk include Li-Frau...
  How do MAK-4 and MAK-5 exert their antioxidant properties?	Hybrid F1 mice were injected with urethane (300 mg/kg) at 8 days of age. A group was then put on a MAK-supplemented diet, another group was fed a standard pellet diet. At 36 weeks of age the mice were sacrificed and the livers examined for the presence of tumors mouse (Panel A) and for the number of nodules per mouse (Panel B) (* p < 0.05, ** P < 0.001). Statistical analysis was performed by Two Way ANOVA Test followed by Post Hoc Bonferroni analysis. We than measured the influence of the MAK-4+5 combination on the expression of the three liver-specific connexins (cx26, cx32, and cx43). The level of cx26 expression was similar in all the groups of mice treated with the MAK-supplemented diet and in the control (Figure 4, Panel A) . A significant, time-dependent increase in cx32 was observed in the liver of all the groups of MAK treated mice compared to the normal diet-fed controls. Cx32 expression increased 2-fold after 1 week of treatment, and 3-to 4-fold at 3 months (Figure 4, Pane...
  What are the primary indications for a decompressive craniectomy, and what role does neurocritical care play in determining the suitability of a patient for this procedure?	Decompressive craniectomy is a valid neurosurgical strategy now a day as an alternative to control an elevated intracranial pressure (ICP) and controlling the risk of uncal and/or subfalcine herniation, in refractory cases to the postural, ventilator, and pharmacological measures to control it. The neurocritical care and the ICP monitorization are key determinants to identify and postulate the inclusion criteria to consider a patient as candidate to this procedure, as it is always considered a rescue surgical technique. Head trauma and ischemic or hemorrhagic cerebrovascular disease with progressive deterioration due to mass effect are some of the cases that may require a decompressive craniectomy with its different variants. However, this procedure per se can have complications described in the postcraniectomy syndrome and may occur in short, medium, or even long term. 1,2 The paradoxical herniation is a condition in which there is a deviation of the midline with mass effect, even t...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.03
• bf16: True
• prompts: {'question': 'task: search result | query: ', 'passage_text': 'title: none | text: '}
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.03
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: {'question': 'task: search result | query: ', 'passage_text': 'title: none | text: '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	cosine_ndcg@10
-1	-1	-	-	0.8898
0.0128	5	-	0.0743	0.8908
0.0256	10	-	0.0549	0.8996
0.0384	15	0.0696	0.0418	0.9052
0.0512	20	-	0.0372	0.9067
0.0640	25	-	0.0346	0.9082
0.0768	30	0.0344	0.0327	0.9095
0.0896	35	-	0.0309	0.9116
0.1024	40	-	0.0296	0.9136
0.1152	45	0.0308	0.0285	0.9155
0.1280	50	-	0.0275	0.9171
0.1408	55	-	0.0268	0.9181
0.1536	60	0.0274	0.0262	0.9194
0.1663	65	-	0.0257	0.9205
0.1791	70	-	0.0250	0.9226
0.1919	75	0.0231	0.0245	0.9239
0.2047	80	-	0.0242	0.9253
0.2175	85	-	0.0239	0.9262
0.2303	90	0.0227	0.0238	0.9272
0.2431	95	-	0.0232	0.9283
0.2559	100	-	0.0226	0.9294
0.2687	105	0.024	0.0223	0.9299
0.2815	110	-	0.0220	0.9303
0.2943	115	-	0.0218	0.9305
0.3071	120	0.0175	0.0216	0.9308
0.3199	125	-	0.0213	0.9312
0.3327	130	-	0.0209	0.9319
0.3455	135	0.0195	0.0206	0.9324
0.3583	140	-	0.0204	0.9325
0.3711	145	-	0.0201	0.9324
0.3839	150	0.0199	0.0200	0.9321
0.3967	155	-	0.0198	0.9324
0.4095	160	-	0.0196	0.9330
0.4223	165	0.0184	0.0195	0.9336
0.4351	170	-	0.0194	0.9343
0.4479	175	-	0.0193	0.9347
0.4607	180	0.0206	0.0192	0.9348
0.4734	185	-	0.0191	0.9350
0.4862	190	-	0.0191	0.9347
0.4990	195	0.0176	0.0191	0.9346
0.5118	200	-	0.0188	0.9348
0.5246	205	-	0.0186	0.9354
0.5374	210	0.0199	0.0185	0.9352
0.5502	215	-	0.0184	0.9350
0.5630	220	-	0.0183	0.9351
0.5758	225	0.0176	0.0182	0.9355
0.5886	230	-	0.0181	0.9354
0.6014	235	-	0.0181	0.9357
0.6142	240	0.0167	0.0181	0.9359
0.6270	245	-	0.0180	0.9358
0.6398	250	-	0.0179	0.9356
0.6526	255	0.0158	0.0178	0.9361
0.6654	260	-	0.0178	0.9359
0.6782	265	-	0.0177	0.9363
0.6910	270	0.016	0.0177	0.9362
0.7038	275	-	0.0177	0.9362
0.7166	280	-	0.0176	0.9363
0.7294	285	0.0177	0.0176	0.9363
0.7422	290	-	0.0175	0.9366
0.7550	295	-	0.0175	0.9361
0.7678	300	0.0168	0.0174	0.9361
0.7806	305	-	0.0174	0.9362
0.7933	310	-	0.0173	0.9361
0.8061	315	0.0198	0.0172	0.9361
0.8189	320	-	0.0172	0.9363
0.8317	325	-	0.0171	0.9369
0.8445	330	0.0197	0.0171	0.9369
0.8573	335	-	0.0171	0.9367
0.8701	340	-	0.0170	0.9367
0.8829	345	0.0163	0.0170	0.9369
0.8957	350	-	0.0170	0.9369
0.9085	355	-	0.0170	0.9369
0.9213	360	0.0156	0.0170	0.9372
0.9341	365	-	0.0169	0.9371
0.9469	370	-	0.0169	0.9368
0.9597	375	0.0194	0.0169	0.9370
0.9725	380	-	0.0169	0.9369
0.9853	385	-	0.0169	0.9370
0.9981	390	0.018	0.0169	0.9369
-1	-1	-	-	0.9370

Training Time

• Training: 5.9 hours

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 5.7.0
• Transformers: 4.57.6
• PyTorch: 2.11.0+cu130
• Accelerate: 1.15.0
• Datasets: 4.3.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}