Anakeen/all-MiniLM-L6-v2_nte2_pdts32_lr5e-05_wr0.5

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the q_a, mlm and simce datasets. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Training Datasets:
  • q_a
  • mlm
  • simce

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Anakeen/all-MiniLM-L6-v2_nte2_pdts32_lr5e-05_wr0.5")
# Run inference
sentences = [
    "\nArticle 4 - Termination for Fraud\nThe Principal may at any time terminate a Member's or Dependent's eligibility under the Group Policy:\na. in Writing and with 31-day notice, if the individual submits any claim that contains false or fraudulent elements under state or federal law; or\nb. in Writing and with 31-day notice, upon finding in a civil or criminal case that a Member or Dependent has submitted claims that contain false or fraudulent elements under state or federal law; or\nc. in Writing and with 31-day notice, when a Member or Dependent has submitted a claim which, in good faith judgement and investigation, a Member or Dependent knew or should have known, contains false or fraudulent elements under state or federal law.\n",
    "\nArticle 4 - Termination for Fraud\nThe Principal may at any time terminate a Member's or Dependent's eligibility under the Group Policy:\na. in Writing and with 31-day notice, if the individual submits any claim that contains false or fraudulent elements under state or federal law; or\nb. in Writing and with 31-day notice, upon finding in a civil or criminal case that a Member or Dependent has submitted claims that contain false or fraudulent elements under state or federal law; or\nc. in Writing and with 31-day notice, when a Member or Dependent has submitted a claim which, in good faith judgement and investigation, a Member or Dependent knew or should have known, contains false or fraudulent elements under state or federal law.\n",
    '\nArticle 7 - Certificates\nThe Principal will give the Policyholder Certificates for delivery to insured Members. The delivery of such Certificates will be in either paper or electronic format. The Certificates will be evidence of insurance and will describe the basic features of the coverage. They will not be considered a part of this Group Policy.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Datasets

q_a

• Dataset: q_a
• Size: 27 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 3 tokens mean: 5.52 tokens max: 21 tokens	min: 5 tokens mean: 78.96 tokens max: 254 tokens

• Samples:

  query	positive
  Active Work; Actively at Work	A Member will be considered Actively at Work if he or she is able and available for active performance of all of his or her regular duties. Short term absence because of a regularly scheduled day off, holiday, vacation day, jury duty, funeral leave, or personal time off is considered Active Work provided the Member is able and available for active performance of all of his or her regular duties and was working the day immediately prior to the date of his or her absence.
  Activities of Daily Living (ADL)	a. Bathing - the ability to wash oneself in the tub or shower or by sponge with or without equipment or adaptive devices. b. Dressing - the ability to put on and take off garments and medically necessary braces or artificial limbs usually worn and to fasten or unfasten them. c. Eating/Feeding - the ability to get nourishment into the body by any means once it has been prepared and made available. d. Toileting - the ability to get to and from and on and off the toilet, to maintain a reasonable level of personal hygiene and to care for clothing. e. Transferring - the ability to move in and out of a chair or bed with or without equipment such as canes, quad canes, walkers, crutches, grab bars or other support devices including mechanical or motorized devices. f. Continence - the ability to voluntarily control bowel and bladder function, or in the event of incontinence, the ability to maintain a reasonable level of personal hygiene.
  Activities of Daily Living (ADL) Disabled; Activities of Daily Living (ADL) Disability	A Member will be considered disabled under this provision if, as a result of sickness or injury, the Member has lost the ability to safely and completely perform two or more Activities of Daily Living without another person's assistance or verbal cueing or the Member has a deterioration or loss in intellectual capacity and needs another person's assistance or verbal cueing for his or her protection or for the protection of others and the Member is Totally Disabled.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

mlm

• Dataset: mlm
• Size: 122 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 7 tokens mean: 132.83 tokens max: 256 tokens	min: 7 tokens mean: 135.05 tokens max: 256 tokens

• Samples:

  query	positive
  POLICY [MASK] S655 COVERAGE: Life POLICY RIDER [MASK] INSURANCE [MASK] RHODE ISLAND [MASK] DOE Effective on the later of [MASK] Date of Issue of this Group [MASK] or March 1, 2005, the following will apply to your Policy: From time to [MASK] The Principal [MASK] offer or provide certain employer groups who apply for coverage with The Principal a Financial Services Hotline and Grief Support Services or [MASK] other value added service for the employees of that employer group. In addition, The Principal may arrange for third party [MASK] [MASK] [MASK] optometrists, health clubs), to provide discounted goods and services to those employer groups who apply [MASK] coverage with The Principal or who become insureds/enrollees of The Principal. While The Principal has arranged these [MASK] services [MASK] third party provider discounts, [MASK] third party [MASK] providers [MASK] liable to [MASK] applicants/insureds/enrollees for the provision of such goods and/or services. The Principal is not responsible [MASK] the provision of such goods [MASK] services nor is it [MASK] for the failure of the provision of the same. Further, [MASK] Principal is not liable to the [MASK] for the negligent provision of such goods and/or services by the third party [MASK] providers. EXCEPT AS SPECIFICALLY DESCRIBED IN THIS RIDER, ALL [MASK] BENEFITS AND PROVISIONS WILL BE AS DESCRIBED IN THE GROUP POLICY. PRINCIPAL LIFE INSURANCE COMPANY DES [MASK] IOWA 50392-0001	POLICY NO: S655 COVERAGE: Life POLICY RIDER GROUP INSURANCE EMPLOYER: RHODE ISLAND JOHN DOE Effective on the later of the Date of Issue of this Group Policy or March 1, 2005, the following will apply to your Policy: From time to time The Principal may offer or provide certain employer groups who apply for coverage with The Principal a Financial Services Hotline and Grief Support Services or any other value added service for the employees of that employer group. In addition, The Principal may arrange for third party service providers (i.e., optometrists, health clubs), to provide discounted goods and services to those employer groups who apply for coverage with The Principal or who become insureds/enrollees of The Principal. While The Principal has arranged these goods, services and/or third party provider discounts, the third party service providers are liable to the applicants/insureds/enrollees for the provision of such goods and/or services. The Principal is not responsible for the provision of such goods and/or services nor is it liable for the failure of the provision of the same. Further, The Principal is not liable to the applicants/insureds/enrollees for the negligent provision of such goods and/or services by the third party service providers. EXCEPT AS SPECIFICALLY DESCRIBED IN THIS RIDER, ALL OTHER BENEFITS AND PROVISIONS WILL BE AS DESCRIBED IN THE GROUP POLICY. PRINCIPAL LIFE INSURANCE COMPANY DES MOINES, IOWA 50392-0001
  PRINCIPAL LIFE INSURANCE [MASK] (called The Principal in [MASK] Group Policy) Des Moines, Iowa 50392-0002 This group insurance policy is issued to: RHODE ISLAND JOHN [MASK] (called the Policyholder in this Group Policy) The Date of Issue is November 1, 2007. In [MASK] for the Policyholder's application [MASK] [MASK] of all premiums when due, The Principal agrees to provide: MEMBER [MASK] INSURANCE MEMBER [MASK] DEATH AND DISMEMBERMENT INSURANCE DEPENDENT LIFE [MASK] [MASK] to the terms and conditions [MASK] in this Group Policy. GROUP POLICY NO. GL [MASK] RENEWABLE TERM - NON-PARTICIPATING CONTRACT STATE OF ISSUE: RHODE ISLAND	PRINCIPAL LIFE INSURANCE COMPANY (called The Principal in this Group Policy) Des Moines, Iowa 50392-0002 This group insurance policy is issued to: RHODE ISLAND JOHN DOE (called the Policyholder in this Group Policy) The Date of Issue is November 1, 2007. In return for the Policyholder's application and payment of all premiums when due, The Principal agrees to provide: MEMBER LIFE INSURANCE MEMBER ACCIDENTAL DEATH AND DISMEMBERMENT INSURANCE DEPENDENT LIFE INSURANCE subject to the terms and conditions described in this Group Policy. GROUP POLICY NO. GL S655 RENEWABLE TERM - NON-PARTICIPATING CONTRACT STATE OF ISSUE: RHODE ISLAND
  [MASK] Work; Actively at Work A Member will be [MASK] Actively at Work if he or she is [MASK] and available for active performance of all of his or [MASK] regular duties. Short term [MASK] because of a regularly scheduled day off, holiday, vacation [MASK] jury duty, funeral leave, or personal [MASK] off is considered Active Work provided the Member is able [MASK] available for active performance of all of [MASK] or her regular duties and was working the day immediately prior to the date of his or her absence.	Active Work; Actively at Work A Member will be considered Actively at Work if he or she is able and available for active performance of all of his or her regular duties. Short term absence because of a regularly scheduled day off, holiday, vacation day, jury duty, funeral leave, or personal time off is considered Active Work provided the Member is able and available for active performance of all of his or her regular duties and was working the day immediately prior to the date of his or her absence.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

simce

• Dataset: simce
• Size: 122 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 7 tokens mean: 136.61 tokens max: 256 tokens	min: 7 tokens mean: 136.61 tokens max: 256 tokens	min: 19 tokens mean: 133.96 tokens max: 256 tokens

• Samples:

  anchor	positive	negative
  POLICY NO: S655 COVERAGE: Life POLICY RIDER GROUP INSURANCE EMPLOYER: RHODE ISLAND JOHN DOE Effective on the later of the Date of Issue of this Group Policy or March 1, 2005, the following will apply to your Policy: From time to time The Principal may offer or provide certain employer groups who apply for coverage with The Principal a Financial Services Hotline and Grief Support Services or any other value added service for the employees of that employer group. In addition, The Principal may arrange for third party service providers (i.e., optometrists, health clubs), to provide discounted goods and services to those employer groups who apply for coverage with The Principal or who become insureds/enrollees of The Principal. While The Principal has arranged these goods, services and/or third party provider discounts, the third party service providers are liable to the applicants/insureds/enrollees for the provision of such goods and/or services. The Principal is not responsible for the provision of such goods and/or services nor is it liable for the failure of the provision of the same. Further, The Principal is not liable to the applicants/insureds/enrollees for the negligent provision of such goods and/or services by the third party service providers. EXCEPT AS SPECIFICALLY DESCRIBED IN THIS RIDER, ALL OTHER BENEFITS AND PROVISIONS WILL BE AS DESCRIBED IN THE GROUP POLICY. PRINCIPAL LIFE INSURANCE COMPANY DES MOINES, IOWA 50392-0001	POLICY NO: S655 COVERAGE: Life POLICY RIDER GROUP INSURANCE EMPLOYER: RHODE ISLAND JOHN DOE Effective on the later of the Date of Issue of this Group Policy or March 1, 2005, the following will apply to your Policy: From time to time The Principal may offer or provide certain employer groups who apply for coverage with The Principal a Financial Services Hotline and Grief Support Services or any other value added service for the employees of that employer group. In addition, The Principal may arrange for third party service providers (i.e., optometrists, health clubs), to provide discounted goods and services to those employer groups who apply for coverage with The Principal or who become insureds/enrollees of The Principal. While The Principal has arranged these goods, services and/or third party provider discounts, the third party service providers are liable to the applicants/insureds/enrollees for the provision of such goods and/or services. The Principal is not responsible for the provision of such goods and/or services nor is it liable for the failure of the provision of the same. Further, The Principal is not liable to the applicants/insureds/enrollees for the negligent provision of such goods and/or services by the third party service providers. EXCEPT AS SPECIFICALLY DESCRIBED IN THIS RIDER, ALL OTHER BENEFITS AND PROVISIONS WILL BE AS DESCRIBED IN THE GROUP POLICY. PRINCIPAL LIFE INSURANCE COMPANY DES MOINES, IOWA 50392-0001	e. Benefits Payable If death occurs while Coverage During Disability is in force, The Principal will pay the Member's beneficiary the Member Life Insurance benefit amount that would have been paid had the Member remained insured under the Schedule of Insurance in force on the date Total Disability began. Such benefit shall be subject to any reduction provided under the Schedule of Insurance. However, NO BENEFIT WILL BE PAID if Written proof of ADL Disability or Total Disability is not sent to The Principal within one year of the date ADL Disability or Total Disability began. Failure to give Written proof within the time specified will not invalidate or reduce any claim if Written proof is given as soon as reasonably possible. Further, if a death benefit is paid under this section of the Group Policy, it will be in place of all other Member Life Insurance benefits provided under this Group Policy.
  PRINCIPAL LIFE INSURANCE COMPANY (called The Principal in this Group Policy) Des Moines, Iowa 50392-0002 This group insurance policy is issued to: RHODE ISLAND JOHN DOE (called the Policyholder in this Group Policy) The Date of Issue is November 1, 2007. In return for the Policyholder's application and payment of all premiums when due, The Principal agrees to provide: MEMBER LIFE INSURANCE MEMBER ACCIDENTAL DEATH AND DISMEMBERMENT INSURANCE DEPENDENT LIFE INSURANCE subject to the terms and conditions described in this Group Policy. GROUP POLICY NO. GL S655 RENEWABLE TERM - NON-PARTICIPATING CONTRACT STATE OF ISSUE: RHODE ISLAND	PRINCIPAL LIFE INSURANCE COMPANY (called The Principal in this Group Policy) Des Moines, Iowa 50392-0002 This group insurance policy is issued to: RHODE ISLAND JOHN DOE (called the Policyholder in this Group Policy) The Date of Issue is November 1, 2007. In return for the Policyholder's application and payment of all premiums when due, The Principal agrees to provide: MEMBER LIFE INSURANCE MEMBER ACCIDENTAL DEATH AND DISMEMBERMENT INSURANCE DEPENDENT LIFE INSURANCE subject to the terms and conditions described in this Group Policy. GROUP POLICY NO. GL S655 RENEWABLE TERM - NON-PARTICIPATING CONTRACT STATE OF ISSUE: RHODE ISLAND	Article 11 - Electronic Transactions Any transaction relating to this Group Policy may be conducted by electronic means if performance of the transaction is consistent with applicable state and federal law. Any notice required by the provisions of this Group Policy given by electronic means will have the same force and effect as notice given in writing.
  Active Work; Actively at Work A Member will be considered Actively at Work if he or she is able and available for active performance of all of his or her regular duties. Short term absence because of a regularly scheduled day off, holiday, vacation day, jury duty, funeral leave, or personal time off is considered Active Work provided the Member is able and available for active performance of all of his or her regular duties and was working the day immediately prior to the date of his or her absence.	Active Work; Actively at Work A Member will be considered Actively at Work if he or she is able and available for active performance of all of his or her regular duties. Short term absence because of a regularly scheduled day off, holiday, vacation day, jury duty, funeral leave, or personal time off is considered Active Work provided the Member is able and available for active performance of all of his or her regular duties and was working the day immediately prior to the date of his or her absence.	Article 3 - Dependent Life Insurance A person will be eligible for Dependent Life Insurance on the latest of: a. the date the person is eligible for Member Life Insurance; or b. the date the person first acquires a Dependent; or c. the date the person enters a class for which Dependent Life Insurance is provided under this Group Policy; or d. the date Dependent Life Insurance is added to this Group Policy. If a Member's Dependent is employed and is covered under group term life coverage or coverages provided by the Dependent's employer, the date such coverage terminates because the Dependent is no longer eligible under his/her employer's coverage will be considered the date the Member first acquires that Dependent (and any other Dependent who was also covered under such group coverage or coverages).

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• num_train_epochs: 2
• warmup_ratio: 0.5

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.5
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Framework Versions

• Python: 3.9.6
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.3.1
• Accelerate: 0.31.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}