TdelaSelle/PatriLaCSE

SentenceTransformer based on bowphs/LaBerta

This is a sentence-transformers model finetuned from bowphs/LaBerta. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: bowphs/LaBerta
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TdelaSelle/PatriLaCSE")
# Run inference
queries = [
    'et item: introivit eschatologie introeo semel in sancta, aeterna redemptione inventa désir invenio.',
]
documents = [
    'et item: introivit eschatologie introeo semel in sancta, aeterna redemptione inventa désir invenio.',
    'et illud quod scriptum est: non est speciosa laus in ore peccatoris, in ore certe fidelium utique speciosa est.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [2, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[1.0000, 0.1047]])

Evaluation

Metrics

Information Retrieval

• Dataset: patrilacse-ir
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.51
cosine_accuracy@3	0.6065
cosine_accuracy@5	0.6447
cosine_accuracy@10	0.6811
cosine_precision@1	0.51
cosine_precision@3	0.2394
cosine_precision@5	0.1579
cosine_precision@10	0.0859
cosine_recall@1	0.3774
cosine_recall@3	0.4971
cosine_recall@5	0.5397
cosine_recall@10	0.5811
cosine_ndcg@10	0.5217
cosine_mrr@10	0.5675
cosine_map@100	0.478

Training Details

Training Dataset

Unnamed Dataset

• Size: 198,457 training samples
• Columns: sentence_1 and sentence_2
• Approximate statistics based on the first 100 samples:

  	sentence_1	sentence_2
  type	string	string
  modality	text	text
  details	min: 8 tokens mean: 27.41 tokens max: 72 tokens	min: 8 tokens mean: 27.41 tokens max: 72 tokens

• Samples:

  sentence_1	sentence_2
  in quo omnes peccauerunt, sola omnium hominum caro intellegenda est, quae inde transfusa est, a toto partem significante scriptura.	in quo omnes peccauerunt, sola omnium hominum caro intellegenda est, quae inde transfusa est, a toto partem significante scriptura.
  si enim ex dei patris persona ad christum dicitur cui sententiae utrum sequentia consonent uix est adserere eundem dicit primogenitum quem unigenitum: primogenitum, quia etiam nos filii dei sumus, unigenitum uero, quoniam solus ille de substantia patris et patri aequalis atque coaeternus est.	si enim ex dei patris persona ad christum dicitur cui sententiae utrum sequentia consonent uix est adserere eundem dicit primogenitum quem unigenitum: primogenitum, quia etiam nos filii dei sumus, unigenitum uero, quoniam solus ille de substantia patris et patri aequalis atque coaeternus est.
  prius affectus sancti, sine quibus nec virtus nomen nec opus meritum nec cogitatio consequitur fructum.	prius affectus sancti, sine quibus nec virtus nomen nec opus meritum nec cogitatio consequitur fructum.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 32,
      "mini_batch_num_tokens": null,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• learning_rate: 1e-05
• weight_decay: 0.01
• num_train_epochs: 1.0
• warmup_ratio: 0.06
• bf16: True
• dataloader_drop_last: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1.0
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.06
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	patrilacse-ir_cosine_ndcg@10
0.0258	20	0.0804	-
0.0516	40	0.0045	-
0.0774	60	0.0018	-
0.1032	80	0.0007	-
0.1290	100	0.0006	-
0.1548	120	0.0008	-
0.1806	140	0.0007	-
0.2065	160	0.0007	-
0.2323	180	0.0009	-
0.2581	200	0.0004	-
0.2839	220	0.0003	-
0.3097	240	0.0002	-
0.3355	260	0.001	-
0.3613	280	0.0007	-
0.3871	300	0.0005	-
0.4129	320	0.0004	-
0.4387	340	0.0004	-
0.4645	360	0.0009	-
0.4903	380	0.0005	-
0.5161	400	0.0007	-
0.5419	420	0.0005	-
0.5677	440	0.0003	-
0.5935	460	0.0002	-
0.6194	480	0.0003	-
0.6452	500	0.0004	-
0.6710	520	0.0002	-
0.6968	540	0.0004	-
0.7226	560	0.0002	-
0.7484	580	0.0002	-
0.7742	600	0.0004	-
0.8	620	0.0004	-
0.8258	640	0.0003	-
0.8516	660	0.0003	-
0.8774	680	0.0002	-
0.9032	700	0.0004	-
0.9290	720	0.0004	-
0.9548	740	0.0003	-
0.9806	760	0.0005	-
-1	-1	-	0.5217

Training Time

• Training: 5.9 minutes

Framework Versions

• Python: 3.10.20
• Sentence Transformers: 5.7.0
• Transformers: 4.57.6
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.6.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}