Ehsanl/me5-large-inst-nlr

SentenceTransformer based on intfloat/multilingual-e5-large-instruct

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large-instruct on the json dataset. It maps inputs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-large-instruct
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Evaluation

Metrics

Information Retrieval

• Dataset: legal-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.41
cosine_accuracy@5	0.682
cosine_accuracy@10	0.758
cosine_precision@1	0.41
cosine_precision@10	0.1274
cosine_recall@1	0.2215
cosine_recall@10	0.5047
cosine_ndcg@10	0.4276
cosine_mrr@10	0.5215
cosine_map@100	0.3582

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• learning_rate: 1e-05
• lr_scheduler_type: cosine
• warmup_steps: 592
• weight_decay: 0.01
• bf16: True
• per_device_eval_batch_size: 512
• push_to_hub: True
• hub_private_repo: True
• hub_model_id: Ehsanl/me5-large-inst-nlr
• hub_strategy: checkpoint
• load_best_model_at_end: True
• dataloader_num_workers: 2
• dataloader_pin_memory: False
• ddp_find_unused_parameters: False
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 512
• num_train_epochs: 3.0
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 592
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 512
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: True
• hub_private_repo: True
• hub_model_id: Ehsanl/me5-large-inst-nlr
• hub_strategy: checkpoint
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: False
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	legal-val_cosine_ndcg@10
0.0005	1	6.9345	-	-
0.0051	10	6.5904	-	-
0.0101	20	6.5405	-	-
0.0152	30	6.4785	-	-
0.0203	40	6.3686	-	-
0.0253	50	6.2295	-	-
0.0304	60	5.9314	-	-
0.0355	70	5.6377	-	-
0.0405	80	5.3884	-	-
0.0456	90	5.3833	-	-
0.0507	100	5.2380	-	-
0.0557	110	5.1489	-	-
0.0608	120	5.1054	-	-
0.0659	130	5.1459	-	-
0.0709	140	5.0190	-	-
0.0760	150	5.1052	-	-
0.0811	160	4.9734	-	-
0.0861	170	4.9099	-	-
0.0912	180	4.9292	-	-
0.0963	190	4.8553	-	-
0.1013	200	4.8303	-	-
0.1064	210	4.7476	-	-
0.1114	220	4.7810	-	-
0.1165	230	4.7275	-	-
0.1216	240	4.7798	-	-
0.1266	250	4.7553	-	-
0.1317	260	4.6994	-	-
0.1368	270	4.7646	-	-
0.1418	280	4.7063	-	-
0.1469	290	4.6860	-	-
0.1520	300	4.5707	-	-
0.1570	310	4.6649	-	-
0.1621	320	4.5316	-	-
0.1672	330	4.5702	-	-
0.1722	340	4.5718	-	-
0.1773	350	4.5302	-	-
0.1824	360	4.6567	-	-
0.1874	370	4.4909	-	-
0.1925	380	4.5185	-	-
0.1976	390	4.5552	-	-
0.2026	400	4.4308	-	-
0.2077	410	4.4451	-	-
0.2128	420	4.4714	-	-
0.2178	430	4.4771	-	-
0.2229	440	4.4498	-	-
0.2280	450	4.4611	-	-
0.2330	460	4.3519	-	-
0.2381	470	4.4713	-	-
0.2432	480	4.4024	-	-
0.2482	490	4.3919	-	-
0.2533	500	4.3316	2.0683	0.3504
0.2584	510	4.3246	-	-
0.2634	520	4.3759	-	-
0.2685	530	4.3493	-	-
0.2736	540	4.3497	-	-
0.2786	550	4.2904	-	-
0.2837	560	4.3097	-	-
0.2888	570	4.3087	-	-
0.2938	580	4.2688	-	-
0.2989	590	4.2476	-	-
0.3040	600	4.2668	-	-
0.3090	610	4.2278	-	-
0.3141	620	4.2182	-	-
0.3191	630	4.2489	-	-
0.3242	640	4.2550	-	-
0.3293	650	4.2964	-	-
0.3343	660	4.3190	-	-
0.3394	670	4.2399	-	-
0.3445	680	4.1597	-	-
0.3495	690	4.2672	-	-
0.3546	700	4.2180	-	-
0.3597	710	4.2011	-	-
0.3647	720	4.1931	-	-
0.3698	730	4.1991	-	-
0.3749	740	4.2092	-	-
0.3799	750	4.2044	-	-
0.3850	760	4.1337	-	-
0.3901	770	4.1630	-	-
0.3951	780	4.1857	-	-
0.4002	790	4.1443	-	-
0.4053	800	4.1820	-	-
0.4103	810	4.0890	-	-
0.4154	820	4.1168	-	-
0.4205	830	4.1253	-	-
0.4255	840	4.1401	-	-
0.4306	850	4.0797	-	-
0.4357	860	4.1509	-	-
0.4407	870	4.0855	-	-
0.4458	880	4.0931	-	-
0.4509	890	4.0778	-	-
0.4559	900	4.0985	-	-
0.4610	910	4.0401	-	-
0.4661	920	4.0068	-	-
0.4711	930	4.0838	-	-
0.4762	940	4.0577	-	-
0.4813	950	3.9791	-	-
0.4863	960	3.9822	-	-
0.4914	970	4.0811	-	-
0.4965	980	4.0294	-	-
0.5015	990	4.0536	-	-
0.5066	1000	4.0272	1.8259	0.3996
0.5117	1010	4.0741	-	-
0.5167	1020	4.0633	-	-
0.5218	1030	4.0132	-	-
0.5268	1040	3.9364	-	-
0.5319	1050	3.9959	-	-
0.5370	1060	4.0334	-	-
0.5420	1070	3.9527	-	-
0.5471	1080	3.9823	-	-
0.5522	1090	4.0388	-	-
0.5572	1100	3.9964	-	-
0.5623	1110	3.9131	-	-
0.5674	1120	3.9778	-	-
0.5724	1130	3.9594	-	-
0.5775	1140	4.0259	-	-
0.5826	1150	3.9364	-	-
0.5876	1160	3.9169	-	-
0.5927	1170	3.9703	-	-
0.5978	1180	3.9276	-	-
0.6028	1190	3.8777	-	-
0.6079	1200	3.9097	-	-
0.6130	1210	3.9464	-	-
0.6180	1220	3.9298	-	-
0.6231	1230	3.9301	-	-
0.6282	1240	3.8984	-	-
0.6332	1250	3.9419	-	-
0.6383	1260	3.9158	-	-
0.6434	1270	3.8775	-	-
0.6484	1280	3.8590	-	-
0.6535	1290	3.8953	-	-
0.6586	1300	3.9215	-	-
0.6636	1310	3.9956	-	-
0.6687	1320	3.9052	-	-
0.6738	1330	3.8727	-	-
0.6788	1340	3.8240	-	-
0.6839	1350	3.8701	-	-
0.6890	1360	3.8709	-	-
0.6940	1370	3.9179	-	-
0.6991	1380	3.8752	-	-
0.7042	1390	3.8466	-	-
0.7092	1400	3.9109	-	-
0.7143	1410	3.8571	-	-
0.7194	1420	3.8906	-	-
0.7244	1430	3.8358	-	-
0.7295	1440	3.8916	-	-
0.7345	1450	3.7927	-	-
0.7396	1460	3.4740	-	-
0.7447	1470	3.2860	-	-
0.7497	1480	3.3238	-	-
0.7548	1490	3.2031	-	-
0.7599	1500	3.2032	1.6583	0.4201
0.7649	1510	3.2668	-	-
0.7700	1520	3.2168	-	-
0.7751	1530	3.2706	-	-
0.7801	1540	3.2233	-	-
0.7852	1550	3.2142	-	-
0.7903	1560	3.3751	-	-
0.7953	1570	3.1927	-	-
0.8004	1580	3.2709	-	-
0.8055	1590	3.1908	-	-
0.8105	1600	3.1674	-	-
0.8156	1610	3.2348	-	-
0.8207	1620	3.1716	-	-
0.8257	1630	3.0472	-	-
0.8308	1640	3.1482	-	-
0.8359	1650	3.1924	-	-
0.8409	1660	3.0583	-	-
0.8460	1670	3.2053	-	-
0.8511	1680	3.1079	-	-
0.8561	1690	3.2054	-	-
0.8612	1700	3.0135	-	-
0.8663	1710	3.1975	-	-
0.8713	1720	3.2074	-	-
0.8764	1730	3.1560	-	-
0.8815	1740	3.1503	-	-
0.8865	1750	3.0803	-	-
0.8916	1760	2.9967	-	-
0.8967	1770	3.0067	-	-
0.9017	1780	3.0268	-	-
0.9068	1790	3.1135	-	-
0.9119	1800	3.0523	-	-
0.9169	1810	3.0117	-	-
0.9220	1820	3.0953	-	-
0.9271	1830	3.0295	-	-
0.9321	1840	3.0173	-	-
0.9372	1850	2.9733	-	-
0.9422	1860	3.1502	-	-
0.9473	1870	3.0656	-	-
0.9524	1880	2.9975	-	-
0.9574	1890	3.0351	-	-
0.9625	1900	2.9345	-	-
0.9676	1910	3.1698	-	-
0.9726	1920	2.9501	-	-
0.9777	1930	2.9606	-	-
0.9828	1940	2.8870	-	-
0.9878	1950	2.9953	-	-
0.9929	1960	3.0165	-	-
0.9980	1970	2.9291	-	-
1.0030	1980	3.4793	-	-
1.0081	1990	3.7514	-	-
1.0132	2000	3.7655	1.6714	0.4276

Training Time

• Training: 23.8 hours
• Evaluation: 49.3 minutes
• Total: 1.0 days

Framework Versions

• Python: 3.12.14
• Sentence Transformers: 6.1.0
• Transformers: 5.17.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.15.0
• Datasets: 5.0.1
• Tokenizers: 0.23.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}