SentenceTransformer based on intfloat/multilingual-e5-base
This is a sentence-transformers model finetuned from intfloat/multilingual-e5-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: intfloat/multilingual-e5-base
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
Model Sources
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BotnoiNLPteam/me5_icd10_test")
sentences = [
'Contact with other powered hand tools and household machinery at residential institution While engaged in leisure activity',
'สัมผัสกับเครื่องมือที่มีเครื่องยนต์และเครื่องจักรกลอื่นในบ้าน ที่พักอาศัยรวมขณะทำกิจกรรมยามว่าง',
'หลอดเลือดอักเสบรูมาตอยด์',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
Evaluation
Metrics
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.8761 |
cosine_accuracy@3 |
0.9407 |
cosine_accuracy@5 |
0.9569 |
cosine_accuracy@10 |
0.9718 |
cosine_precision@1 |
0.8761 |
cosine_precision@3 |
0.3136 |
cosine_precision@5 |
0.1914 |
cosine_precision@10 |
0.0972 |
cosine_recall@1 |
0.8761 |
cosine_recall@3 |
0.9407 |
cosine_recall@5 |
0.9569 |
cosine_recall@10 |
0.9718 |
cosine_ndcg@10 |
0.9264 |
cosine_mrr@10 |
0.9115 |
cosine_map@100 |
0.9127 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.8768 |
cosine_accuracy@3 |
0.94 |
cosine_accuracy@5 |
0.9576 |
cosine_accuracy@10 |
0.9716 |
cosine_precision@1 |
0.8768 |
cosine_precision@3 |
0.3133 |
cosine_precision@5 |
0.1915 |
cosine_precision@10 |
0.0972 |
cosine_recall@1 |
0.8768 |
cosine_recall@3 |
0.94 |
cosine_recall@5 |
0.9576 |
cosine_recall@10 |
0.9716 |
cosine_ndcg@10 |
0.9265 |
cosine_mrr@10 |
0.9118 |
cosine_map@100 |
0.913 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.8761 |
cosine_accuracy@3 |
0.9376 |
cosine_accuracy@5 |
0.9557 |
cosine_accuracy@10 |
0.9714 |
cosine_precision@1 |
0.8761 |
cosine_precision@3 |
0.3125 |
cosine_precision@5 |
0.1911 |
cosine_precision@10 |
0.0971 |
cosine_recall@1 |
0.8761 |
cosine_recall@3 |
0.9376 |
cosine_recall@5 |
0.9557 |
cosine_recall@10 |
0.9714 |
cosine_ndcg@10 |
0.9257 |
cosine_mrr@10 |
0.9109 |
cosine_map@100 |
0.9121 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.8705 |
cosine_accuracy@3 |
0.9366 |
cosine_accuracy@5 |
0.9537 |
cosine_accuracy@10 |
0.9711 |
cosine_precision@1 |
0.8705 |
cosine_precision@3 |
0.3122 |
cosine_precision@5 |
0.1907 |
cosine_precision@10 |
0.0971 |
cosine_recall@1 |
0.8705 |
cosine_recall@3 |
0.9366 |
cosine_recall@5 |
0.9537 |
cosine_recall@10 |
0.9711 |
cosine_ndcg@10 |
0.923 |
cosine_mrr@10 |
0.9073 |
cosine_map@100 |
0.9086 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.8666 |
cosine_accuracy@3 |
0.9329 |
cosine_accuracy@5 |
0.9503 |
cosine_accuracy@10 |
0.9709 |
cosine_precision@1 |
0.8666 |
cosine_precision@3 |
0.311 |
cosine_precision@5 |
0.1901 |
cosine_precision@10 |
0.0971 |
cosine_recall@1 |
0.8666 |
cosine_recall@3 |
0.9329 |
cosine_recall@5 |
0.9503 |
cosine_recall@10 |
0.9709 |
cosine_ndcg@10 |
0.921 |
cosine_mrr@10 |
0.9048 |
cosine_map@100 |
0.9059 |
Training Details
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: epoch
per_device_train_batch_size
: 16
per_device_eval_batch_size
: 16
num_train_epochs
: 5
warmup_ratio
: 0.1
bf16
: True
batch_sampler
: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir
: False
do_predict
: False
eval_strategy
: epoch
prediction_loss_only
: True
per_device_train_batch_size
: 16
per_device_eval_batch_size
: 16
per_gpu_train_batch_size
: None
per_gpu_eval_batch_size
: None
gradient_accumulation_steps
: 1
eval_accumulation_steps
: None
learning_rate
: 5e-05
weight_decay
: 0.0
adam_beta1
: 0.9
adam_beta2
: 0.999
adam_epsilon
: 1e-08
max_grad_norm
: 1.0
num_train_epochs
: 5
max_steps
: -1
lr_scheduler_type
: linear
lr_scheduler_kwargs
: {}
warmup_ratio
: 0.1
warmup_steps
: 0
log_level
: passive
log_level_replica
: warning
log_on_each_node
: True
logging_nan_inf_filter
: True
save_safetensors
: True
save_on_each_node
: False
save_only_model
: False
restore_callback_states_from_checkpoint
: False
no_cuda
: False
use_cpu
: False
use_mps_device
: False
seed
: 42
data_seed
: None
jit_mode_eval
: False
use_ipex
: False
bf16
: True
fp16
: False
fp16_opt_level
: O1
half_precision_backend
: auto
bf16_full_eval
: False
fp16_full_eval
: False
tf32
: None
local_rank
: 0
ddp_backend
: None
tpu_num_cores
: None
tpu_metrics_debug
: False
debug
: []
dataloader_drop_last
: False
dataloader_num_workers
: 0
dataloader_prefetch_factor
: None
past_index
: -1
disable_tqdm
: False
remove_unused_columns
: True
label_names
: None
load_best_model_at_end
: False
ignore_data_skip
: False
fsdp
: []
fsdp_min_num_params
: 0
fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap
: None
accelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed
: None
label_smoothing_factor
: 0.0
optim
: adamw_torch
optim_args
: None
adafactor
: False
group_by_length
: False
length_column_name
: length
ddp_find_unused_parameters
: None
ddp_bucket_cap_mb
: None
ddp_broadcast_buffers
: False
dataloader_pin_memory
: True
dataloader_persistent_workers
: False
skip_memory_metrics
: True
use_legacy_prediction_loop
: False
push_to_hub
: False
resume_from_checkpoint
: None
hub_model_id
: None
hub_strategy
: every_save
hub_private_repo
: False
hub_always_push
: False
gradient_checkpointing
: False
gradient_checkpointing_kwargs
: None
include_inputs_for_metrics
: False
eval_do_concat_batches
: True
fp16_backend
: auto
push_to_hub_model_id
: None
push_to_hub_organization
: None
mp_parameters
:
auto_find_batch_size
: False
full_determinism
: False
torchdynamo
: None
ray_scope
: last
ddp_timeout
: 1800
torch_compile
: False
torch_compile_backend
: None
torch_compile_mode
: None
dispatch_batches
: None
split_batches
: None
include_tokens_per_second
: False
include_num_input_tokens_seen
: False
neftune_noise_alpha
: None
optim_target_modules
: None
batch_eval_metrics
: False
batch_sampler
: no_duplicates
multi_dataset_batch_sampler
: proportional
Training Logs
Epoch |
Step |
Training Loss |
dim_128_cosine_map@100 |
dim_256_cosine_map@100 |
dim_512_cosine_map@100 |
dim_64_cosine_map@100 |
dim_768_cosine_map@100 |
0.4352 |
1000 |
0.5774 |
- |
- |
- |
- |
- |
0.8703 |
2000 |
0.0914 |
- |
- |
- |
- |
- |
1.0 |
2298 |
- |
0.8254 |
0.8378 |
0.8427 |
0.8159 |
0.8439 |
1.3055 |
3000 |
0.0486 |
- |
- |
- |
- |
- |
1.7406 |
4000 |
0.018 |
- |
- |
- |
- |
- |
2.0 |
4596 |
- |
0.8674 |
0.8731 |
0.8756 |
0.8615 |
0.8781 |
2.1758 |
5000 |
0.0123 |
- |
- |
- |
- |
- |
2.6110 |
6000 |
0.0047 |
- |
- |
- |
- |
- |
3.0 |
6894 |
- |
0.8908 |
0.8940 |
0.8952 |
0.8868 |
0.8959 |
3.0461 |
7000 |
0.0031 |
- |
- |
- |
- |
- |
3.4813 |
8000 |
0.0013 |
- |
- |
- |
- |
- |
3.9164 |
9000 |
0.0012 |
- |
- |
- |
- |
- |
4.0 |
9192 |
- |
0.9049 |
0.9065 |
0.9075 |
0.8999 |
0.9079 |
4.3516 |
10000 |
0.0005 |
- |
- |
- |
- |
- |
4.7868 |
11000 |
0.0002 |
- |
- |
- |
- |
- |
5.0 |
11490 |
- |
0.9086 |
0.9121 |
0.9130 |
0.9059 |
0.9127 |
Framework Versions
- Python: 3.8.10
- Sentence Transformers: 3.0.1
- Transformers: 4.42.0.dev0
- PyTorch: 2.3.0+cu121
- Accelerate: 0.30.1
- Datasets: 2.19.1
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}