SentenceTransformer based on FacebookAI/roberta-base
This is a sentence-transformers model finetuned from FacebookAI/roberta-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: FacebookAI/roberta-base
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("SBERT-roberta_pf")
# Run inference
sentences = [
'section: Experimental Setup, text: The training data used for speech recognition -CSR -is different from the Treebank in two aspects: • the Treebank is only a subset of the usual CSR training data; • the Treebank tokenization is different from that of the CSR corpus; among other spurious small differences, the most frequent ones are of the type presented in',
'section: Multiple choice next sentence prediction (NSP), text: We have collected a new dataset with 54k multiple choice questions where the objective is to predict the correct continuation for a given context sentence from four possible answer choices.',
'section: Comparing with Previous Latent Semantic Models, text: 𝐹 𝑖 (or its translation candidate 𝐸), and 𝐲 be the projected feature vector, i.e., 𝐲 = 𝐖 T 𝐱.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Training Details
Training Dataset
Unnamed Dataset
- Size: 11,354 training samples
- Columns:
text
andlabel
- Approximate statistics based on the first 1000 samples:
text label type string int details - min: 11 tokens
- mean: 38.95 tokens
- max: 234 tokens
- 0: ~14.30%
- 1: ~10.70%
- 2: ~23.90%
- 3: ~0.40%
- 4: ~2.60%
- 5: ~1.50%
- 6: ~1.90%
- 7: ~44.70%
- Samples:
text label section: INTRODUCTION, text: Arguments for the importance of prosody in language abound in the literature.
0
section: Results, text: This overlap ensures that actions that might otherwise occur on clip boundaries will also occur as part of a clip.
7
section: Introduction, text: In Section 4 the experimental setup and results are detailed.
6
- Loss:
BatchAllTripletLoss
Evaluation Dataset
Unnamed Dataset
- Size: 1,419 evaluation samples
- Columns:
text
andlabel
- Approximate statistics based on the first 1000 samples:
text label type string int details - min: 10 tokens
- mean: 40.49 tokens
- max: 221 tokens
- 0: ~15.50%
- 1: ~11.10%
- 2: ~20.50%
- 3: ~0.50%
- 4: ~2.60%
- 5: ~1.50%
- 6: ~2.10%
- 7: ~46.20%
- Samples:
text label section: Introduction, text: It is common in Natural Language Processing (NLP) that the categories into which text is classified do not have fully objective definitions.
0
section: Automatic Evaluation Results, text: With respect to the BLEU score, this difference is 1.58 points absolute for the word based evaluation (27% relative increase), and 2.47 points absolute for the morphemebased evaluation (21% relative increase).
2
section: Neural Descriptor Fields, text: The sum of the maxpooled mask probabilities of all slots can be used for counting, and the loss can be back propagated to optimize NDF as well as the embeddings.
7
- Loss:
BatchAllTripletLoss
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: stepslearning_rate
: 1e-05weight_decay
: 0.1load_best_model_at_end
: Truepush_to_hub
: True
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: stepsprediction_loss_only
: Trueper_device_train_batch_size
: 8per_device_eval_batch_size
: 8per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 1eval_accumulation_steps
: Nonelearning_rate
: 1e-05weight_decay
: 0.1adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1.0num_train_epochs
: 3max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.0warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Falsefp16
: Falsefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Trueignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torchoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Trueresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Falsehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseeval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falsebatch_sampler
: batch_samplermulti_dataset_batch_sampler
: proportional
Training Logs
Epoch | Step | Training Loss | loss |
---|---|---|---|
0.3521 | 500 | 4.3466 | 4.0196 |
0.7042 | 1000 | 3.9809 | 3.3573 |
1.0563 | 1500 | 3.8231 | 3.7082 |
1.4085 | 2000 | 3.5722 | 3.6799 |
1.7606 | 2500 | 3.6224 | 3.4086 |
2.1127 | 3000 | 3.1266 | 3.2109 |
2.4648 | 3500 | 3.1252 | 3.3558 |
2.8169 | 4000 | 3.1115 | 3.1682 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.9.2
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.3.1+cu121
- Accelerate: 0.31.0
- Datasets: 2.19.2
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
BatchAllTripletLoss
@misc{hermans2017defense,
title={In Defense of the Triplet Loss for Person Re-Identification},
author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
year={2017},
eprint={1703.07737},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
- Downloads last month
- 4
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for ClementineBleuze/SBERT-roberta_pf
Base model
FacebookAI/roberta-base
Finetuned
this model