--- language: [] library_name: sentence-transformers tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:10330 - loss:MultipleNegativesRankingLoss base_model: indobenchmark/indobert-base-p2 datasets: [] metrics: - pearson_cosine - spearman_cosine - pearson_manhattan - spearman_manhattan - pearson_euclidean - spearman_euclidean - pearson_dot - spearman_dot - pearson_max - spearman_max pipeline_tag: sentence-similarity model-index: - name: SentenceTransformer based on indobenchmark/indobert-base-p2 results: - task: type: semantic-similarity name: Semantic Similarity dataset: name: sts dev type: sts-dev metrics: - type: pearson_cosine value: -0.051616661741529624 name: Pearson Cosine - type: spearman_cosine value: -0.059260236757554256 name: Spearman Cosine - type: pearson_manhattan value: -0.06426082223860986 name: Pearson Manhattan - type: spearman_manhattan value: -0.06596359759097158 name: Spearman Manhattan - type: pearson_euclidean value: -0.06368615893415144 name: Pearson Euclidean - type: spearman_euclidean value: -0.06528449816144678 name: Spearman Euclidean - type: pearson_dot value: -0.027898791319537007 name: Pearson Dot - type: spearman_dot value: -0.02595347491107127 name: Spearman Dot - type: pearson_max value: -0.027898791319537007 name: Pearson Max - type: spearman_max value: -0.02595347491107127 name: Spearman Max --- # SentenceTransformer based on indobenchmark/indobert-base-p2 This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2) - **Maximum Sequence Length:** 200 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 200, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("sentence_transformers_model_id") # Run inference sentences = [ 'Waduk wadaslintang sebenarnya terbagi menjadi dua kabupaten yaitu kabupaten kebumen dan kabupaten wonosobo.', 'Kabupaten kebumen dan kabupaten wonosobo bertentaggaan.', 'Musim ini di ajang PBL 2020 Hendra melawan tim Pune 7 aces.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Semantic Similarity * Dataset: `sts-dev` * Evaluated with [EmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator) | Metric | Value | |:-------------------|:-----------| | pearson_cosine | -0.0516 | | spearman_cosine | -0.0593 | | pearson_manhattan | -0.0643 | | spearman_manhattan | -0.066 | | pearson_euclidean | -0.0637 | | spearman_euclidean | -0.0653 | | pearson_dot | -0.0279 | | spearman_dot | -0.026 | | pearson_max | -0.0279 | | **spearman_max** | **-0.026** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 10,330 training samples * Columns: sentence_0, sentence_1, and label * Approximate statistics based on the first 1000 samples: | | sentence_0 | sentence_1 | label | |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence_0 | sentence_1 | label | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:---------------| | Pada tahun 1436, pulau Timor mempunyai 12 kota bandar namun tidak disebutkan namanya. | Pulau Timor memiliki 10 kota bandar. | 2 | | Komoditas pertanian yang ada di desa ini antara lain: bunga potong, sayur mayur, waluh (lejet) terutama Paprika (Capsicum annuum L.). Komoditas ini menjadi sumber perekonomian utama di desa ini karena harganya yang lumayan dibandingkan sayuran lain. | Komoditas pertanian di desa ini lebih mahal dibandingkan sayuran lain. | 1 | | Setelah batas waktu pencalonan pada tanggal 15 Juli 2003, sembilan kota telah mencalonkan diri untuk mengadakan Olimpiade 2012. Kota-kota tersebut adalah Havana, Istanbul, Leipzig, London, Madrid, Moskwa, New York City, Paris, dan Rio de Janeiro. Pada 18 Mei 2004, Komite Olimpiade Internasional (IOC), sebagai hasil penilaian teknis, mengurangi jumlah kota kandidat menjadi lima: London, Madrid, Moskwa, New York, dan Paris. | Jumlah kota kandidat tuan rumah olimpide bertambah pada 18 Mei 2004. | 2 | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim" } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 32 - `multi_dataset_batch_sampler`: round_robin #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 32 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1 - `num_train_epochs`: 3 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.0 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: round_robin
### Training Logs | Epoch | Step | Training Loss | sts-dev_spearman_max | |:------:|:----:|:-------------:|:--------------------:| | 0.0991 | 32 | - | -0.0592 | | 0.1981 | 64 | - | -0.0425 | | 0.2972 | 96 | - | -0.0467 | | 0.3963 | 128 | - | -0.0428 | | 0.4954 | 160 | - | -0.0512 | | 0.5944 | 192 | - | -0.0473 | | 0.6935 | 224 | - | -0.0412 | | 0.7926 | 256 | - | -0.0435 | | 0.8916 | 288 | - | -0.0405 | | 0.9907 | 320 | - | -0.0425 | | 1.0 | 323 | - | -0.0420 | | 1.0898 | 352 | - | -0.0346 | | 1.1889 | 384 | - | -0.0333 | | 1.2879 | 416 | - | -0.0325 | | 1.3870 | 448 | - | -0.0312 | | 1.4861 | 480 | - | -0.0316 | | 1.5480 | 500 | 0.077 | - | | 1.5851 | 512 | - | -0.0260 | ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.0.1 - Transformers: 4.41.2 - PyTorch: 2.3.0+cu121 - Accelerate: 0.31.0 - Datasets: 2.19.2 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```