Hugging Face's logo

SwarajSolanke-turtle
/
make-model-mapping-TSDAE

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
dense
Generated from Trainer
dataset_size:86450
loss:DenoisingAutoEncoderLoss
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased. It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: google-bert/bert-base-uncased
  • Maximum Sequence Length: 32 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("SwarajSolanke-turtle/bert-base-uncased-tsdae-fw")
# Run inference
sentences = [
    'subaru -',
    'subaru impreza - petrol 2000',
    'honda br - v s diesel 1498',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9939, 0.9908],
#         [0.9939, 1.0000, 0.9881],
#         [0.9908, 0.9881, 1.0000]])

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.966

Training Details

Training Dataset

Unnamed Dataset

  • Size: 86,450 training samples
  • Columns: damaged and original
  • Approximate statistics based on the first 100 samples:
    damaged original
    type string string
    modality text text
    details
    • min: 4 tokens
    • mean: 11.63 tokens
    • max: 22 tokens
    • min: 8 tokens
    • mean: 16.18 tokens
    • max: 25 tokens
  • Samples:
    damaged original
    1.5 petrol 1482 hyundai alcazar 1.5 executive mt petrol 1482
    mahindra. 300 1.5 w8 tone diesel 1497 mahindra. xuv 300 1.5 w8 dual tone diesel 1497
    mahindra scorpio vlx mhawk bs iv airbag diesel 2179 mahindra scorpio vlx 2.2 mhawk bs iv airbag diesel 2179
  • Loss: DenoisingAutoEncoderLoss with these parameters:
    {
    "decoder_name_or_path": "google-bert/bert-base-uncased",
    "need_retokenization": false
}

Evaluation Dataset

Unnamed Dataset

  • Size: 1,765 evaluation samples
  • Columns: damaged and original
  • Approximate statistics based on the first 100 samples:
    damaged original
    type string string
    modality text text
    details
    • min: 5 tokens
    • mean: 11.19 tokens
    • max: 20 tokens
    • min: 9 tokens
    • mean: 15.5 tokens
    • max: 29 tokens
  • Samples:
    damaged original
    bentley continental gt 6.0 speed convertible 4 petrol bentley continental gt 6.0 speed convertible 4 seater petrol 5998
    tata 1.2 rtn bs-vi 1199 tata altroz xt 1.2 rtn bs-vi petrol 1199
    maruti s-cross hybrid diesel 1248 maruti s-cross hybrid diesel 1248
  • Loss: DenoisingAutoEncoderLoss with these parameters:
    {
    "decoder_name_or_path": "google-bert/bert-base-uncased",
    "need_retokenization": false
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 64
  • num_train_epochs: 10
  • learning_rate: 1e-05
  • warmup_steps: 0.1
  • bf16: True
  • per_device_eval_batch_size: 64
  • seed: 46

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 64
  • num_train_epochs: 10
  • max_steps: -1
  • learning_rate: 1e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 64
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 46
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss fw-variant-dev_cosine_accuracy
-1 -1 - - 0.8845
0.0370 50 10.9495 - -
0.0740 100 8.5518 - -
0.1110 150 7.2139 - -
0.1480 200 6.5304 - -
0.1850 250 5.8942 - -
0.2221 300 5.0836 - -
0.2591 350 4.4827 - -
0.2961 400 4.0965 - -
0.3331 450 3.7756 - -
0.3701 500 3.5197 3.3155 0.8885
0.4071 550 3.2203 - -
0.4441 600 3.0355 - -
0.4811 650 2.8250 - -
0.5181 700 2.6778 - -
0.5551 750 2.4930 - -
0.5922 800 2.3938 - -
0.6292 850 2.2241 - -
0.6662 900 2.1228 - -
0.7032 950 2.0260 - -
0.7402 1000 1.9064 1.8352 0.9680
0.7772 1050 1.8245 - -
0.8142 1100 1.7392 - -
0.8512 1150 1.6638 - -
0.8882 1200 1.6003 - -
0.9252 1250 1.5114 - -
0.9623 1300 1.4978 - -
0.9993 1350 1.4336 - -
1.0363 1400 1.3469 - -
1.0733 1450 1.3045 - -
1.1103 1500 1.2647 1.2372 0.9670
1.1473 1550 1.2286 - -
1.1843 1600 1.2249 - -
1.2213 1650 1.1645 - -
1.2583 1700 1.1292 - -
1.2953 1750 1.1289 - -
1.3323 1800 1.0791 - -
1.3694 1850 1.1114 - -
1.4064 1900 1.0625 - -
1.4434 1950 1.0366 - -
1.4804 2000 1.0352 0.9793 0.9680
1.5174 2050 1.0102 - -
1.5544 2100 0.9745 - -
1.5914 2150 0.9928 - -
1.6284 2200 0.9608 - -
1.6654 2250 0.9365 - -
1.7024 2300 0.9378 - -
1.7395 2350 0.9319 - -
1.7765 2400 0.8982 - -
1.8135 2450 0.8977 - -
1.8505 2500 0.9045 0.8560 0.9635
1.8875 2550 0.9055 - -
1.9245 2600 0.8841 - -
1.9615 2650 0.8696 - -
1.9985 2700 0.8483 - -
2.0355 2750 0.8080 - -
2.0725 2800 0.8050 - -
2.1095 2850 0.7986 - -
2.1466 2900 0.7870 - -
2.1836 2950 0.7727 - -
2.2206 3000 0.7667 0.7722 0.9645
2.2576 3050 0.7718 - -
2.2946 3100 0.7570 - -
2.3316 3150 0.7626 - -
2.3686 3200 0.7846 - -
2.4056 3250 0.7362 - -
2.4426 3300 0.7447 - -
2.4796 3350 0.7596 - -
2.5167 3400 0.7430 - -
2.5537 3450 0.7309 - -
2.5907 3500 0.7371 0.7235 0.9650
2.6277 3550 0.7318 - -
2.6647 3600 0.7213 - -
2.7017 3650 0.7066 - -
2.7387 3700 0.7074 - -
2.7757 3750 0.7072 - -
2.8127 3800 0.6965 - -
2.8497 3850 0.7158 - -
2.8868 3900 0.7055 - -
2.9238 3950 0.7028 - -
2.9608 4000 0.6974 0.6875 0.9660
2.9978 4050 0.6916 - -
3.0348 4100 0.6541 - -
3.0718 4150 0.6413 - -
3.1088 4200 0.6396 - -
3.1458 4250 0.6463 - -
3.1828 4300 0.6338 - -
3.2198 4350 0.6310 - -
3.2568 4400 0.6276 - -
3.2939 4450 0.6183 - -
3.3309 4500 0.6227 0.6574 0.9640
3.3679 4550 0.6204 - -
3.4049 4600 0.6309 - -
3.4419 4650 0.6304 - -
3.4789 4700 0.6149 - -
3.5159 4750 0.6114 - -
3.5529 4800 0.6018 - -
3.5899 4850 0.6260 - -
3.6269 4900 0.6206 - -
3.6640 4950 0.6125 - -
3.7010 5000 0.6178 0.6358 0.9645
3.7380 5050 0.6171 - -
3.7750 5100 0.5940 - -
3.8120 5150 0.6041 - -
3.8490 5200 0.5918 - -
3.8860 5250 0.6052 - -
3.9230 5300 0.5833 - -
3.9600 5350 0.5919 - -
3.9970 5400 0.5953 - -
4.0340 5450 0.5496 - -
4.0711 5500 0.5613 0.6063 0.9650
4.1081 5550 0.5507 - -
4.1451 5600 0.5444 - -
4.1821 5650 0.5467 - -
4.2191 5700 0.5590 - -
4.2561 5750 0.5520 - -
4.2931 5800 0.5634 - -
4.3301 5850 0.5425 - -
4.3671 5900 0.5593 - -
4.4041 5950 0.5281 - -
4.4412 6000 0.5457 0.5993 0.9675
4.4782 6050 0.5426 - -
4.5152 6100 0.5353 - -
4.5522 6150 0.5282 - -
4.5892 6200 0.5553 - -
4.6262 6250 0.5390 - -
4.6632 6300 0.5279 - -
4.7002 6350 0.5438 - -
4.7372 6400 0.5378 - -
4.7742 6450 0.5395 - -
4.8113 6500 0.5429 0.5766 0.9645
4.8483 6550 0.5333 - -
4.8853 6600 0.5329 - -
4.9223 6650 0.5273 - -
4.9593 6700 0.5200 - -
4.9963 6750 0.5304 - -
5.0333 6800 0.4883 - -
5.0703 6850 0.4996 - -
5.1073 6900 0.4942 - -
5.1443 6950 0.4987 - -
5.1813 7000 0.4914 0.5701 0.9655
5.2184 7050 0.4953 - -
5.2554 7100 0.4942 - -
5.2924 7150 0.4792 - -
5.3294 7200 0.4825 - -
5.3664 7250 0.4894 - -
5.4034 7300 0.4950 - -
5.4404 7350 0.4933 - -
5.4774 7400 0.4789 - -
5.5144 7450 0.4901 - -
5.5514 7500 0.4782 0.5646 0.9640
5.5885 7550 0.4881 - -
5.6255 7600 0.4883 - -
5.6625 7650 0.4712 - -
5.6995 7700 0.4925 - -
5.7365 7750 0.4812 - -
5.7735 7800 0.4849 - -
5.8105 7850 0.4895 - -
5.8475 7900 0.4922 - -
5.8845 7950 0.4830 - -
5.9215 8000 0.4814 0.5485 0.9675
5.9585 8050 0.4705 - -
5.9956 8100 0.4926 - -
6.0326 8150 0.4516 - -
6.0696 8200 0.4440 - -
6.1066 8250 0.4471 - -
6.1436 8300 0.4525 - -
6.1806 8350 0.4415 - -
6.2176 8400 0.4440 - -
6.2546 8450 0.4438 - -
6.2916 8500 0.4527 0.5425 0.9655
6.3286 8550 0.4530 - -
6.3657 8600 0.4457 - -
6.4027 8650 0.4347 - -
6.4397 8700 0.4426 - -
6.4767 8750 0.4549 - -
6.5137 8800 0.4475 - -
6.5507 8850 0.4416 - -
6.5877 8900 0.4426 - -
6.6247 8950 0.4496 - -
6.6617 9000 0.4473 0.5347 0.9655
6.6987 9050 0.4571 - -
6.7358 9100 0.4449 - -
6.7728 9150 0.4440 - -
6.8098 9200 0.4438 - -
6.8468 9250 0.4498 - -
6.8838 9300 0.4387 - -
6.9208 9350 0.4459 - -
6.9578 9400 0.4389 - -
6.9948 9450 0.4489 - -
7.0318 9500 0.4162 0.5264 0.9670
7.0688 9550 0.4080 - -
7.1058 9600 0.4086 - -
7.1429 9650 0.4139 - -
7.1799 9700 0.4108 - -
7.2169 9750 0.4104 - -
7.2539 9800 0.4087 - -
7.2909 9850 0.4200 - -
7.3279 9900 0.4085 - -
7.3649 9950 0.3993 - -
7.4019 10000 0.4196 0.5217 0.9665
7.4389 10050 0.4242 - -
7.4759 10100 0.4132 - -
7.5130 10150 0.4064 - -
7.5500 10200 0.4182 - -
7.5870 10250 0.4140 - -
7.6240 10300 0.4266 - -
7.6610 10350 0.4160 - -
7.6980 10400 0.4192 - -
7.7350 10450 0.4140 - -
7.7720 10500 0.4216 0.5198 0.9670
7.8090 10550 0.4114 - -
7.8460 10600 0.4192 - -
7.8830 10650 0.4032 - -
7.9201 10700 0.4100 - -
7.9571 10750 0.4123 - -
7.9941 10800 0.4018 - -
8.0311 10850 0.3920 - -
8.0681 10900 0.3777 - -
8.1051 10950 0.3943 - -
8.1421 11000 0.3795 0.5113 0.9665
8.1791 11050 0.3890 - -
8.2161 11100 0.3941 - -
8.2531 11150 0.3896 - -
8.2902 11200 0.3861 - -
8.3272 11250 0.3837 - -
8.3642 11300 0.3879 - -
8.4012 11350 0.3901 - -
8.4382 11400 0.3815 - -
8.4752 11450 0.3909 - -
8.5122 11500 0.3780 0.5074 0.9670
8.5492 11550 0.3905 - -
8.5862 11600 0.3900 - -
8.6232 11650 0.3854 - -
8.6603 11700 0.3873 - -
8.6973 11750 0.3825 - -
8.7343 11800 0.3884 - -
8.7713 11850 0.3996 - -
8.8083 11900 0.3847 - -
8.8453 11950 0.3811 - -
8.8823 12000 0.3716 0.5050 0.9655
8.9193 12050 0.3829 - -
8.9563 12100 0.3823 - -
8.9933 12150 0.3884 - -
9.0303 12200 0.3662 - -
9.0674 12250 0.3702 - -
9.1044 12300 0.3541 - -
9.1414 12350 0.3674 - -
9.1784 12400 0.3639 - -
9.2154 12450 0.3817 - -
9.2524 12500 0.3649 0.5002 0.9660
9.2894 12550 0.3733 - -
9.3264 12600 0.3744 - -
9.3634 12650 0.3719 - -
9.4004 12700 0.3587 - -
9.4375 12750 0.3685 - -
9.4745 12800 0.3621 - -
9.5115 12850 0.3699 - -
9.5485 12900 0.3674 - -
9.5855 12950 0.3607 - -
9.6225 13000 0.3623 0.4984 0.9665
9.6595 13050 0.3675 - -
9.6965 13100 0.3667 - -
9.7335 13150 0.3612 - -
9.7705 13200 0.3644 - -
9.8075 13250 0.3673 - -
9.8446 13300 0.3653 - -
9.8816 13350 0.3665 - -
9.9186 13400 0.3603 - -
9.9556 13450 0.3536 - -
9.9926 13500 0.3704 0.4973 0.9655
10.0 13510 - 0.4974 0.9660
-1 -1 - - 0.9660

Training Time

  • Training: 54.8 minutes

Framework Versions

  • Python: 3.12.0
  • Sentence Transformers: 6.1.0
  • Transformers: 5.20.1
  • PyTorch: 2.14.1+cu130
  • Accelerate: 1.15.0
  • Datasets: 5.1.0
  • Tokenizers: 0.23.3

Additional Resources

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

DenoisingAutoEncoderLoss 

@inproceedings{wang-2021-TSDAE,
    title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
    author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    pages = "671--688",
    url = "https://arxiv.org/abs/2104.06979",
}
Downloads last month
-
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for SwarajSolanke-turtle/make-model-mapping-TSDAE

Base model

google-bert/bert-base-uncased
Finetuned
(7027)
this model

Papers for SwarajSolanke-turtle/make-model-mapping-TSDAE

Evaluation results