Hugging Face's logo

KatjaK
/
GND-few-shot-retriever

Sentence Similarity
sentence-transformers
Safetensors
xlm-roberta
feature-extraction
dense
Generated from Trainer
dataset_size:5139569
loss:MultipleNegativesRankingLoss
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-m3
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 1024 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Umweltgutachten',
    'Zentralblatt der Bauverwaltung Nachrichten d. Reichs- u. Staatsbehörden',
    'Materialfluss Materialfluss <Landsberg> / Portrait einer Branche',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3120, 0.2367],
#         [0.3120, 1.0000, 0.2667],
#         [0.2367, 0.2667, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

  • Size: 5,139,569 training samples
  • Columns: anchor and positive
  • Approximate statistics based on the first 1000 samples:
    anchor positive
    type string string
    details
    • min: 3 tokens
    • mean: 20.6 tokens
    • max: 101 tokens
    • min: 3 tokens
    • mean: 20.99 tokens
    • max: 100 tokens
  • Samples:
    anchor positive
    The art of Star Wars - das Erwachen der Macht Harry Potter: magische Orte aus den Filmen
    Fachdidaktik Kunst und Design Lehren und Lernen mit Portfolios Mit Kindern moderne Kunst entdecken kreative Ideen auch für Fachfremde, 2. - 4. Schuljahr ; [mit Farbabbildungen und Kopiervorlagen auf CD-ROM]
    Der Ditz alles was man über Niederländer wissen sollte Umgangsformen Protokoll und Etikette, privat und im Beruf
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Evaluation Dataset

Unnamed Dataset

  • Size: 3,736 evaluation samples
  • Columns: anchor and positive
  • Approximate statistics based on the first 1000 samples:
    anchor positive
    type string string
    details
    • min: 3 tokens
    • mean: 19.09 tokens
    • max: 96 tokens
    • min: 3 tokens
    • mean: 18.28 tokens
    • max: 96 tokens
  • Samples:
    anchor positive
    Bibliotheksstatistik staatliche Allgemeinbibliotheken und Gewerkschaftsbibliotheken der Deutschen Demokratischen Republik in Zahlen ; Bericht und Tabellen zu den Gesamtergebnissen in der Republik und in den Bezirken einschließlich der Wissenschaftlichen Allgemeinbibliotheken der Bezirke Fortschrittsbericht Bohrtechnik, Erdöl und Erdgasgewinnung und verwandte Gebiete
    Bibliotheksstatistik staatliche Allgemeinbibliotheken und Gewerkschaftsbibliotheken der Deutschen Demokratischen Republik in Zahlen ; Bericht und Tabellen zu den Gesamtergebnissen in der Republik und in den Bezirken einschließlich der Wissenschaftlichen Allgemeinbibliotheken der Bezirke Zentralkatalog der DDR ZKZ ; Zeitschriften u. Serien d. Auslandes ZKZ
    Zentralkatalog der DDR ZKZ ; Zeitschriften u. Serien d. Auslandes ZKZ Fortschrittsbericht Bohrtechnik, Erdöl und Erdgasgewinnung und verwandte Gebiete
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 64
  • per_device_eval_batch_size: 64
  • learning_rate: 1e-05
  • num_train_epochs: 2

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 64
  • per_device_eval_batch_size: 64
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 1e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 2
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • tp_size: 0
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss
0.0062 500 2.1697 -
0.0125 1000 2.0869 2.2847
0.0187 1500 2.0643 -
0.0249 2000 2.0358 2.2490
0.0311 2500 2.0275 -
0.0374 3000 2.0191 2.2716
0.0436 3500 2.0037 -
0.0498 4000 1.9998 2.2407
0.0560 4500 1.976 -
0.0623 5000 1.9776 2.2088
0.0685 5500 1.9705 -
0.0747 6000 1.9796 2.2011
0.0809 6500 1.952 -
0.0872 7000 1.9462 2.2119
0.0934 7500 1.9466 -
0.0996 8000 1.9162 2.1985
0.1058 8500 1.9171 -
0.1121 9000 1.9207 2.1956
0.1183 9500 1.9041 -
0.1245 10000 1.908 2.2035
0.1307 10500 1.9105 -
0.1370 11000 1.8817 2.1951
0.1432 11500 1.9109 -
0.1494 12000 1.9033 2.1413
0.1557 12500 1.8991 -
0.1619 13000 1.8875 2.1770
0.1681 13500 1.8777 -
0.1743 14000 1.8822 2.1892
0.1806 14500 1.8696 -
0.1868 15000 1.8628 2.1856
0.1930 15500 1.8828 -
0.1992 16000 1.8597 2.1755
0.2055 16500 1.8762 -
0.2117 17000 1.8724 2.1548
0.2179 17500 1.8685 -
0.2241 18000 1.8733 2.1681
0.2304 18500 1.852 -
0.2366 19000 1.8412 2.1868
0.2428 19500 1.859 -
0.2490 20000 1.8433 2.1439
0.2553 20500 1.852 -
0.2615 21000 1.8537 2.1446
0.2677 21500 1.803 -
0.2740 22000 1.8233 2.1722
0.2802 22500 1.8294 -
0.2864 23000 1.8297 2.1857
0.2926 23500 1.8306 -
0.2989 24000 1.8336 2.1519
0.3051 24500 1.8203 -
0.3113 25000 1.8247 2.1728
0.3175 25500 1.8077 -
0.3238 26000 1.7915 2.1985
0.3300 26500 1.8087 -
0.3362 27000 1.8128 2.1808
0.3424 27500 1.8139 -
0.3487 28000 1.8118 2.1518
0.3549 28500 1.8113 -
0.3611 29000 1.8077 2.1721
0.3673 29500 1.8193 -
0.3736 30000 1.7986 2.1783
0.3798 30500 1.8002 -
0.3860 31000 1.807 2.1628
0.3922 31500 1.7917 -
0.3985 32000 1.7757 2.1827
0.4047 32500 1.8038 -
0.4109 33000 1.7838 2.1703
0.4172 33500 1.7748 -
0.4234 34000 1.8039 2.1684
0.4296 34500 1.7674 -
0.4358 35000 1.7597 2.1902
0.4421 35500 1.7828 -
0.4483 36000 1.7747 2.1980
0.4545 36500 1.776 -
0.4607 37000 1.8078 2.1805
0.4670 37500 1.7814 -
0.4732 38000 1.7741 2.1616
0.4794 38500 1.7755 -
0.4856 39000 1.7576 2.1644
0.4919 39500 1.7562 -
0.4981 40000 1.7602 2.1709
0.5043 40500 1.757 -
0.5105 41000 1.7693 2.1938
0.5168 41500 1.7655 -
0.5230 42000 1.7617 2.1970
0.5292 42500 1.7687 -
0.5355 43000 1.7565 2.1630
0.5417 43500 1.7492 -
0.5479 44000 1.7579 2.1694
0.5541 44500 1.7629 -
0.5604 45000 1.7364 2.1836
0.5666 45500 1.7444 -
0.5728 46000 1.7308 2.1785
0.5790 46500 1.7351 -
0.5853 47000 1.7588 2.1523
0.5915 47500 1.7235 -
0.5977 48000 1.7317 2.1694
0.6039 48500 1.7415 -
0.6102 49000 1.7588 2.1867
0.6164 49500 1.7103 -
0.6226 50000 1.7269 2.1812
0.6288 50500 1.7254 -
0.6351 51000 1.7398 2.1663
0.6413 51500 1.7178 -
0.6475 52000 1.7454 2.1856
0.6537 52500 1.7188 -
0.6600 53000 1.7204 2.1688
0.6662 53500 1.72 -
0.6724 54000 1.7356 2.1861
0.6787 54500 1.7338 -
0.6849 55000 1.7254 2.1700
0.6911 55500 1.7243 -
0.6973 56000 1.7308 2.1781
0.7036 56500 1.7215 -
0.7098 57000 1.7115 2.1784
0.7160 57500 1.7071 -
0.7222 58000 1.7155 2.1884
0.7285 58500 1.7236 -
0.7347 59000 1.7078 2.1825
0.7409 59500 1.7056 -
0.7471 60000 1.724 2.1695
0.7534 60500 1.7077 -
0.7596 61000 1.6948 2.1562
0.7658 61500 1.6858 -
0.7720 62000 1.7207 2.1835
0.7783 62500 1.7086 -
0.7845 63000 1.7173 2.1645
0.7907 63500 1.717 -
0.7970 64000 1.7032 2.1732
0.8032 64500 1.6992 -
0.8094 65000 1.7061 2.1841
0.8156 65500 1.6926 -
0.8219 66000 1.6881 2.1970
0.8281 66500 1.6905 -
0.8343 67000 1.6924 2.1774
0.8405 67500 1.6926 -
0.8468 68000 1.6888 2.1826
0.8530 68500 1.6975 -
0.8592 69000 1.6884 2.1740
0.8654 69500 1.6843 -
0.8717 70000 1.6766 2.1717
0.8779 70500 1.6887 -
0.8841 71000 1.6827 2.1788
0.8903 71500 1.6922 -
0.8966 72000 1.6796 2.2068
0.9028 72500 1.6682 -
0.9090 73000 1.6731 2.1866
0.9152 73500 1.6747 -
0.9215 74000 1.6751 2.1737
0.9277 74500 1.6545 -
0.9339 75000 1.6828 2.1894
0.9402 75500 1.6649 -
0.9464 76000 1.6692 2.1638
0.9526 76500 1.6596 -
0.9588 77000 1.6649 2.1874
0.9651 77500 1.6731 -
0.9713 78000 1.6845 2.1805
0.9775 78500 1.69 -
0.9837 79000 1.6622 2.1604
0.9900 79500 1.6581 -
0.9962 80000 1.6763 2.1873
1.0024 80500 1.6351 -
1.0086 81000 1.5969 2.2085
1.0149 81500 1.598 -
1.0211 82000 1.5877 2.2237
1.0273 82500 1.6039 -
1.0335 83000 1.6065 2.2369
1.0398 83500 1.5922 -
1.0460 84000 1.5896 2.2240
1.0522 84500 1.5905 -
1.0585 85000 1.5725 2.2057
1.0647 85500 1.5865 -
1.0709 86000 1.5912 2.2256
1.0771 86500 1.5947 -
1.0834 87000 1.5734 2.2342
1.0896 87500 1.5942 -
1.0958 88000 1.5872 2.2254
1.1020 88500 1.5838 -
1.1083 89000 1.5681 2.2428
1.1145 89500 1.5838 -
1.1207 90000 1.5668 2.2431
1.1269 90500 1.5891 -
1.1332 91000 1.6029 2.2416
1.1394 91500 1.579 -
1.1456 92000 1.5797 2.2316
1.1518 92500 1.5944 -
1.1581 93000 1.581 2.2542
1.1643 93500 1.5601 -
1.1705 94000 1.5653 2.2278
1.1767 94500 1.562 -
1.1830 95000 1.5985 2.2356
1.1892 95500 1.5599 -
1.1954 96000 1.5738 2.2325
1.2017 96500 1.5809 -
1.2079 97000 1.5816 2.2242
1.2141 97500 1.5852 -
1.2203 98000 1.5611 2.2319
1.2266 98500 1.567 -
1.2328 99000 1.5615 2.2454
1.2390 99500 1.5696 -
1.2452 100000 1.5812 2.2469
1.2515 100500 1.566 -
1.2577 101000 1.5751 2.2255
1.2639 101500 1.5864 -
1.2701 102000 1.5796 2.2340
1.2764 102500 1.5646 -
1.2826 103000 1.573 2.2555
1.2888 103500 1.5824 -
1.2950 104000 1.5531 2.2520
1.3013 104500 1.5672 -
1.3075 105000 1.5619 2.2355
1.3137 105500 1.576 -
1.3200 106000 1.5767 2.2266
1.3262 106500 1.563 -
1.3324 107000 1.5627 2.2367
1.3386 107500 1.5589 -
1.3449 108000 1.5512 2.2533
1.3511 108500 1.5725 -
1.3573 109000 1.5454 2.2460
1.3635 109500 1.5609 -
1.3698 110000 1.549 2.2515
1.3760 110500 1.5587 -
1.3822 111000 1.5839 2.2551
1.3884 111500 1.5547 -
1.3947 112000 1.5433 2.2389
1.4009 112500 1.558 -
1.4071 113000 1.5542 2.2350
1.4133 113500 1.5641 -
1.4196 114000 1.5567 2.2444
1.4258 114500 1.5537 -
1.4320 115000 1.5537 2.2395
1.4382 115500 1.5501 -
1.4445 116000 1.5488 2.2517
1.4507 116500 1.5518 -
1.4569 117000 1.5631 2.2351
1.4632 117500 1.5626 -
1.4694 118000 1.5568 2.2289
1.4756 118500 1.5591 -
1.4818 119000 1.5448 2.2264
1.4881 119500 1.5463 -
1.4943 120000 1.5345 2.2319
1.5005 120500 1.5645 -
1.5067 121000 1.5457 2.2289
1.5130 121500 1.5509 -
1.5192 122000 1.5562 2.2302
1.5254 122500 1.5469 -
1.5316 123000 1.5514 2.2322
1.5379 123500 1.5686 -
1.5441 124000 1.5437 2.2453
1.5503 124500 1.5304 -
1.5565 125000 1.5609 2.2427
1.5628 125500 1.5416 -
1.5690 126000 1.5418 2.2385
1.5752 126500 1.5458 -
1.5815 127000 1.5735 2.2241
1.5877 127500 1.5601 -
1.5939 128000 1.546 2.2267
1.6001 128500 1.5419 -
1.6064 129000 1.5579 2.2396
1.6126 129500 1.5383 -
1.6188 130000 1.5451 2.2371
1.6250 130500 1.5505 -
1.6313 131000 1.5374 2.2264
1.6375 131500 1.5357 -
1.6437 132000 1.5223 2.2416
1.6499 132500 1.5312 -
1.6562 133000 1.5438 2.2300
1.6624 133500 1.5366 -
1.6686 134000 1.5354 2.2329
1.6748 134500 1.5316 -
1.6811 135000 1.5452 2.2388
1.6873 135500 1.5548 -
1.6935 136000 1.5448 2.2342
1.6997 136500 1.5281 -
1.7060 137000 1.529 2.2372
1.7122 137500 1.5254 -
1.7184 138000 1.5163 2.2371
1.7247 138500 1.537 -
1.7309 139000 1.5531 2.2455
1.7371 139500 1.5269 -
1.7433 140000 1.5299 2.2372
1.7496 140500 1.5331 -
1.7558 141000 1.5494 2.2293
1.7620 141500 1.5337 -
1.7682 142000 1.5268 2.2365
1.7745 142500 1.5331 -
1.7807 143000 1.5314 2.2318
1.7869 143500 1.5375 -
1.7931 144000 1.5375 2.2257
1.7994 144500 1.5386 -
1.8056 145000 1.5314 2.2387
1.8118 145500 1.5446 -
1.8180 146000 1.5257 2.2421
1.8243 146500 1.5275 -
1.8305 147000 1.5329 2.2389
1.8367 147500 1.5362 -
1.8430 148000 1.5556 2.2322
1.8492 148500 1.5214 -
1.8554 149000 1.5178 2.2337
1.8616 149500 1.5156 -
1.8679 150000 1.5244 2.2318
1.8741 150500 1.5283 -
1.8803 151000 1.5386 2.2400
1.8865 151500 1.5326 -
1.8928 152000 1.5339 2.2408
1.8990 152500 1.5201 -
1.9052 153000 1.5297 2.2402
1.9114 153500 1.5325 -
1.9177 154000 1.5503 2.2428
1.9239 154500 1.5382 -
1.9301 155000 1.534 2.2397
1.9363 155500 1.5245 -
1.9426 156000 1.5198 2.2380
1.9488 156500 1.5301 -
1.9550 157000 1.5324 2.2399
1.9612 157500 1.5171 -
1.9675 158000 1.5298 2.2399
1.9737 158500 1.5309 -
1.9799 159000 1.5053 2.2406
1.9862 159500 1.5237 -
1.9924 160000 1.5228 2.2414
1.9986 160500 1.539 -

Framework Versions

  • Python: 3.9.18
  • Sentence Transformers: 5.1.0
  • Transformers: 4.51.3
  • PyTorch: 2.6.0+cu124
  • Accelerate: 1.10.1
  • Datasets: 4.0.0
  • Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
15
Safetensors
Model size
0.6B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for KatjaK/GND-few-shot-retriever

Base model

BAAI/bge-m3
Finetuned
(568)
this model

Papers for KatjaK/GND-few-shot-retriever