agraharr/telecom-bge-base-matryoshka

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("agraharr/telecom-bge-base-matryoshka")
# Run inference
sentences = [
    'Which aspects of the repeater’s operation does the input intermodulation requirement apply to?',
    'The input intermodulation requirement applies to both the uplink and downlink operations of the NR repeater. Specifically, if the base station (BS) side is confirmed to meet co-location requirements, it must also adhere to the input intermodulation co-location requirements for the downlink. Similarly, if the user equipment (UE) side meets co-location requirements, it must comply with the input intermodulation co-location requirements for the uplink. This dual applicability ensures comprehensive protection against intermodulation interference.<|im_end|>',
    'The NG_RAN_PRN-Core feature enables User Equipment (UE) to acquire NPN-relevant Cell Global Identity (CGI) information from neighboring intra-frequency or inter-frequency NR NPN cells. This is achieved by the UE reading the System Information (SI) of the neighboring cell and subsequently reporting the acquired CGI information back to the network. The feature is defined in the technical specification TS 38.331, which outlines the procedures and requirements for this capability. Notably, this feature is conditionally mandatory; if the UE supports Non-Public Networks (NPN), it must also support the NG_RAN_PRN-Core feature. This capability is crucial for ensuring that the UE can effectively communicate and interact with NPN environments, enhancing overall network connectivity and performance.<|im_end|>',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7323, 0.0072],
#         [0.7323, 1.0000, 0.0721],
#         [0.0072, 0.0721, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: telecom-eval
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9309

Training Details

Training Dataset

Unnamed Dataset

• Size: 41,056 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 7 tokens mean: 22.64 tokens max: 53 tokens	min: 3 tokens mean: 111.16 tokens max: 325 tokens	min: 3 tokens mean: 112.85 tokens max: 512 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  What is the purpose of Wired Equivalent Privacy (WEP) in wireless networks? [IEEE 802.11]	To provide privacy and encryption for data	To terminate the authentication process
  What is the purpose of an optical hybrid in coherent communications?	Combine two optical signals	Convert optical signals to electrical signals
  Which of the following EVPN services is used for point-to-point Layer 2 communication without MAC learning?	3. EVPN E-Line	4. EVPN VLAN-based

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• disable_tqdm: True
• per_device_eval_batch_size: 16
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: True
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	telecom-eval_cosine_accuracy
-1	-1	-	0.8999
0.1949	500	0.5379	-
0.3897	1000	0.3217	-
0.5846	1500	0.3156	-
0.7794	2000	0.2729	-
0.9743	2500	0.2682	-
1.0	2566	-	0.9261
1.1691	3000	0.1888	-
1.3640	3500	0.1704	-
1.5588	4000	0.1695	-
1.7537	4500	0.1743	-
1.9486	5000	0.1710	-
2.0	5132	-	0.9261
2.1434	5500	0.1257	-
2.3383	6000	0.1001	-
2.5331	6500	0.1195	-
2.7280	7000	0.1105	-
2.9228	7500	0.1002	-
3.0	7698	-	0.9309

Training Time

• Training: 2.8 hours
• Evaluation: 45.2 seconds
• Total: 2.8 hours

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.4.1
• Transformers: 5.7.0
• PyTorch: 2.5.1+cu124
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}