TesNik369/bge-m3-tqa-merged-v2

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the tqa-bge-m3-finetune-triplets-verified dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • tqa-bge-m3-finetune-triplets-verified

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TesNik369/bge-m3-tqa-merged-v2")
# Run inference
queries = [
    'Whales evolved from a four-legged ancestor.',
]
documents = [
    'The evolution of whales is a good example of how fossils can help us understand evolution. Scientists have long known that mammals first evolved on land about 200 million years ago. Its been a mystery, however, how whales evolved. Whales are mammals that live completely in the water. Did they evolve from earlier land mammals? Or did they evolve from animals that already lived in the water? Starting in the late 1970s, a growing number of fossils have allowed scientists to piece together the story of whale evolution. The fossils represent ancient, whale-like animals. They show that an ancient land mammal made its way back to the sea more than 50 million years ago. It became the ancestor of modern whales. In doing so, it lost its legs and became adapted to life in the water. In Figure 7.10 you can see an artists rendition of such a whale ancestor. It had legs and could walk on land, but it was also a good swimmer. Watch this short video to learn more about the amazing story of whale evolution based on the fossils:',
    'Humans evolved from primates, and apes and humans have a primate common ancestor. About 7 million years ago, chimpanzees (our closest living relatives) and humans shared their last common ancestor.',
    'The earliest amphibians evolved from a lobe-finned fish ancestor. This occurred about 365 million years ago. Amphibians were the first terrestrial vertebrates. They lived on land as adults, but they had to return to the water to reproduce. The earliest reptiles evolved from an amphibian ancestor. This occurred at least 300 million years ago. Reptiles were the first vertebrates that did not need water to reproduce. Thats because they laid waterproof amniotic eggs. These eggs allowed the embryo inside to breathe without drying out. Mammals and birds both evolved from reptile-like ancestors. The first mammals appeared about 200 million years ago. The earliest birds evolved about 150 million years ago.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7697, 0.6626, 0.6452]])

Evaluation

Metrics

Information Retrieval

• Dataset: tqa-dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9487
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9487
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9487
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9811
cosine_mrr@10	0.9744
cosine_map@100	0.9744

Training Details

Training Dataset

tqa-bge-m3-finetune-triplets-verified

• Dataset: tqa-bge-m3-finetune-triplets-verified
• Size: 725 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  modality	text	text	text	text	text	text	text
  details	min: 5 tokens mean: 16.28 tokens max: 41 tokens	min: 37 tokens mean: 210.79 tokens max: 512 tokens	min: 9 tokens mean: 197.94 tokens max: 512 tokens	min: 27 tokens mean: 189.24 tokens max: 512 tokens	min: 23 tokens mean: 197.05 tokens max: 512 tokens	min: 21 tokens mean: 194.99 tokens max: 512 tokens	min: 25 tokens mean: 193.8 tokens max: 512 tokens

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5
  Types of mutagens include	Mutations have many possible causes. Some mutations occur when a mistake is made during DNA replication or transcription. Other mutations occur because of environmental factors. Anything in the environment that causes a mutation is known as a mutagen. Examples of mutagens are shown in Figure 5.21. They include ultraviolet rays in sunlight, chemicals in cigarette smoke, and certain viruses and bacteria.	There are three types of radioactive decay: alpha, beta, and gamma decay. In all three types, nuclei emit radiation, but the nature of that radiation differs from one type of decay to another. You can watch a video about the three types at this URL: (17:02). MEDIA Click image to the left or use the URL below. URL:	There are several types of radioactive decay, including alpha, beta, and gamma decay. In all three types, nuclei emit radiation, but the nature of the radiation differs. The Table 1.1 shows the radiation emitted in each type of decay. Type Alpha decay Beta decay Gamma decay Radiation Emitted alpha particle (2 protons and 2 neutrons) + energy beta particle (1 electron or 1 positron) + energy energy (gamma ray)	There are several types of pathogens that cause diseases in human beings. They include bacteria, viruses, fungi, and protozoa. The different types are described in Table 21.1. The table also lists several diseases caused by each type of pathogen. Many infectious diseases caused by these pathogens can be cured with medicines. For example, antibiotic drugs can cure most diseases caused by bacteria.	There are three types of mechanical waves. They differ in how they travel through a medium. The three types are transverse, longitudinal, and surface waves. All three types are described in detail below.	Most cancers are caused by mutations. Mutations are random errors in genes. Mutations that lead to cancer usually occur in genes that control the cell cycle. Because of the mutations, abnormal cells are allowed to divide. Some mutations that lead to cancer may be inherited. However, most of the mutations are caused by environmental factors. Anything in the environment that can cause cancer is called a carcinogen. Common carcinogens include certain chemicals and some types of radiation. Many different chemicals can cause cancer. For example, tobacco contains dozens of chemicals, including nicotine, that have been shown to cause cancer. Figure 21.5 shows some of these chemicals. Smoking tobacco or using smokeless tobacco increases the risk of cancer of the lung, mouth, throat, and urinary bladder. Types of radiation that cause cancer include ultraviolet (UV) radiation and radon. UV radiation is part of sunlight. It is the leading cause of skin cancer. Radon is a naturally occurring radio...
  lakes are not like the ocean because they are made of fresh water and they do not have tides or currents.	Lakes are larger bodies of water. Lakes are usually fresh water, although the Great Salt Lake in Utah is just one exception. Water usually drains out of a lake through a river or a stream and all lakes lose water to evaporation. Lakes form in a variety of different ways: in depressions carved by glaciers, in calderas (Figure 1.1), and along tectonic faults, to name a few. Subglacial lakes are even found below a frozen ice cap. As a result of geologic history and the arrangement of land masses, most lakes are in the Northern Hemisphere. In fact, more than 60% of all the worlds lakes are in Canada most of these lakes were formed by the glaciers that covered most of Canada in the last Ice Age (Figure 1.2). Lakes are not permanent features of a landscape. Some come and go with the seasons, as water levels rise and fall. Over a longer time, lakes disappear when they fill with sediments, if the springs or streams that fill them diminish, (a) Crater Lake in Oregon is in a volcanic caldera. L...	After a heavy rain, you may find puddles of water standing in low spots. The same principle explains why water collects in ponds and lakes. Water travels downhill, so a depression in the ground fills with standing water. A pond is a small body of standing water. A lake is a large body of standing water. Most lakes have freshwater, but a few are salty. The Great Salt Lake in Utah is an example of a saltwater lake. The water in a large lake may be so deep that sunlight cannot penetrate all the way to the bottom. Without sunlight, water plants and algae cannot live on the bottom of the lake. Thats because plants need sunlight for photosynthesis. The largest lakes in the world are the Great Lakes. They lie between the U.S. and Canada, as shown in Figure 13.7. How great are they? They hold 22 percent of all the worlds fresh surface water!	Ponds are small bodies of fresh water that usually have no outlet; ponds are often are fed by underground springs. Like lakes, ponds are bordered by hills or low rises so the water is blocked from flowing directly downhill.	Freshwater biomes are defined by their low salt concentration, usually less than 1%. Plants and animals in freshwater regions are adjusted to the low salt content and would not be able to survive in areas of high salt concentration, such as the ocean. There are different types of freshwater biomes: ponds and lakes ( Figure 1.2), streams and rivers, and wetlands. Ponds and lakes range in size from just a few square meters to thousands of square kilometers. Streams and rivers are bodies of flowing water moving in one direction. They can be found everywhere. They get their starts at headwaters, which may be springs, melting snow, or even lakes, and then travel all the way to their mouths, emptying into another water channel or the ocean. Wetlands are areas of standing water that support aquatic plants. Wetlands include marshes, swamps, and bogs. Lake Tahoe in Northern California is a freshwater biome.	Coastal areas have a milder climate than inland areas. They are warmer in the winter and cooler in the summer. Thats because land near an ocean is influenced by the temperature of the oceans. The temperature of ocean water is moderate and stable. Why? There are two major reasons: 1. Water is much slower to warm up and cool down than land. As a result, oceans never get as hot or as cold as land. 2. Water flows through all the worlds oceans. Warm water from the equator mixes with cold water from the poles. The mixing of warm and cold water makes the water temperature moderate. Even inland temperatures are milder because of oceans. Without oceans, there would be much bigger temperature swings all over Earth. Temperatures might plunge hundreds of degrees below freezing in the winter. In the summer, lakes and seas might boil! Life as we know it could not exist on Earth without the oceans.	The depression that allows water to collect to form a lake may come about in a variety of ways. The Great Lakes, for example, are glacial lakes. A glacial lake forms when a glacier scrapes a large hole in the ground. When the glacier melts, the water fills the hole and forms a lake. Over time, water enters the lake from the sources mentioned above as well. Other lakes are crater lakes or rift lakes, which are pictured in Figure 13.8. Crater lakes form when volcanic eruptions create craters that fill with water. Rift lakes form when movements of tectonic plates create low places that fill with water.
  Which type of tissue gives bones their strength?	Bones are organs. Like other organs, they are made up of more than one kind of tissue. There are four different kinds of tissues in bones, as shown in Figure 16.11. From the outside of the bone to the center, the tissues are periosteum, compact bone, spongy bone, and bone marrow. Periosteum is a tough, fibrous membrane that covers and protects the outer surfaces of bone. Compact bone lies below periosteum. It is very dense and hard. Compact bone gives bones their strength. Spongy bone lies below compact bone. It is less dense than compact bone. Spongy bone contains many tiny holes, or pores, which provide spaces for blood vessels and bone marrow. Bone marrow is a soft connective tissue inside pores and cavities in spongy bone. Bone marrow makes blood cells.	You can help keep your bones and skeletal system healthy by eating well and getting enough exercise. Weight- bearing exercises help keep bones strong. Weight-bearing exercises and activities work against gravity. Such activities include basketball, tennis, gymnastics, karate, running, and walking. When the body is exercised regularly by performing weight-bearing activity, bones respond by adding more bone cells to increase their bone density.	Bones are the main organs of the skeletal system. In adults, the skeleton consists of a whopping 206 bones, many of them in the hands and feet. You can see many of the bones of the human skeleton in Figure 16.10. The skeletal system also includes cartilage and ligaments. Cartilage is a tough, flexible connective tissue that contains the protein collagen. It covers the ends of bones where they meet. The gray tissue in Figure 16.10 is cartilage. A ligament is a band of fibrous connective tissue. Ligaments connect bones of the skeleton and hold them together.	Some people think bones are like chalk: dead, dry, and brittle. In reality, bones are very much alive. They consist of living tissues and are supplied with blood and nerves.	Along with the skin, the integumentary system includes the nails and hair. Both the nails and hair contain the tough protein, keratin. The keratin forms fibers, which makes your nails and hair tough and strong. Keratin is similar in toughness to chitin, the carbohydrate found in the exoskeleton of arthropods.	Specialized cells are organized into tissues. A tissue is a group of specialized cells of the same kind that perform the same function. There are four basic types of human tissues: connective, epithelial, muscle, and nervous tissues. The four types are shown in Figure 16.2. Connective tissue consists of cells that form the bodys structure. Examples include bone and cartilage, which protect and support the body. Blood is also a connective tissue. It circulates and connects cells throughout the body. Epithelial tissue consists of cells that cover inner and outer body surfaces. Examples include skin and the linings of internal organs. Epithelial tissue protects the body and its internal organs. It also secretes substances such as hormones and absorbs substances such as nutrients. Muscle tissue consists of cells that can contract, or shorten. Examples include skeletal muscle, which is attached to bones and makes them move. Other types of muscle include cardiac muscle, which makes the heart...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 2
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• seed: 3407
• fp16: True
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 3407
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	tqa-dev_cosine_ndcg@10
0.1739	1	0.8825	-
1.0	6	-	0.9811
2.0	12	-	0.9811
-1	-1	-	0.9811

Training Time

• Training: 15.3 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}