matt-wisdom/KEmbed-naija-v2

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Zahra Buhari: ”I no fit speak Hausa well”',
    'Zahra Buhari attend BBC Hausa literary contest event\n\nDi event wey happun on Friday na to showcase di winners of one literary contest wey tori pipo for BBC Hausa put together.\n\nFor di event, she tok say she and her siblings no dey too use Hausa tok. \n\n"Everyday my papa go correct my hausa wen I dey tok with am", she tok for hausa as she dey laugh.\n\nPresident Muhammadu Buhari wey be Zahra papa na ogbonge Hausa man wey come from Daura for Katsina State and her mama, Aisha Buhari na Fulani from Adamawa state. She marry businessman Ahmed Indimi for ceremony wey be say na Hausa dem use tok for di ceremony. \n\nShe say, she and her family dey tok well-well for "Engausa", di language wey be mix of English and Hausa. \n\nShe add say wetin make am no sabi speak her language well na sake of say she go school wey be say na oyinbo pipo full dia.\n\nTori pipo for Premium Times wey dey di event report am say "She bin tell tori pipo for backstage say most times wen dem dey tok for house na her mama language wey be Fulfude dem dey use."',
    "Mgbe BBC biarutere na ya bụ ebe ọdachị mere, anyị chọpụtara na ya bụ nwaanyị ụmụ aka ya atọ nwụrụ bụ Oriakụ Blessing onye Nkerefi n'Enugwu steet mana dị ya bụ onye Ohaozara nke dị n'Ebonyi steeti.\n\nOtu enyi nwaanyị ahụ bụ Nurse Janet Kalu kọwara na afọ 2018 ka Blessing nwere nsogbu akwụghị ụgwọ ụlọ n'Ugwuagba ahụ, nke mere na onye ụlọ chụpụrụ ya na ụmụ ya atọ ahụ iro.\n\nN'ihi enweghị ego ọ ga eji kwụọ ụgwọ ụlọ ọzọ, ya bụ nwaanyị onye ya na di ya ebikọghị ọnụ duziri ụmụ ya ga biri na ụlọ igwe a na-akpọ container ebe ọ na akwa akwa.\n\nOnye welder nọ na-azụ shọp nwaanyị ahụ bụ Ifeanyi Nakwa nwaanyị na ere nrị na mmanya na akụkụ ya kwuru na otela ha bidoro dọwa Blessing aka n'ntị ka oghara ịdị na-akpọchịnye ụmụaka  ya n'ime ụlọ igwe container were pụọ.\n\nHa kọwara na ụbọchị ọkụ ahụ gbara, na Blessing kpọchinyere ụmụ ya atọ bụ Esther, Chiemerie na nke atọ n'ime ụlọ, sinye nrị n'ọkụ were gaba igota ihe n'ahịa mgbuka.\n\nJanet Kalu bụ ọyị nne ụmụaka ahụ bụ Blessing.\n\nIfeanyi Kwara arịrị na oge okụ ahụ malitere, ha enweghị ike itika ya bụ igwe na nnukwu igodo ojiri gbachie ụlọ ganye na okụ agbagbuo ụmụ aka atọ ahụ.\n\nOtu onye na ndị na elekọta ogbe Ugwuagba Obosi ahụ Mazi Amechi Egwuonwu kọwara ka ya siri gota out tanker mmiri were na aagbanwụ ya bụ ọkụ mana tupu akụkaa ụzọ nwaanyị ahụ gbachiri, ụmụ ya atọ erela ka unyi.\n\nỌkụ ọgbụgba anapụla nwaanyị gbara afọ 92 ụlọ ya n'Enugwu\n\nỌtụtụ ka nọ na anya mmiri yọrọ ndị mmadụ ka ha kwụsị idi na akpọchinye ụmụ ha n'ụlọ ma ọ bụ n'ime ụgbọala ka ha were gbanahụ ụdị mberede a.\n\nAkụkọ ndị ga-amasị gị:",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8242, 0.5078],
#         [0.8242, 0.9961, 0.5273],
#         [0.5078, 0.5273, 1.0000]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

• Dataset: nigerian-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.81
cosine_accuracy@3	0.907
cosine_accuracy@5	0.931
cosine_accuracy@10	0.945
cosine_precision@1	0.81
cosine_precision@3	0.3023
cosine_precision@5	0.1862
cosine_precision@10	0.0945
cosine_recall@1	0.81
cosine_recall@3	0.907
cosine_recall@5	0.931
cosine_recall@10	0.945
cosine_ndcg@10	0.8824
cosine_mrr@10	0.8618
cosine_map@100	0.8633

Training Details

Training Dataset

Unnamed Dataset

• Size: 26,833 training samples

• Columns: anchor, positive, and negative

• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 25.97 tokens max: 73 tokens	min: 24 tokens mean: 239.76 tokens max: 256 tokens	min: 41 tokens mean: 239.6 tokens max: 256 tokens

• Samples:

  anchor	positive	negative
  Zaɓen Ghana: Nana Akufo-Addo ya lashe zaɓen ƙasar	Shugaba Akufo-Addo yana fatan yin nasara karo na biyu Shugaba Nana Akufo Addo ya yi nasara da kashi 50.8 cikin 100 a cewar shugabar hukumar zaɓen Jean Mensa. Alƙaluman sun nuna cewa Nana Akufo-Addo na jam'iyyar NPP ya samu ƙuri'a 6,730,413. Shi kuma John Dramani Mahama na jam'iyyar NDC ya samu ƙuri'a 6,214,889 wato kashi 47.36 cikin 100. Fiye da mutum miliyan 17 ne suka kaɗa kuri'unsu a zaben. Zaɓen shi ne na farko da Ghana ta yi da na'ura, wani abu da ƴan ƙasar za su yi alfahari da shi. Akufo-Addo zai ci gaba da shugabancin ƙsar a karo na biyu. Ya zuwa ranar Laraba da yamma, an bayyana sakamakon zaɓen larduna 15 kamar haka:	Da farko bulon na warin sinadarin ammonia, amma ya kan daina wari bayan kwana biyu

  Daliban sun hada fitsari da yashi da kwayar halitta ta bakteriya wajen hada bulon, idan aka adana su a daki mai madaidaicin yanayi.

  "Ana hada wannan bulon ne kamar yadda ake hadadutsen cikin teku," inji Dyllon Randall, wanda shi ne malamin da ke sa ido a kan aikin binciken da daliban a jami'ar Cape Town.

  Wasu bulullukan kamar jan bulo na bukatar zafi domin a gasa su, lamarin da kan haifar da dumamar yanayi saboda yawan iskar carbon dioxide da ake fitarwa.

  'Karfinsa kamar dutse'

  Daliban masu nazarin kimiyyar hade-hade a jami'ar ta Cape Town (UCT) sun rika samun fitsarin da suke amfani da shi ne wajen tara fitsari a bayin maza.

  Bulon na daukar kimanin kwana hudu zuwa shida kafin su sami karfi da inganci

  A wajen hada bulon, da farko a kan sami takin zamani ne, kamin daga baya a yi amfani da sauran fitsarin wajen samar da abin da jami'ar ke kira "bulo da aka samu daga kwayoyin halitta".

  Shin fitsari ... | | COVID-19 force Carlos Ahenkorah to resign as Ghana Deputy Trade and Industry Minister | Why Ghana minister resign by force

  Carlos Kingsley Ahenkorah resign fordisobeying coronavirus self-isolation measures after testing positive for di virus, President Nana Akufo-Addo tok for statement on Friday.

  Oga Ahenkorah no get any choice but to resign afta im go visit one voter registration centre upon say e don test positive for covid 19.

  Voter registration exercise bin dey go on for Ghana to compile new electoral roll before December elections.

  Many pipo for Ghana don condemn di action of di Minister.

  Ahenkorah wey also be Member of parliament, confam say dem bin admit am for isolation centre of Korle-Bu Teaching hospital.

  E tell local media say dem advise am to self-Isolate afta test result show say e dey positive for di disease but e come decide to visit one registration centre for im constituency inside Ghana industrial city of Tema, because e wan settle one problem for dia.

  E claim say im obey di social distancing rules and im no dey for di midst of pipo.

  As oga ... | All over di world, coronavirus don make hand sanitizer difficult to see to buy

  Ghana goment impose ban on all public gatherings like conferences, funerals, festivals, political rallies, Church services, Islamic worship for 4 weeks starting Monday, March 16 sake of de outbreak.

  President Nana Akufo-Addo plus health officials for de country direct citizens, supermarkets, shopping malls, restaurants den stuff say dem for observe enhanced hygiene, ensure regular use of hand sanitizers and running water plus soap for washing of hands.

  Coronavirus: See how to make your own hand sanitizer

  After dis announcement, price of sanitisers shoot up across de country.

  One consumer, Abigail Lamptey talk BBC Pidgin say "I go ask for small hand sanitiser wey dem dey sell at Ghc3, but now dem dey sell am Ghc15. I ask dem say why, dem say sake of coronavirus"

  Another consumer, Sarah talk reveal say "if you go Kaneshie and Makola markets de Ghc2 sanitisers dey cost Ghc10 now."

  Per calculations wey B... | | Alex Iwobi: Iwobi fọwọ́ sí àdéhùn tuntun pẹ̀lú Arsenal | Alex Iwobi darapọ̀ mọ́ ikọ̀ Arsenal nígbà tí ó wà ní ọmọ ọdún mẹ́san

Iwobi tọwọ bọ iwe adehun ti yoo pari ni ọdun 2023 pẹlu ẹgbẹ agbabọọlu naa.

End of Twitter post, 1

Iwobi to jẹ ọmọ ọdun mejilelogun ti gbayo mẹsan wọle fun Arsenal ninu ifẹsẹwọnsẹ 98 to ko pa ninu rẹ lati ọdun 2015 to ti bẹrẹ si ṣoju ikọ naa.

Àwọn iròyin tì ẹ leè ní ìfẹ̀ síí:

Lati ọmọ ọdun mẹsan ni Iwobi ti darapọ mọ Arsenal, o ṣoju ikọ naa fun igba akọkọ nigba ti o pe ọmọ ọdun mọkandinlogun.

Iwobi dupẹ lọwọ akọnimọọgba tuntun ikọ naa Unai Emery fun igbagbọ rẹ ninu oun, bẹẹni ko gbagbe awọn ololufẹ Arsenal fun atilẹyin wọn. | Alukoro ọlọpaa Gambo Isah tofi ọrọ yi sita ni ohun ko mọ ibi ti awọn akọroyin kan ti ri iroyin pe ohun sọ pe ọwọ ti tẹ awọn afurasi naa.

  O ni ohun ti atẹjade ti oun fi sita sọ ni pe awọn kọlu awọn ajinigbe naa ni.

  "Mi o fi igba kankan sọ fun ẹnikẹni pe a ti mu awọn ajinigbe naa .O jọ pe ọrọ ti a fi sita ni ko ye wọn nitori a ni a da wọn lna nii kii ṣe wi pe a tiu ri wọn mu.''

  Dida lọna ti Isah loun n sọ ni eleyi to ni ṣe pẹlu ikọlu to waye laarin awọn ọlọpaa ati awọn to ji olori ilu Daura naa gbe.O ni awọn ajinigbe naa papa bọ mọ awọn lọwọ.

  Ni irọlẹ Ọjọru ni awọn afurasi ajinigbe naa gbe Umar Isah ni ile rẹ to wa ni Daura ni ipinlẹ Katsina ni nnkan bi ago meje aṣalẹ.

  Oga Isah salaye pe kete ti awọn ọlọpaa gburo ijinigbe naa ni wọn tọ awọn ajinigbe naa lọ titi ti wọn fi de ikorita Kwosanda nibi ti wọn ti doju ija ko wọn.

  Lori boya awọn ajinigbe naa ti bere owo lati fi tu olori naa silẹ, Isah ni ohun ko le fesi si ibeere naa .

  Amọ ṣa ileeṣẹ ọlọpaa ni Katsina sọ fun BBC pe owo ti... |

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "GISTEmbedLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 3,813 evaluation samples

• Columns: anchor and positive

• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 23.81 tokens max: 69 tokens	min: 19 tokens mean: 225.68 tokens max: 256 tokens

• Samples:

  anchor	positive
  What is the capital of Turkey?	Ankara ( , ; ), historically known as Ancyra and Angora, is the capital of Turkey. Located in the central part of Anatolia, the city has a population of 5.1 million in its urban center and over 5.7 million in Ankara Province, making it Turkey's second-largest city after Istanbul. Serving as the capital of the ancient Celtic state of Galatia (280–64 BC), and later of the Roman province with the same name (25 BC–7th century), the city is very old, with various Hattian, Hittite, Lydian, Phrygian, Galatian, Greek, Persian, Roman, Byzantine, and Ottoman archeological sites. The Ottomans made the city the capital first of the Anatolia Eyalet (1393 – late 15th century) and then the Angora Vilayet (1867–1922). The historical center of Ankara is a rocky hill rising over the left bank of the Ankara River, a tributary of the Sakarya River. The hill remains crowned by the ruins of Ankara Castle. Although few of its outworks have survived, there are well-preserved examples of Roman and Ottoman arch...
  In what state was Simon Lalong governor?	Atẹjade kan lori opo ayelujara twita fun ọkan ninu oluranlọwọ fun aarẹ, Bashir Ahmad sọ pe awọn gomina mẹfa se ipade ikọkọ pẹlu aarẹ ni ọjọbọ. Ko sọ ohun ti ipade naa dale lori. Awọn gomina ti o wa nibẹ ni, Atiku Bagudu ti ipinlẹ Kebbi, Udom Emmanuel lati ipinlẹ Akwa Ibom ati Aminu Masari ti ipinle Katsina. Awọn ẹlomiran ni awọn gomina Dafidi Umahi ti ipinlẹ Ebonyi, Simon Lalong ti o jẹ gomina ipinlẹ Plateau ati alaga ẹgbẹ awọn gomina orilẹede Naijiria, Abdulaziz Yari ti o jẹ gomina ipinle Zamfara. Nipa ti awọn Gomina, o tẹ bayi pe: Ipade lori ọrọ aabo orilẹede. Atẹjade Basir Ahmad lori ẹrọ Twitta wi bayi pe:
  2000 ọ̀dọ́ ìpínlẹ̀ Oṣun jàǹfàní ìkọ́ṣẹ́ lọwọ́ ìjọba àpapọ̀	Iṣẹ́ ọnà, àsè gbígbà, ilé ṣíṣe lọ́ṣọ́ àti kẹ́míkà ṣíṣe kún ara iṣẹ́ ọwọ tí wọ́n kọ́ àwọn ọ̀dọ́ náà

  Awọn ẹgbẹrun meji naa ni wọn ṣa kaakiri awọn ijọba ibilẹ mẹwaa lẹkun idibo aringbungbun Ọṣun ati ijọba ibilẹ Ede South nipinlẹ naa.

  Awọn alakoso eto ọhun, Liberty Olawale Badmus ati Ọmọwe Saka Ominiwe, ni afojusun eto ikọṣẹ ọfẹ naa ni lati fun awọn ọdọ ni anfani igbagbọ ninu ara wọn bi o ti ṣe wa lawọn orilẹ-ede kaakiri agbaye.

  Àwọn iròyin tì ẹ leè ní ìfẹ̀ síí:

  O tẹnumọ pataki ṣiṣamulo awọn ẹkọ ti wọn kọ lati mu ayipada rere ba ara wọn ati orilẹ-ede Naijiria lapapọ.

  Ọmọwe Saka Ominiwe ni bi igbesẹ naa ba n tẹsiwaju, yoo mu iṣoro airiṣẹ di ohun igbagbe lorilẹ-ede Naijiria.

  Wọn ni eto ọhun ti kọkọ waye ni ipinlẹ Eko lati gbe igbesẹ fun riro awọn ọdọ lagbara lati da duro pẹlu bi iṣẹ ọba ati aladani ṣe di wahala nitori bi ọrọ aje ṣe ri.

  Gbogbo awọn ọdọ naa ni wọn kọ ni ẹkọ iṣẹ ọna, ṣiṣe ile lọṣọ, fọtọ yiya ati ṣiṣe eroja kẹmika ati bẹẹbẹẹ lọ.

  Russia 2018:Super Eagles ti kẹ́kọ̀ọ́ nínú ... |

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "GISTEmbedLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• gradient_accumulation_steps: 32
• learning_rate: 2e-05
• num_train_epochs: 2
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• bf16: True
• dataloader_num_workers: 2
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• gradient_accumulation_steps: 32
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	nigerian-val_cosine_ndcg@10
0.2385	25	6.5260	-	-
0.4769	50	2.7151	-	-
0.7154	75	1.8633	-	-
0.9538	100	1.8174	-	-
0.9920	104	-	0.8248	0.8802
1.1908	125	1.5465	-	-
1.4292	150	1.4435	-	-
1.6677	175	1.2961	-	-
1.9061	200	1.3278	-	-
1.9824	208	-	0.7767	0.8824

Training Time

• Training: 7.9 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}