Trelis/multi-qa-MiniLM-L6-dot-v1-ft-triplets-2-cst-epoch-overlap

SentenceTransformer based on sentence-transformers/multi-qa-MiniLM-L6-dot-v1

This is a sentence-transformers model finetuned from sentence-transformers/multi-qa-MiniLM-L6-dot-v1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/multi-qa-MiniLM-L6-dot-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Dot Product

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Trelis/multi-qa-MiniLM-L6-dot-v1-ft-triplets-2-cst-epoch-overlap")
# Run inference
sentences = [
    'What is the minimum number of males and females required on the field of play in mixed gender competitions?',
    '5. 3. 1 this does not apply for players sent to the sin bin area. 5. 4 in mixed gender competitions, the maximum number of males allowed on the field of play is three ( 3 ), the minimum male requirement is one ( 1 ) and the minimum female requirement is one ( 1 ). 6 team coach and team officials 6. 1 the team coach ( s ) and team officials may be permitted inside the perimeter but shall be required to be positioned either in the interchange area or at the end of the field of play for the duration of the match. 6. 2 the team coach ( s ) and team officials may move from one position to the other but shall do so without delay. while in a position at the end of the field of play, the team coach ( s ) or team official must remain no closer than five ( 5 ) metres from the dead ball line and must not coach or communicate ( verbal or non - verbal ) with either team or the referees.',
    'tap and tap penalty the method of commencing the match, recommencing the match after half time and after a try has been scored. the tap is also the method of recommencing play when a penalty is awarded. the tap is taken by placing the ball on the ground at or behind the mark, releasing both hands from the ball, tapping the ball gently with either foot or touching the foot on the ball. the ball must not roll or move more than one ( 1 ) metre in any direction and must be retrieved cleanly, without touching the ground again. the player may face any direction and use either foot. provided it is at the mark, the ball does not have to be lifted from the ground prior to a tap being taken. team a group of players constituting one ( 1 ) side in a competition match. tfa touch football australia limited touch any contact between the player in possession and a defending player. a touch includes contact on the ball, hair or clothing and may be made by a defending player or by the player in possession. touch count the progressive number of touches that each team has before a change of possession, from zero ( 0 ) to six ( 6 ).',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 2
• lr_scheduler_type: constant
• warmup_ratio: 0.3

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: constant
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.3
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	loss
0.0066	2	4.2577	-
0.0131	4	4.0287	-
0.0197	6	4.1022	-
0.0262	8	3.8676	-
0.0328	10	3.836	-
0.0393	12	3.5016	-
0.0459	14	3.5338	-
0.0525	16	3.2488	-
0.0590	18	3.1999	-
0.0656	20	3.1266	-
0.0721	22	2.7272	-
0.0787	24	2.9005	-
0.0852	26	2.4328	-
0.0918	28	2.0929	-
0.0984	30	1.9004	-
0.1049	32	2.0227	-
0.1115	34	1.8577	-
0.1180	36	1.6767	-
0.1246	38	1.6381	-
0.1311	40	1.391	-
0.1377	42	1.6477	-
0.1443	44	1.7922	-
0.1508	46	1.3904	-
0.1574	48	1.7869	-
0.1639	50	1.6701	-
0.1705	52	1.1823	-
0.1770	54	0.9928	-
0.1836	56	1.4254	-
0.1902	58	1.4535	-
0.1967	60	1.3876	-
0.2033	62	1.411	-
0.2098	64	0.9778	-
0.2164	66	0.9914	-
0.2230	68	0.9946	-
0.2295	70	0.9828	-
0.2361	72	0.7432	-
0.2426	74	1.315	-
0.2492	76	0.6955	-
0.2525	77	-	0.5755
0.2557	78	0.7919	-
0.2623	80	1.001	-
0.2689	82	1.086	-
0.2754	84	0.8213	-
0.2820	86	0.9834	-
0.2885	88	1.0083	-
0.2951	90	0.6879	-
0.3016	92	0.672	-
0.3082	94	0.663	-
0.3148	96	1.0285	-
0.3213	98	1.0634	-
0.3279	100	0.9316	-
0.3344	102	0.88	-
0.3410	104	0.9057	-
0.3475	106	0.7303	-
0.3541	108	0.8927	-
0.3607	110	0.6541	-
0.3672	112	0.6616	-
0.3738	114	0.9198	-
0.3803	116	0.4953	-
0.3869	118	0.7159	-
0.3934	120	0.6596	-
0.4	122	0.8359	-
0.4066	124	0.7748	-
0.4131	126	0.5949	-
0.4197	128	0.4204	-
0.4262	130	1.0151	-
0.4328	132	0.4979	-
0.4393	134	0.6496	-
0.4459	136	0.6039	-
0.4525	138	0.6333	-
0.4590	140	0.5494	-
0.4656	142	0.5599	-
0.4721	144	0.353	-
0.4787	146	0.6529	-
0.4852	148	0.5215	-
0.4918	150	0.6408	-
0.4984	152	0.6084	-
0.5049	154	0.8947	0.2713
0.5115	156	0.515	-
0.5180	158	0.4075	-
0.5246	160	0.7453	-
0.5311	162	0.4385	-
0.5377	164	0.5747	-
0.5443	166	0.725	-
0.5508	168	0.6338	-
0.5574	170	0.2453	-
0.5639	172	0.4578	-
0.5705	174	0.3541	-
0.5770	176	0.5326	-
0.5836	178	0.3699	-
0.5902	180	0.1714	-
0.5967	182	0.3149	-
0.6033	184	0.561	-
0.6098	186	0.2852	-
0.6164	188	0.2715	-
0.6230	190	0.5521	-
0.6295	192	0.2852	-
0.6361	194	0.5103	-
0.6426	196	0.3866	-
0.6492	198	0.4369	-
0.6557	200	0.3936	-
0.6623	202	0.5084	-
0.6689	204	0.4912	-
0.6754	206	0.2659	-
0.6820	208	0.6209	-
0.6885	210	0.3022	-
0.6951	212	0.2738	-
0.7016	214	0.5555	-
0.7082	216	0.3672	-
0.7148	218	0.3489	-
0.7213	220	0.6139	-
0.7279	222	0.4402	-
0.7344	224	0.2829	-
0.7410	226	0.3669	-
0.7475	228	0.729	-
0.7541	230	0.2565	-
0.7574	231	-	0.1849
0.7607	232	0.2596	-
0.7672	234	0.2359	-
0.7738	236	0.4406	-
0.7803	238	0.2629	-
0.7869	240	0.3583	-
0.7934	242	0.5298	-
0.8	244	0.6225	-
0.8066	246	0.3853	-
0.8131	248	0.4741	-
0.8197	250	0.3991	-
0.8262	252	0.5629	-
0.8328	254	0.2935	-
0.8393	256	0.3563	-
0.8459	258	0.3628	-
0.8525	260	0.2416	-
0.8590	262	0.1493	-
0.8656	264	0.2488	-
0.8721	266	0.4055	-
0.8787	268	0.1286	-
0.8852	270	0.4217	-
0.8918	272	0.3529	-
0.8984	274	0.1921	-
0.9049	276	0.1736	-
0.9115	278	0.4308	-
0.9180	280	0.0992	-
0.9246	282	0.3927	-
0.9311	284	0.3451	-
0.9377	286	0.4504	-
0.9443	288	0.3065	-
0.9508	290	0.2844	-
0.9574	292	0.4308	-
0.9639	294	0.1754	-
0.9705	296	0.2608	-
0.9770	298	0.4232	-
0.9836	300	0.3234	-
0.9902	302	0.24	-
0.9967	304	0.2112	-
1.0033	306	0.6322	-
1.0098	308	0.2987	0.1357
1.0164	310	0.4052	-
1.0230	312	0.1458	-
1.0295	314	0.2593	-
1.0361	316	0.193	-
1.0426	318	0.29	-
1.0492	320	0.299	-
1.0557	322	0.0841	-
1.0623	324	0.0534	-
1.0689	326	0.2166	-
1.0754	328	0.2431	-
1.0820	330	0.2621	-
1.0885	332	0.0986	-
1.0951	334	0.4274	-
1.1016	336	0.2388	-
1.1082	338	0.0899	-
1.1148	340	0.158	-
1.1213	342	0.1748	-
1.1279	344	0.1226	-
1.1344	346	0.1815	-
1.1410	348	0.2312	-
1.1475	350	0.4114	-
1.1541	352	0.2258	-
1.1607	354	0.1519	-
1.1672	356	0.1536	-
1.1738	358	0.103	-
1.1803	360	0.2901	-
1.1869	362	0.1629	-
1.1934	364	0.1541	-
1.2	366	0.1986	-
1.2066	368	0.2492	-
1.2131	370	0.2137	-
1.2197	372	0.1954	-
1.2262	374	0.1947	-
1.2328	376	0.2114	-
1.2393	378	0.4277	-
1.2459	380	0.3636	-
1.2525	382	0.4151	-
1.2590	384	0.2258	-
1.2623	385	-	0.1095
1.2656	386	0.2794	-
1.2721	388	0.2504	-
1.2787	390	0.3785	-
1.2852	392	0.2448	-
1.2918	394	0.3936	-
1.2984	396	0.1686	-
1.3049	398	0.2301	-
1.3115	400	0.1533	-
1.3180	402	0.2516	-
1.3246	404	0.1238	-
1.3311	406	0.1629	-
1.3377	408	0.1395	-
1.3443	410	0.1093	-
1.3508	412	0.0899	-
1.3574	414	0.1793	-
1.3639	416	0.0648	-
1.3705	418	0.2402	-
1.3770	420	0.2711	-
1.3836	422	0.1457	-
1.3902	424	0.1338	-
1.3967	426	0.3074	-
1.4033	428	0.0738	-
1.4098	430	0.1702	-
1.4164	432	0.111	-
1.4230	434	0.249	-
1.4295	436	0.1143	-
1.4361	438	0.2255	-
1.4426	440	0.3167	-
1.4492	442	0.0751	-
1.4557	444	0.1101	-
1.4623	446	0.2098	-
1.4689	448	0.2086	-
1.4754	450	0.0978	-
1.4820	452	0.3184	-
1.4885	454	0.1347	-
1.4951	456	0.2259	-
1.5016	458	0.1651	-
1.5082	460	0.2183	-
1.5148	462	0.1315	0.0771
1.5213	464	0.2672	-
1.5279	466	0.1783	-
1.5344	468	0.0408	-
1.5410	470	0.4634	-
1.5475	472	0.1762	-
1.5541	474	0.0553	-
1.5607	476	0.2445	-
1.5672	478	0.1988	-
1.5738	480	0.1985	-
1.5803	482	0.1484	-
1.5869	484	0.1403	-
1.5934	486	0.1993	-
1.6	488	0.1486	-
1.6066	490	0.2899	-
1.6131	492	0.2464	-
1.6197	494	0.1352	-
1.6262	496	0.1233	-
1.6328	498	0.0413	-
1.6393	500	0.091	-
1.6459	502	0.0828	-
1.6525	504	0.1488	-
1.6590	506	0.1246	-
1.6656	508	0.2795	-
1.6721	510	0.067	-
1.6787	512	0.168	-
1.6852	514	0.2215	-
1.6918	516	0.0854	-
1.6984	518	0.2192	-
1.7049	520	0.1479	-
1.7115	522	0.1924	-
1.7180	524	0.2075	-
1.7246	526	0.208	-
1.7311	528	0.1743	-
1.7377	530	0.0817	-
1.7443	532	0.1513	-
1.7508	534	0.3422	-
1.7574	536	0.1101	-
1.7639	538	0.2815	-
1.7672	539	-	0.0693
1.7705	540	0.1837	-
1.7770	542	0.0879	-
1.7836	544	0.0746	-
1.7902	546	0.2052	-
1.7967	548	0.1416	-
1.8033	550	0.1141	-
1.8098	552	0.0312	-
1.8164	554	0.139	-
1.8230	556	0.1078	-
1.8295	558	0.1302	-
1.8361	560	0.0124	-
1.8426	562	0.2641	-
1.8492	564	0.1625	-
1.8557	566	0.1907	-
1.8623	568	0.0	-
1.8689	570	0.1721	-
1.8754	572	0.1178	-
1.8820	574	0.0345	-
1.8885	576	0.0924	-
1.8951	578	0.0513	-
1.9016	580	0.0929	-
1.9082	582	0.1502	-
1.9148	584	0.0338	-
1.9213	586	0.1348	-
1.9279	588	0.0297	-
1.9344	590	0.0306	-
1.9410	592	0.1416	-
1.9475	594	0.0427	-
1.9541	596	0.1916	-
1.9607	598	0.1969	-
1.9672	600	0.0765	-
1.9738	602	0.1035	-
1.9803	604	0.261	-
1.9869	606	0.0845	-
1.9934	608	0.0566	-
2.0	610	0.704	-

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.3
• PyTorch: 2.1.1+cu121
• Accelerate: 0.31.0
• Datasets: 2.17.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification}, 
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}