as9122/instructor-xl-stance-mixed

SentenceTransformer based on hkunlp/instructor-xl

This is a sentence-transformers model finetuned from hkunlp/instructor-xl. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: hkunlp/instructor-xl
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'T5EncoderModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': False})
  (2): Dense({'in_features': 1024, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Show literature challenging Artificial intelligence will ultimately enhance human creativity rather than replace it.',
    'Economic pressures will drive the replacement of human creators. AI can produce vast quantities of high-quality creative content for a fraction of the cost and time, making human professionals unable to compete in the marketplace.',
    "The 'filter bubble' thesis is overstated; empirical research shows that social media often exposes people to more diverse, cross-cutting political views than their offline interactions, which can challenge rather than reinforce their beliefs.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7428, 0.4338],
#         [0.7428, 1.0000, 0.4565],
#         [0.4338, 0.4565, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 31,800 training samples
• Columns: anchor, positive, negative, instruction_text, boost_words, and claim
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative	instruction_text	boost_words	claim
  type	string	string	string	string	list	string
  details	min: 11 tokens mean: 25.07 tokens max: 43 tokens	min: 30 tokens mean: 53.01 tokens max: 86 tokens	min: 24 tokens mean: 54.17 tokens max: 120 tokens	min: 4 tokens mean: 5.75 tokens max: 9 tokens	size: 3 elements	min: 8 tokens mean: 20.32 tokens max: 35 tokens

• Samples:

  anchor	positive	negative	instruction_text	boost_words	claim
  Extract statements advocating for Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.	The claim is true because Alfred Nobel and Bertha von Suttner maintained a long and close correspondence in which she persistently advocated for peace, providing a direct and sustained channel for her to influence his thinking.	The claim is false because Nobel's philosophy on peace was notably different from Suttner's. Nobel held a more cynical and pragmatic view, once remarking that his dynamite factories would end war sooner than her peace congresses. The prize's criteria reflect his practical mindset, not a wholesale adoption of her idealistic pacifism.	Extract statements advocating for	['suttner', 'nobel', 'bertha']	Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.
  Retrieve arguments in favor of Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.	The claim is true because their letters contain direct discussions about creating a peace prize. In an 1893 letter to Suttner, Nobel expressed his desire to establish a prize for the person who had done the most to advance the pacification of Europe, showing a clear causal link between her advocacy and his idea.	The claim is an oversimplification and likely false because Nobel had his own independent and pre-existing motivations for an interest in peace, primarily his personal anguish over the military use of his inventions. The prize was a way to reconcile his own conflicting legacy, not simply an adoption of Suttner's ideas.	Retrieve arguments in favor of	['suttner', 'nobel', 'bertha']	Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.
  Find evidence backing Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.	The claim is true because Nobel explicitly read and praised Suttner's influential anti-war novel, "Die Waffen nieder!" ("Lay Down Your Arms!"), demonstrating that her specific work and pacifist message directly impacted his views on the subject.	The claim of 'direct' influence is false because the final text of Nobel's will and the creation of the prizes were a surprise to nearly everyone, including Suttner. If her influence had been so direct and formative, it is likely she would have had more specific knowledge of his ultimate plans before they were revealed.	Find evidence backing	['suttner', 'nobel', 'bertha']	Bertha von Suttner, a 19th-century peace activist and author, directly influenced Alfred Nobel's decision to create the Nobel Peace Prize.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 2
• learning_rate: 2e-05
• warmup_steps: 0.1
• gradient_accumulation_steps: 2
• bf16: True
• gradient_checkpointing: True
• remove_unused_columns: False

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 2
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 2
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: False
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0050	10	0.5099
0.0101	20	0.5399
0.0151	30	0.5483
0.0201	40	0.5209
0.0252	50	0.5185
0.0302	60	0.5430
0.0352	70	0.5905
0.0403	80	0.4935
0.0453	90	0.5462
0.0503	100	0.4765
0.0553	110	0.5693
0.0604	120	0.5354
0.0654	130	0.5261
0.0704	140	0.5290
0.0755	150	0.6048
0.0805	160	0.5459
0.0855	170	0.4844
0.0906	180	0.4622
0.0956	190	0.5328
0.1006	200	0.5414
0.1057	210	0.4627
0.1107	220	0.4761
0.1157	230	0.4688
0.1208	240	0.4816
0.1258	250	0.5219
0.1308	260	0.4848
0.1358	270	0.4522
0.1409	280	0.4396
0.1459	290	0.5258
0.1509	300	0.4351
0.1560	310	0.3776
0.1610	320	0.3685
0.1660	330	0.4476
0.1711	340	0.3930
0.1761	350	0.4399
0.1811	360	0.5114
0.1862	370	0.4537
0.1912	380	0.3510
0.1962	390	0.3916
0.2013	400	0.3884
0.2063	410	0.3887
0.2113	420	0.3290
0.2164	430	0.3710
0.2214	440	0.5399
0.2264	450	0.3434
0.2314	460	0.4026
0.2365	470	0.4085
0.2415	480	0.3931
0.2465	490	0.3618
0.2516	500	0.4182
0.2566	510	0.3877
0.2616	520	0.5206
0.2667	530	0.3248
0.2717	540	0.2880
0.2767	550	0.3231
0.2818	560	0.3942
0.2868	570	0.3697
0.2918	580	0.2718
0.2969	590	0.4321
0.3019	600	0.3333
0.3069	610	0.3739
0.3119	620	0.3254
0.3170	630	0.3031
0.3220	640	0.3243
0.3270	650	0.3142
0.3321	660	0.3366
0.3371	670	0.3124
0.3421	680	0.2669
0.3472	690	0.3253
0.3522	700	0.2979
0.3572	710	0.3731
0.3623	720	0.3672
0.3673	730	0.2686
0.3723	740	0.2930
0.3774	750	0.3106
0.3824	760	0.2907
0.3874	770	0.2552
0.3925	780	0.2228
0.3975	790	0.3208
0.4025	800	0.2806
0.4075	810	0.3060
0.4126	820	0.3567
0.4176	830	0.2460
0.4226	840	0.2272
0.4277	850	0.2586
0.4327	860	0.3009
0.4377	870	0.2921
0.4428	880	0.2696
0.4478	890	0.2891
0.4528	900	0.2252
0.4579	910	0.2069
0.4629	920	0.3262
0.4679	930	0.2075
0.4730	940	0.2313
0.4780	950	0.2412
0.4830	960	0.2690
0.4881	970	0.2494
0.4931	980	0.2903
0.4981	990	0.2453
0.5031	1000	0.2454
0.5082	1010	0.2801
0.5132	1020	0.2039
0.5182	1030	0.2272
0.5233	1040	0.2536
0.5283	1050	0.2100
0.5333	1060	0.2501
0.5384	1070	0.2821
0.5434	1080	0.2489
0.5484	1090	0.2181
0.5535	1100	0.2315
0.5585	1110	0.2505
0.5635	1120	0.2299
0.5686	1130	0.2357
0.5736	1140	0.2717
0.5786	1150	0.1304
0.5836	1160	0.2265
0.5887	1170	0.2509
0.5937	1180	0.2092
0.5987	1190	0.1632
0.6038	1200	0.2063
0.6088	1210	0.1782
0.6138	1220	0.1526
0.6189	1230	0.1882
0.6239	1240	0.2270
0.6289	1250	0.1742
0.6340	1260	0.1844
0.6390	1270	0.1908
0.6440	1280	0.1660
0.6491	1290	0.1646
0.6541	1300	0.1128
0.6591	1310	0.1897
0.6642	1320	0.2455
0.6692	1330	0.2417
0.6742	1340	0.1507
0.6792	1350	0.1822
0.6843	1360	0.1600
0.6893	1370	0.1666
0.6943	1380	0.2700
0.6994	1390	0.2267
0.7044	1400	0.1802
0.7094	1410	0.1689
0.7145	1420	0.1589
0.7195	1430	0.2079
0.7245	1440	0.2486
0.7296	1450	0.2169
0.7346	1460	0.2419
0.7396	1470	0.2183
0.7447	1480	0.1406
0.7497	1490	0.1453
0.7547	1500	0.1527
0.7597	1510	0.1847
0.7648	1520	0.2497
0.7698	1530	0.2041
0.7748	1540	0.1166
0.7799	1550	0.1825
0.7849	1560	0.1570
0.7899	1570	0.1770
0.7950	1580	0.1301
0.8	1590	0.1608
0.8050	1600	0.1869
0.8101	1610	0.2282
0.8151	1620	0.1446
0.8201	1630	0.1689
0.8252	1640	0.1846
0.8302	1650	0.1788
0.8352	1660	0.2224
0.8403	1670	0.1572
0.8453	1680	0.1638
0.8503	1690	0.2027
0.8553	1700	0.1270
0.8604	1710	0.1698
0.8654	1720	0.1734
0.8704	1730	0.1922
0.8755	1740	0.1792
0.8805	1750	0.2164
0.8855	1760	0.1515
0.8906	1770	0.1727
0.8956	1780	0.1255
0.9006	1790	0.1588
0.9057	1800	0.1547
0.9107	1810	0.1595
0.9157	1820	0.1829
0.9208	1830	0.1488
0.9258	1840	0.1833
0.9308	1850	0.1490
0.9358	1860	0.1300
0.9409	1870	0.1703
0.9459	1880	0.1936
0.9509	1890	0.1240
0.9560	1900	0.1829
0.9610	1910	0.2300
0.9660	1920	0.1825
0.9711	1930	0.1815
0.9761	1940	0.2364
0.9811	1950	0.2092
0.9862	1960	0.1514
0.9912	1970	0.1658
0.9962	1980	0.1464
1.0010	1990	0.1514
1.0060	2000	0.1770
1.0111	2010	0.1545
1.0161	2020	0.1625
1.0211	2030	0.1775
1.0262	2040	0.2215
1.0312	2050	0.1535
1.0362	2060	0.1553
1.0413	2070	0.1783
1.0463	2080	0.1474
1.0513	2090	0.1201
1.0564	2100	0.1606
1.0614	2110	0.1594
1.0664	2120	0.1985
1.0714	2130	0.1254
1.0765	2140	0.2004
1.0815	2150	0.1798
1.0865	2160	0.1929
1.0916	2170	0.1706
1.0966	2180	0.1522
1.1016	2190	0.1715
1.1067	2200	0.1175
1.1117	2210	0.1244
1.1167	2220	0.1651
1.1218	2230	0.1631
1.1268	2240	0.1397
1.1318	2250	0.1741
1.1369	2260	0.1353
1.1419	2270	0.2062
1.1469	2280	0.1937
1.1519	2290	0.1643
1.1570	2300	0.1636
1.1620	2310	0.1311
1.1670	2320	0.1505
1.1721	2330	0.1439
1.1771	2340	0.1902
1.1821	2350	0.1905
1.1872	2360	0.1686
1.1922	2370	0.2103
1.1972	2380	0.2438
1.2023	2390	0.1074
1.2073	2400	0.1573
1.2123	2410	0.1026
1.2174	2420	0.1434
1.2224	2430	0.1548
1.2274	2440	0.1310
1.2325	2450	0.1462
1.2375	2460	0.1737
1.2425	2470	0.1368
1.2475	2480	0.1689
1.2526	2490	0.1306
1.2576	2500	0.1871
1.2626	2510	0.1398
1.2677	2520	0.1615
1.2727	2530	0.1907
1.2777	2540	0.1457
1.2828	2550	0.2284
1.2878	2560	0.1385
1.2928	2570	0.1707
1.2979	2580	0.1904
1.3029	2590	0.1615
1.3079	2600	0.1942
1.3130	2610	0.1665
1.3180	2620	0.1844
1.3230	2630	0.1531
1.3281	2640	0.1575
1.3331	2650	0.1901
1.3381	2660	0.1214
1.3431	2670	0.1458
1.3482	2680	0.1424
1.3532	2690	0.1380
1.3582	2700	0.1721
1.3633	2710	0.1977
1.3683	2720	0.1343
1.3733	2730	0.2174
1.3784	2740	0.1630
1.3834	2750	0.1423
1.3884	2760	0.2155
1.3935	2770	0.2150
1.3985	2780	0.1799
1.4035	2790	0.1219
1.4086	2800	0.1215
1.4136	2810	0.1695
1.4186	2820	0.2008
1.4236	2830	0.1185
1.4287	2840	0.1701
1.4337	2850	0.2147
1.4387	2860	0.2015
1.4438	2870	0.1395
1.4488	2880	0.1397
1.4538	2890	0.1552
1.4589	2900	0.1396
1.4639	2910	0.1649
1.4689	2920	0.1566
1.4740	2930	0.1356
1.4790	2940	0.2107
1.4840	2950	0.1463
1.4891	2960	0.1725
1.4941	2970	0.1273
1.4991	2980	0.1615
1.5042	2990	0.1492
1.5092	3000	0.1591
1.5142	3010	0.1752
1.5192	3020	0.1555
1.5243	3030	0.1711
1.5293	3040	0.1362
1.5343	3050	0.1613
1.5394	3060	0.1337
1.5444	3070	0.1038
1.5494	3080	0.1645
1.5545	3090	0.2168
1.5595	3100	0.1397
1.5645	3110	0.2055
1.5696	3120	0.1868
1.5746	3130	0.1747
1.5796	3140	0.1461
1.5847	3150	0.1623
1.5897	3160	0.1672
1.5947	3170	0.1752
1.5997	3180	0.1190
1.6048	3190	0.1260
1.6098	3200	0.1013
1.6148	3210	0.1621
1.6199	3220	0.1972
1.6249	3230	0.1542
1.6299	3240	0.1260
1.6350	3250	0.2469
1.6400	3260	0.1345
1.6450	3270	0.1634
1.6501	3280	0.1823
1.6551	3290	0.1264
1.6601	3300	0.2212
1.6652	3310	0.1529
1.6702	3320	0.1622
1.6752	3330	0.1691
1.6803	3340	0.1321
1.6853	3350	0.1586
1.6903	3360	0.1749
1.6953	3370	0.1498
1.7004	3380	0.1421
1.7054	3390	0.1334
1.7104	3400	0.1736
1.7155	3410	0.1476
1.7205	3420	0.1568
1.7255	3430	0.1754
1.7306	3440	0.1318
1.7356	3450	0.1704
1.7406	3460	0.1249
1.7457	3470	0.1381
1.7507	3480	0.2418
1.7557	3490	0.1413
1.7608	3500	0.1825
1.7658	3510	0.1796
1.7708	3520	0.1428
1.7758	3530	0.1380
1.7809	3540	0.1593
1.7859	3550	0.1682
1.7909	3560	0.1359
1.7960	3570	0.1810
1.8010	3580	0.2006
1.8060	3590	0.2232
1.8111	3600	0.1477
1.8161	3610	0.1301
1.8211	3620	0.1285
1.8262	3630	0.1441
1.8312	3640	0.1246
1.8362	3650	0.1026
1.8413	3660	0.1626
1.8463	3670	0.1605
1.8513	3680	0.1213
1.8564	3690	0.1666
1.8614	3700	0.1574
1.8664	3710	0.1656
1.8714	3720	0.1174
1.8765	3730	0.1639
1.8815	3740	0.2174
1.8865	3750	0.1231
1.8916	3760	0.1492
1.8966	3770	0.1451
1.9016	3780	0.1382
1.9067	3790	0.1359
1.9117	3800	0.1257
1.9167	3810	0.2029
1.9218	3820	0.1401
1.9268	3830	0.1760
1.9318	3840	0.1785
1.9369	3850	0.1375
1.9419	3860	0.1296
1.9469	3870	0.1382
1.9519	3880	0.1354
1.9570	3890	0.1886
1.9620	3900	0.1518
1.9670	3910	0.1622
1.9721	3920	0.1397
1.9771	3930	0.2438
1.9821	3940	0.1824
1.9872	3950	0.1874
1.9922	3960	0.1974
1.9972	3970	0.1645

Training Time

• Training: 1.1 hours

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.4.1
• Transformers: 5.5.4
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}