SetFit with sentence-transformers/all-MiniLM-L6-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/all-MiniLM-L6-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/all-MiniLM-L6-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 256 tokens
Number of Classes: 4 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
question	'Cash withdrawal was incorrect at ATM.' 'What is the server?' 'What is the invoice?'
request	'I need you to show the policy.' 'Can you summarise the compliance posture?' 'Can you find the port?'
assertion	'It seems that the report schedule.' 'I believe the balance is secure.' 'This is the compliance posture.'
expressive	'Great, the host is working!' 'I am frustrated with the port.' 'Amazing, the compliance posture is fixed!'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("AutoCyberAI/crp-intent-setfit")
# Run inference
preds = model("Can you find the port?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	4	7.4883	53

Label	Training Sample Count
assertion	64
expressive	64
question	64
request	64

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0003	1	0.4095	-
0.0163	50	0.3272	-
0.0326	100	0.2618	-
0.0488	150	0.1939	-
0.0651	200	0.1145	-
0.0814	250	0.0689	-
0.0977	300	0.0714	-
0.1139	350	0.0542	-
0.1302	400	0.0306	-
0.1465	450	0.0145	-
0.1628	500	0.0044	-
0.1790	550	0.0026	-
0.1953	600	0.0022	-
0.2116	650	0.0015	-
0.2279	700	0.0013	-
0.2441	750	0.0012	-
0.2604	800	0.001	-
0.2767	850	0.0008	-
0.2930	900	0.0009	-
0.3092	950	0.0008	-
0.3255	1000	0.0006	-
0.3418	1050	0.0006	-
0.3581	1100	0.0006	-
0.3743	1150	0.0006	-
0.3906	1200	0.0005	-
0.4069	1250	0.0005	-
0.4232	1300	0.0005	-
0.4395	1350	0.0005	-
0.4557	1400	0.0004	-
0.4720	1450	0.0004	-
0.4883	1500	0.0004	-
0.5046	1550	0.0004	-
0.5208	1600	0.0004	-
0.5371	1650	0.0004	-
0.5534	1700	0.0004	-
0.5697	1750	0.0003	-
0.5859	1800	0.0003	-
0.6022	1850	0.0003	-
0.6185	1900	0.0003	-
0.6348	1950	0.0003	-
0.6510	2000	0.0003	-
0.6673	2050	0.0003	-
0.6836	2100	0.0003	-
0.6999	2150	0.0003	-
0.7161	2200	0.0003	-
0.7324	2250	0.0003	-
0.7487	2300	0.0003	-
0.7650	2350	0.0003	-
0.7812	2400	0.0003	-
0.7975	2450	0.0003	-
0.8138	2500	0.0002	-
0.8301	2550	0.0002	-
0.8464	2600	0.0003	-
0.8626	2650	0.0002	-
0.8789	2700	0.0002	-
0.8952	2750	0.0002	-
0.9115	2800	0.0003	-
0.9277	2850	0.0002	-
0.9440	2900	0.0002	-
0.9603	2950	0.0002	-
0.9766	3000	0.0002	-
0.9928	3050	0.0002	-

Framework Versions

Python: 3.11.6
SetFit: 1.1.3
Sentence Transformers: 5.6.1
Transformers: 4.57.6
PyTorch: 2.13.0+cpu
Datasets: 5.0.0
Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}