---
library_name: setfit
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
base_model: sentence-transformers/paraphrase-mpnet-base-v2
metrics:
- accuracy
widget:
- text: travel book a train ticket
- text: how much is the average house
- text: do i need a jacket
- text: i like the songs of yeshudas please play it
- text: tell me the current time
pipeline_tag: text-classification
inference: true
model-index:
- name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.7743480574773816
      name: Accuracy
---

# SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 35 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label                    | Examples                                                                                                                                                                                            |
|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| alarm_query              | <ul><li>'do i have any alarms set for six am tomorrow'</li><li>'what is the wake up time for my alarm i have set for the flight this weekend'</li><li>'please tell me what alarms are on'</li></ul> |
| alarm_set                | <ul><li>'set an alarm for six thirty am'</li><li>'add an alarm for tomorrow morning at six am'</li><li>'wake me up at five am'</li></ul>                                                            |
| audio_volume_mute        | <ul><li>'can you please stop speaking'</li><li>'turn off sound'</li><li>'shut down the sound'</li></ul>                                                                                             |
| calendar_query           | <ul><li>'how long will my lunch meeting be on tuesday'</li><li>'what time is my doctor appointment on march thirty first'</li><li>'what days do i have booked'</li></ul>                            |
| calendar_remove          | <ul><li>'clear everything off my calendar for the rest of the year'</li><li>'please clear my calendar'</li><li>'remove from my calendar meeting at nine am'</li></ul>                               |
| calendar_set             | <ul><li>'new event'</li><li>'remind me of the event in my calendar'</li><li>"mark april twenty as my brother's birthday"</li></ul>                                                                  |
| cooking_recipe           | <ul><li>'tell me the recipe of'</li><li>'how is rice prepared'</li><li>'what ingredient can be used instead of saffron'</li></ul>                                                                   |
| datetime_query           | <ul><li>'what is the time in canada now'</li><li>"what's the time in australia"</li><li>'display the local time of london at this moment'</li></ul>                                                 |
| email_query              | <ul><li>'do i have any unread emails'</li><li>'what about new mail'</li><li>'olly do i have any new emails'</li></ul>                                                                               |
| email_sendemail          | <ul><li>'dictate email'</li><li>'reply an email to jason that i will not come tonight'</li><li>'please send an email to cassy who is there on my family and friend list'</li></ul>                  |
| general_quirky           | <ul><li>'where was will ferrell seen last night'</li><li>'do you think i should go to the theater today'</li><li>'what is the best chocolate chip cookies recipe'</li></ul>                         |
| iot_coffee               | <ul><li>'i need a drink'</li><li>'please activate my coffee pot for me'</li><li>'prepare a cup of coffee for me'</li></ul>                                                                          |
| iot_hue_lightchange      | <ul><li>'please make the lights natural'</li><li>'make the room light blue'</li><li>'hey olly chance the current light settings'</li></ul>                                                          |
| iot_hue_lightoff         | <ul><li>'siri please turn the lights off in the bathroom'</li><li>'turn my bedroom lights off'</li><li>'no lights in the kitchen'</li></ul>                                                         |
| lists_createoradd        | <ul><li>'add business contacts to contact list'</li><li>'please create a new list for me'</li><li>"i want to make this week's shopping list"</li></ul>                                              |
| lists_query              | <ul><li>'give me all available lists'</li><li>'give me the details on purchase order'</li><li>'find the list'</li></ul>                                                                             |
| lists_remove             | <ul><li>'replace'</li><li>"delete my to do's for this week"</li><li>'get rid of tax list from nineteen ninety'</li></ul>                                                                            |
| music_likeness           | <ul><li>'store opinion on song'</li><li>'are there any upcoming concerts by'</li><li>'enter song suggestion'</li></ul>                                                                              |
| music_query              | <ul><li>'is the song by shakira'</li><li>'which film the music comes from what is the name of the music'</li><li>'which song is this one'</li></ul>                                                 |
| news_query               | <ul><li>'news articles on a particular subject'</li><li>'get me match highlights'</li><li>'show me the latest news from the guardian'</li></ul>                                                     |
| play_audiobook           | <ul><li>'continue the last chapter of the audio book i was listening to'</li><li>'open davinci code audiobook'</li><li>'resume the playback of a child called it'</li></ul>                         |
| play_game                | <ul><li>'bring up papa pear saga'</li><li>'play ping pong'</li><li>'play racing'</li></ul>                                                                                                          |
| play_music               | <ul><li>'play mf doom anything'</li><li>'play only all music released between the year one thousand nine hundred and ninety and two thousand'</li><li>'nobody knows'</li></ul>                      |
| play_podcasts            | <ul><li>'play all order of the green hand from previous week'</li><li>'i want to see the next podcast available'</li><li>"search for podcasts that cover men's issues"</li></ul>                    |
| play_radio               | <ul><li>'can you turn on the radio'</li><li>'play country radio'</li><li>'tune to classic hits'</li></ul>                                                                                           |
| qa_currency              | <ul><li>'let me know about the exchange rate of rupee to dirham'</li><li>'how much is one dollar in pounds'</li><li>'what is the most current exchange rate in china'</li></ul>                     |
| qa_definition            | <ul><li>'define elaborate'</li><li>'look up the definition of blunder'</li><li>'give details of rock sand'</li></ul>                                                                                |
| qa_factoid               | <ul><li>'where are the rocky mountains'</li><li>'what is the population of new york'</li><li>'where is new zealand located on a map'</li></ul>                                                      |
| recommendation_events    | <ul><li>'are there any fun events in la today'</li><li>"what's happening around me"</li><li>'are there any crafts fairs happening in this area'</li></ul>                                           |
| recommendation_locations | <ul><li>'what is the nearest pizza shop'</li><li>'please look up local restaurants that are open now'</li><li>'tell me what clothing stores are within five miles of me'</li></ul>                  |
| social_post              | <ul><li>"tweet at united airlines i'm angry you lost my bags"</li><li>'send a funny message to all of my friends'</li><li>'tweet my current location'</li></ul>                                     |
| takeaway_query           | <ul><li>'could you please confirm if paradise does takeaway'</li><li>"i've canceled the order placed at mcd did it go through"</li><li>"please find out of charley's steakhouse delivers"</li></ul> |
| transport_query          | <ul><li>'directions please'</li><li>'what time does the train to place leave'</li><li>'look up the map to stores near me'</li></ul>                                                                 |
| transport_ticket         | <ul><li>'find me a train ticket to boston'</li><li>'can you please book train tickets for two for this friday'</li><li>'order a train ticket to boston'</li></ul>                                   |
| weather_query            | <ul><li>'will i need to shovel my driveway this morning'</li><li>'does the weather call for rain saturday'</li><li>'is there any rain in the forecast for the next week'</li></ul>                  |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.7743   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("aisuko/st-mpnet-v2-amazon-mi")
# Run inference
preds = model("do i need a jacket")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median | Max |
|:-------------|:----|:-------|:----|
| Word count   | 1   | 6.7114 | 19  |

| Label                    | Training Sample Count |
|:-------------------------|:----------------------|
| alarm_query              | 10                    |
| alarm_set                | 10                    |
| audio_volume_mute        | 10                    |
| calendar_query           | 10                    |
| calendar_remove          | 10                    |
| calendar_set             | 10                    |
| cooking_recipe           | 10                    |
| datetime_query           | 10                    |
| email_query              | 10                    |
| email_sendemail          | 10                    |
| general_quirky           | 10                    |
| iot_coffee               | 10                    |
| iot_hue_lightchange      | 10                    |
| iot_hue_lightoff         | 10                    |
| lists_createoradd        | 10                    |
| lists_query              | 10                    |
| lists_remove             | 10                    |
| music_likeness           | 10                    |
| music_query              | 10                    |
| news_query               | 10                    |
| play_audiobook           | 10                    |
| play_game                | 10                    |
| play_music               | 10                    |
| play_podcasts            | 10                    |
| play_radio               | 10                    |
| qa_currency              | 10                    |
| qa_definition            | 10                    |
| qa_factoid               | 10                    |
| recommendation_events    | 10                    |
| recommendation_locations | 10                    |
| social_post              | 10                    |
| takeaway_query           | 10                    |
| transport_query          | 10                    |
| transport_ticket         | 10                    |
| weather_query            | 10                    |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- body_learning_rate: (2e-05, 1e-05)
- head_learning_rate: 0.01
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: True

### Training Results
| Epoch   | Step     | Training Loss | Validation Loss |
|:-------:|:--------:|:-------------:|:---------------:|
| 0.0001  | 1        | 0.1814        | -               |
| 0.0067  | 50       | 0.1542        | -               |
| 0.0134  | 100      | 0.0953        | -               |
| 0.0202  | 150      | 0.0991        | -               |
| 0.0269  | 200      | 0.0717        | -               |
| 0.0336  | 250      | 0.0653        | -               |
| 0.0403  | 300      | 0.0412        | -               |
| 0.0471  | 350      | 0.0534        | -               |
| 0.0538  | 400      | 0.013         | -               |
| 0.0605  | 450      | 0.0567        | -               |
| 0.0672  | 500      | 0.0235        | -               |
| 0.0739  | 550      | 0.0086        | -               |
| 0.0807  | 600      | 0.0086        | -               |
| 0.0874  | 650      | 0.0786        | -               |
| 0.0941  | 700      | 0.0092        | -               |
| 0.1008  | 750      | 0.0081        | -               |
| 0.1076  | 800      | 0.0196        | -               |
| 0.1143  | 850      | 0.0138        | -               |
| 0.1210  | 900      | 0.0081        | -               |
| 0.1277  | 950      | 0.0295        | -               |
| 0.1344  | 1000     | 0.0074        | -               |
| 0.1412  | 1050     | 0.0025        | -               |
| 0.1479  | 1100     | 0.0036        | -               |
| 0.1546  | 1150     | 0.0021        | -               |
| 0.1613  | 1200     | 0.0168        | -               |
| 0.1681  | 1250     | 0.0024        | -               |
| 0.1748  | 1300     | 0.0039        | -               |
| 0.1815  | 1350     | 0.0155        | -               |
| 0.1882  | 1400     | 0.0057        | -               |
| 0.1949  | 1450     | 0.0027        | -               |
| 0.2017  | 1500     | 0.0018        | -               |
| 0.2084  | 1550     | 0.0012        | -               |
| 0.2151  | 1600     | 0.0032        | -               |
| 0.2218  | 1650     | 0.0017        | -               |
| 0.2286  | 1700     | 0.0012        | -               |
| 0.2353  | 1750     | 0.002         | -               |
| 0.2420  | 1800     | 0.0025        | -               |
| 0.2487  | 1850     | 0.0014        | -               |
| 0.2554  | 1900     | 0.0033        | -               |
| 0.2622  | 1950     | 0.0007        | -               |
| 0.2689  | 2000     | 0.0006        | -               |
| 0.2756  | 2050     | 0.001         | -               |
| 0.2823  | 2100     | 0.001         | -               |
| 0.2891  | 2150     | 0.0007        | -               |
| 0.2958  | 2200     | 0.0011        | -               |
| 0.3025  | 2250     | 0.0009        | -               |
| 0.3092  | 2300     | 0.0006        | -               |
| 0.3159  | 2350     | 0.001         | -               |
| 0.3227  | 2400     | 0.0005        | -               |
| 0.3294  | 2450     | 0.0012        | -               |
| 0.3361  | 2500     | 0.0005        | -               |
| 0.3428  | 2550     | 0.0007        | -               |
| 0.3496  | 2600     | 0.0018        | -               |
| 0.3563  | 2650     | 0.0008        | -               |
| 0.3630  | 2700     | 0.0009        | -               |
| 0.3697  | 2750     | 0.0007        | -               |
| 0.3764  | 2800     | 0.0013        | -               |
| 0.3832  | 2850     | 0.0004        | -               |
| 0.3899  | 2900     | 0.0005        | -               |
| 0.3966  | 2950     | 0.0005        | -               |
| 0.4033  | 3000     | 0.0006        | -               |
| 0.4101  | 3050     | 0.0005        | -               |
| 0.4168  | 3100     | 0.0004        | -               |
| 0.4235  | 3150     | 0.0007        | -               |
| 0.4302  | 3200     | 0.0009        | -               |
| 0.4369  | 3250     | 0.0007        | -               |
| 0.4437  | 3300     | 0.0007        | -               |
| 0.4504  | 3350     | 0.0004        | -               |
| 0.4571  | 3400     | 0.0004        | -               |
| 0.4638  | 3450     | 0.0009        | -               |
| 0.4706  | 3500     | 0.0006        | -               |
| 0.4773  | 3550     | 0.0006        | -               |
| 0.4840  | 3600     | 0.0005        | -               |
| 0.4907  | 3650     | 0.0005        | -               |
| 0.4974  | 3700     | 0.0003        | -               |
| 0.5042  | 3750     | 0.0004        | -               |
| 0.5109  | 3800     | 0.0004        | -               |
| 0.5176  | 3850     | 0.0005        | -               |
| 0.5243  | 3900     | 0.0007        | -               |
| 0.5311  | 3950     | 0.0005        | -               |
| 0.5378  | 4000     | 0.0006        | -               |
| 0.5445  | 4050     | 0.0004        | -               |
| 0.5512  | 4100     | 0.0006        | -               |
| 0.5579  | 4150     | 0.0005        | -               |
| 0.5647  | 4200     | 0.0004        | -               |
| 0.5714  | 4250     | 0.0003        | -               |
| 0.5781  | 4300     | 0.0003        | -               |
| 0.5848  | 4350     | 0.0005        | -               |
| 0.5916  | 4400     | 0.0002        | -               |
| 0.5983  | 4450     | 0.0006        | -               |
| 0.6050  | 4500     | 0.0004        | -               |
| 0.6117  | 4550     | 0.0005        | -               |
| 0.6184  | 4600     | 0.0003        | -               |
| 0.6252  | 4650     | 0.0005        | -               |
| 0.6319  | 4700     | 0.0007        | -               |
| 0.6386  | 4750     | 0.0003        | -               |
| 0.6453  | 4800     | 0.0004        | -               |
| 0.6521  | 4850     | 0.0004        | -               |
| 0.6588  | 4900     | 0.0004        | -               |
| 0.6655  | 4950     | 0.0003        | -               |
| 0.6722  | 5000     | 0.0003        | -               |
| 0.6789  | 5050     | 0.0004        | -               |
| 0.6857  | 5100     | 0.0003        | -               |
| 0.6924  | 5150     | 0.0005        | -               |
| 0.6991  | 5200     | 0.0002        | -               |
| 0.7058  | 5250     | 0.0004        | -               |
| 0.7126  | 5300     | 0.0003        | -               |
| 0.7193  | 5350     | 0.0007        | -               |
| 0.7260  | 5400     | 0.0002        | -               |
| 0.7327  | 5450     | 0.0002        | -               |
| 0.7394  | 5500     | 0.0005        | -               |
| 0.7462  | 5550     | 0.0003        | -               |
| 0.7529  | 5600     | 0.0003        | -               |
| 0.7596  | 5650     | 0.0003        | -               |
| 0.7663  | 5700     | 0.0004        | -               |
| 0.7731  | 5750     | 0.0004        | -               |
| 0.7798  | 5800     | 0.0004        | -               |
| 0.7865  | 5850     | 0.0003        | -               |
| 0.7932  | 5900     | 0.0003        | -               |
| 0.7999  | 5950     | 0.0004        | -               |
| 0.8067  | 6000     | 0.0004        | -               |
| 0.8134  | 6050     | 0.0004        | -               |
| 0.8201  | 6100     | 0.0003        | -               |
| 0.8268  | 6150     | 0.0002        | -               |
| 0.8336  | 6200     | 0.0005        | -               |
| 0.8403  | 6250     | 0.0003        | -               |
| 0.8470  | 6300     | 0.0003        | -               |
| 0.8537  | 6350     | 0.0002        | -               |
| 0.8604  | 6400     | 0.0003        | -               |
| 0.8672  | 6450     | 0.0004        | -               |
| 0.8739  | 6500     | 0.0002        | -               |
| 0.8806  | 6550     | 0.0003        | -               |
| 0.8873  | 6600     | 0.0003        | -               |
| 0.8941  | 6650     | 0.0002        | -               |
| 0.9008  | 6700     | 0.0002        | -               |
| 0.9075  | 6750     | 0.0002        | -               |
| 0.9142  | 6800     | 0.0002        | -               |
| 0.9209  | 6850     | 0.0003        | -               |
| 0.9277  | 6900     | 0.0002        | -               |
| 0.9344  | 6950     | 0.0002        | -               |
| 0.9411  | 7000     | 0.0002        | -               |
| 0.9478  | 7050     | 0.0002        | -               |
| 0.9546  | 7100     | 0.0002        | -               |
| 0.9613  | 7150     | 0.0003        | -               |
| 0.9680  | 7200     | 0.0002        | -               |
| 0.9747  | 7250     | 0.0003        | -               |
| 0.9814  | 7300     | 0.0002        | -               |
| 0.9882  | 7350     | 0.0003        | -               |
| 0.9949  | 7400     | 0.0003        | -               |
| **1.0** | **7438** | **-**         | **0.0755**      |

* The bold row denotes the saved checkpoint.
### Framework Versions
- Python: 3.10.13
- SetFit: 1.0.3
- Sentence Transformers: 2.7.0
- Transformers: 4.39.3
- PyTorch: 2.1.2
- Datasets: 2.18.0
- Tokenizers: 0.15.2

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->