File size: 5,434 Bytes
0b3bb4e
 
 
 
 
 
 
f057b32
 
0b3bb4e
 
442c8ce
f057b32
220be08
 
70cd4d1
f7fcbfb
0b3bb4e
70cd4d1
 
 
9f2d78b
bf2c834
9f2d78b
 
0b3bb4e
81ab2c6
 
 
 
 
 
b9ad619
 
 
 
 
 
824f730
d862046
824f730
b9ad619
 
 
 
 
 
 
 
 
 
 
 
 
81ab2c6
0b3bb4e
 
 
 
 
 
 
 
81ab2c6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0b3bb4e
 
 
 
81ab2c6
d862046
81ab2c6
 
 
 
 
 
 
 
 
 
 
 
0b3bb4e
 
9f2d78b
81ab2c6
 
 
 
 
 
 
 
 
 
 
 
9f2d78b
 
ed2d450
9f2d78b
 
81ab2c6
 
9f2d78b
 
 
 
 
 
 
bf2c834
9f2d78b
 
 
 
 
bf2c834
9f2d78b
 
 
 
bf2c834
9f2d78b
 
 
 
81ab2c6
0b3bb4e
 
 
 
 
 
 
 
 
 
 
 
 
f057b32
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
---
license: apache-2.0
tags:
- setfit
- sentence-transformers
- text-classification
pipeline_tag: text-classification
datasets:
- argilla/alpaca-gigo-detector
---

# 🚮 🦙 Alpaca GarbageCollector

> [Announcement tweet](https://twitter.com/dvilasuero/status/1643234487386374148?s=20)

A cross-lingual SetFit model to **detect bad instructions from Alpaca Datasets** and other instruction-following datasets. 
`GarbageCollector` can greatly speed up the validation of instruction-datasets across many languages, flagging examples that need to be fixed or simply discarded.

Data quality is key for  LLMs, but open-source LLMs are being built with data of "unknown" quality. This model can help practitioners to find and fix frequent issues (e.g., the model hallucinating stock prices, describing non-existing images, etc.) 

The model has been fine-tuned with 1,000 labeled examples from the AlpacaCleaned dataset labeled with [Argilla](https://www.argilla.io/). It leverages a multilingual sentence transformer `paraphrase-multilingual-mpnet-base-v2`, inspired by the findings from the SetFit paper (Section 6. Multilingual experiments.), where they trained models in English that performed well across languages.

<div style="text-align:center">
    <img src="https://huggingface.co/argilla/alpaca-hallucihunter-multilingual/resolve/main/front-image.png" alt="Alpaca Cleaned"">
</div>


It's a binary classifier with two labels:

- `ALL GOOD`, a given instruction, input, and output are correct,
- `BAD INSTRUCTION`, there's an issue with the instruction, and/or input and output.

This model can be used as follows (see full usage instructions below):

```python
from setfit import SetFitModel

# Download from Hub
model = SetFitModel.from_pretrained(
  "argilla/alpaca-garbage-collector-multilingual"
)

text = """
INSTRUCTION: 
Gebt mir drei Adjektive, um dieses Foto zu beschreiben.
INPUT: 
[photo]
OUTPUT: 
Auffällig, lebhaft, ruhig.
"""
model.predict([text])
```
Output: `BAD INSTRUCTION`


## Usage

To use this model for inference, first install the SetFit library:

```bash
python -m pip install setfit
```

Load your Alpaca Dataset:

```bash
from datasets import Dataset, load_dataset

import pandas as pd

# this can be a translation (e.g., Spanish, Camoscio Italian Alpaca, etc.)
dataset = pd.read_json("https://github.com/gururise/AlpacaDataCleaned/raw/main/alpaca_data_cleaned.json")

dataset["id"] = [i for i in range(len(dataset))]

ds = Dataset.from_pandas(dataset)
```

Create a text field containing the instruction, input and output to use for inference:

```python
def transform(r):
  return {
      "text": f"INSTRUCTION:\n{r['instruction']}\nINPUT:\n{r['input']}\nOUTPUT:\n{r['output']}\n"
  }
ds = ds.map(transform)
```

Load the model:

```python
from setfit import SetFitModel

# Download from Hub
model = SetFitModel.from_pretrained("argilla/alpaca-garbage-collector-multilingual")
```

Perform inference and prediction col to your dataset:
```python
labels = ["ALL GOOD", "BAD INSTRUCTION"]

def get_predictions(texts):
    probas = model.predict_proba(texts, as_numpy=True)
    for pred in probas:
        yield [{"label": label, "score": score} for label, score in zip(labels, pred)]

ds = ds.map(lambda batch: {"prediction": list(get_predictions(batch["text"]))}, batched=True)
```

Load the data into Argilla for exploration and validation. First, you [need to launch Argilla](https://www.argilla.io/blog/launching-argilla-huggingface-hub). Then run:
```python
# Replace api_url with the url to your HF Spaces URL if using Spaces
# Replace api_key if you configured a custom API key
rg.init(
    api_url="https://your-agilla-instance.hf.space", 
    api_key="team.apikey"
)

rg_dataset = rg.DatasetForTextClassification().from_datasets(ds)
rg.log(records=rg_dataset, name="alpaca_to_clean")
```

## Live demo

You can explore the dataset using [this Space](https://huggingface.co/spaces/argilla/alpaca-hallucihunter) (credentials: `argilla` / `1234`):


## Examples

This model has been tested with English, German, and Spanish. This approach will be used by ongoing efforts for improving the quality of Alpaca-based datasets, and updates will be reflected here.

Here are some examples of highest scored examples of `BAD INSTRUCTION`.


### English

<div style="text-align:center">
    <img src="https://huggingface.co/argilla/alpaca-hallucihunter-multilingual/resolve/main/front-image.png" alt="Alpaca Cleaned"">
</div>

### German

<div style="text-align:center">
    <img src="https://huggingface.co/argilla/alpaca-hallucihunter-multilingual/resolve/main/german-alpaca.png" alt="Alpaca Cleaned"">
</div>

### Spanish
<div style="text-align:center">
    <img src="https://huggingface.co/argilla/alpaca-hallucihunter-multilingual/resolve/main/spanish-alpaca.png" alt="Alpaca Cleaned"">
</div>



## BibTeX entry and citation info

```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
doi = {10.48550/ARXIV.2209.11055},
url = {https://arxiv.org/abs/2209.11055},
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Efficient Few-Shot Learning Without Prompts},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```