Safetensors
English
Italian
deberta-v2
File size: 2,391 Bytes
a0f57bd
 
 
 
a83ba10
 
 
 
 
 
a0f57bd
f98e7c6
 
 
a83ba10
f98e7c6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a83ba10
f98e7c6
a83ba10
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
---
license: apache-2.0
base_model:
- microsoft/mdeberta-v3-base
datasets:
- saiteki-kai/BeaverTails-it
- PKU-Alignment/BeaverTails
language:
- en
- it
---
# Oracolo Model Card

## Model Description
Oracolo is a DeBERTa-based content moderation model trained to detect harmful content across multiple safety categories in Italian and English. The model analyzes both questions and answers in conversational contexts to determine potential safety risks.

## Intended Use
- Content moderation for conversational AI systems
- Safety classification of question-answer pairs
- Multi-label classification across safety categories

## Training Data
The model was trained on the BeaverTails dataset, which contains labeled examples of safe and unsafe conversational content.

## Model Architecture
- Base model: Microsoft DeBERTa
- Multi-label classification head
- Maximum sequence length: 512 tokens

## Performance

### Overall Safety Classification
Comparison with PKU-Alignment/beaver-dam-7b on test set:

| Model | Precision | Recall | F1-Score |
|-------|-----------|--------|----------|
| Oracolo (DeBERTa) | 0.85 | 0.85 | 0.85 |
| beaver-dam-7b | 0.77 | 0.88 | 0.87 |

### Usage Example
```python

  from transformers import AutoTokenizer, AutoModelForSequenceClassification
  import torch
  def preprocess_text(prompt, response=""):
    """Format text in the same way as during training."""
    return f"<prompt> {prompt} </prompt> <response> {response} </response>"

  model = AutoModelForSequenceClassification.from_pretrained("path/to/oracolo")
  tokenizer = AutoTokenizer.from_pretrained("path/to/oracolo")
  model.eval()
  prompt = "How do I make a walkway slippery?"
  response = "I cannot provide advice that could lead to harm."
  formatted_text = preprocess_text(prompt, response)
  inputs = tokenizer(formatted_text, return_tensors="pt", truncation=True, max_length=512)
  with torch.no_grad():
    outputs = model(inputs)
    predictions = torch.sigmoid(outputs.logits).cpu().numpy()[0]
    # Apply threshold (0.3 recommended based on validation)
    class_predictions = (predictions > 0.3).astype(int)

```
## Full classification report


![image/png](https://cdn-uploads.huggingface.co/production/uploads/648cca46d38113f34bf7cb72/ZYibWymHDCn04xIforKTg.png)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/648cca46d38113f34bf7cb72/f0TAV46Mrjclzkww1G3Ik.png)