File size: 2,614 Bytes
7fae170 8def050 7fae170 8def050 6e7ed4d 8def050 5ea6c15 b352642 8def050 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
---
datasets:
- PKU-Alignment/PKU-SafeRLHF
language:
- en
tags:
- reinforcement-learning-from-human-feedback
- reinforcement-learning
- beaver
- safety
- llama
- ai-safety
- deepspeed
- rlhf
- alpaca
library_name: safe-rlhf
---
# 🦫 Beaver's Reward Model
## Model Details
The Beaver reward model is a preference model trained using the [PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) dataset.
It can play a role in the safe RLHF algorithm, helping the Beaver model become more helpful.
- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team.
- **Model Type:** An auto-regressive language model based on the transformer architecture.
- **License:** Non-commercial license.
- **Fine-tuned from model:** [LLaMA](https://arxiv.org/abs/2302.13971), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca).
## Model Sources
- **Repository:** <https://github.com/PKU-Alignment/safe-rlhf>
- **Beaver:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0>
- **Dataset:** <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF>
- **Reward Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-reward>
- **Cost Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost>
- **Dataset Paper:** <https://arxiv.org/abs/2307.04657>
- **Paper:** *Coming soon...*
## How to Use the Reward Model
```python
from transformers import AutoTokenizer
from safe_rlhf.models import AutoModelForScore
model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v1.0-reward', device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v1.0-reward', use_fast=False)
input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'
input_ids = tokenizer(input, return_tensors='pt')
output = model(**input_ids)
print(output)
# ScoreModelOutput(
# scores=tensor([[[-19.6476],
# [-20.2238],
# [-21.4228],
# [-19.2506],
# [-20.2728],
# [-23.8799],
# [-22.6898],
# [-21.5825],
# [-21.0855],
# [-20.2068],
# [-23.8296],
# [-21.4940],
# [-21.9484],
# [-13.1220],
# [ -6.4499],
# [ -8.1982],
# [ -7.2492],
# [ -9.3377],
# [-13.5010],
# [-10.4932],
# [ -9.7837],
# [ -6.4540],
# [ -6.0084],
# [ -5.8093],
# [ -6.6134],
# [ -5.8995],
# [ -9.1505],
# [-11.3254]]], grad_fn=<ToCopyBackward0>),
# end_scores=tensor([[-11.3254]], grad_fn=<ToCopyBackward0>)
# )
```
|