|
--- |
|
datasets: |
|
- PKU-Alignment/PKU-SafeRLHF |
|
language: |
|
- en |
|
tags: |
|
- reinforcement-learning-from-human-feedback |
|
- reinforcement-learning |
|
- beaver |
|
- safety |
|
- llama |
|
- ai-safety |
|
- deepspeed |
|
- rlhf |
|
- alpaca |
|
library_name: safe-rlhf |
|
--- |
|
|
|
# 🦫 Beaver's Reward Model |
|
|
|
## Model Details |
|
|
|
The Beaver reward model is a preference model trained using the [PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) dataset. |
|
It can play a role in the safe RLHF algorithm, helping the Beaver model become more helpful. |
|
|
|
- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team. |
|
- **Model Type:** An auto-regressive language model based on the transformer architecture. |
|
- **License:** Non-commercial license. |
|
- **Fine-tuned from model:** [LLaMA](https://arxiv.org/abs/2302.13971), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca). |
|
|
|
## Model Sources |
|
|
|
- **Repository:** <https://github.com/PKU-Alignment/safe-rlhf> |
|
- **Beaver:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0> |
|
- **Dataset:** <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF> |
|
- **Reward Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-reward> |
|
- **Cost Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-cost> |
|
- **Dataset Paper:** <https://arxiv.org/abs/2307.04657> |
|
- **Paper:** <https://arxiv.org/abs/2310.12773> |
|
|
|
## How to Use the Reward Model |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer |
|
from safe_rlhf.models import AutoModelForScore |
|
|
|
model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v3.0-reward', torch_dtype=torch.bfloat16, device_map='auto') |
|
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v3.0-reward') |
|
|
|
input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?' |
|
|
|
input_ids = tokenizer(input, return_tensors='pt') |
|
output = model(**input_ids) |
|
print(output) |
|
|
|
# ScoreModelOutput( |
|
# scores=tensor([[[-14.0000], |
|
# [ -2.6094], |
|
# [ -2.6562], |
|
# [ -2.0312], |
|
# [ -1.2188], |
|
# [ -1.6250], |
|
# [ -2.4688], |
|
# [ -2.7500], |
|
# [ -3.0000], |
|
# [ -6.0000], |
|
# [ -5.0625], |
|
# [ -7.0938], |
|
# [ -6.9688], |
|
# [ -4.3125], |
|
# [ -4.2188], |
|
# [ -3.7969], |
|
# [ -3.6875], |
|
# [ -3.3750], |
|
# [ -2.8906], |
|
# [ -3.9219], |
|
# [ -2.1406], |
|
# [ -1.7578], |
|
# [ 0.4629], |
|
# [ 2.1719], |
|
# [ 4.4062], |
|
# [ 7.1562], |
|
# [ 7.7188], |
|
# [ 10.7500]]], grad_fn=<ToCopyBackward0>), |
|
# end_scores=tensor([[10.7500]], grad_fn=<ToCopyBackward0>), |
|
# last_hidden_state=tensor([[[ 0.4805, -0.4863, -0.9258, ..., -0.0718, 0.8555, 0.6641], |
|
# [ 0.2021, 2.0156, 3.5156, ..., -0.9844, -1.1484, 1.3203], |
|
# [ 1.0938, 1.4609, 1.7891, ..., -3.2031, -0.8555, -1.2969], |
|
# ..., |
|
# [ 1.5859, 0.1348, 0.0322, ..., -1.3672, -1.5234, 1.5156], |
|
# [ 0.9102, 0.6367, -0.8555, ..., -1.2109, -0.6953, 1.5312], |
|
# [ 1.7188, 0.4434, -0.5586, ..., -1.1484, -0.7461, 2.2031]]], |
|
# dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>), |
|
# end_last_hidden_state=tensor([[ 1.7188, 0.4434, -0.5586, ..., -1.1484, -0.7461, 2.2031]], |
|
# dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>), |
|
# end_index=tensor([27]) |
|
# ) |
|
``` |
|
|
