File size: 5,166 Bytes
0641a71 2946e2f 0641a71 4656afb 0641a71 0185e3a 2d50cb3 df86f11 0185e3a 2d50cb3 0185e3a 0641a71 0185e3a 0641a71 4656afb 0641a71 0185e3a 2946e2f 0185e3a 2946e2f 0185e3a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
---
license: cc
tags:
- trl
- transformers
- reinforcement-learning
- LLM detoxification
datasets:
- ProlificAI/social-reasoning-rlhf
language:
- en
pipeline_tag: conversational
---
# TRL Model
This is a [TRL language model](https://github.com/huggingface/trl). It has been fine-tuned with reinforcement learning to
guide the model outputs according to a value, function, or human feedback. The model can be used for text generation.
This project aims to reduce the level of toxicity in the outputs generated by the LAMINI Flan T5 248M language model using
Reinforcement Learning with Artificial Intelligence Feedback technique (RLAIF). Reinforcement Learning with Human Feedback (RLHF)
is a method to align models with a particular kind of data. RLHF creates a latent reward model using human feedback and finetunes
a model using Proximal Policy Optimization. RLAIF on the other hand replaces human feedback with a high-performance AI agent. The model
has been fine-tuned on the [Social Reasoning Dataset](https://huggingface.co/datasets/ProlificAI/social-reasoning-rlhf/viewer/default/train?p=38&row=3816) by
ProlificAI for 191 steps and 1 epoch using the Proximal Policy Optimisation (PPO) algorithm. The [Roberta hate text detection](https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target)
model was used as the Proximal Policy Optimisation (PPO) reward model.
The power of this model lies in its size; it is barely 500 MBs in size and performs well given its size. The intended use of this model should be conversation, text generation, or context-based Q&A.
This model might not perform well on tasks like mathematics, sciences, coding, etc. It might hallucinate on such tasks. After quantization, this model could be easily run on edge devices like smartphones and microprocessors.
The training log of the model can be found in this [weights and biases](https://wandb.ai/tnarahul/trl/runs/nk30wukt/overview?workspace=user-tnarahul) page.
Note: This model is a fine-tuned version of [LaMini Flan T5 248M](https://huggingface.co/MBZUAI/LaMini-Flan-T5-248M), which in turn is a fine-tuned version of the Flan T5 model released by Google. The Flan T5 follows the encoder-decoder architecture, unlike other GPT-like models that are decoder-only.
## Usage
To use this model for inference, first install the TRL library:
```bash
python -m pip install trl
```
You can then generate text as follows:
```python
from trl import AutoModelForSeq2SeqLMWithValueHead
from transformers import pipeline, AutoTokenizer
import torch
checkpoint = "ARahul2003/lamini_flan_t5_detoxify_rlaif"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
base_model = AutoModelForSeq2SeqLMWithValueHead.from_pretrained(checkpoint,
device_map='cpu', #or 'auto'/'cuda:0'
torch_dtype=torch.float32)
pipe = pipeline('text2text-generation',
model = base_model,
tokenizer = tokenizer,
max_length = 512,
do_sample=True,
temperature=0.3,
top_p=0.95,
)
prompt = 'Hello! How are you?'
print(pipe(prompt)[0]['generated_text'])
```
If you want to use the model for training or to obtain the outputs from the value head, load the model as follows:
```python
from transformers import AutoTokenizer
from trl import AutoModelForCausalLMWithValueHead
tokenizer = AutoTokenizer.from_pretrained("ARahul2003/lamini_flan_t5_detoxify_rlaif")
model = AutoModelForCausalLMWithValueHead.from_pretrained("ARahul2003/lamini_flan_t5_detoxify_rlaif")
inputs = tokenizer("Hello, my llama is cute", return_tensors="pt")
outputs = model(**inputs, labels=inputs["input_ids"])
```
If you want to use the model for inference in a gradio app, consider the following code:
```python
from trl import AutoModelForSeq2SeqLMWithValueHead
from transformers import pipeline, AutoTokenizer
import torch
import gradio as gr
title = "LaMini Flan T5 248M"
checkpoint = "ARahul2003/lamini_flan_t5_detoxify_rlaif"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
base_model = AutoModelForSeq2SeqLMWithValueHead.from_pretrained(checkpoint,
device_map='cpu', #or 'auto'
torch_dtype=torch.float32)
pipe = pipeline('text2text-generation',
model = base_model,
tokenizer = tokenizer,
max_length = 512,
do_sample=True,
temperature=0.3,
top_p=0.95,
)
def chat_with_model(inp_chat, chat_history = None):
prompt = f"{inp_chat}" #f"User: {inp_chat} Bot:"
responses = pipe(prompt)
return responses[0]['generated_text']
examples = [
'Hi!',
'How are you?',
'Please let me know your thoughts on the given place and why you think it deserves to be visited: \n"Barcelona, Spain"'
]
gr.ChatInterface(
fn=chat_with_model,
title=title,
examples=examples
).launch()
```
Make sure to keep all the tensors on the same device (CPU/GPU). |