File size: 2,472 Bytes
17cd953 748da8d 17cd953 c93d072 17cd953 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
tags:
- flan
- opt
- peft
datasets:
- SirNeural/flan_v2
metrics:
- perplexity
base_model: facebook/opt-2.7b
---
## FLAN-OPT-2.7b-LoRA
OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
This model is [facebook/opt-2.7b](https://hf.co/facebook/opt-2.7b) finetuned with low-rank adapters (https://arxiv.org/abs/2106.09685) on the FLAN datasets (https://arxiv.org/pdf/2210.11416.pdf).
Low-rank adapters (r=16) finetuned over 1.1m new tokens of a FLAN task mixture, with the start of each example cut off if it was too large to fit within a 256 token context.
The model reaches a train ppl of 5.09 and an eval ppl of 4.36.
### Inference Example (Chain-of-Thought prompt):
```python
# %pip install -qq transformers git+https://github.com/huggingface/peft accelerate bitsandbytes
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "crumb/FLAN-OPT-2.7b-LoRA"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, load_in_8bit=True, low_cpu_mem_usage=True, device_map='auto')
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
import torch
prompt = """
Q: Answer the following yes/no question by reasoning step-by-step. Could a dandelion suffer from hepatitis?
A: Hepatitis only affects organisms with livers. Dandelions don’t have a liver. The answer is no.
Q: Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?
A: A haiku is a japanese three-line poem. That is short enough to fit in 280 characters. The answer is yes.
Q: Answer the following yes/no question by reasoning step-by-step. Can you reach space with a Cessna?
A:
""".strip()
inputs = tokenizer([prompt], return_tensors='pt')
with torch.autocast("cuda", dtype=torch.float16):
outputs = model.generate(
input_ids=inputs.input_ids.cuda(),
attention_mask=inputs.attention_mask.cuda(),
max_new_tokens=32,
top_k=4,
penalty_alpha=0.6
)
print("\n".join(tokenizer.decode(outputs[0]).split("\n")[:prompt.count("\n")+1]))
# Cessna is a single-engine aircraft. Cessna cannot reach space. The answer is no.
``` |