File size: 2,897 Bytes
3710c26 936181c 3710c26 936181c b3087e8 936181c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
---
datasets:
- tatsu-lab/alpaca
language:
- en
pipeline_tag: text2text-generation
library_name: transformers
license: other
---
# Model Details
- **Model name:** Flan-UL2-Alpaca-LORA
- **Model type:** - Text2Text Generation
- **Parent Model:** [google/flan-ul2](https://huggingface.co/google/flan-ul2)
- **Training dataset:** [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)
- **Language:** English
- **Framework:** PyTorch
- **Model version:** 1.0
We take the instruction-tuned Flan models (trained on Academic datasets) and perform style transfer using the Alpaca dataset.
We fine-tuned the google/flan-ul2 model on the Alpaca datset using [PEFT-LORA](https://huggingface.co/docs/diffusers/main/en/training/lora).
# License
- Parent model ([google/flan-ul2](https://huggingface.co/google/flan-ul2)): Apache 2.0
- Dataset ([Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)) : cc-by-4.0
- Text-Davinci-3 (Used to generate Alpaca): [OpenAI License](https://openai.com/policies/terms-of-use)
# How to Use
```
import torch
from transformers import pipeline
# Chose the model inference precision
dtype = torch.float16 # options are torch.bfloat16, torch.float32
model = pipeline(model="VMware/flan-ul2-alpaca-lora",device_map = 'auto',torch_dtype=dtype )
prompt = "YOUR PROMPT HERE"
output = model(prompt, max_length=2048, do_sample=True)
```
Using Alpaca prompt template might generate better outputs for certain prompts as the model was trained using the template.
```
# Chose the model inference precision
import torch
from transformers import pipeline
dtype = torch.float16 # options are torch.bfloat16, torch.float32
model = pipeline(model="VMware/flan-ul2-alpaca-lora",device_map = 'auto',torch_dtype=dtype )
prompt_template = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:"
prompt = "YOUR PROMPT HERE"
output = model(prompt_template.format(instruction= prompt), max_length=2048, do_sample=True)
```
# Training Details
The model was trained on 3xV100 GPUs using PEFT-LORA and Deepspeed
* Hyperparameters:
* learning_rate = 3e-4
* per_device_batch_size = 2
* gradient_accumulation_steps = 21
* epochs = 3
```
# Limitations and Bias
The model is based on a large and diverse dataset, but it may still have limitations and biases in certain areas. Some limitations include:
- Language: The model is designed to work with English text only and may not perform as well in other languages.
In addition, the model may have some bias in terms of the data it was trained on. The dataset includes questions from a variety of sources, but it may not be representative of all populations or perspectives. As a result, the model may perform better or worse for certain types of questions or on certain types of texts.
# Contribution |