|
--- |
|
datasets: |
|
- Alpaca69B/semeval2016-full-absa-reviews-english-translated-resampled-train |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- absa |
|
- qlora |
|
--- |
|
# Model Card |
|
|
|
## Model Details |
|
|
|
- **Model Name:** Alpaca69B/llama-2-7b-absa-semeval-2016 |
|
- **Base Model:** NousResearch/Llama-2-7b-chat-hf |
|
- **Fine-Tuned On:** Alpaca69B/semeval2016-full-absa-reviews-english-translated-resampled-train |
|
- **Fine-Tuning Techniques:** LoRA attention, 4-bit precision base model loading, gradient checkpointing, etc. |
|
- **Training Resources:** Low resource usage |
|
|
|
## Model Description |
|
|
|
This model is an aspect based sentiment analysis model fine-tuned from the Llama-2-7b-chat model on an adjusted semeval-2016 dataset. |
|
|
|
## Fine-Tuning Techniques |
|
|
|
### LoRA Attention |
|
- LoRA attention dimension: 64 |
|
- Alpha parameter for LoRA scaling: 16 |
|
- Dropout probability for LoRA layers: 0.1 |
|
|
|
### bitsandbytes (4-bit precision) |
|
- Activated 4-bit precision base model loading |
|
- Compute dtype for 4-bit base models: "float16" |
|
- Quantization type: "nf4" |
|
- Nested quantization for 4-bit base models: Disabled |
|
|
|
### TrainingArguments |
|
- Output directory: "./results" |
|
- Number of training epochs: 1 |
|
- Enabled fp16/bf16 training: False |
|
- Batch size per GPU for training: 4 |
|
- Batch size per GPU for evaluation: 4 |
|
- Gradient accumulation steps: 1 |
|
- Enabled gradient checkpointing: True |
|
- Maximum gradient norm (gradient clipping): 0.3 |
|
- Initial learning rate: 2e-4 |
|
- Weight decay: 0.001 |
|
- Optimizer: paged_adamw_32bit |
|
- Learning rate scheduler: cosine |
|
- Maximum training steps: -1 (overrides num_train_epochs) |
|
- Ratio of steps for linear warmup: 0.03 |
|
- Group sequences into batches with the same length: True |
|
- Save checkpoint every X update steps: 0 (disabled) |
|
- Log every X update steps: 25 |
|
|
|
### SFT (Sequence-level Fine-Tuning) |
|
- Maximum sequence length: Not specified |
|
- Packing multiple short examples in the same input sequence: False |
|
- Load the entire model on GPU 0 |
|
|
|
## Evaluation |
|
|
|
The model's performance and usage can be observed in the provided [Google Colab notebook](https://colab.research.google.com/drive/1ArLpQFfXJiHcAT3VuYZndDqvskM6SkOM?usp=sharing). |
|
|
|
## Model Usage |
|
|
|
To use the model, follow the provided code snippet: |
|
|
|
```python |
|
from transformers import AutoTokenizer |
|
import transformers |
|
import torch |
|
|
|
model = "Alpaca69B/llama-2-7b-absa-semeval-2016" |
|
tokenizer = AutoTokenizer.from_pretrained(model) |
|
pipeline = transformers.pipeline( |
|
"text-generation", |
|
model=model, |
|
torch_dtype=torch.float16, |
|
device_map="auto", |
|
) |
|
|
|
def process_user_prompt(input_sentence): |
|
sequences = pipeline( |
|
f'### Human: {input_sentence} ### Assistant: aspect: ', |
|
do_sample=True, |
|
top_k=10, |
|
num_return_sequences=1, |
|
eos_token_id=tokenizer.eos_token_id, |
|
max_length=200, |
|
) |
|
result_dict = process_output(sequences[0]['generated_text']) |
|
return result_dict |
|
|
|
def process_output(output): |
|
result_dict = {} |
|
|
|
# Extract user_prompt |
|
user_prompt_start = output.find("### Human:") |
|
user_prompt_end = output.find("aspect: ") + len("aspect: ") |
|
result_dict['user_prompt'] = output[user_prompt_start:user_prompt_end].strip() |
|
|
|
# Extract cleared_generated_output |
|
cleared_output_end = output.find(")") |
|
result_dict['cleared_generated_output'] = output[:cleared_output_end+1].strip() |
|
|
|
# Extract review |
|
human_start = output.find("Human:") + len("Human:") |
|
assistant_start = output.find("### Assistant:") |
|
result_dict['review'] = output[human_start:assistant_start].strip() |
|
|
|
# Extract aspect and sentiment |
|
aspect_start = output.find("aspect: ") + len("aspect: ") |
|
sentiment_start = output.find("sentiment: ") |
|
aspect_text = output[aspect_start:sentiment_start].strip() |
|
result_dict['aspect'] = aspect_text |
|
|
|
sentiment_end = output[sentiment_start:].find(")") + sentiment_start |
|
sentiment_text = output[sentiment_start+len("sentiment:"):sentiment_end].strip() |
|
result_dict['sentiment'] = sentiment_text |
|
|
|
return result_dict |
|
|
|
|
|
output = process_user_prompt('the first thing that attracts attention is the warm reception and the smiling receptionists.') |
|
print(output) |
|
|
|
``` |
|
|
|
## Fine-Tuning Details |
|
|
|
Details of the fine-tuning process are available in the [fine-tuning Colab notebook](https://colab.research.google.com/drive/1PQfBsDyM8TSSBchL6PPA4o6rOyLFLUnu?usp=sharing). |
|
|
|
|
|
**Note:** Ensure that you have the necessary dependencies and resources before running the model. |