metadata
datasets:
- Alpaca69B/semeval2016-full-absa-reviews-english-translated-resampled-train
language:
- en
pipeline_tag: text-generation
tags:
- absa
- qlora
Model Card
Model Details
- Model Name: Alpaca69B/llama-2-7b-absa-semeval-2016
- Base Model: NousResearch/Llama-2-7b-chat-hf
- Fine-Tuned On: Alpaca69B/semeval2016-full-absa-reviews-english-translated-resampled-train
- Fine-Tuning Techniques: LoRA attention, 4-bit precision base model loading, gradient checkpointing, etc.
- Training Resources: Low resource usage
Model Description
This model is an aspect based sentiment analysis model fine-tuned from the Llama-2-7b-chat model on an adjusted semeval-2016 dataset.
Fine-Tuning Techniques
LoRA Attention
- LoRA attention dimension: 64
- Alpha parameter for LoRA scaling: 16
- Dropout probability for LoRA layers: 0.1
bitsandbytes (4-bit precision)
- Activated 4-bit precision base model loading
- Compute dtype for 4-bit base models: "float16"
- Quantization type: "nf4"
- Nested quantization for 4-bit base models: Disabled
TrainingArguments
- Output directory: "./results"
- Number of training epochs: 1
- Enabled fp16/bf16 training: False
- Batch size per GPU for training: 4
- Batch size per GPU for evaluation: 4
- Gradient accumulation steps: 1
- Enabled gradient checkpointing: True
- Maximum gradient norm (gradient clipping): 0.3
- Initial learning rate: 2e-4
- Weight decay: 0.001
- Optimizer: paged_adamw_32bit
- Learning rate scheduler: cosine
- Maximum training steps: -1 (overrides num_train_epochs)
- Ratio of steps for linear warmup: 0.03
- Group sequences into batches with the same length: True
- Save checkpoint every X update steps: 0 (disabled)
- Log every X update steps: 25
SFT (Sequence-level Fine-Tuning)
- Maximum sequence length: Not specified
- Packing multiple short examples in the same input sequence: False
- Load the entire model on GPU 0
Evaluation
The model's performance and usage can be observed in the provided Google Colab notebook.
Model Usage
To use the model, follow the provided code snippet:
from transformers import AutoTokenizer
import transformers
import torch
model = "Alpaca69B/llama-2-7b-absa-semeval-2016"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
def process_user_prompt(input_sentence):
sequences = pipeline(
f'### Human: {input_sentence} ### Assistant: aspect: ',
do_sample=True,
top_k=10,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
max_length=200,
)
result_dict = process_output(sequences[0]['generated_text'])
return result_dict
def process_output(output):
result_dict = {}
# Extract user_prompt
user_prompt_start = output.find("### Human:")
user_prompt_end = output.find("aspect: ") + len("aspect: ")
result_dict['user_prompt'] = output[user_prompt_start:user_prompt_end].strip()
# Extract cleared_generated_output
cleared_output_end = output.find(")")
result_dict['cleared_generated_output'] = output[:cleared_output_end+1].strip()
# Extract review
human_start = output.find("Human:") + len("Human:")
assistant_start = output.find("### Assistant:")
result_dict['review'] = output[human_start:assistant_start].strip()
# Extract aspect and sentiment
aspect_start = output.find("aspect: ") + len("aspect: ")
sentiment_start = output.find("sentiment: ")
aspect_text = output[aspect_start:sentiment_start].strip()
result_dict['aspect'] = aspect_text
sentiment_end = output[sentiment_start:].find(")") + sentiment_start
sentiment_text = output[sentiment_start+len("sentiment:"):sentiment_end].strip()
result_dict['sentiment'] = sentiment_text
return result_dict
output = process_user_prompt('the first thing that attracts attention is the warm reception and the smiling receptionists.')
print(output)
Fine-Tuning Details
Details of the fine-tuning process are available in the fine-tuning Colab notebook.
Note: Ensure that you have the necessary dependencies and resources before running the model.