Rakesh7n/Qwen2.5-0.5_alpaca-finance_finetuned

Model Description

This model is a fine-tuned version of Qwen/Qwen-0.5B on the Alpaca-1K finance dataset using LoRA (Low-Rank Adaptation). It is specifically optimized for financial domain tasks and conversations.

Model Details

Base Model: Qwen-0.5B
Training Type: LoRA fine-tuning
Domain: Finance
Language: English
Training Precision: FP32

Training Data

This model was fine-tuned on the Alpaca-1K finance dataset, which consists of 1,000 carefully curated financial conversations and instructions. The dataset covers various financial topics including:

Investment analysis
Market trends
Financial planning
Risk assessment
Economic indicators

Training Procedure

Fine-tuning Method: LoRA (Low-Rank Adaptation)
Precision: FP32
LoRA Hyperparameters:
- Rank: 64
- Alpha: 32
- Target modules: ["q_proj","k_proj","v_proj","o_proj","gate_proj","down_proj","up_proj","lm_head"]
Hardware Constraints: Limited GPU resources

Limitations

The model's knowledge is limited to the financial domain
Performance on non-financial tasks may be suboptimal
Training was conducted under GPU constraints which may impact model performance
Current loss metrics indicate room for potential optimization

Performance

Training Loss: ~1.52
Validation Loss: ~1.5
Note: The model was trained with hardware constraints and a focus on efficiency using LoRA rank 64. While the loss metrics indicate room for improvement, the model maintains consistent performance between training and validation sets, suggesting stable learning within the given constraints.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig
peft_model_id = "Rakesh7n/Qwen2.5-0.5_alpaca-finance_finetuned"  
model = AutoModelForCausalLM.from_pretrained(peft_model_id, torch_dtype=torch.float16, device_map='auto')
peft_config = PeftConfig.from_pretrained(peft_model_id)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)