README.md

---
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
library_name: peft
---

# SCoReLoRA: Self-Correct via Reinforcement Learning

SCoReLoRA is an innovative approach to fine-tuning language models using Low-Rank Adaptation (LoRA) combined with reinforcement learning techniques for self-correction. This method aims to improve the model's ability to generate more accurate and refined responses through a two-stage training process.

## Features

- Implements a two-stage training process for self-correction
- Utilizes reinforcement learning to improve model outputs
- Compatible with Hugging Face's Transformers library and PEFT
- Supports quantized models for efficient fine-tuning
- Includes evaluation metrics for self-correction performance

## How It Works

SCoreLora uses a two-stage training process:

1. **Stage I**: The model is trained to generate initial responses and then correct them, minimizing the KL divergence between the base model and the fine-tuned model.

2. **Stage II**: The model is further trained using reinforcement learning techniques, with rewards based on the quality of self-corrections.

The training process utilizes shaped rewards and KL divergence to balance between improvement and staying close to the original model's behavior.

## Evaluation

The implementation includes functions to evaluate the model's self-correction capabilities, measuring metrics such as:

- Accuracy before and after correction
- Improvement rate
- Rate of successful corrections
- Rate of erroneous corrections

## Reference

- [Training Language Models to Self-Correct via Reinforcement Learning](https://arxiv.org/abs/2409.12917)