D-Khalid/qwen2-7b-instruct-compliance-cot

Uploaded model

• Developed by: D-Khalid
• License: apache-2.0
• Finetuned from model : unsloth/Qwen2-7B-Instruct-bnb-4bit

This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

Step 1: Set Up the Notebook

1. Create a New Notebook: Start a fresh notebook in your preferred environment (Colab, Kaggle, etc.).

2. Enable a GPU: This is essential for the model to run.

3. Add Your Hugging Face Token: Securely add your HF_TOKEN as a secret. This is needed to download your model from the Hub.

Step 2: Install and Load Your Model

This single code cell handles everything. It will install the necessary libraries, log in to Hugging Face, and download the fine-tuned model.

How to Use This Fine-Tuned Model (LoRA)

This repository contains a LoRA (Low-Rank Adaptation) adapter for the unsloth/Qwen2-7B-Instruct-bnb-4bit model. It has been fine-tuned for compliance-related question answering with a Chain-of-Thought reasoning process.

## 1. Environment Setup

This model can be run in a Kaggle or Google Colab notebook with a GPU accelerator (T4 or newer recommended).

First, you need to add your Hugging Face token as a secret to your environment.

• In Kaggle: Go to Add-ons > Secrets and add a new secret with the name HF_TOKEN.
• In Colab: Go to the Secrets (🔑) tab and add a new secret with the name HF_TOKEN.

## 2. Installation and Loading

The following code will install the necessary libraries, log in to your Hugging Face account, and load the fine-tuned model. Unsloth automatically handles loading the base model and applying the LoRA adapter from this repository.

For Kaggle Notebook follow this

# Install the Unsloth library
!pip install "unsloth[kaggle-new]" # Use "[colab-new]" for Colab

# Import necessary libraries
from unsloth import FastLanguageModel
from transformers import pipeline
import torch
from kaggle_secrets import UserSecretsClient # Or `from google.colab import userdata` for Colab
import huggingface_hub

# Log in to Hugging Face
user_secrets = UserSecretsClient()
hf_token = user_secrets.get_secret("HF_TOKEN")
huggingface_hub.login(token=hf_token)

# Load the fine-tuned model from the Hub
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "D-Khalid/qwen2-7b-instruct-compliance-cot", # This repository ID
    load_in_4bit = True,
)

print("✅ Model loaded successfully!")

For Google Collab notebook follow this

# Install the Unsloth library
!pip install "unsloth[colab-new]" # Use "[kaggle-new]" for Kaggle

# Import necessary libraries and log in to Hugging Face
from unsloth import FastLanguageModel
from transformers import pipeline
import torch
# Replace this with the correct library for your environment (e.g., from kaggle_secrets import UserSecretsClient)
from google.colab import userdata 
import huggingface_hub

hf_token = userdata.get('HF_TOKEN')
huggingface_hub.login(token=hf_token)

# --- Load your fine-tuned model from the Hub ---
# Unsloth automatically handles loading the base model and applying your adapter.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "D-Khalid/qwen2-7b-instruct-compliance-cot", # Your repo ID
    load_in_4bit = True,
)

3. Running Inferance (same for both platform notebook)

Once the model is loaded, you can run inference using the correct chat template

# Create the generation pipeline
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

# Prepare your prompt using the chat template
system_prompt = "You are a helpful compliance assistant. Provide a detailed chain of thought before the final response."
user_query = "What are the key steps to ensure CCPA compliance for a mobile app?"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_query},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Generate the response
outputs = pipe(prompt, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=0.95)

print("\n--- Generated Response ---")
# This correctly extracts ONLY the model's generated text
print(outputs[0]['generated_text'].split("<|im_start|>assistant\n")[-1].replace("<|im_end|>", "").strip())