language: - en library_name: peft base_model: google/gemma-2-2b-it tags: - lora - peft - empathy - mental-health - conversational - chat
Gemma-2B Empathy LoRA Adapter
Overview
This repository contains a LoRA (Low-Rank Adaptation) adapter fine-tuned on top of the google/gemma-2-2b-it base model to improve empathetic and emotionally supportive conversational responses.
The adapter modifies the response style and tone of the base model without altering its underlying knowledge or reasoning capabilities.
This model is intended as a supportive conversational assistant, not as a replacement for professional mental health care.
Base Model
- Base model:
google/gemma-2-2b-it - Model type: Instruction-tuned large language model
- Fine-tuning method: Parameter-Efficient Fine-Tuning (PEFT) using LoRA
Only the LoRA adapter weights are provided in this repository.
The base model must be loaded separately.
Fine-Tuning Method
This adapter was trained using LoRA (Low-Rank Adaptation) via the PEFT framework.
Key characteristics:
- Base model parameters are frozen
- Approximately 0.12% of total parameters are trainable
- LoRA layers are applied to attention projection layers
- Training focuses on behavioral adaptation, not factual learning
This approach allows efficient fine-tuning on limited compute while preserving base model performance.
Datasets Used
Two publicly available datasets were combined to balance emotional sensitivity and professional tone:
1. Empathetic Dialogues
- Source: https://huggingface.co/datasets/Estwld/empathetic_dialogues_llm
- Used to learn emotional mirroring, conversational flow, and response grounding
2. CounselChat
- Source: https://huggingface.co/datasets/nbertagnolli/counsel-chat
- Used to learn calm, supportive, and non-judgmental response framing from mental health professionals
Approximately 12,000 processed dialogue pairs were used for training.
No dataset files are redistributed in this repository.
Training Configuration (Summary)
- Fine-tuning type: Supervised Fine-Tuning (SFT)
- Epochs: 2
- Learning rate: 2e-4
- Effective batch size: 16 (via gradient accumulation)
- Optimizer: AdamW
- Quantization: 4-bit loading for memory efficiency
- Hardware: Single GPU (Tesla T4 class)
Training converged stably without divergence or overfitting.
Intended Use
This model is intended for:
- Empathetic conversational agents
- Mental health support tools (non-clinical)
- Reflection, validation, and emotional acknowledgment use cases
- Educational or research purposes related to affective AI
Not Intended For
This model must NOT be used for:
- Medical or psychological diagnosis
- Crisis intervention or suicide prevention
- Emergency response or clinical decision-making
- Replacing licensed mental health professionals
Applications should implement external safety checks and escalation logic.
Safety & Limitations
- The model may generate incomplete or overly verbose responses
- It does not have real understanding of human emotions
- It may fail in high-risk or crisis scenarios
- Safety, moderation, and escalation must be handled at the application level
Users are responsible for deploying this model responsibly.
Usage Example
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")
model = PeftModel.from_pretrained(
base_model,
"<your-username>/<your-repo-name>"
)
model.eval()