gemma-empathy-lora

A small LoRA adapter that tunes google/gemma-3-27b-it toward warmer, more empathetic conversational responses. Trained on 1 June 2026 as the first component of MindMirror, a Thai voice companion for CBT-style emotional support.

Status: early proof-of-concept. Not evaluated. Not recommended for use. The training set was very small and no held-out evaluation was run. This adapter is published for transparency about the project's development history, not as a usable model.

---

Why this exists

MindMirror needs two things that are trained separately: a model that sounds like a counselor, and a model that can hear Thai speech.

Ultravox freezes the LLM backbone and trains only the audio adapter, which means all personality and conversational behaviour has to live in the backbone itself. So the plan was to develop and validate the counseling behaviour entirely in text first, then merge that adapter into Gemma and use the merged model as the Ultravox backbone.

This adapter is the text-only half of that plan — the first attempt at it.

Model details

Type	LoRA adapter (PEFT)
Base model	google/gemma-3-27b-it
Task	causal language modeling
Rank / alpha	r=16, α=32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Modality	text only (no audio)

Training configuration

Steps	228 (3 epochs)
Batch size	1 per device
Learning rate	2e-4
Loss	33.03 → 3.14
Mean token accuracy	0.469 → 0.886
Date	1 June 2026

Working back from steps and epochs, the training set was on the order of ~76 examples. That is small enough that the accuracy figure above should be read as fitting the training data, not as a measure of generalization.

Training data

CBT and emotional-support dialogue data. The exact corpus used for this run was not recorded at the time, so it is not stated here rather than guessed. Candidate sources considered during this phase of the project were CACTUS, ESConv and EmpatheticDialogues, with Thai translation applied.

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("google/gemma-3-27b-it", device_map="auto")
model = PeftModel.from_pretrained(base, "Funk888/gemma-empathy-lora")

To use as an Ultravox backbone, merge into bf16 weights first — a LoRA cannot be merged directly into a 4-bit or AWQ-quantized model:

merged = model.merge_and_unload()

Evaluation

None. No held-out set, no baseline, no human rating. Training loss and token accuracy both improved, but with a training set this small neither number distinguishes learning from memorization.

Metrics that would be meaningful for a model like this:

• Human or rubric-based rating of counseling quality on held-out conversations
• Rate of Socratic questioning versus direct advice-giving — the literature flags advice-giving and excessive positivity as the main failure modes of empathetic LLMs
• Comparison against the un-tuned base model on the same prompts

Limitations

• No evaluation, no baseline. Any claim about this adapter's quality is unverified.
• Training set was tiny (~76 examples over 3 epochs). High risk of memorization.
• Training data not documented. The exact corpus was not recorded, which is itself the main process failure of this run.
• Text only. This adapter has never been used together with audio input. Combining it with a separately-trained audio projector is untested and may interact badly.
• Not safe for real users. Emotional-support models can validate harmful thinking, miss crisis signals, or give advice when they should be asking questions. None of that was tested here. Do not deploy this in any setting where a person might rely on its responses.

Acknowledgements

Base model is google/gemma-3-27b-it, distributed under the Gemma Terms of Use, which apply to any use of this adapter with that backbone.

Built with PEFT 0.14.0.

Author

Funk Jaitus (@Funk888)