ChrisLalk/mistral-7b-therapy-speaker-id

Mistral-7B-Instruct-v0.3 -- Speaker Identification for German Psychotherapy Transcripts (QLoRA)

Status: Gated Repo

This is a QLoRA fine-tuned adapter for mistralai/Mistral-7B-Instruct-v0.3, adapted as a binary sequence classifier to identify the speaker (Therapist or Patient) of a given utterance in a two-person German psychotherapy session transcript, using surrounding dialogue as context.

Model Details

• Base model: mistralai/Mistral-7B-Instruct-v0.3 (7B), loaded with a SequenceClassification head (MistralForSequenceClassification)
• Adaptation method: QLoRA (4-bit NF4 quantization + LoRA)
• LoRA config: r=16, alpha=8, dropout=0.05, target modules q_proj, k_proj, v_proj, o_proj
• Task: Binary text classification -- T (Therapist) vs. P (Patient)
• Language: German
• Labels: {0: "T", 1: "P"}
• License: Apache 2.0 (inherited from base model); adapter weights not currently released

Intended Use

Supports speaker diarization / re-attribution in transcribed two-person psychotherapy sessions, e.g. as a post-processing step after ASR diarization, or to reconstruct speaker turns when diarization metadata is missing or unreliable.

Not intended for:

• Clinical decision-making, diagnosis, or use as a medical device
• Identification of individual speakers (this model distinguishes roles, not identities)
• Languages other than German, or therapy modalities/settings substantially different from the training data
• Use as a general-purpose text classifier outside the documented prompt format

Training Data

The model was fine-tuned on German two-person psychotherapy transcripts drawn from multiple public sources. The training data contained approximately 1,500 transcripts of psychotherapy sessions.

No training data or verbatim excerpts are included in or redistributed with this repository.

Each training example consists of:

• no_speaker: the target utterance with the speaker label stripped
• dialog: the surrounding conversational context, with the target utterance marked by --

Class weighting [T: 2, P: 1] was applied during training to counteract the therapist-utterance class being under-represented with about 2:1-ratio and to reduce bias toward over-predicting "Patient".

Training Procedure

• Framework: Transformers Trainer (custom subclass) + PEFT + bitsandbytes
• Quantization: 4-bit NF4, double quantization, compute dtype float16
• Epochs: 2 (early stopping on eval_roc_auc, patience 10)
• Batch size: per-device batch size x2 gradient accumulation steps
• Learning rate: 3e-5, 500 warmup steps, weight decay 0.01
• Optimizer: paged_adamw_8bit
• Loss: weighted BCEWithLogitsLoss (one-hot labels, class weights [2, 1])
• Max sequence length: 2048 tokens
• Precision: fp16 mixed precision
• Hardware: 1xH100 (80GB GPU)

Prompt format

Du bist eine sehr intelligente Psychiaterin.
Untersuche den Verlauf einer Therapiesitzung. Bitte ordne die folgende AUSSAGE im
Psychotherapietranskript dem Sprecher zu. Spricht der Therapeut (T) oder der Patient (P)?
Du erhaeltst den KONTEXT der AUSSAGE unten. Die AUSSAGE wird innerhalb des KONTEXTES
markiert ("--").

## AUSSAGE: {utterance}

## KONTEXT:
{context_with_utterance_marked_by_--}

Evaluation

Evaluated on a held-out test set (3,710 labeled utterances):

Metric	Value
Accuracy	0.928
F1 (weighted binary)	0.948
ROC-AUC	0.978
Eval loss	0.264

Per-class breakdown:

Class	Precision	Recall	F1-score	Support
T (Therapist)	0.88	0.89	0.88	1121
P (Patient)	0.95	0.95	0.95	2589
Accuracy			0.93	3710
Macro avg	0.91	0.92	0.91	3710
Weighted avg	0.93	0.93	0.93	3710

The model performs noticeably better on Patient utterances than Therapist utterances, due to the class imbalance.

How to Use

import torch
from transformers import AutoTokenizer, MistralForSequenceClassification, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "mistralai/Mistral-7B-Instruct-v0.3"
adapter_id = "ChrisLalk/mistral-7b-therapy-speaker-id"  # gated repo

id2label = {0: "T", 1: "P"}
label2id = {"T": 0, "P": 1}

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id, model_max_length=2048)
tokenizer.pad_token = tokenizer.eos_token

base_model = MistralForSequenceClassification.from_pretrained(
    base_model_id,
    num_labels=2,
    id2label=id2label,
    label2id=label2id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
)
base_model.config.pad_token_id = tokenizer.pad_token_id

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = """Du bist eine sehr intelligente Psychiaterin.
Untersuche den Verlauf einer Therapiesitzung. Bitte ordne die folgende AUSSAGE im
Psychotherapietranskript dem Sprecher zu. Spricht der Therapeut (T) oder der Patient (P)?
Du erhaeltst den KONTEXT der AUSSAGE unten. Die AUSSAGE wird innerhalb des KONTEXTES
markiert ("--").

## AUSSAGE: Und wie fuehlen Sie sich dabei, wenn Sie daran zurueckdenken?

## KONTEXT:
... -- Und wie fuehlen Sie sich dabei, wenn Sie daran zurueckdenken? -- ...
"""

inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048).to(model.device)
with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

pred = id2label[int(probs.argmax())]
print(pred, probs.tolist())

Limitations, Risks & Open Items

• Memorization/extraction risk. A loss-gap membership-inference test found no disproportionate fit to training examples vs. held-out data. A prefix-continuation extraction attempt (reconstructing a generative model from the fine-tuned attention weights + the untouched base LM head) found zero verbatim n-gram overlap on 200 train and 200 held-out examples
• Even though it has been trained on various therapeutic approaches and settings, it may not generalize to other therapeutic approaches, dialects, or transcript formats.
• Requires the exact prompt template above; performance on free-form or reformatted inputs is not evaluated.
• Do not use probability scores as a substitute for validation on your own data.

Citation

@misc{[ChrisLalk]2026,
  author = {[Christopher Lalk]},
  title  = {Speaker Identification in German Psychotherapy Transcripts via QLoRA-finetuned Mistral-7B},
  year   = {2026},
}