BiGraph-Diffuse

LoRA adapter for BiGraph-Diffuse, a retrieval-augmented diffusion language model for empathetic mental health counseling.

Model Overview

This is a LoRA adapter fine-tuned on LLaDA-8B-Instruct, a discrete diffusion language model. The adapter is trained on counseling dialogues to generate empathetic, psychologically grounded counselor responses.

Architecture

• Base Model: LLaDA-8B-Instruct (discrete diffusion LM)
• Adapter: LoRA (rank=32, alpha=64, dropout=0.1)
• Target Modules: q_proj, k_proj, v_proj, o_proj
• Task: Causal language modeling with masked diffusion loss

Full architecture includes BiGraph-RAG, a bipartite graph retrieval system that augments generation with relevant psychological knowledge. Code available at the GitHub repo.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model_path = "path/to/LLaDA-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
tokenizer.padding_side = "left"

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Chekhov0919/BiGraph-Diffuse")
model.eval()

# Generate with diffusion
# See GitHub repo for full inference code with BiGraph-RAG integration

For the complete inference pipeline with BiGraph-RAG retrieval, refer to the GitHub repository.

Training

Setting	Value
Base Model	LLaDA-8B-Instruct
Dataset	CPsyCounD (counseling dialogues)
LoRA rank	32
LoRA alpha	64
LoRA dropout	0.1
Batch size	2 × 32 (gradient accumulation)
Learning rate	3e-5
Epochs	5
LR scheduler	Cosine
Mask token ID	126336

Citation

Please stay tuned — citation information will be added upon publication.

License

MIT