ABCREATIVEAKSHAY/dementia-symptom-classifier

Dementia Symptom Classifier

Multimodal fusion model for detecting dementia symptoms from patient speech.

Architecture

• Audio encoder: WavLM (frozen) — microsoft/wavlm-base
• Text encoder: DistilBERT (frozen) — distilbert-base-uncased
• Fusion: BiLSTM (2 layers, bidirectional) + Multi-head attention (4 heads)
• Classification: 4 multi-label binary classifiers

Symptom Classes

Class	Description
repetition	Patient is repeating themselves
confusion	Patient seems confused or disoriented
distress	Patient is anxious, scared, or in pain
topic_switch	Patient suddenly changed topics

Inference API

Send a POST request with JSON body:

{
  "text": "where is my daughter",
  "audio": "<base64-encoded 16kHz mono float32 WAV>"
}

audio is optional — if omitted, only the text encoder is used.

Example (JavaScript)

const response = await fetch(
  "https://api-inference.huggingface.co/models/ABCREATIVEAKSHAY/dementia-symptom-classifier",
  {
    method: "POST",
    headers: { "Authorization": "Bearer hf_YOUR_TOKEN" },
    body: JSON.stringify({ text: "माझी मुलगी कुठे आहे" })
  }
);
const result = await response.json();
// { repetition: 0.12, confusion: 0.85, distress: 0.03, topic_switch: 0.05 }

Example (Python)

import requests

response = requests.post(
    "https://api-inference.huggingface.co/models/ABCREACTIVEAKSHAY/dementia-symptom-classifier",
    headers={"Authorization": "Bearer hf_YOUR_TOKEN"},
    json={"text": "माझी मुलगी कुठे आहे"}
)
print(response.json())

Local Usage

import torch
from model import MultimodalFusionModel

model = MultimodalFusionModel(freeze_encoders=True)
state = torch.load("best_model.pt", map_location="cpu")
model.load_state_dict(state)
model.eval()

ae, te = model.compute_embeddings([waveform], ["hello, where am I?"])
logits = model(audio_embeds=ae.unsqueeze(0), text_embeds=te.unsqueeze(0))
probs = torch.sigmoid(logits)

Files

File	Size	Description
best_model.pt	631 MB	PyTorch checkpoint
model.py	6.2 KB	Model architecture
inference.py	2.5 KB	HF Inference API handler
embeddings.pkl	223 KB	Precomputed embeddings