DOI: 10.5281/zenodo.16418462
title: NeuroJustice

description: # Multimodal Supervision Framework for Offenders with Mental Health Challenges

This repository presents the implementation of **CrossFusionNet**, a novel multimodal architecture, and its associated **Supervision Harmonization Protocol (SHP)** for improving offender management, particularly for individuals with mental health challenges. The project introduces new approaches to integrate heterogeneous data sources, handle supervision noise, and provide adaptive, real-time offender profiling.

## 🔍 Overview

Traditional criminal justice supervision systems often struggle with the dynamic and complex needs of mentally ill offenders due to their reliance on simplistic, single-modality data and static evaluation methods. This project addresses those limitations with:

- **CrossFusionNet**: A deep neural architecture that uses modality-specific encoders and a learnable cross-modal attention mechanism to retain modality fidelity while enabling semantic alignment.- **SHP (Supervision Harmonization Protocol)**: A robust training scheme that mitigates weak supervision, noisy labels, and partial modality availability through dynamic weighting, self-distillation, and uncertainty regularization.

## 🧠 Key Features

- Multimodal integration of visual, audio, and textual data.- Cross-modal attention and fusion via joint interaction tensors.- Structured loss functions for modality-specific and global consistency.- Dynamic reliability scoring and knowledge distillation between modalities.- Latent consistency enforcement using frequency attention and alignment penalties.

## 🏗️ Architecture

### CrossFusionNet

- **Input**: Modality-specific streams (e.g., spatio-temporal, textual, audio).- **Encoders**: Per-modality neural encoders generate embeddings.- **Fusion**: Joint interaction tensors and attention-based global fusion.- **Output**: Distributional embeddings for downstream tasks.

> *See Figure 1 on page 7 of the paper for a detailed architecture diagram.*

### Supervision Harmonization Protocol (SHP)

- **Dynamic Reliability Weighting**: Adjusts contribution of each modality using entropy-based trust scores and inter-modality agreement.- **Knowledge Distillation**: Teacher-student strategy where high-confidence modalities guide weaker ones.- **Latent Consistency Enforcement**: Penalties for uncertainty and alignment errors in shared latent space.

> *See Figures 2–4 on pages 9–11 for schematics of loss structure and consistency enforcement.*

## 📊 Datasets

Evaluated on four multimodal benchmarks:

- **VQA** – Visual Question Answering- **MSCOCO** – Image captioning and object detection- **Flickr30k** – Text-image retrieval- **AVSpeech** – Audio-visual speech recognition

## 🏆 Performance

Outperforms state-of-the-art models (e.g., CLIP, ViLT, LXMERT, BLIP) across all datasets and metrics, with significant gains in:

- Accuracy- F1 Score- AUC- Recall

See **Tables 1–4** (pages 14–15) for full benchmark and ablation results.

## ⚙️ Implementation Details

- Framework: PyTorch- Optimizer: AdamW- Training: 50 epochs, cosine annealing, early stopping, mixed-precision- Hardware: 8 × NVIDIA A100 GPUs- Data Preprocessing: Resize to 224×224, BPE tokenization, spectrogram extraction

## 📁 File Structure

```bash├── data/                   # Dataset loaders and preprocessors├── models/                 # CrossFusionNet and SHP modules├── training/               # Training loops and loss functions├── utils/                  # Metrics, visualizations, config files├── scripts/                # Scripts for experiments└── README.md               # Project overview