AstraNexus Multimodal Email Encoder (DriptoBhattacharyya/astranexus-mm-encoder)

LoRA-fine-tuned late-fusion multimodal encoder for unsupervised email clustering. Text branch = MiniLM (all-MiniLM-L6-v2), vision branch = google/siglip-large-patch16-384, joined by a learned 256-d fusion head. Trained with supervised contrastive loss on 600 teacher-labeled emails (topic labels distilled from Qwen2.5-7B-Instruct), 6 epochs on 2×T4.

Why this exists

Off-the-shelf fixed-weight fusion can't win both text-clear and image-decisive emails (no single α is best for both). This encoder learns the fusion, so it clusters hard cases (generic subject + topic-revealing attachment) correctly.

Results — independent hard eval (120 emails, image-decisive)

	ARI	NMI
Off-the-shelf fused (gate)	0.168	0.519
Fine-tuned (this model)	0.252	0.511

Δ ARI +0.084. The gate is an honest off-the-shelf baseline on the same encoders; the lift is purely from the LoRA + learned fusion.

Files

• text_lora/ — PEFT-LoRA adapter for the MiniLM text tower
• vision_lora/ — PEFT-LoRA adapter for the SigLIP vision tower
• proj.pt — learned fusion head (Linear(t+v, 256) -> GELU -> Linear(256, 256))

Usage

See astranexus/cluster/ft_encoder.py in the AstraNexus repo — loads both adapters + the fusion head and exposes encode(emails) -> np.ndarray.

Reproducibility note

Trained on Kaggle (torch 2.10, 2×T4). The SigLIP LoRA adapter keys use the vision_model. module layout from the training-time transformers; newer transformers (5.x) flattened SigLIP, which shifts both the adapter key paths and the frozen base-model numerics. ft_encoder._load_adapter_robust remaps the keys, but for faithful results pin transformers to the training line (4.x) and install torchvision (matches the image processor). The eval numbers above were measured in the training environment.