DS4AI-UPB/vanguard-xlmr-clip-film-exist2026

VANGUARD: Through the Eyes of the Beholder - Biometric and Demographic Conditioning for Multimodal Sexism Detection

CLEF 2026 · EXIST Lab · Task 2 · Team VANGUARD

Ana-Maria Luisa Mocanu · Sebastian Mocanu · Ciprian-Octavian Truică · Elena-Simona Apostol

---

Model Description

This is the non-ensemble deep model - the best single configuration from Team VANGUARD's submission to EXIST 2026 Task 2. It is a five-stream human-centered multimodal framework for sexism detection in memes that fuses textual, visual, demographic, and physiological modalities through a FiLM-conditioned cross-attention architecture.

Rather than predicting a single ground-truth label, the model is trained to learn from annotator disagreement, conditioning its predictions on who saw the meme and how they physically reacted to it. Meme text and visual descriptions are extracted by Gemma 4 (via Ollama). Text and image representations are produced by LoRA-adapted XLM-RoBERTa and CLIP ViT-B/32 encoders, fused via multi-head cross-attention, and then modulated by a 56-dimensional human-context vector (annotator demographics + physiological sensor embedding) via Feature-wise Linear Modulation (FiLM). Subtask 2.1 is framed as a label distribution learning problem using soft-label KL divergence over the full annotator distribution.

This configuration - no augmentation, no SVM ensemble - achieves the best overall validation and All-split test scores, outperforming every variant that adds augmentation or the ensemble.

---

Architecture Overview

Figure 1: Overview of the five-stream FiLM-conditioned cross-attention architecture. Five input streams, VLM-extracted embedded text, VLM-generated visual description, meme RGB image, annotator demographics, and physiological sensor vector, are fused through cross-attention and FiLM modulation into a shared 512-dimensional fused state, producing predictions for all three subtasks.

---

Files

File	Description
best_model.pt	Trained model weights (~1.4 GB). Contains LoRA adapters for both XLM-RoBERTa and CLIP, plus all fusion, FiLM, and task head parameters
config.json	Model hyperparameters and architecture configuration
best_thresholds.json	Per-subtask decision thresholds, grid-searched on validation to maximize macro F1
sensor_scaler.joblib	StandardScaler fitted on the training split for the 4-D physiological feature vector; must be applied before inference

Note: best_model.pt stores LoRA adapter weights and all custom components. The frozen backbones (FacebookAI/xlm-roberta-base and openai/clip-vit-base-patch32) are loaded separately from Hugging Face at inference time.

---

Usage

Requirements

pip install "numpy>=2.0,<2.5" "torchao>=0.16.0"
pip install transformers peft huggingface_hub joblib scikit-learn pillow
pip install git+https://github.com/DS4AI-UPB/VANGUARD-CLEF2026-EXIST.git

You also need the base models available locally or via HF:

• FacebookAI/xlm-roberta-base
• openai/clip-vit-base-patch32

Loading the model

import torch
import json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, CLIPImageProcessor
from exist_2026.train.nn.meme_classifier import LoRAMemeMultitaskModel

REPO = "DS4AI-UPB/vanguard-xlmr-clip-film-exist2026"
config     = json.load(open(hf_hub_download(REPO, "config.json")))
thresholds = json.load(open(hf_hub_download(REPO, "best_thresholds.json")))
weights    = hf_hub_download(REPO, "best_model.pt")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = LoRAMemeMultitaskModel(
    text_model=config["text_model"],
    image_model=config["image_model"],
    lora_r=config["lora_r"],
    lora_alpha=config["lora_alpha"],
)
model.load_state_dict(torch.load(weights, map_location=device))
model.to(device).eval()

tokenizer      = AutoTokenizer.from_pretrained(config["text_model"])
clip_processor = CLIPImageProcessor.from_pretrained(config["image_model"])

Preparing sensor features

The model expects a 4-dimensional physiological feature vector per meme — [log_reaction_time, fixation_count, saccade_count, hr_std] — standardized using the StandardScaler fitted on the training split. Without real annotator biometric data, pass zeros directly; FiLM conditioning is a mathematical identity in this case since its weights are zero-initialized.

import numpy as np

# With real biometric data:
# raw = np.array([[log_reaction_time, fixation_count, saccade_count, hr_std]], dtype=np.float32)
# sensorial = torch.tensor(sensor_scaler.transform(raw), dtype=torch.float32).to(device)

# Without biometric data:
sensorial = torch.zeros(1, 4).to(device)

Full inference example

For a complete inference example using the pipeline from the repository, see runnable/evaluate.py in the GitHub repo.

---

VLM enrichment (preprocessing)

Meme text and visual descriptions are extracted by Gemma 4 (gemma4:e4b via Ollama, temperature 0.0, top_p 0.1) before training and inference. This step is not embedded in the model weights, the repo's runnable/preprocess.py handles it.

---

Intended Use

• Intended: Research on human-centered multimodal sexism detection; reproducing EXIST 2026 Task 2 results; studying label distribution learning and biometric conditioning in subjective NLP tasks.
• Out of scope: Production content moderation without further safety validation; domains outside meme analysis; tasks other than EXIST 2026 Task 2 subtasks without fine-tuning.

---

Limitations

• Physiological signals are meme-level averages - features are averaged across all 16 annotators per meme, discarding individual-level variation. A viewer-specific model could better capture subjectivity.
• Sensor multicollinearity - fixation count and saccade count are nearly perfectly correlated (ρ ≈ 1.0), so the 4-D sensor vector likely carries less independent information than its dimensionality suggests.
• EEG signals - not individually significant in statistical testing; they enter only through the classical SVM ensemble (not this model), leaving open whether richer temporal EEG modeling could unlock additional signal.
• No single component survives multiple-comparison control - ablations over 5 seeds with Holm-Bonferroni correction across 63 tests find no individually statistically detectable benefit from FiLM conditioning, the image stream, or the contrastive loss. System gains appear to arise from integration rather than any single module. We report this openly.
• English-Spanish gap - hard ICM-Norm on test is 0.5496 (EN) vs. 0.4306 (ES), likely because XLM-RoBERTa and CLIP pretraining better captures English idioms of online sexism.
• Subtask 2.3 collapse - the 6-class fine-grained head collapses to a near-trivial solution under multi-task training; only single-task training on 2.3 breaks this pattern.

---

Citation

@InProceedings{Mocanu_2026_EXIST_CLEF,
    author    = {Mocanu, Ana-Maria Luisa and Mocanu, Sebastian and Truică, Ciprian-Octavian and Apostol, Elena-Simona},
    title     = {Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection},
    booktitle = {Conference and Labs of the Evaluation Forum (CLEF), EXIST 2026 Lab, Task 2},
    month     = {September},
    year      = {2026}
}

---

Links

Resource	Link
Paper	WIP — will be updated when proceedings are published
arXiv	WIP
Code	GitHub — DS4AI-UPB/VANGUARD-CLEF2026-EXIST
Dataset	EXIST 2026 (official)