ArGuard-CLIP-MARBERTv2

A cross-attention dual encoder for Arabic hateful meme detection. It pairs the Arabic text encoder MARBERTv2 with a frozen CLIP ViT-L/14 image encoder and is trained jointly on both subtasks of ArGuard 2026, Track A:

• A1: binary Hateful / Not Hateful
• A2: multi-label hate categories (10 classes)

This was the NAMAA team's official A2 submission. It is described in our paper NAMAA at ArGuard Shared Tasks: Multimodal Fusion, Vision–Language Model Adaptation, and In-Context Prompting for Arabic Hateful Meme Detection (ArabicNLP 2026).

Results

Macro-F1, from Tables 1 and 4 of the paper:

System	A1 dev	A2 dev	A1 test	A2 test
CLIP–MARBERTv2 (this model)	0.7613	0.4236	0.7374	0.3136*
SigLIP–AraBERT-Twitter	0.7663	0.3880	0.7345	0.3851
Soft-vote ensemble of the two	0.7756	0.4236	0.7443	0.3136
Qwen3-VL-8B QLoRA	0.73	–	0.7610*	0.2119

* official shared-task score.

Architecture

meme OCR text ──► MARBERTv2 (fine-tuned) ──► Linear 768→512 + LayerNorm ──► T ─┐
                                                                               ├─► T' = MHA(T, V, V)   ┐
meme image ─────► CLIP ViT-L/14 (frozen) ──► Linear 1024→512 + LayerNorm ─► V ─┘   V' = MHA(V, T, T)   │
                                                                                                       ▼
                         h = [masked-mean(T') ; mean(V')]  (1024) ──► LayerNorm ─► Linear 1024→512 ─► z
                         p_A1 = σ(w·z + b)          p_A2 = σ(W z + b)   (10 labels)

• Each bidirectional cross-attention block has 8 heads, d = 512, residual connections, LayerNorm, and a 512→1024→512 feed-forward layer.
• The loss is BCE(A1) + BCE_w(A2), where w holds inverse-frequency positive-class weights.
• The 471.7M parameters are stored in fp32.

Repository contents

Path	Description
best/training_state.pt	Recommended. The checkpoint selected on the development set (epoch 8, step 1971, best combined A1 + A2 macro-F1).
best/meta.json	The tuned decision thresholds and development scores for best/.
last/training_state.pt	The final checkpoint (epoch 12, step 2628).
last/meta.json	Thresholds re-tuned for last/.

Each training_state.pt is a full training state: model (the state dict), optimizer, scheduler, scaler, rng, cfg (the training config), and thresholds / best. For inference, only model, cfg and thresholds are needed.

Decision thresholds (best/)

Thresholds were tuned on the development set by searching 0.05–0.95 in steps of 0.01.

A1	Contempt	Dehumanization	Exclusion	Humor	Incitement	Inferiority	Mocking	Other	Sarcasm	Slurs
0.83	0.87	0.57	0.05	0.67	0.64	0.29	0.67	0.35	0.26	0.56

The A2 heads follow the alphabetical label order shown above. Five taxonomy classes (Extremism, Historical, Insults, Stereotyping, Threat) have no training support and are never predicted.

Training

Setting	Value
Text encoder	UBC-NLP/MARBERTv2 (fine-tuned), max 96 tokens
Vision encoder	openai/clip-vit-large-patch14 (frozen), 224 px
Fusion	proj dim 512, 8 heads, fusion dropout 0.1, head dropout 0.2
Optimiser	AdamW, lr 1e-5 (text encoder) / 1e-4 (fusion + heads), weight decay 0.01
Schedule	cosine, 10% warmup, 12 epochs, batch size 16, grad clip 1.0, FP16
Loss	BCE (A1) + inverse-frequency-weighted BCE (A2), λ_A2 = 1.0
Selection	best combined A1 + A2 macro-F1 on the development set
Seed	42
Text preprocessing	removes diacritics and tatweel, normalises Alef / Ya / Hamza, reduces character runs of 3+ to 2

Usage

The checkpoint holds a plain PyTorch state dict. Rebuild the module described above and load the weights from model:

import json, torch
from huggingface_hub import hf_hub_download

REPO = "NAMAA-Space/ArGuard-CLIP-MARBERTv2"
state = torch.load(hf_hub_download(REPO, "best/training_state.pt"), map_location="cpu", weights_only=False)
cfg, weights = state["cfg"], state["model"]
thr = json.load(open(hf_hub_download(REPO, "best/meta.json")))["thresholds"]

# model = DualEncoder(cfg)            # text_encoder, vision_encoder, proj_t, proj_v, t2i, i2t, cls, head_a1, head_a2
# model.load_state_dict(weights)
# p_a1, p_a2 = model(...)
# hateful = p_a1 >= thr["a1_thr"]
# labels  = [l for l, p, t in zip(LABELS, p_a2, thr["a2_thr"]) if p >= t]

Limitations and ethical considerations

• The model was trained on about 3.5k memes with a single seed. A2 is long-tailed and very sparse: some classes have only a handful of training examples, and per-class thresholds tuned on the development set did not fully transfer to the test set.
• The vision tower is frozen, while the VLM baselines adapt both modalities, so comparisons reflect different training budgets.
• The model handles hateful content and can reproduce social, cultural or dialectal biases. Use it to support culturally informed human moderation, not to replace it. The shared-task data is for research use under its access terms.

NAMAA at ArGuard 2026 models

Model	Description
ArGuard-Qwen3-VL-8B-QLoRA-A1	Best A1 system (official A1 submission)
ArGuard-CLIP-MARBERTv2	Dual encoder for A1 + A2 (official A2 submission)
ArGuard-SigLIP-AraBERT-Twitter	Dual encoder for A1 + A2 (best A2 test score)

Citation

@inproceedings{djamai-etal-2026-namaa,
    title     = "{NAMAA} at {ArGuard} Shared Tasks: Multimodal Fusion, Vision{--}Language Model Adaptation, and In-Context Prompting for {A}rabic Hateful Meme Detection",
    author    = "Djamai, Abdelbasset and Al Jallad, Khloud and Emad Eldin, Fatimah and Nacar, Omer",
    booktitle = "Proceedings of the Fourth Arabic Natural Language Processing Conference: Shared Tasks",
    year      = "2026",
    address   = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
}

@inproceedings{alam-etal-2026-arguard,
    title     = "{ArGuard} Shared Task: Harmful Content Detection in {A}rabic Memes and {LLM} Prompts",
    author    = "Alam, Firoj and Biswas, Md. Rafiul and Kmainasi, Mohamed Bayan and Shahroor, Ali Ezzat and Mubarak, Hamdy and Mikros, Georgios and Hasnat, Abul and Zaghouani, Wajdi",
    booktitle = "Proceedings of the Fourth Arabic Natural Language Processing Conference: Shared Tasks",
    year      = "2026",
    address   = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
}

Authors: Abdelbasset Djamai, Khloud Al Jallad, Fatimah Emad Eldin, Omer Nacar (NAMAA Community). Contact: o.najar@tuwaiq.edu.sa