CVML-CFU/HyVol-PlusPlus

HyVol++

Hypergraph Fusion for Multimodal Retrieval

Anindya Nag · Ambuj Mehrish · Sebastiano Vascon

Ca' Foscari University of Venice

---

Pretrained weights for Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization.

HyVol++ turns HyVol into a general training-time hypergraph fusion framework. Observed modalities (video, audio, subtitles) are vertices, and caption-guided mutual top-k neighbourhoods connect related training clips, so a clip can draw on evidence that related clips have observed. The hypergraph is discarded after training, so the backbone, scoring function and inference cost stay exactly the same.

✨ Two new configurations

	What it learns	Propagation
wHyVol (weighted HyVol)	a reliability weight per view + a symmetric caption affinity per pair of related clips	additive
wRHG (weighted relational hypergraph)	both weights + typed relations: document, same-modality, cross-modality and joint-candidate hyperedges	relation-conditioned, multiplicative

Missing modalities get no vertex: they are omitted, not imputed.

📦 Checkpoints

Model	Backbone	File
wHyVol	GRAM	wHyVol/GRAM_wHyVol/wHyVol_GRAM.pt
wHyVol	HyperGRAM	wHyVol/HyperGRAM_wHyVol/wHyVol_HyperGRAM.pt
wRHG	GRAM	wRHG/GRAM_wRHG/wRHG_GRAM.pt
wRHG	HyperGRAM	wRHG/HyperGRAM_wRHG/wRHG_HyperGRAM.pt

• Initialisation: VAST (EVA-CLIP ViT-g/14 vision, BEATs audio, BERT-base text)
• Training data: one epoch on the 150K-clip subset of VAST-27M (136,694 clips recovered with video and audio)
• Checkpoint selection: best text-to-video R@1 of the matching (ITM) score on MSR-VTT 1k, T-VAS

🚀 Usage

from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    repo_id="CVML-CFU/HyVol-PlusPlus",
    filename="wRHG/GRAM_wRHG/wRHG_GRAM.pt",
)

Load each checkpoint with the codebase of its backbone: GRAM or HyperGRAM. No extra module is needed at test time.

📊 Results

Zero-shot on eleven benchmarks: eight text-video (MSR-VTT, DiDeMo, ActivityNet, VATEX, Shot2Story, MSVD, FLARE, VALOR-32K) and three text-audio (AudioCaps, Clotho, SoundDescs).

Backbone	Mean T2V R@1 (17 settings)	+ wRHG	Gain
GRAM	63.4	68.2	+4.8
HyperGRAM	64.8	68.4	+3.6

• With GRAM, wRHG improves all 34 text-to-video and video-to-text settings.
• With HyperGRAM, it improves 33 of 34.
• The largest video-to-text gain is +13.3 R@1 on ActivityNet.

📝 Citation

If you use these weights, please cite:

@inproceedings{nag2027hyvol,
  title     = {Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval},
  author    = {Nag, Anindya and Mehrish, Ambuj and Vascon, Sebastiano},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}