Title: VAFL: Vector-field Assisted Functional Layer for Equal-Compute Multi-modal Learning
Primary area: generative models
Keywords: multi-modal learning, energy-based models, Langevin dynamics, equal-compute protocol, unified transformer, WikiText-2, CIFAR-10, Speech Commands

Abstract: We study whether energy-based refinement can improve multi-modal learning under strict equal-compute constraints. We present VAFL (Vector-field Assisted Functional Layer), a lightweight post-processing mechanism that applies K=2 Langevin dynamics steps to refine hidden representations from a unified transformer backbone. Our approach addresses a critical challenge in multi-modal learning: achieving consistent improvements across diverse modalities without substantial computational overhead. 

On a unified 6-layer transformer (d=384, h=6), VAFL demonstrates: (1) Text perplexity reduction from 22.5 to 20.8 on WikiText-2 (7.6% improvement), (2) Image MSE reduction from 0.0320 to 0.0290 on CIFAR-10 patches (9.4% improvement), (3) Audio MSE reduction from 0.0450 to 0.0410 on Speech Commands mel-spectrograms (8.9% improvement). These gains require only 50M additional FLOPs (4.2% increase from 1.20G to 1.25G), satisfying the equal-compute constraint (<5% overhead).

We introduce SOMA (Synergistic Optimization for Multi-modal Assessment), a composite metric balancing quality (inverse loss), diversity (Distinct-2), and stability (refinement delta p95). VAFL achieves 5.9% SOMA improvement (0.680→0.720) while maintaining inference latency under 52ms. Ablation studies confirm K=2 as optimal: K=1 underperforms (SOMA=0.698), while K>2 shows diminishing returns with linear cost increase. Our energy function uses a compact 256-dim hidden layer (0.3M parameters), demonstrating that effective multi-modal refinement doesn't require architectural scaling.

TL;DR: We propose VAFL, an energy-based refinement layer using K=2 Langevin dynamics steps that improves multi-modal generation quality by 7.6% (text), 9.4% (image), and 8.9% (audio) with only 4.2% additional compute.