Title: Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Alignment

Curriculum Learning

Contrastive Local Attention
Abstract: Most multimodal models treat every negative pair alike, ignoring the ambiguous negatives that differ from the positive by only a small detail.  We propose Boundary-A ware Curriculum with Local Attention(BACL), a lightweight add-on that turns these borderline cases into a curriculum signal.  A Boundary-aware Negative Sampler gradually raises difficulty, while a Contrastive Local Attention loss highlights where the mismatch occurs.  The two modules are fully differentiable and work with any off-the-shelf dual encoder.  Theory predicts a fast $\tilde{\mathcal{O}}(1/n)$ error rate; practice shows up to +32 \% R@1 over CLIP and new SOTA on four large-scale benchmarks, all without extra labels.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Boundary-Aware Curriculum with Local Attention (BACL), a novel framework for improving multimodal alignment by leveraging ambiguous negatives—pairs that are near the decision boundary and difficult to classify. ​ BACL consists of two key components:\n\nBoundary-aware Negative Sampler (BNS): Dynamically schedules training samples from easy to hard negatives based on a learnable boundary score. ​\nContrastive Local Attention (CLA): Highlights token-level mismatches between positive pairs and their hardest negatives, enforcing fine-grained alignment.\n\nThe results section shows improvement for both components.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe idea is novel; this paper identifies ambiguous negatives as a valuable supervisory signal and proposes a curriculum-based approach to exploit them effectively.\n\nStrong empirical results and also shows detailed ablation study.\n\nI also appreciate that this paper conduct on diverse datasets, covering diver modalities; such as image, text, audio. \n\n-------------------------------------------------------------------------------------------\n\nWeaknesses\n\n1, I am not quite familiar with this direction; but their concept is close to hard negative mining. i am not sure if this paper properly cited or compared with those works \n\n2, how is your training iters/epoches compared with original baseline? If I let base model (random sample approach) see the same amount of  ambiguous negative samples, will they also improve performance? Talking about distribution of number of training samples over different difficult levels is useful for us to understand your approach \n\n3, the current approach needs a pretrained model first to get feature (L104), will this be a problem? will this bound the final learning performance? \n\n\n4, it seems that the CLA part is written assuming you have a cross-modal Transformer. if so, how is this approach be used in dual-encoder cases where the attention is not shared?\n\nQUESTIONS:\nplease see weakness section. Currently, I am not quite sure about about score; If authors can address my concerns and I donot see big issue from the other reviewers, I will raise my score\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Boundary-aware Curriculum Learning (BACL) to learn from hard negatives in contrastive pre-training. BACL consists of two components: a Boundary-aware Negative Sampler (BNS) and a Contrastive Local Attention (CLA) loss. The former one tries to shift towards harder examples over time and the latter one amplifies attention gaps between a positive pair and its hardest negative. Experiments are performed across diverse benchmarks to show the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper tackles an important problem in contrastive pre-training: how to effectively utilise hard negatives during training. The proposed BACL method explicitly exploit the hard / confusing negatives via curriculum learning.\n- Lightweight yet effective methods that can be plugged to existing foundation model training frameworks and yield gains.\n- This paper provides a theoretical analysis that provides insights why curriculum mining of hard negatives is beneficial.\n- Ablation tests for BNS, CLA, and BNS+CLA are reported for better understanding of the proposed method.\n\nWeaknesses:\n- The CLA component is built on top of the assumption of an architecture that produces a cross-modal attention map between image and text modalities. This doesn’t apply in a standard CLIP setup that has no built-in cross-attention between image and text. The proposed CLA component has to introduce an additional cross-attention module or a joint forward pass to compute A(+) and A(-). This likely would make the training much slower, please provide additional details and speed comparison about CLA for standard contrastive learning. \n- The BNS module adds a policy network on top of standard training, which slightly increases implementation and computational complexity. Though it’s claimed to be lightweight, more results about the training speed should be attached. \n- Other than the training speed impact as mentioned in the above two points, the additional CLA and BNS components will also consume more memory. This makes the maximum training batch size smaller with the same training devices. Please attach quantitative results for the memory cost and hit on maximum batch sizes that could fit with the same devices. Reporting those could be very useful for the practitioners. \n- This paper focuses on retrieval tasks on LAION-400M, WebVid-10M, WavText and VAST-27M. Results on commonly used benchmarks (0-shot retrieval on ImageNet, COCO, etc.) should also be reported to allow comparison with previous methods.\n\nQUESTIONS:\nSee above weaknesses section for questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BACL, a novel boundary-aware curriculum learning framework that leverages ambiguous negative samples as a valuable supervisory signal for multimodal alignment. It incorporates a boundary-aware negative sampler (BNS) to dynamically schedule the difficulty of negative samples and a contrastive local attention (CLA) mechanism to enforce token-level disambiguation. The approach improves decision boundary tightness, leading to significant performance gains on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.  The proposed BACL framework effectively leverages ambiguous negative samples, an underexplored but important aspect in multimodal alignment. \n\n2. By combining the BNS and CLA, the method achieves both global alignment and token-level disambiguation, leading to more precise representations.\n\n3. The paper provides strong theoretical justification for the framework to support its empirical success.\n\n4.  BACL shows strong improvements across multiple benchmarks, indicating its effectiveness.\n\nWeaknesses\n\n1.While CLA operates on token-level attention maps, more interpretability analysis or additional visualizations could provide stronger insights into how the model identifies and addresses fine-grained mismatches.\n\n2. The current experiments are limited to smaller-scale datasets, the improvement on large-scale datasets remains unexplored.\n\nQUESTIONS:\n1. Ambiguous negatives can sometimes represent inherent ambiguity in the data (e.g., captions that plausibly describe different images). How does BACL ensure that such genuinely ambiguous cases are not penalized incorrectly?\n\n2. The paper mentions applying BACL to language-only instruction tuning in future work. How would the negative sampling and fine-grained token-level disambiguation techniques translate to this setting?\n\nLIMITATIONS:\nThis work has included the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}