Title: Adaptive and Multi-scale Affinity Alignment for Hierarchical Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: representation learning

contrastive learning

optimization
Abstract: Contrastive self-supervised learning has emerged as a powerful paradigm for extracting meaningful representations without labels. While effective at capturing broad categorical distinctions, current methods often struggle to preserve the fine-grained and hierarchical relationships inherent in real-world data. From the perspective of semantic alignment, conventional contrastive learning aligns representations to semantic structure at a global level, treating the entire embedding space uniformly and frequently overlooking rich local structural information. In this paper, we propose \emph{Adaptive Multi-scale Affinity alignment (AMA-alignment)}, a framework that introduces localized contrastive objectives and a dynamic multi-scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space. Although our model is inherently more complex due to its \emph{multi-scale} and \emph{adaptive} design, we provide the theoretical guarantees indicating that its convergence rate remains comparable to that of standard smooth non-convex optimization. We conduct a set of experiments on diverse benchmarks to show that AMA-alignment can effectively preserve hierarchical structure; moreover, AMA-alignment also outperforms existing contrastive methods on a range of downstream tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Adaptive Multi-scale Affinity Alignment (AMA-alignment), an innovative contrastive learning framework that significantly enhances representation learning by dynamically integrating hierarchical semantic structures through a novel three-level optimization strategy. The method uniquely combines local affinity alignment within semantically coherent regions, multi-scale decomposition of the embedding space, and adaptive weighting of challenging regions, achieving state-of-the-art performance while maintaining theoretical guarantees comparable to standard contrastive approaches. Through extensive experiments, AMA-alignment demonstrates superior capability in preserving both global categorical distinctions and fine-grained semantic relationships, outperforming existing methods by over 5% on challenging fine-grained classification tasks while remaining computationally efficient during inference. The framework's robustness to noise and fragmented graph structures, along with its compatibility with various graph neural architectures, establishes it as a versatile solution for complex graph representation learning, though its performance remains sensitive to region radius parameters and currently focuses on static graph scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\nThe manuscript is highly readable.\n\nTheoretical strength is relatively strong.\n\nThe experiments are relatively rich and sufficient.\n\nWeaknesses：\n\nIt lacks comparison with algorithms from the past three years.\n\nIn Table 3, why does adaptive learning perform worse than fixed weights?\n\nHow is the convergence of Equation (15) guaranteed?\n\nQUESTIONS:\nThe paper claims AMA-alignment achieves comparable convergence to standard non-convex optimization (Theorem 3.3), yet introduces multi-scale adaptive complexity. How does the convergence rate scale with the number of scales and regions?\n\nWhile AMA-alignment improves fine-grained classification (Table 1), does the local affinity alignment (Eq. 12) risk overfitting to region-specific noise? How does the method balance local semantic preservation with global consistency, especially for ambiguous samples near region boundaries?\n\nThe comparisons focus on contrastive methods (SimCLR, MoCo) and graph techniques. Why not include hierarchical contrastive approaches or hyperbolic embeddings? \n\nThe performance relies on key parameters: region radius, KL penalty, and temperature. How sensitive are results to these choices? Does the adaptive weighting (Eq. 17) mitigate sensitivity, or does it introduce new hyperparameters needing tuning?\n\nLIMITATIONS:\nPlease see Limitations in the Appendix and Supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a key limitation in contrastive self-supervised learning (CL): while effective at capturing broad categorical distinctions, standard CL methods often fail to preserve fine-grained and hierarchical semantic relationships inherent in real-world data. The authors propose Adaptive Multi-scale Affinity alignment (AMA-alignment), which introduces two core innovations: (1) localized contrastive objectives that align semantic structures within adaptively defined regions across multiple scales, enabling hierarchical representation learning; and (2) a distributionally robust optimization strategy that dynamically identifies and prioritizes poorly aligned regions where representation affinity deviates significantly from semantic affinity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) The paper introduces Adaptive Multi-scale Affinity alignment (AMA-alignment), which fundamentally advances contrastive learning by incorporating hierarchical region-aware optimization. This addresses a critical gap in conventional methods that overlook fine-grained semantic relationships. The concept of local affinity graphs and the derived local contrastive loss provide a rigorous mathematical foundation for multi-scale representation learning.\n(2) Despite the complexity of the multi-scale adaptive design, the authors provide strong theoretical guarantees. Theorem D.10 establishes that the convergence rate remains comparable to standard smooth non-convex optimization, which significantly bolsters the method’s credibility. The equivalence between the local contrastive loss and probabilistic graph alignment further solidifies the theoretical contributions.\nWeaknesses:\n(1) Dated baseline selection disproportionately relies on legacy methods rather than contemporary state-of-the-art approaches across both supervised and unsupervised paradigms. While demonstrating gains over established references, this creates an incomplete competitive landscape that fails to reflect current representation learning advancements.\n\nQUESTIONS:\nQuestions\nThe comparative experiments in the current results appear somewhat outdated (e.g., data from 2020). Are there updated comparative studies based on recent relevant literature? If available, could these revised comparative analyses be provided?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper tackles the problem of hierarchical contrastive learning. It introduces localized contrastive objectives and a dynamic multi-scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space. Theoretical analysis of convergence is provided. Results on five datasets verify the effectiveness of the method.

STRENGTHS AND WEAKNESSES:
### Strengths
+ Incorporating hierarchical structure into contrastive learning is an interesting topic.
+ The proposed method is theoretically proven to converge at a rate comparable to standard smooth non-convex methods.
+ The method shows promising performance on five datasets.

### Weaknesses
- The method is evaluated only with the ResNet-50 backbone. Its generalizability to other architectures such as ViT is not studied. 
- The ablation study presented in the main paper is limited and lacks sufficient detail. In particular, it seems that the inclusion of Equation (13) leads to performance degradation. This is not consistent with the statement "local contrast loss is essential".
- The method is only validated on hierarchical image classification tasks. Its applicability to other structured prediction tasks, such as hierarchical semantic segmentation [ref1], should be studied. 
- The batch size used in the experiments is not reported. How does the number of batch size affect learning performance ?

[ref1] Deep Hierarchical Semantic Segmentation. CVPR 2022.

QUESTIONS:
see weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents AMA-alignment, a method that evaluates the contrastive objective at multiple scales adaptively using a min-max objective, maximizing over a semantic region specific objective, ensuring that the most challenging regions are addressed adaptively. The motivation stems from coupling of semantic and representation affinities in order to minimize the divergence between the two at varying scales. Subsequently, local sub-graphs are sampled and incorporated into the main objective with theoretical motivation.\n\nSTRENGTHS AND WEAKNESSES:\n## Stregths\n* The work is adequately rigorous in theory and the motivations are well founded. \n* The framework with the alignment between semantic and representational matrices is novel.\n* Empirical evaluation shows promise.\n\n## Weaknesses\n* The assumption that the sampled sub-regions (especially in the unsupervised setting) may be semantically related may require empirical evidence.\n* The method may be analogous to hard-negative mining wherein samples semantically similar to the query are chosen from a pre-trained network using similarity based thresholds which contain inherent drawbacks which may be extensible to this method.\n* There may be drastic shifts in the regions if the fine-tuning is conducted over a different data distribution and the initial region specific objective may hinder the performance of the network depending on the strength of pre-training.\n* Computational overhead isn't discussed.\n* Certain implementation details are missing ($\\rho$ and $\\eta_{\\textbf{q}}$) and the robustness to these parameters isn't discussed. The work could benefit from an ablation over these parameters. How many samples per scale were used in practice and what criteria were used to determine \"coarse\" vs \"fine\" scales?\n* Other ablations over the number of scales and choice of radius are missing.\n* The work may be akin to multi-crop augmentation strategy [1] wherein the smaller crops are embedded closely with the main image. If so, is the procedural and computational overhead justified? Might the comparisons be fairer against [1, 2]?\n\n[1] Caron, Mathilde, et al. \"Unsupervised learning of visual features by contrasting cluster assignments.\" Advances in neural information processing systems 33 (2020): 9912-9924.\n\n[2] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\nQUESTIONS:\n* Was the network pre-trained over 200 epochs and subsequently fine-tuned using the method?\n* Could the authors present visual illustrations of the semantic relevance of the samples in regions?\n* How was the region radius determined in practice?\n* Might the method be applicable to text-image contrastive learning settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}