Title: Variational Supervised Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Contrastive Learning

Variational Inference

Representation Learning

Evidence Lower Bound

Confidence-Adaptive Temperature

Image Classification

Temperature control
Abstract: Contrastive learning has proven to be highly efficient and adaptable in shaping representation spaces across diverse modalities by pulling similar samples together and pushing dissimilar ones apart. However, two key limitations persist: (1) Without explicit regulation of the embedding distribution, semantically related instances can inadvertently be pushed apart unless complementary signals guide pair selection, and (2) excessive reliance on large in-batch negatives and tailored augmentations hinders generalization. To address these limitations, we propose Variational Supervised Contrastive Learning (VarCon), which reformulates supervised contrastive learning as variational inference over latent class variables and maximizes a posterior-weighted evidence lower bound (ELBO) that replaces exhaustive pair-wise comparisons for efficient class-aware matching and grants fine-grained control over intra-class dispersion in the embedding space. Trained exclusively on image data, our experiments on CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1K show that VarCon (1) achieves state-of-the-art performance for contrastive learning frameworks, reaching 79.36% Top-1 accuracy on ImageNet-1K and 78.29% on CIFAR-100 with a ResNet-50 encoder while converging in just 200 epochs; (2) yields substantially clearer decision boundaries and semantic organization in the embedding space, as evidenced by KNN classification, hierarchical clustering results, and transfer-learning assessments; and (3) demonstrates superior performance in few-shot learning than supervised baseline and superior robustness across various augmentation strategies.


Peer reviews:
comments: SUMMARY:
The authors have derived a supervised contrastive learning solution using variational inference. Furthermore, they have introduced an adaptive temperature based on the model’s confidence.

STRENGTHS AND WEAKNESSES:
Strengths:

- Interesting idea with the confidence-adaptive temperature.

Weaknesses:

- Only a small improvement; the results from the second model are within the standard error.

- A large portion of the article focuses on less advanced areas, such as the ELBO derivations and the number of training epochs, which are not central to the main contribution.

- More focus should be placed on demonstrating the effects of the introduced solutions.

QUESTIONS:
- Is it necessary to include the ELBO derivations in the main paper? Similarly, are the ablation studies essential to present in the main text?
- Did you select the models for comparison in Table 1 based on architectural similarity, or because they are state-of-the-art for the task?
- How does the confidence-adaptive temperature differ from other importance-weighted approaches for similar objectives? Are there potentially better methods for this task?
- Have you analyzed the learned latent representations and compared them to baseline models?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-05

comments: SUMMARY:
VarCon is a framework that reinterprets supervised contrastive learning through the lens of variational inference over latent class variables. By deriving an ELBO for class-conditional likelihoods, VarCon replaces pairwise instance comparisons with efficient class-level objectives. Its loss combines a KL-divergence term and a log-posterior term, with confidence-adaptive temperature scaling to modulate supervision strength. The method outperforms prior supervised contrastive learning counterpart, SupCon, on multiple image benchmarks.

STRENGTHS AND WEAKNESSES:
## Strengths
- The paper is well-motivated and clearly written.
- It provides a rigorous probabilistic foundation for supervised contrastive learning by deriving the objective from a variational ELBO.
- The authors conduct a thorough evaluation of the impact of key hyperparameters.

## Weaknesses
- Although X-CLR is mentioned, the authors do not include empirical comparisons between VarCon and X-CLR, despite both methods requiring label supervision. This omission limits the ability to contextualize VarCon’s performance gains.

(minor) Tables 3 and 4 appear out of order.

Overall, I find this to be a strong paper with a solid contribution. However, the lack of discussion and empirical comparison with X-CLR, another supervised contrastive method, detracts from its completeness. A more thorough evaluation against X-CLR would strengthen the paper and better contextualize its improvements.

QUESTIONS:
- How does VarCon compare against X-CLR empirically?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Variational Supervised Contrastive Learning (VarCon), a new method incorporating variational inference with (supervised) contrastive learning. \n\nThe VarCon loss is derived by extracting the contrastive-related components from the ELBO of $\\log p_\\theta(z|r)$, resulting in a KL augmented CE loss, where the classification probability is based on the representation similarity, akin to contrastive learning.\n\nThe authors then propose a confidence-adaptive temperature for the KL's target distribution, which tightens pull strength for hard samples and relaxes it for confident ones.\n\nExperiments on image classification tasks show that VarCon outperforms several existing contrastive learning baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Theoretically Grounded and Intuitively Sound: The VarCon loss function is derived and simplified from the ELBO. This theoretical foundation leads to an intuitive combination of KL divergence and negative log-likelihood terms, which effectively regularizes embedding distributions while simultaneously maximizing classification likelihood.\n2. Demonstrated Superior Performance: VarCon consistently outperforms both existing self-supervised and supervised contrastive learning baselines across diverse datasets.\n\n### Weaknesses\n1. Inherent Supervision Requirement: \nThe VarCon method is fundamentally supervised. In contrast, a major advantage of exsiting contrastive learning methods (SimCLR, etc.) is their power in self-supervised settings.\n2. Lack of In-depth Analysis of the Adaptive Temperature: \nThe confidence-adaptive temperature, $\\tau_2$, is a central innovation, yet its behavior is not fully explored. \nThe ablation study on $\\epsilon$ only shows its effect on overall performance and fails to adequately verify the stated rationale of *\"adjusting supervision intensity\"* (L153). A deeper analysis is needed to fully understand how this adaptive mechanism functions and contributes to the method's success.\n\nQUESTIONS:\n1. As described in Weaknesses#2, can you include more ablation study regarding the adpative temperature? For example: \n    - Since $\\epsilon$ is learnable (L150), how does it evolve during training and thus affects the representations? \n    - Can adaptive temperature indeed improve the intra-class dispersion? Quantitative evidence supporting this claim would be valuable.\n    - Can similar effects by adaptive temperature be achieved with other regularization methods?\n2. Seems like it should be $\\log p_\\theta(z)$ rather than $p_\\theta$ in the LHS of Eq.(3)\n\nLIMITATIONS:\nThe authors have discussed the limitations in their manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Variational Supervised Contrastive Learning (VarCon), a framework that casts supervised contrastive learning as variational inference over latent class variables. The method derives an ELBO-based objective, incorporating a KL divergence term between an auxiliary distribution and the model posterior, alongside a log-likelihood term. Additionally, the paper proposes a confidence-adaptive temperature scaling mechanism to modulate label sharpness based on prediction confidence. The approach is evaluated on standard benchmarks (CIFAR, ImageNet) and shows consistent performance improvements over SupCon, particularly in few-shot learning and robustness under image corruption.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The ELBO-based objective is grounded in variational inference theory and clearly derived.\n\n- Empirical results are broad in scope and demonstrate consistent gains across tasks including few-shot learning, robustness, and transfer.\n\n- The method is architecture-agnostic and demonstrates performance across ResNet and ViT models.\n\nWeaknesses\n\n- The paper lacks component-wise ablations to isolate the contributions of KL divergence, log-likelihood, and confidence-adaptive temperature scaling.\n\n- Key claims such as improved intra-class compactness and difficulty-aware learning are not supported by empirical evidence (e.g., visualization or metrics).\n\n- The novelty is limited as the core idea builds on prior work; SimVAE[1] already explored a similar ELBO-style objective for self-supervised learning.\n\n- Important comparisons to other advanced SupCon extensions that also address sample difficulty, such as PaCo[2] and GPaCo[3], are missing.\n\n- There is limited insight into model behavior beyond accuracy, such as representation structure.\n\n[1] Bizeul, Alice, Bernhard Schölkopf, and Carl Allen. \"A Probabilistic Model behind Self-Supervised Learning.\" Transactions on Machine Learning Research.\n\n[2] Cui, Jiequan, et al. \"Parametric contrastive learning.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[3] Cui, Jiequan, et al. \"Generalized parametric contrastive learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 46.12 (2023): 7463-7474.\n\nQUESTIONS:\n- Could you provide ablations to quantify the individual contribution of the KL term, log-likelihood, and confidence-adaptive temperature?\n\n- Could you visualize confidence-adaptive temperature values during training to confirm that they correlate with sample confidence?\n\n- Can you provide t-SNE/UMAP visualizations or inter/intra-class distance statistics to support your claim of better representation structure?\n\n- Can you provide an empirical comparison against other SupCon extensions that handle sample difficulty, such as PaCo and GPaCo?\n\nLIMITATIONS:\nThe authors do not adequately discuss limitations. At minimum, it should discuss computational overhead, risk of overconfidence, or calibration issues under distribution shift. No societal concerns are discussed, though none are obvious.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}