Title: The Complexity of Finding Local Optima in Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: contrastive learning

computational complexity

local search

PLS

CLS

reductions
Abstract: Contrastive learning is a powerful technique for discovering meaningful data representations by optimizing objectives based on $\textit{contrastive information}$, often given as a set of weighted triplets $\{(x_i, y_i^+, z_{i}^-)\}_{i = 1}^m$ indicating that an "anchor" $x_i$ is more similar to a "positive" example $y_i$ than to a "negative" example $z_i$. The goal is to find representations (e.g., embeddings in $\mathbb{R}^d$ or a tree metric) where anchors are placed closer to positive than to negative examples. While finding $\textit{global}$ optima of contrastive objectives is $\mathsf{NP}$-hard, the complexity of finding $\text{\textit{local}}$ optima---representations that do not improve by local search algorithms such as gradient-based methods---remains open. Our work settles the complexity of finding local optima in various contrastive learning problems by proving $\mathsf{PLS}$-hardness in discrete settings (e.g., maximize satisfied triplets) and $\mathsf{CLS}$-hardness in continuous settings (e.g., minimize Triplet Loss), where $\mathsf{PLS}$ (Polynomial Local Search) and $\mathsf{CLS}$ (Continuous Local Search) are well-studied complexity classes capturing local search dynamics in discrete and continuous optimization, respectively.  Our results imply that no polynomial time algorithm (local search or otherwise) can find a local optimum for various contrastive learning problems, unless  $\mathsf{PLS}\subseteq\mathsf{P}$ (or $\mathsf{CLS}\subseteq \mathsf{P}$ for continuous problems). Even in the unlikely scenario that $\mathsf{PLS}\subseteq\mathsf{P}$ (or  $\mathsf{CLS}\subseteq \mathsf{P}$), our reductions imply that there exist instances where local search algorithms need exponential time to reach a local optimum, even for $d=1$ (embeddings on a line).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies contrastive learning, where the input consists of a set of triplets $(x_i, y_i^+,z_i^- )$. This indicates $x_i$ is more similar to $y_i^+$ than to $z_i^-$. Each triplet is associated with weight $w_i$. The objective l is to learn an embedding such that the embedding of $x_i$ is closer to $y_i$ than to $z_i$ . The paper provides the computational complexity of finding locally optimal solutions with three loss formulations for both discrete and continuous settings. Its main contribution is providing proof that optimizing these contrastive loss functions is either PLS-hard or CLS-hard.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nComprehensive Theoretical study. \n\nWeakness:\nLack of empirical validation. Currently the result show the number of iterations increase exponentially with the number of vertices. How suboptimal the result would be if it is run using SGD for a finite number of epcohs, which is common in learning embedding.\n\nQUESTIONS:\n- Could you clarify the difference between the continuous and discrete settings? I assume the distinction is whether the inputs  $(x_i, y_i^+,z_i^- )$ take values in a continuous or discrete space. But loss is optimized over the space of embedding parameters $\\theta$. $\\theta$ can be continuous even if the   $(x_i, y_i^+,z_i^- )$ are discrete?\n\n- What does the weight $w_i$ represent in practice? How is the importance of a triplet determined?\n\n- I am interested to know if you can provide a practical application of the setting. \n- What would the complexity and suboptimality in an online (dynamic) optimization setting, where the model is updated based on the arrival of each new triplet?\n- Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) also consider learning from pairwise preference, where users indicate that they prefer $y$ to $z$. Can the theoretical analysis in this paper be extended to cover such settings?\n\nLIMITATIONS:\nI do not see any immediate potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper worked on the complexity of finding local minimum of contrastive learning loss. The main idea is to reduce a polynomial local search or continuous local search problem to the triple minimization/maximization problem, then use the existence theory of PLS and CLS to show the corresponding complexities. The bridge is the LocalMaxCut problem, which was proved to be PLS-hard. The author worked on several variants/cases in 1D and $\geq 2$-D of the contrastive loss. i.e LocalConstrastive-Euclidean problem, Local Constrastive-Tree problem, LocalBetweenness-Euclidean problem in both discrete case and continuous case. The main ideas in all the proofs are to add some certain triples that have height weights to show the equivalence between two problems.

STRENGTHS AND WEAKNESSES:
Strengths:

1. I think the author make an attempt to introduce the problem and all related works, since I believe they are not familiar with AI conferences.
2. The presentation is well-structured and clear. 
3. I believe that the results have some valid contribution to the theory of complexity of contrastive learning loss. 

Weaknesses:

1. As a practitioner, I do not find any practical benefit from this work when working with contrastive learning.
2. I am not familiar with most of the literature mentioned in the paper. However, I find the proofs are based on combinatorial structures, but all these structures are quite simple. Thus, I do not appreciate the novelty as well as the difficulty of presented proofs. 
3. No applications.

QUESTIONS:
Based on two points stated in the weaknesses, my decision is to reject, although I am not familiar with most of the literature.

LIMITATIONS:
Yes, it is theoretical work.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the computational complexity of finding local optima in contrastive learning, proving PLS-hardness for discrete objectives and CLS-hardness for continuous objectives, indicating that no polynomial-time algorithm can find local optima unless complexity classes collapse. It provides theoretical foundations for understanding the optimization challenges in contrastive learning and suggests future research directions, including average-case complexity and the quality of local solutions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. By focusing on the complexity of local optima (rather than just global optima), the paper connects empirical observations of slow or stuck convergence in contrastive learning to formal complexity-theoretic barriers.\n2. The paper delivers a comprehensive set of hardness results: PLS-hardness for maximization of contrastive triplets and betweenness objectives in both R^d and on trees, and CLS-hardness for continuous triplet loss minimization. These are rigorously demonstrated via reductions from classic hard problems (LocalMaxCut, QuadraticProgram-KKT).\n\nWeaknesses：\n1. There is little discussion or exploration of practical scenarios or real datasets in experiments --- leaving open the question of whether these hardness results are representative of typical training behavior in common machine learning settings beyond theoretical constructions.\n2. The paper focuses on the complexity in the worst case, but does not analyze the behavior in the average case, which makes it difficult to fully understand the complexity of local optima in contrastive learning.\n3. There are no corresponding methods or experiments proposed to address the complexity caused by the worst-case scenarios.\n\nQUESTIONS:\n1. Could the authors please clarify how worst-case scenario analysis for local optima contributes to either optimization efficiency or convergence reliability, and whether there are any existing mitigation strategies for such scenarios?\n\n2. The experimental evidence (Figure 5) relies on specific worst-case constructions inherited from LocalMaxCut. Is there empirical evidence or theoretical argument about how these scenarios relate to typical or randomly generated instances of contrastive learning? Do the exponential local search behaviors manifest on real datasets?\n\nLIMITATIONS:\nThe paper does not include extensive empirical experiments to validate the theoretical results. While the constructed hard instances demonstrate the potential for exponential time complexity, the paper does not provide a broad empirical study across different datasets and initialization methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the computational complexity of finding local optima in contrastive learning problems, a gap previously unexplored compared to global optimization.\nThey prove that finding local optima for both discrete and continuous contrastive objectives is PLS-hard or CLS-hard, respectively, even in low-dimensional settings.\nThis establishes that no polynomial-time algorithm can guarantee finding such optima unless major complexity class collapses (PLS ⊆ P or CLS ⊆ P) occur.\nThe authors establish PLS- and CLS-hardness of contrastive learning objectives by reducing from well-known hard problems such as LocalMaxCut (for discrete settings) and QuadraticProgram-KKT (for continuous settings). These reductions are designed to preserve local optimality, using specialized constraint gadgets that encode the structure of the original problems within the contrastive framework.\n\nThe authors used a synthetic graph from Monien and Tscheuschner (2010), known to be hard for local search, and transformed it into instances of contrastive learning problems via their reductions.\nTheir goal was to test whether local search behaves similarly across problems; the result showed identical exponential-time convergence, validating that worst-case hardness transfers empirically.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Novelty: First formal study of the complexity of local optima in contrastive learning.\nTechnical depth: Strong PLS and CLS reductions from canonical problems (LocalMaxCut, QuadraticProgram-KKT).\n- Generality: Hardness holds even in low dimensions (e.g., $d=1$), and across Euclidean, tree, and loss-based settings.\n- Theoretical rigor: Reductions are clean, constructions are well-motivated, and results are sharp.\n\nWeaknesses\n- Clarity: Some sections (especially CLS reductions) are dense; The paper assumes background in PLS/CLS; adding more intuition and context would help make it easier to follow for the broader ML community.\n- Empirical scope: Experiments are limited to synthetic worst-case instances. This is acceptable since the paper was submitted under Theory as the primary area, though broader empirical validation could strengthen future versions.\n\nQUESTIONS:\n1. In lines 118--121, you state that the PLS/CLS-hardness results extend to $\\varepsilon$-approximate local optima. Could you clarify for what values of $\\varepsilon = \\varepsilon(n, m)$---where $n = |V|$ (number of data points) and $m = |C|$ (number of triplet constraints)---the reductions remain valid? \n\n2.  Can you explicitly define the geometric configuration of points X, Y, Z in the 1D embedding?\n\n3. In lines 223--228, the proof shows that violating a boundary constraint results in a net loss, but it assumes that the total loss is upper bounded by $\\sum w_{uv}$. Why this upper bound always dominates the gain from violating any single boundary constraint? Including a concrete inequality to justify this would help formalize the argument and strengthen the correctness of the reduction.\n\n4. Theorem 4.1, It would be very helpful if you could list the 12 triplets used to encode a quadratic over $x, y, z$, and share a simple example (either symbolic or numeric) of the corresponding weights. This would really clarify how cross-terms like $xy$ are handled in the reduction.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}