Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling
Authors: Jialin Liu
Lisang Ding
Stanley Osher
Wotao Yin
Venue: ICLR 2026 Poster (2026)
Keywords: Implicit models
Deep equilibrium models
Expressive power
Primary area: learning theory

Abstract: Implicit models, an emerging model class, compute outputs by iterating a single parameter block to a fixed point. This architecture realizes an infinite-depth, weight-tied network that trains with constant memory, significantly reducing memory needs for the same level of performance compared to explicit models. While it is empirically known that these compact models can often match or even exceed the accuracy of larger explicit networks by allocating more test-time compute, the underlying reasons are not yet well understood.

We study this gap through a non-parametric analysis of expressive power. We provide a strict mathematical characterization, showing that a simple and regular implicit operator can, through iteration, progressively express more complex mappings. We prove that for a broad class of implicit models, this process allows the model's expressive power to grow with test-time compute, ultimately matching a much richer function class. The theory is validated across four domains: imaging, scientific computing, operations research, and LLM reasoning, demonstrating that as test-time iterations increase, the complexity of the learned mapping rises, while the solution quality simultaneously improves and stabilizes.

Published as a conference paper at ICLR 2026
EXPRESSIVE POWER OF IMPLICIT MODELS: RICH EQUILIBRIA AND BEST-TIME SCALING
Jialin Liu
Lisang Ding
School Data, Mathematical, and Statistical Sciences University of Central Florida
Department of Mathematics University of California, Los Angeles Los Angeles, CA 90095 1 lisangding@ucla.co
Orlando, FL 32826 jialin. liu@ucf. edu
Stanley Osher
Wotao Yin
Department of Mathematics University of California, Los Angeles Los Angeles, CA 90095 jo@mathh www.matth.ulla..e... ucla.
Decision Intelligence Lab, DAMO Academy Alibaba US Bellevue, WA 98004
Dettao.yineaibaaaa-ie com
ABSTRACT
Implicit models, emerging model class, compute outputs by iterating sin- gle parameter block to a fixed point. This architecture realizes an infinite-depth, weight-tied network that trains with constant memory, significantly reducing memory needs for the same level of performance compared to explicit models. While it is empirically known that these compact models can often match even exceed the accuracy of larger explicit networks by allocating more rest-time com- pute, the underlying reasons are not yet well understood.
We study this gap through a non-parametric analysis of expressive power. We provide strict mathematical characterization, showing that a simple and regular implicit operator can, through iteration, progressively express more complex map- pings. We prove that for broad class of implicit models, this process allows the model's expressive power to grow with test-time compute, ultimately matching a much richer function class. The theory is validated across four domains: imaging, scientific computing, operations research, and L.M reasoning, demonstrating that test-time iterations increase, the complexity of the learned mapping rises, while the solution quality simultancously improves and stabilizes.
INTRODUCTION
Many machine-learning tasks can be cast learning mapping F from input the desired output y-, i.e., F(æ). An emerging alternative is the implicit models: train an operator G whose fixed
point matches the target, i.e., y. Gyy, x) (Bai et al.) 2019 El Ghaoui et al. 2021). At inference time, the fixed point is obtained via root-finding solver. While advanced algorithms (e.g., Anderson acceleration or Broyden's 's method) exist, the canonical approach is the Picard iteration:
= ((yo. x) 3/2 = G()1,x), ys = G(y,,z),
(1)
and expect (x: + (x)x)= = F(x) for all æ. Rather than producing y, in single feed-forward
pass, implicit models reach the target through gradual cquilibrium-sekking updates. By tailoring the structure of G, implicit models have shown strong results across many domains (e.g.. imaging (Gilton et al. 2021), scientific computing (Marwah et al. (023). generative modeling (Pokle et al. 2022
Geng et al. 22023). LLM reasoning (Geiping et al. 2025), etc.).
Behind these successes, the advantages of implicit models include: (i) they realize an infinite-depth. weight-tied network trainable with constant memory, which yields efficient training (Fung et al.) 2022 Geng et al. 2021): (ii) they allow us to "implicitly bake in" domain constraints and structure (e.g., physics, geometry, safety), see Xie et al. y Güngör al. (2023); Oshin et al. (2024); and, most surprisingly, (iii) they can often match or even exceed the accuracy of larger explicit networks by allocating more iterations (Marwah et al. 2023
Wang al. 2024 Geiping et 2025) Point

---

Published as a conference paper at ICLR 2026
(i) stems from the weight-tied architecture and avoiding full back- socck-ropagation. Point (ii) arises from the inherently implicit nature of many real-world, equation- based constraints. In contrast, the mechanism underlying the surprising efffcctiveness of (iii) remains less well understood.
We study this through the lens of expressive power-the the set of input-output maps model family can represent. We ask two questions. First, as a baseline: (Q1) Do implicil models (at least) match the expressive power of explicit ones? Concretely, for target map F x H Ye, does there always exist an implicit operator G such that the iterates of (T) satisfy yr (x) + F(x) for all x? If yes, a more insightful question follows: (Q2) Do implicit models offer an expressive advantage? In particular, can a relatively simple implicit operator G. through iteration, represent a complex explicit map F? A positive answer to (Q2) would directly explain phenomenon (iii).
To our knowledge, these questions remain largely open. While universality has been touched upon in specific settings (Bai et al., |019 Marwah et al. (023) and separation results have demonstrated advantages over explicit models (Wu et al.) (024), a complete characterization of the representable function class of implicit models (and hence a direct answer to questions (QI) and (Q2)) is still missing. Unlike studies focusing on infinite-width limits and kernel connections (Gao et al. 2022. Feng & Kolter 2023 Ling et al. 2024), our work fills this gap from a nonparametric, function perspective, establishing that implicit model's expressive power scales with test-time compute.
(See Appendix for broader contextual discussions.) Specifically:
Expressive boundary. We identify locally Lipschitz mappings as a natural target class and prove: every such map can be expressed as the fixed point of a "regular" (simple and well- behaved) G, and conversely, every such fixed-point map is locally Lipschitz.
Emergent expressive power: Our theory. combined with iterative solvers' dynamics, yields a new viewpoint implicit models: the expressive power is progressively unlocked by iterations. Validation across domains. We validate our theory with case studies in wide range of applica- tions (e.g., image reconstruction, scientific computing, operations research, and LLM reasoning). Note that, while explicit networks are capable of expressing locally Lipschitz target maps (Beneven- tano et al., 2021) by scaling up the model size, implicit models can scale expressivity with test-time compute (i.e., more iterations at inference), which scales up test-time runtime but not parameters.
2
MAIN RESULTS
We now return to (Q1): given a target map F, does there exist an implicit operator G whose fixed- point iteration yields ye (a) + F(x)? A naive construction answers "yes": define, for 0 < 1j < 1,
(ly. x) = (1 - n) + FF()))
(2)
Then the fixed-poin iteration reduces to y:: (1 7) 7) + 1 F(x), hence y: F(x) (1 1)(111 - F(x)). As 0 < < 1, it holds that, for all x. (x)))) - F(x) -> 0 as t oc.
However, ( is merely trivial averaging of y and F(ær); learning such an implicit model is no different from learning F directly. This prompts the natural follow-up: is there any nontrivial implicit representation that is able to indicate the expressive benefits of implicit models?
An illustrative example. Let F(z) 1/x on [-1, 1] \ {0). This function is smooth (differentiable to any order) almost everywhere, but blows up near the singular point x = 0:
dF
1
[(x)] = 00, dx
00, as x - 0.
x
Neural networks approximating 1/2 on (-1,-5) U (6, 1] typically demands higher network com-
plexity-i.e.. increasing depth/width d - 0 to capture the growing steepness near the singularity (Telgarsky] 2017). If we adopt the naive implicit form (2) 9 x) (1 1) + n/2, nothing is gained: the model still inherits the singular behavior jag/dz| = 7/2=" pc.
What would be a nontrivial implicit representation in this setting? Instead of writing (1/z) explicitly, we can regard it as the solution of the equation zy - 1 = 0 (implicit representation). Inspired by 0: ( x) = y - 77 1). Using the general scheme in [. we have Yt = Vt - 17( 1). Subtracting the true solution gives
this, we apply a fixed fixed-point iteration to ry
1
1
1/ Vt-1 1 (1 mz) (
y/
x 9t-1
For any 0 7 and any (0, 1]. we have 0 (1 -n))= < 1 which implies yt + 1/x. (For x < O, simply flip the stepsize sign, 17 to - -..) This implicit formulation is much simpler and more elegant: the operator x) y 'n(ry- 1) has no singularity and no blow-up.
2

---

Published as a conference paper at ICLR 2026
The example indicates: intuitively, implicit representation can realize complicated map with singularities via a much simpler. smoother update operator G. Next, we make it precise: we for- mally define what we mean by "simple" versus "complex." and characterize- beyond the 1/r ex- ample-the class of target functions for which an implicit representation admits such a simple form.
Definition 2.1 (Lipschitz continuity). Let (x, || [I) and (%. || . II) be normed spaces, and let Q : X Y. We say Q is I.-lipschitz (globally Lipschitz) on X if there exists I. > 0 such that
[((())) - Q(=2))l & L || xz for all æ1, #2 E X,
and the smallest such L is the Lipschitz constant (or Lipschitz modulus), denoted as Lip( Q). If the Lipschitz constant L < 1, say Q is L-contractive on X. Given æ X, we say Q is locally Lipschitz at x if there exists a neighborhood U of x on which Q is Tag-Lipschitz continuous for some Ly > O. If Q is locally Lipschitz every X, we say Q is locally Lipschitz on X.
Intuitively, Lipschitz continuity limits how quickly function's value can change. When function is differentiable, its Lipschitz modulus can be characterized by the norm of its first derivative via the mean-value theorem. For example, F(x) 1/2 is locally Lipschitz on [-1, {0} but not globally Lipschitz there, since |dF/daa = 1/2² is unbounded as x + 0, causing local Lipschitz constants blow up near the singularity. In contrast, the implicit update 9(y,x) = y n(xy 1)
has simple partial derivatives |ag/ /Or = 17 y and |āg/ /Oyl = |1 - na0l without singularity.
Locally Lipschitz mappings form much richer class than globally Lipschitz ones. Typical examples (locally Lipschitz everywhere in their domains but not globally Lipschitz on the whole set) include: log x in (0, 1]. tan in 2 -) vx in (0, 1), ((x) in fO00, 11 22 J. etc.
For this reason, we refer to globally Lipschitz maps as "simple" operators and locally Lipschitz maps (which may exhibit large local slopes near certain inputs) as "complex." " Next, we formally state our main result: identifying a broad family of target functions for which implicit representations provide such simple update operators while expressing complex fixed-point mappings.
Assumption 2.2. Let C R.d and F X + IR. be locally Lipschitz. on X.
We do NOT assume the domain X to be bounded, compact, closed, or connected. For instance, X = R {0} excludes the singular point and permits = 1/x to blow up at the interior gap = 0 while remaining locally Lipschitz on X. Another example is X UKeZ (KT 5, km + 2). where F(x) tan x remains locally Lipschitz despite blowing up at the singularity points {kT 2 ]ikez.
We now formalize what we mean by "simple" update rules-namely regular implicit operators.
Definition 2.3 (Regular implicit operator). Let X C IRd be bounded. An operator G : IR" x X + IR" regular if: (i) For any y E IR",, the map x + G (gy. x)) is globally Lipschitz (w.rt. x)) on X. and the Lipschitz constant grows linearly w.r.t. |/ and (ii) For each x E X. there exists u(x) E (0, 1), the map y + (gy x) is u(x) ccontraccv on R". and fu()) is continuous w.r.t. x.
A regular G satisfies: (i) Fixing y. Gly. -) is globally Lipsehitz in æ, this makes it "simple" operator, and (ii) Fixing æ. ((,, x) is contructive in y; by Banach's theorem, this yields a unique fixed point (zr) and guarantees that iterates of (1 converge to it: (x)) (xx) An example of regular G the aforementioned G(y,*) x) = y - Tyay - 1) on æ E (0, 1] with < 1 < 1. Moreover, regularity does not require joint Lipschitz properties. With this definition,
present
main results.
Theorem 2.4 (Sufficiency). Under Assumption 2.2 for any F there exists regular implicit operator G: 1R' x X + RR" whose fixed- map reproduces F: Fix(G(- ((( x)) = F(x) for all a E X.
Theorem 2.5 (Necessity). Let X C Rd and let G R" x X + R² be regular. For every x E X, (-.xz) has unique fixed point y.. and the fixed map x + y, (z) is locally Lipschitz on X.
Proofs are deferred to Appendix A Theorem 2.4 provides an affirmative answer to (Q1) and (Q2) posed in the introduction. It proves that for any locally Lipschitz target F on a bounded domain, there exists a regular implicit operator G. whose iterations converge to the target (x) + F(x) for all œ. This demonstrates that the expressive power of implicit models not only matches that of explicit models but also provides a distinct expressive benefit: a relatively simple (regular) implicit representation can yield a complex fxedd-ooi mapping. Complementarily, Theorem 2.5 shows the boundary is tight: fixed points induced by any regular & are necessarily locally Lipschitz. Together, the two results give exact expressivity characterization for regular implicit models.
3

---

Published as a conference paper at ICLR 2026
10
yV
y
5
-5
5
5
-5
-
-10
10
-10
10
-10
Target map F(()
36(*]
Fixed-point map (x(x)
Figure 1: (Conceptual diagram) A simple implicit update expresses complex map via iteration.
What does our theory imply? Take locally Lipschitz target F (c.g., the curve in Fig. D. Our sults guarantce the existence of a regular implicit operator such that the iteration yr = (y:-1, x)
with Yo 0 converges: (x) + F(x).. Consider the first iterate and its Lipschitz property:
||9 x (O, x'Il
y (x)) = G(O. x)
Lip(y)) = sup a,.r
Lip(G(O,))) )))
x x''
Because a regular operator G globally Lipschitz by definition, y+ is restricted to representing "simple." globally smooth mappings. However, as iterations progress, Y converges toward F.. If the target F features singularities (regions where local slopes become large or unbounded), the effective Lipschitz constant of the iterate y+(-) naturally grows with / to match that complexity:
[g (a)) (=x'))
[F(()) F(x')]l
lim
(100
x - x'|
x - x'|
This dynamic highlights a fundamental distinction: while explicit networks scale model size to approximate locally Lipschitz targets (Beneventano et al. (02I), implicit models scale expressivity with rest-time compute. A regular implicit operator can yield a complex equilibrium, and iterating this single operator at test time progressively unlocks that complexity without adding parameters.
Generalization. Someone may ask: does a large Lipschitz constant of the fixed-point map yar)) imply sensitivity poor generalization (ef. Paabbarauu et al. (202T))² Our view is that this sensi- tivity is inherent to the target F, not to the implicit representation any faithful model, explicit or implicit, must track F's sharp variations. Our case studies in Section B confirm this: the target F in many tasks is indeed steep somewhere and the effective Lipschitz grows as accuracy improves, Cru- cially, the implicit representation can realize such targets with simple operator G, which regularizes training and supports good generalization in practice.
Insights for practitioners. A substantial line of work (c.g., El Ghaoui et al. 202 Winston & Kolter (2020); Jafarpour et al. (2021); Revay et al. 22020;; ; Havens et al. (023) enforces global Lipschitz bound on the fixed-point map y. (z). Typically, the model is parameterized as G(y, x) = (Ay + Bx b). and by imposing specific algebraic structure A and B, one ensures that y. (x) is globally Lipschitz in æ. While this indeed improves robustness, our theory shows constrains expressivity and undercuts the unique advantage of implicit models. Our recommendation is different: rather than imposing uniform Lipschitz constraints, incorporate case-by-case domain- specific knowledge, priors, or constraints (as illustrated in our case studies Sec. B}. This method provides effective regularization, leading to robustness and strong test performance while unlocking the full power of implicit models- repressettpeeeenngneg complex maps with relatively simple operators.
3 CASE STUDIES
In this section, we present four case studies. For the first three tasks, we (i) verify that the target satisfies Assumption 2.2; (ii) specify domain-informed architecture for G: (iii) confirm empirically that, under standard training without explicitly enforcing 9 to be regular, the learned operators G exhibit these properties-ice i... G Lipschitz in æ and iterates Y: converge (see Appendix (F) for train- ing strategies and discussions regarding regularity guarantees); and (iv) demonstrate that expressive power scales with test-time iterations. Finally, we extend this analysis to LLM reasoning to validate our predictions in domain where strict mathematical definitions are less applicable.
3.1 CASE STUDY I: IMAGE RECONSTRUCTION (INVERSE PROBLEMS)
Inverse problems in imaging seek to recover image y. E R from partial, noisy measurements x = Ay. + n E Rd (d < n),, where A a known linear operator and n noise. A common prior is that y. lies near smooth data manifold MI C R". To recover y,, standard estimator solves
min yER*
1
2
+
a dist fist?(y,)) (y,Ml) M)),
(3)
2
2
4

---

Published as a conference paper at ICLR 2026
or, equivalently, variable splitting surrogate
min || Ayl2 + 2
at dist (2, M) +
2 21v =11².
3
(4)
M.
Next we will show that, under mild assumptions, both 33 and (4) admit a unique minimizer for each in a bounded set, and the solution map æ + ()x) is locally Lipschitz. Hence the reconstruction target falls within Assumption 2.2 and covered by our expressivity results in Section 2
Assumption 3.1. Let M. C R" be a compact, C², embedded (possibly nonconvex) submanifold with positive reach > 0. Assume the forward operator IR"" +R Rd (μ, (i--))-b--Lipphie when restricted to M and let max denote the maximal singular value of A.
These assumptions are modest: they are standard in prior work and supported by existing theory. Formal definitions (reach and bi-Lipschitz continuity) and relevant literature appear in Appendix C
Definition 3.2. Define the admissible set of observations x for (3) and
X := x: x= Ay. + n, for some yz E M,
|12l||
80 80
Theorem 3.3. Under Assumption 3.1 there exists a > 0 for all a E X. such that the minimization problem 3 yields unique minimizer y. Let Fia: + denote the associated solution map from imput x to the recovery y. Then Fia is locally Lipschitz continuous on X.
Theorem 3.4. Under Assumption 3.7 there exist a, B > 0 for all x E X such that the minimization (4) yields a unique minimizer (9, 2). Let F1s: x: + ÿ denote the associated solution map
problem
from input to the recovery ÿ. Then Fib is locally Lipschitz continuous
X.
Corollary 3.5. There must be a regular implicit operator G(y, x)) such that (i (.)) = (xx)
for all x X. The same conclusion holds for Fib(x).
Proofs of the theorems are deferred to Appendix C and Corollary 3.5 follows immediately from Theorems (.3 (34 This corollary guarantees the existence of regular implicit models G for image reconstruction. Next, we present how to implement G in this context.
Problem-specificc G.
We adopt algorithm-inspien 4.. Parameterizing these iterative solvers gives problem-tailored implicit models. In particular,
designs that mirror classical solvers for ( and
Option (PGD-style). To solve (3. if MI were known, one would use proximal gradient descent (PGD): y++1 pyo A (Ay: - z)), with parameters 7, >> O, where prox, is the proximal map of (a/2)disst (y, M) (see Appendix C.1. In practice, we replace prox, by learnable neural network denoiser Ha. (parameters and noise level input or) and obtain
Ge(y x) = Ho.o
A (Ay =)). θ == (0,00,7)
(5)
Option II (HQS-style). For (4). a standard solver is half-quadratic splitting (HQS, see Ap- pendix C.2 Similar to Option 1, we replace the proximal map by a learned module and obtain
Ge(g,x)= = Ho.o (4) (A
+ x +Bb))
e = (0, 0,
(6)
Here adopt the long-standing "plug-in denoiser" idea from Venkatakrishnan replaces proximal operator with an off-the-shelf denoiser inside an iterative solver; in our case, we train the entire Ge as an implicit model. Details and bibliography
al. (2013), which
given in Appendix C.2
Experiment settings. We study image deblurring, x A(y.) n. where A is motion-blur operator and n is additive Gaussian noise. Using BSDS500 (Martin et al. 200T), we construct 200 training. 100 validation, and 200 test pairs (xx, y=). yielding datasets Dnvearriin Danvval. and D Inv.tet- Implementation details (data preprocessing, model choices, and training) are in Appendix G
For evaluation, we analyze 100 iterations of the learned dynamics, Vt+1 (x) (3)).
t S 99 and yo = 0, on the test set Dinnee = {(., y² ),200 For each i, we create 5 perturbed ground truths yr 1 < < 5. and for each vi, we apply A, add noise, and then obtain æt.j- The
perturbed pairs {(rt, ( )i, form the perturbed dataset D Diissess Details appear in Appendix G
We track two metrics, including an empirical Lipschitz estimate and reconstruction quality in PSNR (i.e.. Peak Signal-to Noisee Ratio, higher PSNR means more accurate reconstruction):
(x.)) Ye (z:,))) and Pi(i.)) = PSNRRy, vi,)):
L = max max
1S05200 200 19355
x ær,
for 1 < i < 200,0 < S 5, where j = means the original (unperturbed) sample, E;o0 == F;, V. = V. Here, L, estimates how complex the t-th iterate map v-(() is, while P. measures the reconstruction quality on both the original dataset Desy and the perturbed set IT/ (T/
5

---

Published as a conference paper at ICLR 2026
PNSR (dB)
30
Lipschitz
Implicit (PGD)
Implicit (PGD)
Implicit (HOS)
Implicit (HQS)
20
40
60
80
100
20
40
60
80
100
Iteration
Iteration
(a) Empirical Lipschitz Lt of (-(-) vs. iteration. Lt starts small at f=1 and grows to plateau (~-5), indicating increasing expressivity of y-(-).
(b) Reconstruction quality Pt (mean + std over the original and perturbed test samples) inereases and stabilizes: ye(x) converges toward the truth.
Figure 2:
Validation image deblurring. Iterating simple operator Ge produces complex fixed- point mapping: Lipschitz (a) grows, while accuracy (b) improves and stabilizes.
Ground Truth
Observation
Explicit:28..o 49dB
Imp-PGD:30.0332
Imp-Hip H0--HQQS:31330
26.97 = 3.54 dB
28.20 + 3.70 dB
29.18 3.66 dB
Params: 32.64 M
Params: 32.64 M Params: 32.64 M
Figure 3: Visual results for deblurring. The top PSNR values (28.49, 30.03, or 31.53 dB) correspond the single visualized image; the second line shows the average std) over all test samples.
Experiment results. (i) Results in Figure 2 support our theory. Figure 2a plots I, versus 1, while Figure 2b the mean + std of (i. versus f. At = 1, the mapping y1 (x) (O, a) reflects single application of Ge and exhibits low Lipschitz constant: L1 = 0.140 for PGD and Li 0.436 for HQS. As & increases, y: approaches the fixed point and L; grows substantially, saturating around xe 5.0 for both models (Figure 2a). Meanwhile, the PSNR rises and stabilizes, indicating that yo (x) converges toward the ground truth (Figure 2b}. Thus, the increase in L, does not reflect divergence or instability; rather, it captures the greater complexity of the underlying target mapping / Yo which is progressively expressed through iteration. (û) We also provide comparison (both visually and quantitatively) to an explicit model in Figure 3 This baseline uses the identical DRUnet and is trained on the deblurring dataset with end-to-end MSE loss. A visual inspection reveals that implicit models, particularly implicit HQS (6). produce sharper images with better-recovered textures and fewer artifacts than the explicit baseline. This perceptual advantage is corroborated by the quantitative metrics, where the DEQ-HQS model achieves significant PSNR gain of over 2dB on average across the entire test set. (iii) Additional experiments showing a small implicit model outperforming larger explicit ones appear in Appendix G
3.2 CASE STUDY 2: SCIENTIFIC COMPUTING
The Navier-Stokes (NS) equations are foundational to computational fluid dynamics. We focus on the 2D steady-state incompressible case on a periodic domain : [0 2
(u V)u + Vp vAu + S, V w=== on Ω
where - R the velocity field, p - is the pressure, v > is the viscosity, f: Ω + 122 is the external foree. Solving NS equations refers to determining given f. Although global existenceesmoothness of the solution given general forcings famously open, classical results guarantee well posedpssss under suitable conditions on f-
Theorem 3.6 (Temam (1995). There exists constant
O depending only such that, if
|| S CV then 7 admits unique solution u. (f). Let Hi denote the space of admissible forcing and set B, = [f HEEH H: I| I cv²). Then there exists subset Hy, C By that is dense in B., on which the solution map f + u. (.. )) is locally Lipschitz.
Details regarding the function spaces are provided in Appendix

---

Published as a conference paper at ICLR 2026
Vorticity form. Let == V (and hence Vxu,), Under periodic boundary and zero-mean conditions, one can recover the velocity u from vorticity w by solving a Poisson equation (Majda
et al. 2002). We hence focus the solution map in vorticity: f + w..
While Theoremm3.6ggvves a local Lipschitz result in function spaces, our expressivity results (Section 2 are stated for finie-dimensional spaces. To bridge this gap, we discretize the NS equations.
Discretization. Partition Ω2 into N₂ cells 22 {Ci) and define the cell-average restriction
Rx(f)lc = j0] Sc d( (similarly for w). We work with the discrete forcings and vorticities:
= Rr (f) R.N,x2 y == (w) IR No.
and aim to learn x y where y, == (a.,,) is the discrete solution in vorticity form. Back to the continuum setting, let the lifting operator &n be the piecewise-constnt reconstruction &n (a+) Ecea, sc and let P be the orthogonal projection onto divergence- free, zero-mean fields.
Corollary 3.7. F2 : y, is locally Lipschitz on X == fo E R. N. X2, (x)) E H and there exists regular implicit operator ((y, x)) satisfying (i(, = F2(2)) on X.,h-
The corollary instantiates our expressivity theory for steady-state NS, guaranteeing the existence of a regular implicit model G. As in the image-reconttruction case, we now (i) choose a problem- specific parameterization of G and (ii) verify our theory numerically on this architecture.
Parameterization. We use Marwah et al. (2023) as our code base: z, Golz.. (x))). and
y. = -2((z.) The core Ge is implemented as Fourier Neural Operator (FNO) {Li et al. 2021 and both the encoder 24 and decoder 2y use pointwise MLP Details appear in Appendix H
Experiments. We use the dataset of Marwah et al. (023) with viscosity v 0.01, which provides 4500 training pairs and 500 test pairs (x) y²), where x is the discretized force and y* is the corre- sponding vorticity: we denote these sets by Dpdearrin and pde.uss. Details are given in Appendix H]
We test iteration-wise behavior for 50 steps starting from 2o = O: z++1 = Ge 2o (ax) for 0 < t < 49, and (xx) - (v(z.) Analogous to the inverse-problem study, we augment the test set with perturbations. For each (ar:; y/² E pddetest. we construct 15 perturbed vorticities {
we then compute compatible forces { by evaluating the NS operator (see Appendix for details). The perturbed test set is IDV, (dtttteu = { vi,)) : 1 < i S 500, 1 J S 15]. Across iterations we report an empirical Lipsehitz estimate I., and relative reconstruction error Ey:
15
(a.) y. and Ex(i,)) = |1 vi,
L = max max 5505500 151515
|1, I1,i
| +
for 1 < i < 500,0 < j < 15, where j 0 means the original (unperturbed) sample, 0i,0 == 2; y7.0 = y, Therefore, E, evaluates accuracy on both Dpdeses and IV
pde.test
The results in Figure 4 align with our theory. At = 1, the mapping y (x) reflects a single applica- tion of Ge and exhibits low Lipschitz constant: L1 23.1. As iterations proceed toward the fixed point, the complexity grows markedly: Le increases to x 367 by t = 50 (Figure 4ap. Meanwhile, the relative error E, decreases monotonically and stabilizes 0.078 = 0.028 (Figure 4b). indicating convergence to a good approximation of y.. Thus, the learned operator Ge is simple (Lipschitz in ær), while additional test-time ilerations let y, realize progressively more complex mappings. In addition, a comparison with an explicit baseline (vanilla FNO) in FigureB]shows the implicit model produces more accurate solutions, both visually and quantitatively. Additional experiments showing a small implicit model outperforming larger explicit ones appear in Appendix H
3.3 CASE STUDY 3: OPERATIONS RESEARCH
Linear program (LP) is foundamental to operations research, of which a general form is given by
min y, s.t. Ay b, i y S u. VER"
Here, A Rmxnn b IRm C RR. L R E R", and E {= denotes componentwise relations, i.c., each E { := (A, b, c, o,
specifies whether (Ay); cquals or is bounded above by by Let (, u) as the input that discribes the LP in (). In addition, we consider those feasible and bounded LPs (which admit an optimal solution (Bertsimas & Titsikkis 1997))
Inttrouucing additional encoder and decoder is common in practice. Compared to the vanilla formulation y. G(y., .)), does not change our expressivity results in Section 2 Details appear in Appendix B
7

---

Published as a conference paper at ICLR 2026
400
150. 3671
1.5
Lipschitz 4,
200
Relative Errors
1.0
100
50
0.5
550, 0.0784]
(1, 23.31
20
0.0
10
20
30
40
50
0
10
20
30
40
50
Iteration t
Iteration
(a) Empirical Lipschitz Le of the (-step map (+)-) vs. iteration. La starts small t=1 (23.1) and grows substantially, reaching ~ 367 by f=50.
(b) Relative error E. (mean 1 std over original and perturbed inputs) vs. iteration. E, decreases
0.078 0.028 Yt converges towards y..
Iterating simple operator Ge yields complex exed-point mapping: Lipschitz constant (a) increases, while error (b) decreases.
Figure 4: Validation the steady Navier Stokes task.
Navier-Stokes
Ground Truth y.
Prediction by FNO
Error: 0.260
Prediction by implicit FNO
Error: 0.112 0.078 + 0.028
0.179 + 0.050
Figure 5: Visual results for NS equations. The top value (0.260 and 0.) means the relative error (between the prediction and the ground truth) on the single visualized sample: the second line shows the average relative error (+ std) over all test samples. Both models have 2.376 parameters.
X = {(A, b, c, o, u): The resulting LP is feasible and bounded)
Within X, there are some LPs where the solution mapping is not single-valued or not continuous. By excluding these LPs, forms a subset Xsub C X on which F3 is single-valued and locally Lipschitz. The strict definition of Xab and the proof of Theorem 3.8 are provided in Appendix
Theorem 3.8. There is a subset Xgub C X. that is dense in X. on which each LP admits a unique solution ye, and the solution map F3 x y. is locally Lipschitz continuous Xsub-
Corollary 3.9. There exists a regular implicit model (3y x)) with Fix x))) Fs(x)) on Xgub-
Corollary 3.9 follows immediately from Theorems 2.4 3.8 indicates the existence of implicit models with desired properties that solves LP. As in the previous case studies, problem-specific parameterization of G and (ii) verify the theory numerically on this architecture.
(i) choose
Implicit GNN parameterization. We model the implicit operator G for LP with graph neural network (GNN). First, express an LP instance æ = (A, b, c, o, L, u) as a bipartite graph (Figure 6). We create variable nodes {V and constraint nodes (W) m Node features collect the data of the LP: each V, stores (cy. uy): each W, stores (b,, o.). We connect W; to V, if Ai # O, and place A,y on that edge as its feature. Given this representation, an (explicit) GNN can map the I.P solution, i.e., y. GNN(x) where denotes the graph-encoded LP. This approach was proposed in Gasse et al. (2019), and Chen et al. (2023) subsequently showed that (explicit) GNNs offer a universal framework for representing LPs. Built on this, we propose an implicit GNN:
Ge(z., (a:),, y. = -v
where Ge is the core GNN. Q. encodes instance instance-specifee per-variable states to the solution. Both Qs and Qo are small MLPs shared across all nodes. At
specific (static) features from x, and 2y decodes
inference, we repeatedly call Go with initialization 2a 0: Z, = Ge(z,1, 2+(x)) for 1,2, ... T, and finally output y, = (x,). Relative to prior work, our only modification is to attach to each variable node an additional dynamic state 2/ E RR. Details appear Appendix
There is rapidly growing literature implicit GNNs with diverse applications and theories (Gu et al. 2020 Park et al. (2022; Chen et al., 2022a b. Baker et al., [023 Lin et al. 2024 Nastorg et al.
2024 Zhong al. 2024 Yang et al. 2025)_ ). Our LP study complementary to this line of
work: rather than adopting particular implicitGNNi architecture, we start from standard explicit
GNN for I.P and convert it into fixed-point formulation tailored linear programs.
8

---

Published as a conference paper at ICLR 2026
node W1,
1411
node Vi
(0)
min
61 M1
02 V/
Co Va
12
M1
1
M1, 12 S
y2
ls
node
12
M/
A12 1/2
node W
Azg
node V2
Ag2 M/2
A23 9/3
A23
Ma
static parts
20
static parts
A linear program
Graph representation
Implicit GNN iteratively called
Figure The graph representation of LP and implicit GNN applied on this graph
10a
Lipschitz L
Relative Erore
0.6
11. 0.75]
0.5
10a
Perturaing
0.4
Perturaing
0.3
18. 6.1611
Pettrring
10
Parturbinng
0.2
Farturting
0.1
2
2
8
Iteration t
Iteration
(a) Empirical Lipschitz Ly of the t-step map y=(-) vs. iteration. Le starts small t=1 and grows substantially by f=8 for perturbation modes.
(b) Relative error Eg (mean + std over original and perturbed inputs) vs. iteration. E, decreases 0.146 0.091-y, converges towards y...
Figure 7: Numerical validation the linear-progran task.
Experiments. We sample LP instances a (A, b, o, L, u), solve it to obtain an optimal solution y., and form 2,500 training pairs and 1,000 test pairs like (æ, y.). denoted D. and D' DP We also create five perturbed test sets {D L ) by altering exactly one block among (A, b, c, l. or u.) For each (æ,, v.) E DLAses and each perturbation type j. we form perturbed instance Zijs solve it to obtain yī and collect TD(6)
Lf {(xi, 10000 Details Appendix We report:
I
[1(( y (x. and Ex(i,) = (39) y7,ll
Le(i) = max 15151000
|0; 3ial
1y +
for 1 < i < 1000, 0 S < 5, where j = O denotes the unperturbed pair (æ Vi,o) = (x,, yi Results support our theory. (i) Figure palpt Filprer7alplots the five curves Le()) (one for each perturbation type). At = 1, a single application of (9) yields relatively small empirical Lipschitz constants for all per- turbation modes. As iterations proceed toward the fixed point, Lipschitz constants grow markedly. (ii) Figure 7b reports the sta of (i, E, decreases and stabilizes at 0.146, indicating that the growth of I, reflects the higher intrinsic complexity of the solution mapping y, (a) rather than divergence or instability. (ii) Table [ contrasts implicit and explicit GNNs. At matched embedding sizes, implicit GNNs match or beat explicit ones most clearly small/mid sizes (4/8/16). In addi- tion, a smaller implicit model can outperform a larger explicit model on training error. For example, implicit-4 explicit-8 (0.203 0.233) and implicit-8 vs. explicit-16 16 (0. vs. 0.183). This supports our theory that iterating a simple implicit operator can yield strong expressivity.
Discussion generalization. While generalization is our main focus, trend in Table [ is informative: explicit GNNs improve as width increases from 4 to 8 but then overfit (test error signif- icantly rises at 16/32), whereas implicit GNNs improve from 4
16 and only tick up slightly at 32. We attribute this to: (i) LP constraints Ay o b in (8) are specified implicitly rather than as an ex- plicit set; implicit models align naturally with such structure. and (ii) while fixed-point maps y. (x)
Table I: Comparison between explicit GNNs and implicit GNNs on the LP task.
Emb. size
4
8
16
32
# Params.
580
2,088
7,888
30,624
Exp-GNNs
Err (Train)
0.387 + 0.103
0.233 + 0.084
0.183 + 0.070
0.1 + 0.049
(Test)
0.397 0.107
0.273 + 0.104
0.283 0.111
0.318 0.122
Emb. size
16
32
Params.
722
2,350
8,390
31,606
Imp-GNNs
Em (Train)
0.203 + 007
0.162 + 0.094
0.131 + 0.080
0. + 0.073
Lir (Test)
0.218 1 0.117
0.177 1 0.5
0.152 1 0.098
0.156 + 0.109

---

Published as a conference paper at ICLR 2026
can be sensitive to inputs æ, the implicit formulation allows us realize them via a simpler, smaller operator G, which "implicitly" regularizes training and support good generalization in practice.
3.4 CASE STUDY 4: LLM REASONING
While previous case studies focused on domains with strict mathematical defimitions (inverse prob- lems, PDDEs I.Ps), we now investigate if our theory extends to broader applications where metrics like "smoothness" and "Lipschitz continuity" are less formally defined. We examine the looped transformer for LLM reasoning, utilizing the pre-trained model from Geiping et al. (2025). Unlike standard feed-forward transformers, this architecture recycles a shared block Ge to iteratively update a latent "thought" vector 2: 2 = Ge 12 (z)), Y = Qv(z)) where Qo encodes the input æ, and Qy decodes the latent state into the output sequence ye obtained after t recurrent blocks.
Strictly extending our Lipschitz theory to the discrete space of language tokens is challenging, as standard norms do not apply. However, we can empirically test the core prediction of our theory: can the model express increasingly complex mappings as iterations increase? In this context, complexity implies: subtle differences in the input correspond substantial shifts in context. A capable model must effectively distinguish these semantic nuances and produce vastly different responses.
Qualitative Results. Table 2 visualizes the evolution of reasoning on a typical example. At early iterations (1 = 2, 2), the model fails to differentiate context (Physics vs. Finance), producing rep- etitions or shallow associations. Conversely, with more iterations utilizes increased test-time compute to resolve this ambiguity, correctly defining "charge" as electric potential versus financial debt. This confirms the implicit operator's ability to progressively realize complex, context-sensitvv mappings.
6, 8 or more), the model
Table 2: Evolution of model outputs for the inputs "charge and voltage" vs. "charge and pay". showing how the semantic difference emerges over iterations.
Input/Output
Input/Outpat
Comments
explain the difference between charge and vollage.
s': explain the difference between charge
Prempts ditter single word, but the context shifls from Physics to Finance.
Yo (ar): explain difference between charge and voltage.
va(ar'): explain difference between charge and
Both outputs merely echo puts.
(ar): explain the difference between charge and vollage. explain the difference between cur- rent and volluge.
explain the difference between charge and pay. explain the difference between the two.
34 (æ=):
Outputs repeat the inputs with slight variations: contexts are yet separated.
(ae): The difference between voltage and charge is that voltage is the difference between the potential difference between two points, while charge is the difference beiween the elec- tric field hetween two points.
yrs(a"):: ): explain the difference between charge and pay. Explain the difference between charge and pay.
Separation begins: Output moves into a Physics expla- nation (potential difference). while Outgut 2 still stays near prompt.
ye (ue): The difference between churge and voll- age is that voltage is the difference in electric potential between two puints in an electric field, while charge the amount of electric charge in a system.
Wa (ar'): ): Charge umouun of money that person owes to company organization, Pay the amount of money that a person receives from company or organization.
Full separation: Output gives a Physics definition;
Ouipul 2 correctlyy adopts the Financial interpretation (moncy owed vs, received).
Maz Vaz(ar): (ar): Charge is the amount of electric charge present body. Voltage is the difference in electric potential between two points.
Vas (an') " Charge is the amount of money that person or company owes credit card com- pany. Pay is the amount of money that a person or company has paid to credit card company.
Refinement: hoth domains have stable, concise, and curate definitions specialized
to Physics versus Finance.
To quantify this effect, we define an empirical Lipschitz measure based on Levenshtein distance over paired prompts that differ by 1-2 words, and track its geometric mean across iterations. Details are given in Appendix J The observed rise with t indicates increasing sensitivity to subtle input changes, consistent with progressively sharper context separation.
Quantitative Results.
4 CONCLUSIONS AND FUTURE DIRECTIONS
We have provided a strict characterization of the representational capacity of regular implicit mod- els. Our analysis reveals that iterating a simple operator allows the model progressively realize increasingly complex mappings. ultimately covering the entire class of locally Lipschitz functions. This theory is validated through four diverse case studies, showing that the empirical Lipschitz con- stant rises alongside solution quality. Our codes are available online
Available at: httpa://githaa Hugj 11gtt//MP-PPww
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Osher was supported by DARPA
DoE DE-SC0026262. and NSF 2208272.
REFERENCES
Eddie Aamari, Jisu Kim, Frédéric Chazal, Bertrand Michel, Alessandro Rinaldo, and Larry Wasser- man. Estimating the reach of manifold. Electronic Journal of Statistics, ([11):1359-199) 2019.
Iskander Azangulov, George Deligiannidis, and Judith Rousseau. Convergence of diffusion models under the manifold hypothesis in high- high-dimensions. arXiv preprint VrXiv:2409.18800 2024.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Justin Baker, Qingsong Wang, Cory D Hauck, and Bao Wang. Implicit graph neural networks: A monotone operator viewpoint. International Conference on Machine Learning, pp. 1521-1548. PMLR, 2023.
Richard Baraniuk and Michael B Wakin. Random projections of smooth manifolds. Foundations of compulational mathemulics, 9()):11-77, 2009.
Pierfrancesco Beneventano, Patrick Cheridito, Robin Graeber, Arnuff Jentzen, and Benno Kuck- uck. Deep neural network approximation theory for high-dimensional functions. arXiv preprint arXiv:2/ 14523, 2021.
Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena scientific Belmont, MA, 1997.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Gregery T Buzzard, Stanley H Chan, Suhas Srechari, and Charles A Bouman. Plug-and-play un- plugged: Optimization-free reconstruction using consensus equilibrium. SIAM Journal on Imag- ing Sciences, 11(3):2001-200 -2000, 2018
Emmanuel J Candes and Terence Tao. Near-optimal signal recovery from random projections: Uni- versal encoding strategies? IEEE Transactions on Information Theory, 52(12):5406-442) 2006.
Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1): 84-98, 2016.
Qi Chen, Yifei Wang, Yisen Wang, Jianlong Chang, Qi Tian, Jiansheng Yang, and Zhouchen Lin. Efficient and scalable implicit graph neural networks with virtual equilibrium. In IEEE Interna- tional Conference on Big Data (Big Data), pp. 864-873. IEEE, 2022a.
Qi Chen, Yifei Wang. Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induuer graph implicit nonlinear diffusion. In International Conference on Machine Learning, pp. 3648-3661. PMLR, 2022b.
Tianlong Chen, Weiyi Zhang, Zhou Jingyang, Shiyu Chang, Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. Advances in Neural Information Processing Systems, 33:7332-7343, 2020.
2020.
Tianlong Chen, Xiaohan Chen. Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning optimize: A primer and benchmark. Journal of Machine Learning Research, 23(189):1-59) 2022c.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of un- folded ISTA and its practical weights and thresholds. Advances Neural Information Processing Systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2026
Ziang Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. On representing Linear Programs by
Graph Neural Networks. In International Conference on Learning Representations, 2023. URL
http::::///penreiie...... net/ http://www..e//ooueeeeeeeeeeee http:/orummmidd2ooo-0------
Kenneth L Clarkson. Tighter bounds for random projections of manifolds. In Proceedings of the Twenty-Fourth Annual Symposium on Computational Geometry, pp. 39-48, 2008.
Christian Daniele, Silvia Villa, Samuel Vaiter, and Luca Calatroni. Deep equilibrium models for
Poisson imaging inverse problems via Mirror Descent. arXiv preprint arXiv:250711161, 2025,
Manfredo P Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
David L Donoho and Carrie Grimes. Image manifolds which are isometric to Euclidean space.
Journal of Mathematical Imaging and Vision, 23(1):5 2005.
Asen L Dontchev and R Tyrrell Rockafellar. Characterizations of strong regularity for variational inequalities over polyhedral convex sets. SIAM Journal on Optimization, 6(4):1087-1105, 1996.
Asen L Dontchev and R Tyrrell Rockafellar. Implicit functions and solution mappings, volume 543.
Springer, 2009.
Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Matematics of Data Science, 3(3):9309558, 2021.
Herbert Federer. Curvature measures. Transactions of the American Mathematical Society, 93(3): 418-491, 1959.
Zhili Feng and J Zico Kolter. On the Neural Tangent Kernel of equilibrium models. arXiv preprint arXiv:2310. 14062, 2023.
Gerald B. Folland. Advanced Calculus. University of Washington, Seattle, WA, 2nd edition. 2023. Freely available at /attps://sitess math. wwshington.cncommm..mo wdw. wtwpl/ooanddddooa//////daaaaaaaa wow.aaa//Adaaaale..... AAd/Calcy/cc/////
Samy Wu Fung and Benjamin Berkels. A generalization bound for a family of implicit networks.
arXiv preprint arXiv:2410.07427, 2024.
Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. JFB: Jacobian-free backpropagation for implicit networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao. A global convergence theory for deep ReLU implicit networks via over-parameterizaatoo.. In International Confer- ence on Learning Representations, 2022. URL
www.:nnntfoounnnee forum?id
R332S766 6R 1 XxS
Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combi- natorial optimization with graph convolutional neural networks. Advances Neural Information Processing Systems, 32, 2019.
Ruturaj G Gavaskar and Kunal N Chaudhury. Plug-and-play ISTA converges with kernel denoisers. IEEE Signal Processing Letters, 27:610-614, 2020.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.051177 011.552...51112000 2025.
Zhengyang Geng, Xin-Yu Zhang, Shaojic Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems, 34:24247-24206 2021.
Zhengyang Geng, Ashwini Pokle. and J Zico Kolter. One-step diffusion distillation via deep equi- librium models. Advances in Neural Information Processing Systems, 36:41914-41931, 2023.
Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging, 7:1123-1133, 2021.
12

---

Published as a conference paper at ICLR 2026
Alexandros Gkillas, Dimitris Ampeliotis, and Kostas Berberidis. An celimiization-basee Deep Equilibrium Model for hyperspectral image deconvolution with convergence guarantees. arXiv preprint arXiv:2306.06.3 06378, 2023.
Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent EI Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems, 33:1 1984-11995, 2020.
Alper Güngör, Baris Askin, Damla Alptekin Soydan, Can Bariş Top. Emine Ulku Saritas, and Tolga Cukur. DEQ-MPI: A deep equilibrium reconstruction with learned consistency for magnetic particle imaging. IEEE Transactions on Medical Imaging. 43(1):321 :331-334, 2023.
Aaron Havens, Alexandre Araujo, Siddharth Garg, Farshad Khorrami, and Bin Hu. Exploiting con- nections between Lipschitz structures for certifiably robust deep equilibrium models. Advances in Neural Information Processing Systemx, (6:21155-21674 2023.
Chinmay Hegde and Richard G Baraniuk. Signal recovery on incoherent manifolds. IEEE Transac- tions on Information Theory, 58(12)):200 2012.
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Gradient step denoiser for convergent Plug-and-Play. In International Conference on Learning Representations, 2022a. URL https:
//openreview. wttp://wwwwee/.ooommiieeeeeeeeee
for wtt.rrmmmmdffffeeeeee. d3Ckk
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Proximal denoiser for convergent plug- and-play optimization with nonconvex regularization. In International Conference on Machine Learning. pp. 9505, PMLR, 2022b.
Mark A Iwen and Mauro Maggioni. Approximation of points low-dimensional manifolds via random linear projections. Information and Inference: A Journal of the IMA, 2(1):1-31, 2013.
Saber Jafarpour, Alexander Davydov, Anton Proskurnikov, and Francesco Bullo. Robust implicit networks via non- -on-Euclidean contractions. Advances in Neural Information Processing Systems, 34:9857-9868, 2021.
Ulugbek S Kamilov, Hassan Mansour, and Brendt Wohlberg. plug-and-pla priors approach for solving nonlinear imaging inverse problems. IEEE Signal Processing Letters, 24712):187--1876 2017.
Ulugbek Kamilov, Charles A Bouman, Gregery T Buzzard, and Brendt Wohlberg. Plug-and- play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 40(1):85 97, 2023.
Gunther Leobacher and Alexander Steinicke. Existence, uniqueness and regularity of the projection onto differentiable manifolds. Annals of global analysis and geometry, 60(3)5559-587, 2021.
Anat Levin, Yair Weiss, Frédo Durand, and William Freeman. Understanding and evaluating Blind Deconvolution algorithms. In JEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 1964-1971, 2009.
doi: 10.1109/CVPR.20009.0. 5206815.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for parametric par- tial differential equations. In International Conference on Learning Representations, 2021. URL
net ttte//oorumm1d==333000000
Junchao Lin, Zenan Ling, Zhanbo Feng, Jingwen Xu, Minxuan Liao, Feng Zhou, Tianqi Hou, Zhenyu Liao, and Robert C Qiu. IGNN-Solver: graph neural solver for Implicit Graph Neural Networks. arXiv preprint arXiv:2410.08522, 2024.
Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global convergence of aver-parameterized deep equilibrium models. In International Conference on Artificial Intelli- gence and Statistics, pp. 767-787. PMLR, 2023.
Zenan Ling, Longbo Li, Zhanbo Feng, Yixuan Zhang, Feng Zhou, Robert C Qiu, and Zhenyu Liao. Deep Equilibrium Models are almost equivalent to not-so-deep explicit models for high- dimensional Gaussian mixtures. In International Conference on Machine Learning, pp. 30585- 30609. PMLR, 2024.
13

---

Published as a conference paper at ICLR 2026
Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. ALISTA: Analytic weights are as good as learned weights in LISTA. In International Conference on Learning Representations, 2019.
Jiaming Liu, Salman Asif, Brendt Wohlberg, and Ulugbek Kamilov. Recovery analysis for plug-and- play priors using the restricted eigenvalue condition. Advances in Neural Information Processing Systems, 34:5921-5933, 2021.
Andrew J Majda, Andrea L Bertozzi, and A Ogawa. Vorticity and incompressible flow. cambridge texts in applied mathematics. Appl. Mech. Rev., 55(4):B77-B78, 2002.
Olvi L Mangasarian and T-H Shiau. Lipschitzconttinitty of solutions of linear inequalities, programs and complementarity problems. SIAM Journal on Control and Optimization, 25(3):583-555, 1987.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference Computer Vision (ICCV), volume 2, pp. 416-423, July 2001.
Tanya Marwah, Ashwini Pokle, Zico Kolter, Zachary Lipton, Jianfeng Lu, and Andrej Risteski. Deep equilibrium based neural operators for steady-state PDEs. Advances in Neural Information Processing Systems, 36:15716-15733 2023.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for Generative Adversarial Networks. In International Conference on Learning Representations, 2018. http::/:/ppeeeeeeeeeeeeee//rrrrp
Davide Murari, Takashi Furuya, and Carola-Bibiane Schönlieb. Approximation theory for 1- lipschitz resnets. arXiv preprint arXiv:2505. /2003, 2025.
Florent Nacry and Lionel Thibault. Distance function associated to a prox-regular set. Set- Valued and Variational Analysis, pp. 1-20, 2022.
Matthieu Nastorg, Micsele-Alessandro Bucci, Thibault Faney, Jean-Marc Gratien, Guillaume Charpiat, and Mare Schoenauer. An implicit GNN solver for Poisson-like problems. Computers & Mathematics with Applications, 176:270-288, 2024.
Alex Oshin, Hassan Almubarak, and Evangelos A. Theodorou. Differentiable Robust Model Predictive Control. In Robotics: Science and Systems (RSS), 2024. URL https://
rrboticscooferencce.. wwtpor/gg///ooooeee http:///////rrgaaa//paapss//////
Chirag Pabbaraju, Ezra Winston, and J Zico Kolter. Estimating Lipschitz constants of monotone deep equilibrium models. In International Conference on Learning Representations, 2021.
Junyoung Park, Jinhyun Choo, and Jinkyoo Park. Convergent graph solvers. In International Confer- ence on Learning Representations, 2022. URL htttppps//openrevie...eeeeee//
ItkxIQUO11D
Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion
models. Advances Neural Information Processing Systems, 35:37975-37990, 2022.
René Poliquin, R Rockafellar, and Lionel Thibault. Local differentiability of distance functions.
Transactions of the American mathematical Society, 352(11))5211-5544, 2000.
Peter Potaptchik, Iskander Azangulov, and George Deligiannidis. Linear convergence of diffusion models under the manifold hypothesis. arXiv preprint urXiv:2410.09046 2024.
Arash MexmanMey Aboozar Ghaffari, and Emad Fatemizadeh. Plug and play augmented HQS: Convergence analysis and its application in MRI reconstruction. Neurocomputing, 518:1-14,
2023.
Max Revay, Ruigang Wang, and Ian R Manchester. Lipschitz bounded equilibrium networks. arXiv preprint arXiv:2010.0/332 2020.
Stephen M Robinson. Strongly regular generalized equations. Mathematics of Operations Research, 5(1 511):43-62, 1980.
14

---

Published as a conference paper at ICLR 2026
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290(55000)2333-2255 2326, 2000.
Emest Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. Plug-and and play methods provably converge with properly trained denoisers. In International Conference on Machine Learning. pp.
PMLR, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions Neural Networks, 20(1):61-80, 2008
Vineet R Shenoy, Suhas Lohit, Hassan Mansour, Rama Chellappa, and Tim K Marks. Recovering Pulse Waves from video using Deep Unrolling and Deep Equilibrium Models. arXiv preprint arXiv:2503. 17269, 2025.
Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online plug-and-play algorithm for regular- ized image reconstruction. IEEE Transactions on Computational Imaging, 5(3):395 4088, 2019.
Julián Tachella, Matthieu Terris, Samuel Hurault, Andrew Wang. Dongdong Chen, Minh-Hai Nguyen, Maxime Song, Thomas Davies, Leo Davy, Jonathan Dong, Paul Escande, Johannes Hertrich, Zhiyuan Hu, Tobías I. Liaudat, Nils Laurent, Brett Levac. Mathurin Massias, Thomas Moreau, Thibaut Modrzyk. Brayan Monroy, Sebastian Neumayer, Jérémy Scanvic, Florian Sar- ron, Victor Sechaud, Georg Schramm, Romain Vo, and Pierre Weiss. DeepInverse: A Python package for solving imaging inverse problems with deep learning. Technical Report 2505.20160, arXiv, 2025. https: /tttpp://arxii..o. wttp:/xwwwv..../////./oaa///////
20160
Rong Tang and Yun Yang. Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statisties, pp. 1648 1656. PMLR, 2024.
Matus Telgarsky. Neural networks and rational functions. In International Conference on Machine Learning. pp. 3387-3393 PMLR, 2017.
Roger Temam. Navier-Stokes equations and nonlinear functional analysis. SIAM, 1995.
Lan V. Truong. Global convergence rate of deep equilibrium models with general activations. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 945-948. IEEE, 2013.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Michael Wakin. Manifold-based signal recovery and parameter estimation from compressive measurements. arXiv preprint 100. 2010.
Haixin Wang, Jianlong Chang, Yihang Zhai, Xiao Luo, Jinan Sun, Zhouchen Lin, and Qi Tian. Lion: Implicit vision prompt tuning. In Proceedings of the AAAI Conference Artificial Intelligence, 2024.
Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in Neural Information Processing Systems, 33:10718-10725 2020.
Zhoutong Wu, Yimu Zhang, Cong Fang, and Zhouchen Lin. Separation and bias of deep equilib- rium models on expressivity and learning dynamics. Advances in Neural Information Processing Systems, 37:32476-322 32511, 2024.
Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Optimization in- duced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3604-36 2022.
15

---

Published as a conference paper at ICLR 2026
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are Graph Neural
Networks? International Conference on Learning Representations, 2019. URL https:
Chengda Yang. Nonlinear image recovery with half-quadratic regularization, IEEE Transactions on Image Processing, 4(7):932-946, 1995.
Yongyi Yang, Tang Liu, Yangkun Wang, Zengfeng Huang, and David Wipf. Implicit unfolded Graph Neural Networks. Journal of Machine Learning Research, 26(82):1-44 2025.
Youhao Yu and Richard M Dansereau. MsDC-DEQ-Net: Deep Equilibrium Model (DEQ) with multiscale dilated convolution for image compressive sensing (CS). IET Signal Processing, 2024 (1):6666549, 2024.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-pp -layy image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44( (1)::)660-6366 2021.
Yongjian Zhong, Hieu Vu, Tianbao Yang, and Bijaya Adhikari. Efficient and effective implicit dy- namic graph neural network. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4595-4606, 2024.
Zihao Zou, Jiaming Liu, Brendt Wohlberg, and Ulugbek Kamilov. Deep equilibrium learning of explicit regularization functionals for imaging inverse problems. IEEE Open Journal of Signal Processing, 4:390-398, 2023.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Proofs of Main Results
18
A.1 Proof of sufficiency
18
A.2 Proof of necessity
23
Variant Architecture
26
Proofs of Theorems for Inverse Problems
26
Proximal operator near a manifold
33
C.2
Discussions regarding PnP
35
D
Proofs regarding NS Equations
35
Proofs regarding Linear Programming
37
Training Strategies
Experiment Details regarding Image Reconstruction
42
H Experiment Details regarding Scientific Computing
43
Experiment Details regarding I.P
46
Additional Results regarding LLM Reasoning
47
Broader Contextual Discussions
48
LLM Usage Statement
49
17

---

Published as a conference paper at ICLR 2026
A PROOFS OF MAIN RESULTS
The core intuition behind our proof' is an extension of the 1 //x example discussed the introduction.
For Theorem 2.4 (Sufficiency), construct the implicit operator G dynamic interpolation: (y. x) = (1 - E(x)) = which iteratively pulls the state y toward the target t(rgetF(())) with step size (x)). The key theoretical innovation is making this step size adaptive: we construct e(x) to be inversely proportional to the local steepness (Lipschitz constant) of the target y- (æ). In regions where the target function becomes extremely steep or singular (like æ 0 for 1/2),, our constructed E(x) naturally vanishes. This effectively "slows down" the dynamics, ensuring the operator G itself remains globally smooth and contractive.
Theorem 2.5 (Necessity) establishes the converse: we show that for any regular operator, the local steepness of the fixed point mathematically bounded by the operator's parameters (g-contraction modulus μ ()))); and the fixed point map y. (xr) can only become singular if the convergence rate slows down (contraction modulus > 1), perfectly matching the mechanism used in our sufficiency construction.
A.1 PROOF OF SUFFICIENCY
Proof of Theorem 2.4 Given any F satisfying Assumption 2.2 the existence of G is proved by the following construction:
x) F(x) + - =(x)) F(x)).
(10)
The proof will be done by choosing a function - such that
Functions <<x) and 5 F(x)F(() are both globally Lipschitz continuous on X.
* 0 < <(x) < 1 for any x E X.
The existence of such E function is deferred to Theorem A.4
Now let's suppose such s(x) is given and finish the whole proof. First let's check the contractivity of & in (To) as æ fixed. For any y. ÿ E IR., holds that
C a) G(ū, x) = -1 (1- (-)-)(F))))) - (1 s F(x)) = (1 (y D))
Since < E(x) < 1 for x E X, we conclude that ((, w) is contractor for x e X. In addition, the continuity of the contractive factor (1 - =(x)) is directly resulted from the continuity of exx). Finally, we check the Lipschitz continuity as fixed. For any x, & E X and any E RR,,, it holds that
x) - G(y,
(g(y - v) (g(y, -
(F(xz) - y + (1 - =(x))) F(xx)) -
= =- --((x)(y - F(x)) + =())(y - F(+))
- (())(x) + = - - ((()))())
e(x)
With fixed E I the Lipschitz continuity of G .) follows from the Lipschitz continuity of (ar) and F(x). In partieular, by denoting the Lipschitz constants of e (xx) and e(x) Lee and T respectively, we have
[ x) g a))l s Le æll ||yl|
|| + LE 2|| - (Le yll + LEF) ||
øll
where the Lipschitz constant of g. L. == T.elly/ + Tugr grows linearly w.r.t. || which finishes the whole proof.
Li
Below we provide the core theorems used in the proof of Theorem 2.4 We first consider X to be bounded (Theorem A. and then extend the results to the unbounded domain (Theorem A.4)
18

---

Published as a conference paper at ICLR 2026
Theorem A.1. For any F satisfying Assumption 2.2 defined bounded domain X C R2, there exists a function E X + IR such that 0 E(x) < for a E X, and E(x) and F(x) are both globally Lipschitz continuous X.
Proof. Let X be the closure of set X. In this proof, we will first extend F to X. construct the function on X, and finally prove the global Lipsehitz continuity of E(x) and c(x)F(()) on X.
Step 1: Extension to First we extend F to X by the limit relative to X:
lim x=a-E 0,
F(x),
lim F(z) exists, x-2-
F())
otherwise.
Note that even if F is continuously extendable to æ still possible that F is not locally Lipschitz continuous the point æ. A simple example is the function VI, which is continuous locally Lipschitz continuous for all points x > 0 but NOT locally Lipschitz at x = We collect all these points (where F is not locally Lipschitz) into the set D(F):
2 0 and
(FF)= = x x : is not locally Lipschitz continuous at x:
For brevity, we will use II denote D(F). holds that ID a closed set (ref. to Lemma A.2 and X.
Step 2: Constructing a function e X + I2 Now let's define set including all points that are very "safe", i.e., sufficiently far from the discontinuity set ID. In particular, given a positive real number r > 0, the set DD, is define by
D, = {o E X D)
where d(x; I)) means the distance of x and ID, and the closedness of D, can be derived from the continuity the distance function. Since Dy C X and is compact, D, must be compact. Note that D. and D are disjoint, hence J is locally Lipschitz continuous everywhere on ID,, Thanks to the fact that local Lipschitz continuity on a compact set implies global Lipschitz continuity (ref to Lemma A.3). we can conclude that F is bounded and globally Lipschitz continuous DD for all r 0. Therefore, the following two supremums exist, as long as the cardinality (number of elements) of D, is large enough:
[F(x)) F(xx))
sup E..
(Tr, 2 2,
(r(
21 x22
0,
otherwise.
sup |F())]| menh
card (D,) 2 1,
h2(r)=
0.
otherwise.
Here, both h and h2 are non-negative and monotone non non-increasing on (0, o00)) Then we define:
i((r) Tn(r) + (2(r)++1111))
It has the following properties:
Bounded: 0 h(r))
Monotone M(r)) _ (((2) as 0 < rı S r2. (Due to the monotonicity of h and h2)
Naturally extended to 0: lim,-+0, h(()) exists. (Due to the monotonicity h)
(())4.()) < for 2 and = 2,2.
These properties implies that à Riemann integrable on [0, +oo). Then we can define the following function:
é(r) = / h(s)ds
with the following properties:
19

---

Published as a conference paper at ICLR 2026
E(0) 0.
Monotone increasing. This is a straightforward result of the fact that h(s) > 0 for >> 0.
Strictly positive as > This also straightforward as h(s)) for >
1-Lipschitz continuous on [0. +oo). For any T1. with 0 S ri S r2 +oo, we have
(#(r1) - E((r)] = É(r2) - {(11) I," h(s)ds
suu (upp |r1 r2 lri -2/- r20
With such a (r)), we can define =(x) by
(((x) ())
<(x)
+ E [(xx
holds that =(x) for æ E and < s(x) < for x E X\D. As C XXX, we have 0 < <(x) < for E X.
Step 3: Establishing the Lipsethiz continuity. Since the distance function d(xr, D) 1-Lipschitz continuous (Federer) [959] Theorem 4.8 (I)), the Lipschitz continuity of é implies the Lipschitz continuity of c. In particular, for all æ1, X, it holds that
((x)) s(x22)
é {(d(o),13)
é {(d(22,
1-Lipschitz as
1 + E ((( D))
1 1 ((2))
1 1 x
(1+2)
(1 + x))
1 (d(x), D) - (d((22
(Lipschitz continuity of é)
d(< D) d(x;, D)
| x2
(Lipschitz continuity of d)
Therefore, to complete the whole proof, it's enough to show the global Lipschitz continuity of aF on X. As X is compact, and thanks to Lemma A.3 it's enough to show eF is locally Lipschitz everywhere on X.
First, we consider the local Lipschitz continuity of eF on X.D. Due to Lemma A.2 X must be open relative to X. For any æ E IDD there must be a small enough r > 0 such that U = B(x, r) nx c X.D. Pick æ1, U. For any æ1, 22, it holds that
[l((z))F(xz) ((x2)
=(1e(x))F(zz) =<x1))((22) (x2) + = - =(x2) {=(x))][F(x,) - F(æ2)) + [{()) E(x2) - 1(((2))
(11)
Since both & and F are locally Lipschitz and locally bounded everywhere on X\D, they must be Lipschitz and bounded within U. Then the local Lipschitz continuity of eF at x immediately follows from [1). Note that is arbitrarily picked from X,D, hence eF is locally Lipschitz everywhere on X.D.
Next, we consider the local Lipschitz continuity of eF on D. For any E ID, we consider its neighborhood U = (ar, 1) X and pick æı #2 U. Then need consider three cases. The first case is both æ1, Ø2 belong to the discontinuity set D: æτ, Ø2 E D. In this case, it holds that E(x1) = =(x2) = 0 and hence
[(a)) - ((x2) (ea)) = o || - x221-
The second case is that one of the point is in while the other is not, we suppose # E DD, 02 E X.D, then
[(())) - e(x2)
20

---

Published as a conference paper at ICLR 2026
= [((2)) s (((2,, {(((o>)))) (F(z))l
(1rtee h()dds) |
<h(d(a2, d(az. |
(Monontonicity of h))
<h(d(x2, D)) d(222 D) hy(d(z2,,))
(Definition of h2)
<d(x2, D)
(h(r) h2(r) < 1as 0)
=d(x2, D) d( D) S 21 x2l
Finally, we consider the last case where σ1, D2 E M.D
D.
Without loss of generality, we assume
0 < d D) < ((22,DD)
Then the definition of hı and h2 implies that
[[F(x1) - F(x2)
I <ax (11 (d(((.)) D)), (d(x2, | -x2
x2
-h, (d(x), D)) |ل| x2l1.
and
| & h2(d((22, D)).
Consequently, applying 111 and the above inequalities, we have
[l(()))F((1) ((x2)
<<(<))) [(x1) F(æ2)) + (((;)) - =(x2)1 F(x2)]] Sc(x)) h1 (d(21) D)) ||||| æ2l + (s(m))
{xx2) (e(d((22,)))) D))
<s (((x1. ID) hi (d(21, D)) |||| x21l (d((z)) ((12) h2(d(x2,
(1 ( h(s)ds) hi (d(()), D)) | - |211+ (1
dlm h(s)dl h2(d(22, D))
<d(a1, D). h(d(a), D)) hn (d(z), D)) || x2ll
+ [(x) IB) ((2r2 m²) ((2(222 M)) 122(d(22, D))
<d || 22l1 + d(æ DD) - d( (2, D)
D) |||| 22ll ||ا| x211
The last inequality results from /(()) (hi(r) h2(r)) < 1 for all > 0. And the above inequalities imply
- (
< (diam(X) 1) a - 222/
Combining all the results together, we have eF is locally (diam(X) + 1)-Lihschitz at any æ e X. Then the compactness of X concludes the global Lipschitz continuous of eF, which finishes the whole proof.
Follows are some lemmas (as well as their proofs) that we used in the proof of Theorem A.I
Lemma A.2. Let T C. Rd be closed and let F T + R" Denote by D(F) C. T the set of points at which F is not locally Lipschitz. Then TB(F) is closed (in T,, hence in Rd).
Proof. Recall that F is locally Lipschitz (relative to T) at E T if there exist r >> 0 and L > 0 such that
Fu() - F(w) < L 1| w1l Let G = (FF be the set points where F is locally Lipschitz. We first show that is relatively open in T. Fix G and choose r, L. as above. If ar E TTB((, r r/2), then B(x', rr2) C
for all v TTT B(x,
21

---

Published as a conference paper at ICLR 2026
B(x, r); hence the same L works T B(x', r/2). so F is locally Lipschitz at x'. Therefore T B(x, r/2) C G, proving that G is open in T. Consequently, D(F) = TQG is closed in T. Since T is closed in RR, every set closed in is also closed in Rd Hence (FF) closed in IR as well.
Lemma A.3. Let be a compact set. If F is locally Lipschitz everywhere on T, then must be globally Lipschitz T.
Proof. Assume, to the contrary, that F is not globally Lipschitz T. Then we can choose quences (#x (yx C T such that
[F((m)) F(y:)ll k |
0c.
(12)
yall
Local Lipschitzness implies continuity of on T, so by compactness F is bounded: there exists C' < 3o with [[(()]]] S C for all T. Consequently,
for all k. [[F(xx)) F(yx) - 20
and therefore
12 forces [a
ykl -> 0.
By sequential compactness of T, passing subsequence (not relabeled) may assume 2% E T; since || yk + 0, we also have yı: + æ. Since F is locally Lipschitz at æ, for k large enough have
|{(())) F(yx)l L,
D y
for some L > O, which contradicts 12
Therefore F must be globally Lipschitz on T.
Now we relax the condition in Theorem A. and extend it to unbounded domains.
Theorem A.4. For any X Rd (not necessarity bounded) and any locally finctiont F X R. there exists function X - R such that 0 E(x) < for x X, and e(x) and F(x) are both globally Lipschitz continuous on X.
function
Proof. We consider the following grids in Rdd
k
(2kg 2kg ... 2kg), ky 2, i [d].
We define the closed d-dimensional cubic centered at k by
3
3
3.
3
3
2ka + 3 2
Cs 2k:
2k
+
2kz
2' 2k2 +
x .... 2ka
2
According to Theorem A.I there exists a function En defined on Cs X, that
0 < (x) < 1 on Cx X.
(x) is I-Lipschitz on Cę n X.
(x())F(x)) is (diam(C*) + Li-Lisschitz on Ck n X.
Next, we concatenate all these E functions and get global E concatenation function in 1-dimensional space:
X -> R. We define the following
3
3
+
D E
2
2 1 2'2
2
1
1
r E
p(x)
[
+
1 E
2
2
0,
otherwise.
22

---

Published as a conference paper at ICLR 2026
Then define the d-dimensional concatenation function. Let (I1, 22 22, Fd).
d
(x)) = II p(xs).
1
We define the shifting function Pk
(4) (a)) = pl))(x k). On X. we construct
=(x)
Pk (d)
(x))x(<))
KE (E(
Given the constructed E(r), it's enough to prove that: s(x) < 1, s(x) and ((x)F(x) are
globally Lipschitz over X. We will show these claims one by one and finish the proof.
First, let's show 0 < <xx) < 1. As p(d) is non-negative and 0 < (x))) < 1, we have
0 << <(x) <
Pk (d)
(x) =1
where En (() (a) comes from the fact that each term p(z; - 2k;) depends only on its spe- cific index k; and not on the others, and hence we can distribute the summation E, -, a.bj
(E, by). That is,
Σ (4) (x) k
Pk
Σ EEles
p(1o p(11 1 (((1-2kk) -2hi)
x
(E
Σ p(1z 2h22) kgEZ
Σ VeyEZ
plara 2kid)
KIEZ
-1x
Second, we prove the Lipschitz continuity of <(t). In particular, it holds that
|=(x)) -(())
Σ
Pk (4)
(x)
Σ (d) (#))
Pk
Σ
(d) (Ex - (x)) + Σ (Pr (pla((()
Pk (=)))x(())
(4)
Pk
(d)
(x) [k(x)) =x(i)) + Σ
Px (1) (z) Pk (x) [x((#)
(d)
Pk
Svdlla
< x bl 2d Vdl= क|| (1+ 21
El|
where 2dd comes from the fact most 2d grid cubes overlap at each point
Third, using the same argument, we can show that F(x) is globally Lipschitz over X, and the Lipschitz constant is bounded by (3v0+1+ (Recall that diam(Cx) 3dd), which finishes the proof.
A.2 PROOF OF NECESSITY
we adopt a similar idea: first considering a bounded domain and then extending the results to unbounded domains.
For Theorem 2.5
Theorem A.5. Let X C TR. be a bounded domain and let G TR x X - R be regular. Then, for every X, the map y + G(y, æ) has unique fixed point y. (2),, and the resulting fixed-point map y. (x) must be locally Lipschitz on X.
Proof. Let X be the closure of X. In this proof, we will first extend the operator G to R x X. and then analyze its properties on this closed domain.
Step 1: Extension to X. For any y E IR., G(y, m) is globally Lipschitz continuous X, hence its
extension is naturally define by
g(y. x)
lim Oly, for all w XXX.
X32 +
23

---

Published as a conference paper at ICLR 2026
Different from the proof of Theorem A.1 where FF might be not locally Lipschitz æ even if it is continuous at æ, here the extended & must be Lipschitz at a and hence Lipschitz on the overall set X. This can be verified by examining the difference quotient for 1i o 22g and y E RR:
Agly: 01, 02) == [[ æ0) Gly x22)
[
T2
Let Cly, y'' Lipschitz constant on % L(y) == SuPe, Ag æ1, Ø2). For any æ X and
22 E XXX, it holds that
A y: 11 [2] =
lim X3
Agly; 1t,, [t]
sup
S L(y)
22 Ei
For any á # 32 E X we have
Agly: 31, .2] = lim lim
Agly: a1, x2)
sup an a2 Eaauttt
Ag 21, = L(y)
Xami -8 X322 222
Therefore, we obtain an upper bound for gly. '''s Lipschitz constant on X:
sup #1
2g(y;01,302)
maxx
sup xx
Ag 11, #2]
sup ,x
Ag 21, F2).
sup
~
S max (L(g). L(y) L(y)
That is, for any R. (y..)) is globally Lipschitz on X, and the Lipschitz constant is the same with that of X.
In the other hand, let's consider the Lipschitz constant (contraction constant) w.r.t y when fixing a E X
p (
lim u(x) x
Since 0 < (()) < 1 for E X. by taking limit, we have 0 S u( < 1. For those z with p()) < 1, the operator G(,, Φ) is still contractive. But if p(œ) = 1, the operator ((. is not contractive.
Step 2: Defining and D,,, We collect all points æ E X where the operator g(,, 2)) is not contrac- tive:
D = {x X u(xx) = 1}
and define a "safe" set that is sufficiently far from D:
D, x E X d(x, (D) 2 r).
Note that X = Urso Dr and X c D.D We obtain
x x xCD r>0
For any D, with O, can obtain uniform contraction of the operator ((, There is constant /4, E (0, 1) such that
19 x) G(y2, m1 S prl yal
(13)
for all y1, y2 E R2 and æ E D,,, which follows immediately from the continuity of μ(x)) and the compactness of D,. By the Banach fixed-point theorem, the operator ((,,2)) must have a unique fixed point y. for each x D,..
To complete the proof of Theorem 2.5 thanks to the fact that X C Urso Dr, it's enough to show that: For any ID, with 0, there constant C, such that
(x,) y. (x2)) C, 22l
(14)
holds for all æ1, .2 DD, In the following steps, will show (14)
24

---

Published as a conference paper at ICLR 2026
Step 3: A controllable sequence. Fix D,- By defining sequence (y:(z))=20 R":
y*+1(x) ((y x), Yo is constant for all æ.
are able to estimate the upper bound of y (=))l- In particular, we decompose yo y+ by series:
yo y- k0000
(3o yv)
yk yR+1)
Thanks to 13 we have
yv (
(++1(x)] S Pr 11:: - yx(x)) ... S Mr 330 y1 1f - G x)
for all æ E D,. Therefore, it holds that
yo y- (x)xl
11)(x)) yx++((x))|
Mr
11:00 G(yo, x))l =
11yo - G x)|
Now we can conclude the boundedness of || (x)]l for x E Dr by the compactness of IDy;
[x)] - || +
sup I1yo - G x) Py EED,
defined as M, 2
With the same argument, we have (xxx())) S Mr for all k and DD, It implies that L(yx(x)) S L1 L2M,
for some L1, L2 > L(y) grows linearly w.r.t yy/. Consequently, we can estimate an upper bound for the Lipschitz constant of yı (x). In particular, for æ1, E D, it holds that
11 1 (x2)
(Z])) (xy) 2221
= (11())) - G(yx(a2). + G(y (z2) œ1) (())((2)) x2)1l
2z) - ((yx(a2), x1)l + (1) (x2) x. (() x22)
(H111Bu((11) - (=(22)) + (L1 L2M,) ||] xzll
For simplicity, let L, == L1 + Mp == (1b)(x1) - yn (2<))) and h = | x2ll. By recursively applying a+++ S pr + L.h and ag 0. we have
Lr
(((()))-x(xx))) ax S (pr) 00 ( + +1
L,h
x0 -x2ll-
Mry
pr
Step 4: Final proof. As ((,, a)) is contractor w.r.t. y for any ID,, holds that (x)) (ar) for any x D,,. (Here, as for the "convergence," we mean the pointwise convergence, which is enough here. We don't need stronger conditions like the uniform convergence.) For the above æτ, 22, there is K such that
L, (x - y. (x1))l < 1--44 |21 - æell. (12)(x22 - y. (x2))l - - x1 - x2
Lr
for & 2 K. Combining the above results, we obtain
|l.. y- (x:2)l Slyy. (a1) - yı (#1)1l + (m1) y (2:22)) 11 (12)) yo (2:2)
3
|| -x2l
pr
By letting C, 3T,,/(1 (r)), we get 14 which completes the proof.
25

---

Published as a conference paper at ICLR 2026
Remark A.6. Our result relaxes two uniformity requirements in (Dontchev & Rockafellar 2009
Thm. 1A.4): (i) the contraction modulus (u(z) is allowed to vary with æ (it only needs to be contin- in æ), rather than being single global constant; and (ii) for each y, the mapping
( Lipschitz on X with a constant that may grow linearly in l| instead of being uniformly bounded in y. Because these bounds are not uniform, conclude only local (as opposed global) Lipschitz continuity of the fixed-point map a + (x) on X.
Now relax the condition in Theorem A.5 unbounded domains and prove Theorem 2.5 based on Theorem A.5
Proof of Theorem 2.5 We cover the domain R using the grid (2k1, 2kd) for ki E E,
defining closed cubic regions Ck of side length 3 centered at each k:
3
3
3 2
3
3 2
C+
2k
2k1
2k
2k
+
2kd 2kd
2
2"
By applying Theorem A.5 to the bounded set Cs X, guarantee the existence of unique fixed- point map yk., Ck n X R" which is locally Lipschitz continuous on its domain.
Consider any in the intersection of two regions C& n Ce. Since ((., .) is contraction, it admits a unique fixed point in R". Therefore, the local solutions must coincide:
yk. (x) (
This consistency allows us to define a global fixed-point map y. X + RR by setting y, (x) yk.. (x) for any & such that E Cs. Since y. coincides with locally Lipschitz function yk., on every compact neighborhood Ck. y- is locally Lipschitz continuous on X.
B VARIANT ARCHITECTURE
In practice, many works use a variant of the vanilla model y. G(y,, z):
z. (zz,, (z)),, y. Q2(z))
(15)
where G is the core implicit model, 21 a encoding network and Q2 is decoding (readout).
At inference, one iterates 2, = (( (x))) for 1 t s T and finally yr = @2(2r) This often improves empirical performance but does not alter the expressivity in Theorems 2.4 2.5
Corollary B.1. Under Assumption 2.2 for any F there exists regular implicit operator and F(x) for all x X. Con-
globally Lipschitz maps Q1. 22 such that Qu (Fix(g(-, (1(x))) versely, for any regular implicil operator G any globally Lipschitz 21, 2z. the fixed point z, defined by 15 exists uniquely and the induced map a. y, must be locally Lipschitz on X.
Proof. The claim follows directly from Theorems
2.5
Sufficiecyy Given any locally Lipschitz target F on X, Theorem 2.4 ensures the existence of regular G whose fixed-point map equals F. Taking Q1. 22 as both identity maps recovers the sufficiency statement with globally Lipschitz Q1, Q2.
Necessity. Suppose is regular and 21, 22 are globally Lipschitz. Then the composite update G(z, (x)) is still regular in and
By Theorem 2.5 for every X, there is unique ) is locally Lipschitz on X. Finally, applying the globally
fixed point 2, (z) and the map æ + z. (x) Lipschitz readout Q2 preserves local Lipschitz continuity, so y. is locally Lipschitz claimed. The proof is finished.
C PROOFS OF THEOREMS FOR INVERSE PROBLEMS
This section proves that the target solution mappings, Fig and Fis. single-valued and locally Lipschitz on their domain, as stated Theorems 3.3 and 3.4 Before the proofs, we first provide some definitions that used in Assumption 3.1
26

---

Published as a conference paper at ICLR 2026
Given close subset M C R", its reach T is defined in Federer (1959):
== sup(r > 0 Ny E RR with dist(y, M) < v,
there exists unique M. such that yyy =|| dist(y M)).
A set with positive reach is also called "prox-regular" set in the literature (Poliquin et al. 2000)
The Bi-Lipschitz condition refers to: for some < μ S L < +00, holds that
1||y y2ll < Ayy Ayzll S L yell Vy1: y/2 E M.
(16)
According to the definition, it holds that 0 < l Σ L S max < +00. This condition ensures A can be viewed an injective mapping when restricted M. which is important for the recovery guarantee.
Remark for Assumption 3.7 The assumption that data (particularly images) lies on a smooth mani- fold has a long and influential history (Roweis & Saul 2000 Donoho & Grimes 2003), and it is still widely used in recent literature. The compactness of the data manifold can be achieved by standard techniques like normalization. In addition, reach is an important concept for manifold to ensure the uniqueness of its projection (Federer 1959 Aamari et al. 2019). The overall assumptions on man- ifolds, smoothness, compactness and positive reach, is typically used in recent literature regarding image and signal processing (Tang Yang 2024 Potaptchik et al. 2024) [zannuulvv et al. 2024). The on-manifold bi-Lipschitz condition does not require A to be globally invertible; it merely rules out ill-posedness restricted to M. This is closely related to Johnson-Lindense Lindenstrauss (JL)-type em- beddings in compressive sensing: e.g., Baraniuk & Wakin (009) shows that random matrices bi- bi-Lipschitz on diw-dmennional manifolds with high probability, and JL-style conditions are widely
analyzed and used (Candes & Tao 2006) Clarkson 2008
Wakin 2010 Twen & Maggioni
2013
Hegde & Baraniuk 2012).
Proof of Theurem 3.3 For simplicity, we first denote the objective functions in 3 as Fia(y)
1 -111 Ayll² + at
(1) = 2 Then we introduce some definitions that will be useful in our proof:
dist (y. M)
2
(,(M) == {y E R" dist(y. M) <r), (M == (y E RR" dist(y, M) Sr)
Here, (MI is an open tubular neighborhood of the manifold M and (M is its closure. As
r == r, the open set (M)) is named as the reach tube of M, denoted as UM(M). As introduced in
Federer (1959), within the reach tube, some nice properties of the distance function and projection
mapping can be utilized. For any y (M or any E (MI) with r < the projection mapping
pvU) == arg min || yl
ZEM
is single valued and well defined, and dist(z. Hist(y.M) M) yy p(y)ll-
Step 1: Existence of minimizers of Fla- As x E X, there must be an underlying y, E M (hence || |||]| Therefore, it holds that
y. E (M)) and such that ||æ: Ay.
(y.)
-1l Ay.
a dist (s., M) |||| +0= ||
2
In the other hand, for any point outside the tube: y & U, (MI), the objective value is lower bounded by:
Fia 2 0 dist (,, M) > a g
2
As long as we have large enough at:
|||
a ,2
(17)
we can ensure Fya(y) > Fya(y.) for all & (M),, which implies infyer- Fia(y) = inf yeu, (B)) (y). As M is compact, (M) must be compact as well. Consequently, the infi- mum of F is attainable, which concludes the existence of the minimizer of Flas denoted by ÿ. and
27

---

Published as a conference paper at ICLR 2026
that, condition 17 ensures the (M). Finally, we have the conclusion: It holds for all existence of and ÿ E (ML.
Step 2: Bound of minimizers of Fla- For any y E D, (M)), the projection p(y) is uniquely defined, hence we have
Ay 20
Ay
Ay. n² - Av Ap(y) + Ap(y) Ay. - n
Ap(y) Ay. - Ay Ap(y) ||ra||
>xllp(y) y. max 11 p(y)] ||||||
According to the conclusion in Step 1, as long as
a |n
n
(18)
it holds that the minimizer exists and y E (M) for some and hence ý UM(M,, which
allows us to use the above inequalities at the beginning of Step 2. Now we aim to establish an upper bound for (g) y-ll by contradiction. Suppose
pl(p(ÿ) y-ll O 11 p()) + 2||
will obtain
| æ01 2 p (p((()) v.ll mmax 1 -p(ii) - || > ||m]]
which implies
1 2 |nl + 0 Fia(y-).
Fia (ύ) = 1 Aッ æ=|2 + dist (ú. M)
"dis
2
This contradicts with the definition of D: the minimizer of function Fia. Therefore, we obtain:
pl(p(D) - y-1l S mmax 11
p(ú)l + 2||m2|| S O + +2|m||
which is equivalent
(p(()) -y-ll
max 11
+ |||a||
and implies that
19 y. S 1 p())ll + 1p()) y-ll s (14 Fmax
2 r
r
nl
(19)
holds for all ý that minimizes Fia
Step 3: Positive definiteness of the Hessian of Fia. To prove the uniqueness of the solution, will establish the strict convexity of the objective function Fy) within a neighborhood around any point of M. To achieve this, establish the positive definiteness of the Hessian of Fia(y) in this step.
For any y Uy (MI), the projection mapping is single valued and the objective function can be written as
Fia = 2 x Ayll
- p(3))1
f(w)
g(y)
The smoothness of M implies the smoothness of and of the projection mapping, and hence we can take first and second orders of derivatives on g (Leobacher & Steinicke 2021 Theorem 2). Thanks to (Federer 1959 Theorem 4.8), the gradient and Hessian of are given by
Vg(y) = 2 (y- p(y)),
V gyl(y) = 2 Dp(y)),
where Dp denotes the Jacobian of the projection mapping. The overall Hessian of Fia is provided by
2 Fi)())))
A T -Dp(y))
(20)
28

---

Published as a conference paper at ICLR 2026
To further present the properties of the above Hessian, we introduce space decomposition accord- ing to ply):
Rd p(y)) (M) @ Np(y) (M)
where T Tp(y) (M) denotes the tangent space of M at the point p(y) M. and Np(p) (M) represents the normal space. According to (Leobacher & Steinicke, 2021 Theorem C and Definition 7), the matrix Dp(y) is actually restricted to the tangent space. In other words, for any decomposition h. with h = hr + by where hr E p(t)) (Ml and hx E (M(()(M) it holds that
Dp(y)Nn= = O, Dply)hr E (MI).
(21)
In addition, function g(y) is (--5) J-weakly convex where is the reach of M and dist(y, MM) (Narry & Thibault 2022 Section 5), and hence the spectrum of V2g can be lower bounded by
28
(hT, V g(y)h)) 2
|
(22)
Now, let's turn to the first term in the Hessian: A A. It can be shown using the JL condition 16 that, the spectrum of A A restricted to the tangent space can also be lower bounded. In particulan, we pick an arbitrary tangent vector ho E T p(y) (M)) According to the definition of tangent space,
there must be curve y (-6,8) M with ò 0, (()) p(y), and (0) hr. For any
0 S < 6, (1) E M. By applying condition 16 with the pair (r((), 7(0) and divide by 12, we have
2 (l) (0)]2
||A>(1)
2>(0)11
S 1 [I((() 7(0]11
t²
t²
t
By differentiability and the continuity of the operator A, it holds that
7(1)
7(0))
Ay(1) A7(0)
lim t~0
hr,
lim t00
Ahr
t
t
which implies
1r < ||
< I hr 112
(23)
Combining
20 21 22).
and 23 we have
(h, 2 Fl. h))
(hT, Al AhT) + A AhN) (hx, A Ahn)
20
+ (TT, (I Dp(u)) hT) +2a (hT, (I Dp(y)) hN) ta (hN, (I Dp(y)) hx)
2- | |
=(NT-bo))=0
=11bgs|11
12 8
||r||| + +all/Nll? 21|AAT|| -
2
112
8
||r||1 +allxxl?
Tmax |NN
= =[[b
||xl|||
112 -
Tmax
L
hT [hs
-max
a
Therefore, to ensure (h, 42 (i)( > 0 for any h + O, it's enough to ensure the 2 x 2 matrix to be positive definite:
11
8
0
and a a(22 a 8
o²
mnx I > 0.
(24)
In other words, 24 will guarantee the positive definiteness of 4 Fia() for all y E (M and any s T.
Step 4: Uniqueness of minimizers of Fi.. In this step, will combine the results from Steps 2 and 3. Then we are able to prove that the objective function Fia(y) strictly convex in a neighborhood of its minimizers, which implies the uniqueness of the minimizer. To achieve this, it's enough to ensure
| y. S
(25)
29

---

Published as a conference paper at ICLR 2026
for all ÿ E arg miny Fia (y), where satisfies 24
With this condition 25 it holds that
D E (y,,, C (M(.).
Along with the fact that (y,,,8) 8) convex and that 4 (1) is positive definite for all y E U, (M) Fia is strictly convex within B(y.,s) (Boyd & Vandenberghe 2004, Section 3.1.4). As
all minimizers of the strict convex function belong to this convex set, (y., the minimizer y must be unique.
Now the question is: How to guarantee
25 According to T9). Condition
18 along with
max 1
2
n2||
(26)
11
guarantee 25
Finally, it's enough to choose a and such that (88) 24 and 22)
satisfied together. particular, we choose
20 I
- n2ll: T
a
max
112
max
where a merely depends on A and M. but is independent of m. Such a parameter choice implies
26
max
r
||ll < 2
mmax
+
|||/||
2
n
+
12 8.
1
As |2|||
holds that
20 o
L
||||||
8
50 L
30 nex
L
8
50 max
L
40
L
5 mas
12
and therefore 24 satisfied:
112
112
20
max
2
1
122
402
L
2
and
20 2 T
1
122
Ju²
2
0
12
T
122
Finally, by choosing before, condition
(8) is satisfied:
L
2||m2||
1
1
Q 22
20
20 max
|n||
L T < T.
22
mmax
Omnx
20 max
o2
which finishes the proof of the uniqueness of minimizers of Fia.
Step 5: Local Lipschitz continuity of Fie. Previous results from Steps 1-4 indicate that, for any E X. there is a unique ūzz) that minimizes Fia. but the continuity of w.r.t x has not been established. In this step, we will show this continuity via the implicit function theorem. Firstly, as ý minimizes Fias by first-order optimality conditions for smooth minimization, holds that
Fia = A (Ag x) + a( p())) =
-((29)
Now, let's pick point from X. Previous results from Steps 1-4 indicate that, operator H(æ, y) is continuously differentiable within a neighborhood of (æo, (x))), and its Jacobian matrix w.r.t. y
DyHH(a, y) = Fin)
positive definite within that neighborhood of (xu, ğ (x))). Therefore, we are able to apply the implicit function theorem (Folland 2023 Theorem 3.9) and conclude that ÿ(x) is Lipschitz tinuous within a neighborhood of 26. This argument applies for any points wo X. Therefore, Fa)(::) is locally Lipschitz continuous on X.
30

---

Published as a conference paper at ICLR 2026
The proof line of Theorem 3.4 largely follows the proof of Theorem 3.3
Here will highlight
the difference of proofs between the two theorems, so that Theorem 3.4 will be rigorously proved without too much redundancy.
Proof of Theorem 3.4 For simplicity, we denote the objective function in as FIo z):
Fiy 2) := : -- Ayll dist" (x) + 8 || yy 81
and we will study its properties analogously to Fio.
Step 1: Existence of minimizers of Fic- For any r > 0, as
n 2
|n|
p2
it holds that
inf (i 9.2
inf (y.2): dis( and ||= sllss
(y 2)-
(27)
This can be proved by contradiction: (I) Suppose Fie 2) lower than the lighthhand-side of 27 and dist(2,M) r, we have
|2| (2, + 0 _ |m|1 Fio y-)
Fis 2) 0+
2,
which contradicts with the hypothesis regarding (ú. 2). (II) Suppose Fis(ū, E) is lower than the nighthhandsside of (27 and || 9 > have
|r|| Fib(ū. 2) 0 0+0 2, 1 ~l|² |n| Fio(ya.9.)
which also derives a contradiction. Arguments in (I) and (II) together prove 27). Similar to the proof of Theorem
3.3
27 implies the existence of minimizers of Fis (i.e., minimizers are attainable.)
Step 2: Bound of minimizers of Fib. To extend the proof regarding Fia to F16, we consider the following inequality that holds for all y. z E (M
1y p(y)l < |1y p(=)]|
y =|| + || p(z)ll = yyy =|| + dist(z, M) S 2r.
Therefore, we need 2r and
n 2
n
|n|
4|/
(28)
2
22
to ensure ÿ. & (M.).. Following the same argument as the proof of Theorem 3.3 the above
condition 288 implies
(p(0) yyll S mmax (2r) + ||2]||
11
and hence
11
y-l
99 p(0)l +lp(() y-ll 2 1+
0 max
2 =||2
(29)
1
P
and
22 y-1l $
||
ا11 + g/ y. lall S
3
max
r
r
2
||
(30)
holds for all (g, 2) that minimizes F1 x).
Step 3: Positive definiteness of the Hessian of Fib. Function Fis(y, 2)'s Hessian matrix is of size
2n x 2n and can written as a x 2 block w.r.t. y and z:
A A
0
0
- - I
V 2)
+ +
0
I Dp(z)
For any h [u" T
R the quadratic form (h, 2 Fisly. 2)h) can be calculated through:
(h, F 2)h n A Au av (I Dp(z))u Bl el|2
31

---

Published as a conference paper at ICLR 2026
Decompose ur + uy and or + en in T Tp(=) (M) ① Np(=) Using the same argument as the proof of Theorem 3.3
we have
(h, ()),
1 ( 1? 20 max Li| ur //ll/llllll /1ll///l1l) +a (-7-6 | 112 + || 11x + 811 - wll
which implies
(h. 2 (i.
p WT 1 max ||ur]|ll
+ ta (- T - ||rr| 12 |o 112
(Iler vrl1 lay uxil?)
11 ( ur ||2 20 maaxxlllus |||ur||l|/x
+ a
- (--- s
|
|
(1/er|1
|| + (1uxx || - ||
4²2 + 3
-Omal L
-3
ur
m L
3
-B
uN
[|
|NN| ||
|
||x||]
-B
B
UU
7-8
B
[NN]
B
To ensure the positive definiteness of 2 Fio 2), it's enough to ensure B > 0. For simplicity, we define
112 + max T.
- I.
-B
[ 0
B1
B₂
B3
-3]
a + 3
T
B
B2
Then
is positive definite if and only if B3 and its Schur complement S are both
B
B3
positive definite:
B3 0. S - B,
As Bg and B3 are both diagonal, so BB B. straight forward to calculate: BBB, BB =
8a
diag
Then the Schur complement can be calculated:
F- a+B
S 2 B
Tmax L
122
30
T -mxx L o8 a+
8-8
Tmax L
8²
σ maxx
L
at3
Note that B3 0 if.f 3 > 0. Therefore, B if.f.
30
30
0.
8 0,
-² L
(31)
0
3
a B
where 8 = ميبة Finally, we obtain that 31 ensures V Fio(U > for all y E R and all
(M with
Step 4: Uniqueness of minimizers of Fio- Comparable to the Step 4 in Theorem 3.3 we need
2 y-1 < s for all (g. 2) E arg min Fib4y.2 2). Based (30 it's enough to guarantee
2 0 max
2 n S
+
r
(32)
1
1
Now choose
4
2
8
|n||
p=
|||||
50 max
32

---

Published as a conference paper at ICLR 2026
which directly satisfies (32). As ||ral|
<
76 oun
T, we have
8 n
<
19
19 Oz
mm
L2
1 1
19
o
L
18
o
L
max
19 o
max
As long as we take
90
1.2 112
3
max
B 2 max
a, 2
holds that
90 2 1 1
1rd
1 21²
0 0
18
o2
L2
max
which implies B > 30 and hence
3 > 0. Moreover, we can verify the remaining part of 31
30
30
3
3
3 0
3
3/3
2
0
<
4
<p²;
30
a
3
aB
1 9 L
max
maa L
a + B
4
8+8 3
a
8
112
which finishes the proof of
(3I). Finally, it's enough to verify
|n|
|| 22
25 0 max
2r
1 TT
S 3,
max
76 03
which finishes Step 4, and concludes the uniqueness of 2).
Step 5: Local Lipschitz continuity of F16- By largely following Step 5 in the proof of Theorem 3.3 and changing F2 (y) to V2Fin(y, z), one can directly conclude that the mapping Fib is locally I Lipschitz continuous on X.
C.I PROXIMAL OPERATOR NEAR A MANIFOLD
We collect here the definition and basic properties of the proximal map used in the main text and relate them to the convergence condition proposed in Ryu al. (2019).
Theorem C.1 (Contractivity of the proximal residual near C² manifold). Let MI C R" be compact 2 embedded submanifold with reach T > O. For a > 0 define, for each z E (M,),
1 -1 21 + 2
(v, 2) := dist (u, (U.
Then @o must yield a unique minimizer, and hence we are able to define
pr)x,(=) = =arg min Oa (y, 2), Sx(z) = prox, (2)
y
Then S, is a contractive operator within tubular neighborhood of ML In particular, it holds that
-5) - 1+o
σ -= z'1l
(33)
for all x, x e U, (M) where r I T/4 and || - x/1 S 7/4.
Relation to plug-and-play (PnP): Condition (A) of Ryu et al. (2019) assumes a (nearly) contractive denoiser residual-preed
the kind of property (33) guarantees for the proximal residual prox, on a neighborhood of M. In practice, M is unknown; one therefore learns a parameterized operator (e.g., neural network) whose residual is constrained to be (nearly) o-contractive and plugs it into PGD/HQS in place of the exact proximal map. Whereas Ryu et al. (2019) posits Condition (A) to ensure convergence, Theorem C.T shows this condition arises naturally when the prior corresponds to the manifold-penalyy 7 dist (. M).
33

---

Published as a conference paper at ICLR 2026
Proof of Theorem
C.1
We first note that, for any y. if 1 =1 > 1z-ppe (=(()>))l then it holds that
d ( 0 + 2 llz p(z)]1 < dist (v.
y M) + 2 |1 =1 - (). 2)
2
which implies
inf de (y) 2)
v-lly = -|| p(=)1| &o (y: 2)
inf
Letr 12 =-=p==)) -(())l We further notice that, for any y with ly z|| = S r, we are able define
カ= -= p(z)
which satisfies p(ğ) = p(z) and hence it holds that
dist (ist(Ú.M) M)) = - - p())]l =||= p(=)] - 1 - =
z p(y)
=
< = yl + 1/ p(y)l
-1 =||
+ yy p(y)] 11 p(y))
dist(y M)
which implies
0 (g. 2) dist (j. M) + 1 =11 dist (y. M) + マ =112 do (3,
di.
(y.M)
2
Consequently, we conclude that minimizing &a is equal to minimizing it over the line segment between # and its projection p(z):
inf 0o (y, 2) inf 6E + (1 {)p(z),
gE(U.1]
Now define (E) 65 + (1 - (2) 2). We have
0
2 (xz + (1- - (1))((()) p(=)|| 1 ( + (1 -b)(z)) =x{
({)
0
fl - p(=)]12 + マ
pp()]|
2
o =(ot² + (1 - -)²)
1 -||= - p(2)11
Therefore, inf (E() is attainable, and the minimizer is &. sto which implies &o must yield
unique minimizer
yz z +
1+o
Consequently, we have
Sz(2) = yt - x = o
1 (p(x) - =x)
and hence
DS,(z)= 1to (Dp(z) I).
According to (Leobacher & Steinicke 2021 Theorem C), Dp(z) is actually restricted to the tangent space T p(=))())
Dp(z) (I [=ppanen -Lpp()).))))
Pr Pipx))))
where r [p(2)- =|1. v (p(z) 2)/r, and Cp(z).v the shape operator in direction v at p(z). The shape operator's eigenvalues K1, Kid (In this context, d means the dimension of the langent space) are the principal curvatures of M (Do Carmo 2016), which implies the eigenvalues of Dp(z), when restricted the tangent space,
1 1-
1 rkd
34

---

Published as a conference paper at ICLR 2026
All the curvatures are bounded by the reciprocal of the reach: |K/] 1 1/r (Aamari et al. 2019 Therefore, holds that
I Dp(z)
3
T
Tx(e)(M)
Moreover, as Dp(z) is restricted to and acts only on the tangent space (()))())) we have 0 v Dp(z) 3 - I, which implies
-I Dp(z) - 1 13 r
o For I 7/2, we have r S 1 and hence [ S Ito As long as z, E (M where r S 7/4 and || - z'1l I T 744, the two points 2, x' can included in a convex subset (actually a ball) of (M with r = T/2. By the mean value theorem, we finish the proof of (33).
7-r
C.2 DISCUSSIONS REGARDING PNP
Derivation of HQS. Consider
B
min V.ZER"
Ayl1 + di, +
- y =1|
2
2
A typically method to solve it is applying block coordinate descent on it, which is also named "Half- quadratic-splittng (HQS)" in the literature (Yang. 1995
3
yt+1 = arg min yER"
Ay - æ1
21y z: 11
(A A A + B2.)
2
2
a dist (z, Ml) + マ = y 1| prox (1++1) (let = a/
2++1 = arg min =ER
2
Similarly, we can parameterize prox, as neural network Ho,a. Therefore, HQS suggests an implicit model
Go(z, x.) Hoo ( (ATA+ -1 '(4 8z)
where a,, 3} includes all trainable parameters, which derives (
Bibliographical notes. Here we adopt the long-standing "plug-in denoiser" idea. It originated with Plug-and-Play (PnP) ADMM, which replaces proximal operator with an off-the-shel denoiser in- side ADMM (Venkatakrishnan et al. (013). The framework has since been developed and analyzed
extensively- e.g., (Chan et al. 2016 Kamilov et al.) 2017: Buzzard et al.) 20188] [Snn et al.]
2019) and the recent survey (Kamilov et al., 2023). In the PGD setting, one pretrains H. for Gaus-
sian denoising and plugs into (5) (Ryu et al. [0019] Gavaskar & Chaudhury 2020: Liu et al. 2021
Hurault al. 2022b). The same plug-in idea applies HQS via (6) (Zhang et al. 2022 Hurault
ct al. 2022a
ging in, Casti-Meymandee et al., 2023).
contrast to training a denoiser off off-the-sheel and plug-
train the entire Ge via deep equilibrium methods for the target task (the approach
closest to this paper) in both PGD-style (Gilton et al.) 2021; Winston & Kolter 2020 Zou et al.)
2023
Yu & Dansereaul
2024
Daniele
al., 2025 Shenoy al.] 2025) and HQS-style (Gkillas)
et al.
2023).
D PROOFS REGARDING NS EQUATIONS
To rigorously state and prove the theorems, we present some definitions here. First, We denote by ( the Sobolev space of functions which are in L2( together with all their derivatives of order m. Then (I2) C (?) is the collection of functions in 4 (I2) that satisfies the
periodic boundary condition on 52 with zero mean (ref. to (Temam 1995 Remark 1.1)). Then, we
can define the spaces considered in this paper:
= {u <== (H)(9))² : v-u= V := {u u (H)(1)) : v. ==
35

---

Published as a conference paper at ICLR 2026
For the NS equation 7 consider f E Hi and u E V. Moreover, denote V as the dual space of V and have
H V.
We then equip H with the standard L inner product and norm for vector fields:
(u, U)H :- (u(E), v(E))dE, ||u||a (u, (u,u) U/H (% Iu(()))
1/2
The space V is equipped with the L norm on the first-order derivatives of u. In particular,
du ag
dv
u),
(5),
(E)
de
1/2
Ou
2
|u|ly :=v (u, uvv
(x)
de
[\V
de,
and ||- l|w is defined as the dual norm of || 1/v. By Poincare and Canchy-Shwartz inequalities, have
||m|| S cllloll,
and
||l|| S callellar Ve E H
where C1, are constants depending on the domain Q. The above definitions and results are standard in the literature and we largely follow the notation in (Temam 1995 Section 2).
Proof of Theorem 3.6 (Tema 1995 Theorem 10.1) states that, for any f E V. if || S cgp? (with cq > 0 depending only on (2), then the steady NS problem (7) has a unique solution u. Since HI C V and |v c2 this yields uniqueness on
H
{
E HI 11/1 S CD 02
Theorem 10.4), there exists open dense set HI! C IH such that, on each connected component of HH{²), the solution u, depends C on f; in particular, f + us is locally Lipschitz there. Define H, := H!" H2) Since [H((2) is open and dense in IH,, the set HI, is dense in H!!". On HI, the solution is unique and the map f us is locally Lipschitz. This completes the proof.
Moreover, by (Tema 1995
Before moving to Corollary 3.7 let's reclarify lifting and projection operators: Let the lifting (or extension) operator En. IR N, x2 + {1)) be the piecewise-constam reconstruction (x)
Eces, Zclc, and let P {L)( + H be the orthogonal projection onto divergence-free, mean fields. Then move Corollary 3.7
Proof of Corollary 3.7 The mapping F2 mappings: We first map x N X2 continuous version E H by P En, then f can be mapped to its corresponding solution u, by Locally Lipschitz operator as stated in Theorem 3.6 Here we denote this mapping by S r> u.. Then u, is mapped w. by vorticity: U., and finally w, can be mapped to y. by a restriction operator RR
can be viewed as a composition of multiple
F2 R4 (Vx) S O
Then let's analyze the norm of the above operators one by one. Firstly, the restriction operator Rs has norm no greater than 1 as:
/ w()dE
[w)
Σ
|c| c
Ce
(1 (u(E)dEE))
2
/ ((E))
Σ CE
Σ
||w1120
[
Cea
36

---

Published as a conference paper at ICLR 2026
Note that R4, is linear operator, hence its bounded norm immediately leads to its bounded Lipschitz constant:
(w) (w'
(w -') < -1w--'1i
Second, the curl operator V x must be bounded linear operator because the solution u, E V. where first-order derivatives must be I Third, the solution mapping has been discussed Theorem 3.6 it is nonlinear operator, but it is locally Lipschitz continuous. Fourth, the projection operator P. must be linear and have a norm no greater than 1. Finally, the lifting operator is linear and has bounded norm as:
[[ [()(())))))(0) = ICl | = |
CERa
Therefore, except for the nonlinear operator S, the other four operators are all linear and bounded and hence are globally Lipschitz continuous. As long as we can show that the input of S must be taken from the unique solution regime H,, we will complete the proof that F2 is locally Lipschitz
everywhere on Xy This can be proved because x X implies (x)) E Hl,. Finally, by
applying Theorem 2.4 we conclude the existence of G described in Corollary 3.7 which finishes the entire proof.
E PROOFS REGARDING LINEAR PROGRAMMING
Although Lipschitz continuity of LP solution maps has been studied (e.g., (Mangasarian & Shiau 1987| Dontchev & Rockafellar 2009)),
not aware of a reference that states Theorem 3.8 in the precise form needed here- particaaall allowing perturbations of A (rather than treating A as fixed). For completeness, we therefore include self-contained discussion and proof.
To work with standard form, we rewrite the general-form problem (8) in standard form. Suppose there are p equality constraints and q inequality constraints. Without loss of generality, we assume o, equals to "_" " for 1 < S p and o, equals to "<" for + 1 < i < m. Then we denote Ap as the first p rows of matrix A and Ag as the remaining part:
Ap
A :].
A, := Alp + 1: m, :]
And therefore the general form LP 8 can be written as
min c T y, VER"
s.t. Apy by:
A₂y bu 1 5 u.
Let ÿ = y - 1, = by A,y,
y, and t: = u
y, the above problem can be transformed to
A
y'
b
Ap
min T Ú. s.t.
Ag
$
=
bg
Aql
1 2
0,
0,882000
0,100
yER"
I
I
t
u -
By letting
bp Ap
bp Ap by A,
y
é ē:=
0
A =
by
A,
=
ÿ =
8
u 1
u
The problem is equivalently expressed in standard form as
min ē V. S.l. Aÿ = b, 9 2 0.
In fact, every LP can be rewritten in equivalent standard form. While concepts such basic feasible solutions, degeneracy, and complementary slackness are most naturally and cleanly stated in standard form, each admits closely related analogue (with minor adjustments) for the general form. Accordingly-withour loss of generality and to keep the focus on core idcas-we carry out the proof in the standard-form setting:
min e y. S.L. Ay = b, y 220 0, y
with dual
m s.t. A T Sc c.
37

---

Published as a conference paper at ICLR 2026
Here, follow the standard settings in the literature: y.c R",, 2, E RT A R=*" (A) = m (ensured by preprocessing with removing redundant equalities), and m S n. In this context, we define the domain of LP that work on:
: = {(A, b) The resulting standard LP is feasible and bounded)
Note that, to match the rest of the paper, we reserve a for machine learning model inputs (in this context, it is (A, b, c)) and hence write the primal LP variable y and the dual LP variable as 2. This departs from the common (x, y) convention. Note also that in the main text the symbol denotes a latent variable; here, in the appendix regarding LP's technical details, denotes the dual variable. These meanings are unrelated and should be clear from context.
Now let's present some definitions used in this appendix. Fix basis by selecting an index set B C (1,2. ... n with [B] = m such that the m x m submatrix B == Al: B] is nonsingular.
Let N = 21, ..... ,n) B be the complement of the basis and let N := A(; N.. Then the equality constraints read
Byв + NyN = b
Setting yvv = yields ye B- Such a y ys: is called basic solution. If additionally yB 2 0, this basic solution is feasible, then it is called basic feasible solution (BFS). On the dual side, we define the slack variable and its sub-vector restricted to and N:
c AT 2, 8g = CB B
N CN N
A pair (y, 2) is primal-dua optimal (i.e., satisfies KKT for LP) iff
Ay b, c c= A=
y + s 0,
y y 0, 8 8
(34)
for R". If, in addition, there exists basis B such that
yB 2 0, yN 0, 8g = 0, AN 2 0.
(35)
then the tuple (y, 2, 8) is called an optimal BFS with a complementary dual. By the fundamental theorem of linear programming, any feasible instance with finite optimal value (A, b, c) E X admits an optimal BFS with a complementary dual satisfying (34) and (35) together (Bertsimas & Tsitsiklis, 1997].
While conditions (34) and 35 are enough to ensure the existence of the optimal basic solutions, they are not enough to ensure that the optimal solution is unique and local Lipschitz continuous w.r.t. the inputs (A, b, c). To ensure these points, we present two additional conditions based on (34) and 35
yB (Non-degeneracy)
(36)
SN > 0
(Strict complementary slackness)
(37)
All the conditions together are enough to the uniquenss and local Lischitz continuity. Let's introduce consisting of all "good" L. instances:
Xanh b, E X The LP yields tuple (y, z, satisfying
34
35
36 and 37
With all the preparations, we can prove Theorem 3.8 now. Actually, proving Theorem context of standard-form LP is equivalent to proving the following two theorems.
3.8 in the
Theorem E.1. For any LP (A, b) E Xauh. must yield unique optimal solution y. and the solution mapping (A, b, e) + y- is locally Lipschitz continuous everywhere on Xaub-
Theorem E.2. Ysub is dense subset x X.
Theorem follows from Dontchev & Rockafellar (199)), which develops Robinson's notion of strong regularity (Robinson, 1980) for nonlinear programs. For completeness andd to keep notation consistent with linear programming ww restate the relevant lemma in an LP-adapted form and then verify its hypotheses for LP.. We begin by quoting the result from Dontchev & Rockafellar (199)..
Lemma E.3 (Dontchev & Rockafellar (1996)). Consider a parameteric nonlinear program:
min c. (o (w,y)
yER"
38

---

Published as a conference paper at ICLR 2026
s.t. ge y) = us 1sisr gi y) S us r+lgisd
where 9:(0 S i S d) are all C² functions, and c, and u [u1."..,uu are parameters to describe the program, and consider its Lagrangian with multipliers [ Aal R given by
L(w, y, A) go v) + Σ ).ge(w, y)-
Let (g. X) a KKT point at (c, w. ú) and define the index sets at (ÿ, (ū,
4 + 1 < i - d: g. us A, > 0 {I (.....rh
1 r r+ rrtliiid::oo g: us =0 0).
13 = d: (( < <,, = }
If the following conditions hold:
The constraint gradients V yy ū) E Ii U l2 are linearly independent; and
It holds that
( V2 uy L( y, Aly')>
for all y' + 0 in the subspace M y' y T (1 Ø) for all i E 11).
then the KKT solution map (c, w. u) + (y, X) is locally single-valued and Lipschitz around
(c, w, ú, y, X).
Proof of Theorem E. Taking and d m Let a. be the i-th row of A in standard L.,, and let
a y.
i= 1,...,,, m, i= m + 1, 1.....7... m + n,
bis i= 1...., m,
(w. y)
V/s-ms
0, i= m +
m
+ nI
with w collecting the coefficients of A. The Lagrangian in Lemma
E.3 becomes
L(w, y, A) c y +
N a y +
Am+i(- (-9;)-
j=1
Introduce the usual dual/primal-slace variables
z = -iim E Rmm
= Am E 20
rewrite stationarity as V,L = - Ay b, 2 0: dual feasibility is s 2 0; and complementarity is O s 0. Thus the KKT system in Lemma coincides with the standard LP KKT conditions.
0, i.e. c ATz. Primal feasibility is
Assume (A, b, c) Xeub. i.c., the LP admits a tuple (g. 8) satisfying
34
35
36
and 37
(A nondegenerate and strict complementary basic point). this context, the index sets T1, T2, 13 at
y. z, 8) become:
I {1,.... m] U [m + j: 0 = 0, 8, > 0}.
1 m j: 9, 0, 5, 0}.
I3 m + j: 3, > 0, 8, = 0}.
which implies:
For each j. either > 0 8, 0, which implies l2 Ø.
is substantially the basis set: Ia {m + jmijjje B)
39

---

Published as a conference paper at ICLR 2026
1 includes all the indices in the complement of basis: 1 = {1,.... m) U
To verify the hypotheses of Lemma we examine the gradients:
(V y9h her, = {a:) U e-eeEE
In the context of standard LP. N m. Hence, {V yel, consists of n vectors in R" Now create matrix G by stacking these vectors as rows:
G
By properly permuting the columns of G, it becomes
B
N O I
G
where I represents the identity matrix in RR Since B (the basis matrix) and I are both nonsin- gular, G (and hence G) must be nonsingular. Therefore, the rows of G are linearly independent, i.e. yVggkiie,, is linearly independent. With /2 2, the first hypothesis of Lemma E.3 holds. Moreover, because these gradients [V yyii hez, span I",, the M subspace must be trivial: M {0} Therefore, the second hypothesis of Lemma E.3 is automatically satisfied.
By Lemma B33 the KKT solution map is locally single-valued and Lipschitz around the given point, which yields the desired local uniqueness and Lipschitz dependence of y. on (A,b, c) for every (A, b, c) E Xaub-
Theorem E.2 can be proved by fundamental concepts in real analysis.
Proof of Theorem E.2 To prove Xgub is dense in X. it's enough to show that: For any (A, b, c) e X,
can always create sequence of LP {(As, bj, cx)) Xyub such that
A A, by + b, Ck c.
Now let's fix c, E X. As we previously discussed, there must be a tuple (y, 2, s) satisfying and 35
34
Define:
y& y Fe 8% =
K eN,
==
that (Yk, 2k, 84) must satisfy the nondegeneracy and striet complementary slackness: and 37 Accordingly, define
35
36
A= A, by = A
Ck : A 2k + Sk
Then one can verify that the tuple (y, 2, satisfies 34 35
(36) and 37 for the LP instance
(AR, bu, Ck), hence (A,, bu, ca) E Xanh for all & 1. Finally, such perturbed LP instance can be arbitrarily close to (A, b, c) as + oo:
A A =0
mm
bb
bl|
=
A
-Bg
< h
A
lBll
All +0
1
m
||cx
cll
-NN
+
which finishes the proof.
40

---

Published as a conference paper at ICLR 2026
F TRAINING STRATEGIES
Unrolling vs implicit differentiation. There are two training strategies adopted in this paper. One named "unrolling" (minimizing {(yr)):
min ((yr). yi+1 = Goly), 0,1, .. -
and the other is named "implicit differentiation" (minimizing {(y.)): min ((y.); y- - (.).
9
These two strategies closely related. In particular,
As established in prior literature, unrolled training is mathematically equivalent to a Neumann series approximation of the implicit gradient (Geng et al., 202T). Specifically, implicit differ- entiation requires inverting the Jacobian (I Jg,) '; fimite unrolling effectively approximates this inverse via a Neumann series expansion. This is widely adopted technique in the implicit model community to avoid the instability and cost of exact inversion.
Implicit training is simply the limit of unrolled training: as + oc, the gradient Vol(y)) converges to the implicit gradient Vol(y.) (Geng et al. 2021).
Overall, unrolling and root-finding are merely two numerical implementations for approximating the same fixed point, y. (ær), and technically speaking, there is significant gap distinction between the two. Theoretically, infinite unrolling converges exactly to (x)). In practice, unrolling depth simply controls the trade-off between accuracy and computational cost: dynamic strictly analogous to setting the error tolerance in implicit root-finding solvers.
Particularly in our paper, for Case Studies & 2, we employ implicit differentiation (minimizing ((y)) )) via root-fine noot-fnding; for Case Study 3, we adopt unrolling to train implicit GNNs, which serves as truncated Neumann approximation of the implicit GNN gradient; for Case Study 4, we directly use the pretrained model from Geiping et al. (2025).
Guarantees of Regularity and the Expressivity Trade-off. While our experiments demonstrate that standard training (either unrolling and implicit differentiation defined above) empirically results in regular implicit operators, we do not explicitly enforce this property in the loss function. Designing training mechanisms that theoretically guarantee regularity without sacrificing the model's unique expressive capabilities remains an open and interesting future topic.
Recall that regularity (Definition 2.3 comprises two conditions: the Lipschitz continuity of the map x + (y, x) and the contractivity of the map y + Go(y, x). The first condition is largely inherent to standard deep learning architectures; compositions of affine layers with bounded weights and I-Lipschitz activations (e.g.. ReLU) naturally preserve Lipschitz continuity with respect to the input Miyato et al. (2018): Virmaux & Scaman (2018). Therefore, the critical challenge lies in guaranteeing the second condition: contractivity with respect to the state y.
A substantial body of literature has sought to enforce this contractivity by construction (e.g.. El Ghaoui et al. (2021); Winston & Kolter (2020): Jafarpour et al. (202I); Revay et al. (2020); Havens et al. (2023)) These approaches typically impose rigid structural constraints, such as pa- rameterizing the model as a one-layer nonlinear MLP: (y x) = oAy + Bx + b) and strietly bounding the spectral norm of A, or enforcing global monotonicity.
However, these methods generally enforce a uniform contraction modulus μ across the entire domain X. Our theoretical analysis suggests that such uniformity fundamentally undercuts the unique expressive advantage of implicit models. As illustrated in Figure [ for a sequence of continuous iterates yt(x)) to converge to a target F(xr) that is discontinuous or has singularities, the conver- gence cannot be uniform. This implies that the convergence rate-and consequently the operator's contraction modulus
must be adaptive, varying with æ to allow for slower convergence in complex regions. Enforcing a globally uniform μ severs this adaptive capability, thereby severely constraining the model's ''s expressive power.
Therefore, developing novel regularization techniques that can guarantee adaptive contractivity (en- suring < p(x) < 1 locally while allowing it to vary over x) is a critical direction for future research to balance theoretical stability with maximal expressivity.
41

---

Published as a conference paper at ICLR 2026
G EXPERIMENT DETAILS REGARDING IMAGE RECONSTRUCTION
This section complements the main text with additional implementation and dataset details for the inverse-problem experiments.
Experiment settings. We consider an image deblurring task, a A(y-) ) + n, where A is the blur operator and is the Gaussian noise (a 0.03). We blur kernel is the first of the eight kernels from Levin et al. (200)) Ground-truth images y, come from BSDS500 (Martin et al., 2001). We follow the official splits (200 train / 100 validation 200 test) and apply a random 128 x 128 crop to each image. For each y-, we generate the correspond- ing æ: by applying A and adding noise. The resulting pairs (æ:, y-) form three datasets invarrins Dinyyal, and D invttett for training, validation, and testing, respectively. In both PGD and HQS style parameterizations (5) and (6), the operator H. implemented with DRUNet (Zhang et al. 2021)
motion-blur operator, and the
Training, We initialize H using pretrained weights from the Deepinv library (Tachella et al. 2025) and then fine-tune the full implicit models on the BSDS500 training set for this deblurring task. Training follows the vanilla Jacobian- based implicit differentiation and is implemented on top of the official Deepinv framework. All models were trained with Adam (learning rate 10 10-4, batch size 3). Explicit baselines were trained for 20 epochs, and the implicit models for 10 epochs. After each epoch we evaluated on the validation set and saved the checkpoint; the final model used for testing the one with the lowest validation loss. These epoch budgets were sufficient for validation-loss convergence.
PSNR. PSNR (Peak Signal-to-Noise Ratio) is defined between reference y² and reconstructed image y as
n MAX2
PSNR( y²) = 10 log10
[y y 1
where n is the dimension of y and y², and MAX means the max possible pixel value (e.g., 255 for 8-bit, if images are in [0, 1). In our context, it is 1. Higher PSNR means better (more accurate) reconstruction.
Standard test set. Evaluation cropped to 128 x 128. Let Dinvcest { yt ) ) where
the 200 images from the official BSDS500 test split, randomly
A + n4;
N(0, = 0.03.
Here y denotes the clean (ground-truth) image and x; its corresponding blurred alurred-noisy observation under the forward model A.
Perturbed test set. To empirically validate our theory, we created a perturbed version of the test set. To create a diverse and representative set of perturbations, we generate perturbations that cor- respond to different frequency levels. Image frequencies represent different levels of detail, where low frequencies capture smooth, large-scale areas, and high frequencies capture sharp edges and fine textures. By probing the model with perturbations across this spectrum, we can comprehensively evaluate its behavior.
Specifically, we construct each perturbation by targeting a singular vector of the forward operator A. Because is (circular) convolution, its singular vectors are Fourier modes. For each image V. and each frequency magnitude f (0.1, 0.3. 0.5, 0.7, 0.9). we first identify the 2D discrete Fourier frequencies and sort them by their geometric distance from the origin. We then select the frequency coordinate (u, v) the f-th percentile of this sorted list. A one-hot tensor is created in the Fourier domain with a value of 1.0 at the chosen (u, v) position and zeros elsewhere. This sparse frequency representation is transformed back into the image domain by applying the adjoint of the blur operator, A These perturbations are visualized in Figure 8 Adding them to y, respectively yields perturbed clean images yi., (j- 1,..., 5); we then form the corresponding observation
Dijj A(v + n2.
n. N(0,001))
The perturbed evaluation set is
DV Dl { ( vig)) 1 isis 200, 1 15 5
For convenience also define the unperturbed index j 0 by æú,0 æ; and yī.0 = y/-
42

---

Published as a conference paper at ICLR 2026
frequency=0.1
frequency=0.3
frequency=0.5
frequency=0.7
frequency=0.9
Figure 8: Visualized perturbations for inverse problems.
Table 3: Deeper explicit models vs implicit models for image deblurring (PGD architecture). "Exp (xT)" represents an explicit model T times deeper than the implicit baseline. O/M denotes CUDA Out of Memory during training.
Exp (x1)
Exp (x2)
Exp (x4)
Exp (x8)
Exp (x 16)
Exp (x32)
Implicit
Params.
32.641 M
65,282 M
130.56 M
261.13 M
522.26 M
1044.5 M
32.641 M
PSNR
27.14 dB
27.64 dB
27.89 dB
28.11 dB
28.27 dB
O/M
28.21 dB
Table 4: Deeper explicit models vs implicit models for image deblurring (HQS architecture). "Exp (xT)" represents explicit model T times deeper than the implicit baseline. O/M denotes CUDA
Out Memory during training.
Exp (x1)
Exp (x2)
Exp (x4)
Exp (x8)
Exp (x 16)
Exp (x32)
Implicit
Params.
32.641
65.282
130.56
261.13 M
522.26 M
1044.5
32.641 M
PSNR
26.94 dB
28.02 dB
28.35 dB
28.69 dB
28.87 dB
O/M
29.18 dB
Platform. All experiments were run workstation with eight Quadro RTX 6000 GPUs.
Additional Experiments. Implicit models often excel on imaging tasks, but a natural question whether simply stacking more explicit layers (i.e., deepening the model) can close the gap. To probe this, we construct explicit counterparts to implicit models by untying the parameters across iterations:
min Eyyl y-), s.t. = Ggto (31-1, to 1 ,
where each block G has the same architecture as in the implicit case (PGD or HQS), but ( separate for each This is equivalent stacking T blocks form deeper explicit model with more learnable parameters. Unlike implicit models (which can use different iteration counts at train vs. test), these explicit models must use the same for both training and testing. We evaluated T E 21,2, 4, 8, 16, 32} to compare against the corresponding implicit models.
Tables and report results on image deblurring. Across both PGD and HQS settings, deepening explicit models increases parameter counts massively (up to billion) but yields diminishing returns PSNR. Crucially, the implicit models achieve performance comparable or better than explicit models that are 16x deeper, while using fraction of the parameters (32.6 M vs. 522 M). For instance, in the HQS setting, the implicit model (29.18 18 dB) outperforms the explicit model with 16 unrolled blocks (28.87 dB).
Furthermore, training extremely deep explicit models (e.g., memory constraints (O/M). This highlights the distinct efficiency ad advantage of the weight-tied im- plicit approach: it theoretically allows for infinite depth (realized here as 100 test-time iterations) while maintaining constant parameter counts (32.6 M) and memory usage.
32) becomes infeasible due to
H EXPERIMENT DETAILS REGARDING SCIENTIFIC COMPUTING
Model structure and training. Given cell-averaged forces æ E Xx and vorticities y H x Wxl where H means the height and W means the width, we learn
z. Go(Z.; @x)x)
y- Qz.2.),
43

---

Published as a conference paper at ICLR 2026
where 2, ERRxxx is latent field with C' channels. At inference, iterate
2 Go -11 2x(x)),
for 1 S <t S T and finally call yr (zr)
The projection 24 is pointwise linear encoder applied at each grid cell lift into C channels, In particular, g = Qp (2) reads
Wix+ + RR#xxx
where +- = (W1, by) are learnable parameters.
The core map Go 2, 9) stacks L identical FNO layers with input injection:
200)
20 = a (g (WwWe (( 6b0 1.FT(RO..FT(C (FT(2(-11)) 1 111,2,...
ge(z,g) = =1(4)
where = = (W b R are learnable parameters. Each layer: (i) performs a global spec- tral convolution on z: take an FFT of the C'-caanel tensor, keep only a small set of low Fourier modes. Suppose the number of retained Fourier modes is K x K (2D FFT), (z) CK xxxx For each retained mode (k1, 122) multiply the C--dimensional channel vector by learnable dense ma- trix k CCCC (mixing channels) and hence the overall matrix is of size R C
then apply an inverse FFT; (ii) adds a local pointwise transform, adds the injected encoder features (x)), and applies a nonlinearity. This realizes a resolution-invaree that naturally respects periodic boundary conditions.
globally receptive operator
Finally, we decode with the pointwise readout Qu (a small per-cell two-layer MLP) produce E xxxxxx x1 where V are learnable parameters.
All samples use H W 128. Unless stated otherwise, we set the latent width C 32, retain K = 12 Fourier modes per dimension in the FNO blocks, and use L = 3 FNO layers inside Ge- Training differentiates implicitly through the fixed point, and the fixed-point solver uses Anderson acceleration. We optimize with Adam (learning rate x 10-3, 3, batch size 16). For explicit baselines, we train for 500 epochs, which suffices for the training loss to converge.
Perturbed data generation. In this paragraph. we describe how we generate perturbed samples in DV puictees We take the dataset of Marwah et al. (2023) as the unperturbed set PPeexes and create perturbations by linearizing the steady NS equation 7 Each sample (f, w) comprises a forcing term and its vorticity solution
Directly prescribing and solving for is computationally costly; following Marwah et al.] 2023), we instead prescribe w and obtain the corresponding / by evaluating the PDE operator (not by solving the PDE). our setting, the base samples are given; thus we first construct a solution perturbation &w and then compute the induced forcing perturbation o via the linearization, yielding the perturbed pair (f + of, + &w).
Note that, while the dataset is discrete, we use the continuous notation f,w, u in this section to ease reading and to remain consistent with the PDE literature. addition, we use & (EI. as the special domain variable to keep consistent with our main text, and use k = (k1, k2) as the frequency domain variable.
(Generate 5w). Fix target wavenumber ht_ E N and desired T.2 mm2-aauit 0. We construct dw by
dw(E1, = A sin(k,E + k.E2),
A chosen so that ||,0||2><0) 1).
The wavenumber is selected from user-specified frequency percentile Pfrqq relative to the maxi- mum resolvable frequency kmax H12 W/2, namely
k Piteq
kimax
(rounded to the nearest integer mode).
In our code we set the grid size H = W = 128, the perturbation strength 77 = 0.01, and choose pireq E (0.01, 0.03, 0.04, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.96, 0.97, 0.98, 0.99). Accordingly, each original sample yields 15 perturbed samples.
44

---

Published as a conference paper at ICLR 2026
Table 5: Implicit FNO vs deeper Explicit FNO. "Exp(xI)" denotes an explicit model that ( times deeper than the implicit FNO. O/M indicates CUDA out-of-memon error during training.
Exp(x1)
2))
Exp(x4)
Exp(x8)
Exp(x16)
Exp( x3))
Implicit
Params.
2.376 M
4.155 M
7.713 M
14.83M
29.06M
57.52M
2.376 M
Rel. Err.
0.1787
0.1526
0.1410
0.1380
0.1360
O/M
0.0785
Table 6: Implicit FNO wider Explicit FNO. "Exp(x denotes explicit model that is times wider than the implicit FNO. O/M indicates a CUDA out-of-memory error during training.
1xp(x1)
x2)
Exp(x4)
Exp( xxp(x8)
Implicit
Params.
2,376 M
9.504 M
38.01 M
152,0M
2.376 M
Rel. Err.
0.1787
0.1555
0.1401
O/M
0.0785
(Generate velocity from vorticity). Given scalar vorticity field (and its perturbation dw), recover the corresponding velocities u via a streamfunction 1 given by
v (dzb. 0 v)),
D..
Hence vb is obtained by solving the Poisson equation Auh periodic grid, these operators are implemented efficiently in the Fourier domain.
after which we obtain u. On
(Linearization and the perturbed vorticity forcing 6g). Applying the (scalar) curl "V x to both sides of ( yields the steady vorticity form
(u- V) y AW 9, g v x f - - - 02/1, -0//,,
where f (fi, is the body force and g is its curl. Introducing perturbations (ōu, dw, 6g) and expanding
(u + 5u) V (w + ow) - v A(w + 5w) g - 5g.
then subtracting the base equation and discarding higher-order terms gives the first-order relation
8g (u- dw + (8u. V) A
Again, for numerical implementation periodic grid, the differential operators applied effi- ciently in the Fourier domain.
(Recover the vector force öf from its curl ōg). We recover periodic 8f (ff1, satisfying V x 8f = 5g by solving a Poisson equation for an auxiliary streamfunction ao and obtain of exactly as in "Generate velocity from vorticity."
Additional Experiments:
Scaling Explicit Models. A natural question whether the performance gap between implicit and explicit models be bridged simply by scaling up the explicit architec- ture (i.e., stacking more layers or increasing channel width). To investigate this, we compared the implicit FNO against explicit baselines scaled significantly
two dimensions: depth (up 32x) and width (up to 8x). The results, summarized in Table 5 and Table 6 demonstrate that while scaling explicit models yields modest accuracy gains, it faces severe diminishing returns and com- putational bottlenecks (eventually leading to CUDA model achieves markedly better performance (lowest relative error of 0.0785) than even the largest viable explicit models, despite the explicit counterparts using over 10x the number of parameters (e.g., 29.06 for Exp(x x16) vs. 2.376 M for Implicit). This confirms that the implicit formulation provides an expressive advantage that cannot be efficiently replicated by simply allocating more capacity to an explicit solver.
errors). Crucially, the implicit
Note: These findings are broadly consistent with Marwah et al. 2023 We follow their setup with two minor deviations: we use a smaller training batch size (16) due to hardware limits, and while keep T 24 training iterations for the implicit model, inference run T 50, because observe that the trained implicit models remain stable and often benefit from additional fixed-point iterations test time.
45

---

Published as a conference paper at ICLR 2026
1 EXPERIMENT DETAILS REGARDING LP
GNN model details. We implement
(z, Q4 (z)), y- = 2u (z.)
with an L-layer message-passing GNN (Sarsellli et al., 2008 Xu et al. 2019 on the bipartite graph.
Let (i) (resp. N()) be the neighbors of constraint node W; (resp. variable node Vj). With shared MLPs across all nodes and edges, the GNN structure is given by:
Input-embedding:
w = MLP &a (hi, oz),
V2(0) MLP de (c,, 1s, Uj. Zin.j)
Desssage-passingg
LI=
MLP w(O
::
Σ JEN(()
Ag MLP
(v
0
0
()
MLP
Vi
Σ
Ag MLP
(W!(--1)
3
iEN()
((()
Output-embedding:
Pout.j
MLP Ho
We write this compactly as follows.
Zout = (o 2x
1 10 1 0z are trainable parameters in the GNN, = f01, )2) includes the trainable parameters of the input embedding. The input æ includes all static information = (A, b, c, o, u.). Finally, the output embedding y Qu (2) is given by
where θ - (10 si=1
1 ,
yイ MlPp
for every variable node j. All MLPs in Ge. Qx. and Qy two layers with ReLU activations. We sweep widths (or embedding sizes) in {4, 8, 16,32) 32} and report results in the main text.
Note that I is the layer index within the GNN structure, not the iteration number t. All parameters in @ are independent of the iteration number, so this GNN can be applied iteratively. æ is the static features and is the dynamic feature. In addition, removing the dynamic input 2in and decoding directly to recovers the standard (explicit) GNN baseline.
Dataset generation. We largely follow Chen al. (2023) construct the training set Dr Parai and test set DLesst: drawing (A, b, c, o, 1, 2)) i.i.d. from the same distribution. Each LP has 50 variables and 10 constraints. The matrix A is sparse with 100 nonzeros whose locations are chosen uniformly at random and whose values are sampled from a standard normal distribution. Entries of b and e are sampled i.i.d. from Unif[ 1), after which is scaled by 0.01. Variable bounds 1, u are sampled coordinatewise from (0, 10); whenever 1; >> Uj we swap them. Constraint types are sampled independently with Pr(o; = < ") 0.7 and Pr(o; = = ") 0.3. Under this generator, the feasibility probability is approximately 0.53; we retain only feasible instances, yielding 2,500 LPs for training and 1,000 for testing. Solutions are computed with scipy. opt imize.
To build the perturbed datasets LPtest we perturb one component at time while holding the others fixed. For c, draw dc with i.i.d. standard normal entries, normalize, and scale to magnitude 10 4:
òc |8c||
c 10-
We apply the same procedure to b, I, and u. For A, we perturb only existing nonzeros to preserve the sparsity pattern: let S {(i. j&) jx))mm maz(A) be the nonzero locations and draw da [g(S| i..d. standard normal; normalize and scale so ||āa|| = 10-4, then set
A Alyss Aicsi (oa); for ja) E SS,
A/. Ai, otherwise.
46

---

Published as a conference paper at ICLR 2026
This yields five perturbed versions (perturbing A, b, Lipschitz constants L, and relative errors Eg on each version and report the results in the main text.
separately). We evaluate the estimated
Training method. To train our implicit GNNs, employ two-stage curriculum strategy. The model is trained by unrolling its iterative updates for a fixed number of steps. T, and minimizing the loss on the final output:
min 0,0,0
Σ
(yy,,y,)
y- JEDimmis
s.t. 20 = 0
2; Ga(z,-1; (x)))), t= 1,2,
yr = Q2(2r)
We set the final unroll horizon to T - 6, as we observed no significant improvements with longer sequences. Training directly with approach is standard practice in the Learning to Optimize field for training implicit or unrolled models that solve optimization problems (Chen et al. 2022c). This approach begins with a shorter unroll horizon and larger learning rate, using the trained model to warm-start the subsequent stage with a longer horizon and reduced learning rate. This strategy often described as "layerwise training" (Chen et al., 2018 Liu et al., 2019) or "curriculum learning" (Chen al., 2020). In our settings: Stage uses = 3 with learning rate 0.01; Stage 2 uses T = 6 with a learning rate 10-4, Both stages Adam optimizer.
inefficient, so we adopt a two-stage curriculum. This
For a fair comparison, the non-iterative explicit GNNs are trained using the same two-stage learn- ing rate schedule. This regimen proved effective, the training errors for explicit baselines surpassed those reported in prior work (Chen et al. 2023).
At the inference time, T can be chosen the unroll length in the training stage, moderately longer. In our experiments, we use = at the inference time, as we do not observe significant improvement with larger number of iterations.
Remark. While we employ unrolled training rather than the vanilla Jacibian based implicit differ- entiation, classify approach as an "implicit model" because the underlying architecture, weight-tied update numerical implementation:
Ge( 22 (2)). remains identical. The distinction lies solely in the established by Geng et al. (202I), unrolled training mathemati- cally equivalent to a Neumann series approximation of the implicit gradient. Thus, unrolling and root-finding are simply two valid strategies for approximating the same fixed point, y- (x). This equivalence is widely recognized in the Implicit GNN literature, where Neumann approximations are standard for scaling to large graphs (e.g., (Baker et al. 2023)). Since our focus is on expressiv- ity rather than optimization mechanics, we treat both formulations belonging to the same model class.
J ADDITIONAL RESULTS REGARDING LLM REASONING
In the main text (Section 3.4,. we present qualitative example showing how additional recurrent blocks let the model separate nearby prompts into different semantic contexts. Here we complement that illustration with quantitative measure of this effect.
To quantitatively measure this, we define an "Empirical Lipschitz" constant L, using Levenshtein distance d(.,)). -):
(i, d(y)( vx x()
We construct {(x,, 220) dataset of 200 pairs where inputs differ by only 1-2 words but require vastly different semantic contexts. Figure plotts the geometric mean Le, which rises from 29.2 at f = 2 (indicating relative insensitivity) saturate
52.5 by t 16.
with our theory, this growth reflects the model's emergent capacity to map proximal inputs to semantically distinct outputs. Even in the discrete domain of language reasoning, iterating a fixed operator allows the model to scale its expressive power, evolving from simple surface-level processing complex, context-aware reasoning.
Consistent
47

---

Published as a conference paper at ICLR 2026
Lipschitz L..
50
11.. 52.51
40
30
29.2)
4
8
10
12
14
16
Iteration
Figure 9: Empirical Lipschitz of the output sequence y+(-) generated by Geiping et al. (2025 using
I recurrent blocks. Le grows as I increases.
K BROADER CONTEXTUAL DISCUSSIONS
While our work primarily establishes the expressive power of implicit models through the lens of fixed-point iterations, we situate our contributions within the broader landscape of implicit model theory in this section.
Universality and expressivity. Despite foundational importance, a general and systematic theory of expressive power for implicit models remains largely open. To our knowledge, existing results address only specific facets of the problem. For example, while et al. (2019) demonstrated that an operator g exists such that its fixed point reproduces any explicit network f. this existence result crucially does not guarantee that the fixed-point iteration (T) actually converges. Other works have established universality within restricted domains, such as steady-state PDEs Marwah et al. (2023), or proven separation results where implicit models outperform explicit counterparts in specific set- tings Wu et al. (2024). While insightful, none of these studies provide complete characterization of the general function class representable by implicit models, nor do they directly address the fun- damental questions (QI) and (Q2) raised in our introduction.
Contrast with explicit models. First, consider the setting where the model is subject to global Lipschitz constraint (e.g., Lip(fo) S 1), which is common for robustness and stability. In this case, explicit feedforward networks are mathematically strictly limited to representing globally 1- Lipschitz maps (Murari et al.] 2025). Consequently, they are fundamentally incapable of express- ing locally Lipschitz targets whose gradients become arbitrarily large (such 1/x near zero). In contrast, our work demonstrates that implicit models break this barrier: simple, globally regular operator (Lipschitz in contractive in can generate complex, locally Lipschitz fixed-point maps via iteration. This "Simple Operator > Complex Fixed Point" mechanism the core difference claimed in our paper.
Second, we remove constraints, Beneventano et al. (202)) have shown that deep ReLU networks indeed approximate locally Lipsehitz functions arbitrary compact sets. However, achieving high precision for such complex targets requires the explicit model size (depth/width) to grow ar- bitrarily large. Here lies the crucial distinction: explicit models scale expressivity with model size, whereas implicit models are able to scale expressivity with test-time iterations. This allows implicit models to represent increasingly complex functions dynamically without adding parameters.
Training Dynamics and Convergence. A significant body of work focuses on the optimization mechanics of implicit models. Geng et al. (202T) rigorously established the equivalence between unrolled training and implicit differentiation via Neumann series approximations, validating the training methodologies used our case studies. [ing et al. (2023); Truong (2025) provide global convergence guarantees and rate analyses for the training in over-parameterized deep equilibrium models. While these studies ensure that training algorithms can successfully minimize the loss, work addresses the fundamental antecedent question: whether model exists that is capable of representing the target function in the first place.
Generalization. Distinct from expressivity, Fung & Berkels 2024 derive generalization bounds for families of implicit networks, characterizing their ability to perform on unseen data. Our analysis focuses approximation capacity the ability to construct operator that exactly reproduces map-which
target map- which is orthogonal to the sample complexity and generalization bounds discussed in their work.
48

---

Published as a conference paper at ICLR 2026
Infinite-Widthl Limits and Kernel Connections. Recent research has sought to bridge the gap between implicit models, explicit deep networks, and kernel methods. [ao et al. (2022) extend the ave-pearaettiizntioonn theory of explicit networks implicit models, establishing well-posedness and convergence even in finite-width regimes where standard infinite-depth results do not directly apply. In the ffhntee-witth limit, [Feng& & Kolter (2023) formally derive the Neural Tangent Ker- nel (NTK) for equilibrium models, characterizing their training dynamics in the linear regime. On the architectural side, Ling et al. (2024) show that for high-dimensional Gaussian mixtures, deep equilibrium models can be functionally equivalent to shallow explicit networks. In contrast to these kernel-based or citributiion-peciifie analyses, our work adopts a non-parametric function-spoo -ppace perspective; we demonstrate that for general locally Lipschitz targets, the expressive power of im- plicit models not static but scales dynamically with test-time computation, a property distinct from the linear regimes often studied in kernel theory.
L LLM USAGE STATEMENT
ariting-poliss assistant across all sections in the main text and appendix. Its role was limited to grammar fixes, wording/flow improvements, and rephrasing of text that we originally drafted. All model suggestions were reviewed, verified, and, when necessary, edited by the authors to ensure accuracy. The authors take full responsibility for the final manuscript, including any text influenced by LLM assistance.
We used LLMs solely
49


Reviews:
[{"review_id": "Y23UXFFSF2", "reviewer": "Reviewer_FMZf", "summary": "Prior works in deep equilibrium models have shown that these networks can outperform regular feedforward models with the same architecture as well as larger models by using more test-time iterations. This paper identifies a broad class of functions called regular functions for which implicit representations provide simple update operators while expressing complex fixed-point mappings. These regular implicit operators gain expressive power with more test-time iterations and can represent more complex class of locally Lipschitz functions. The key assumption is that the domain is bounded and the function is locally Lipschitz on this bounded domain. Then, the paper states that the first iterate is a simple globally Lipschitz map but the subsequent iterates make the map more complex resulting in locally Lipschitz function. This increase in complexity is measured via increase in Lipschitz constant. This theory has been validated on three case studies: image reconstruction, Navier-stokes PDE (scientific computing), and linear programming (operations research). All the three cases demonstrate that increase in test-time iterations result in a better solution and the composition of the regular operator results in a more complex and expressive function.", "questions": "1. Some typos: \nLine 69 - demonstrating -> deomonstrate\nLine 395 - LInaer programm -> linear program\n\n2. Line 947: $diam(X) + 1$ can be large, Why is it a meaningful upper bound? This uses the assumption that $X$ is bounded, but even then it can be really large.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The key findings are insightful to the community. Prior works have provided proof of universal approximation of implicit models such as deep equilibrium models. The key finding of this paper is the identification of the particular class of functions for which the operator  gains expressive power through additional test-time compute and can match more complex function class the includes singularities and unbounded Lipschitz behavior. This has been proved both theoretically and empirically.\n2. The paper provides valid practical recommendations for implicit models which is to infuse the architecture for operator G with domain priors and constraints instead of constraining the network architecture itself. The paper demonstrates ways to incorporate these priors in the operator through examples in image reconstruction, Navier-stokes PDE and linear programming.", "weaknesses": "1. The paper demonstrates empirically that standard training results in such a regular implicit operator operator in some cases, but to the best of my understanding, it does not explore how or if the training guarantees this. \n2. Another limitation of the theoretical result is the assumption of bounded domain. While the paper notes that this is not a major limitation for the results in the paper, how it affects other real-world problems is not well-understood.\n3. Writing: Many parts of paper are dense and a bit difficult to read. It will be useful if proof sketches are included for the main theorems (especially 2.4 and 2.5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VqXS92CLrn", "reviewer": "Reviewer_Ybb3", "summary": "This paper studies the expressive power of implicit neural models, analyzing how iterative fixed-point dynamics affect model capacity both theoretically and empirically.", "questions": "See the weakness section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper provides a rigorous theoretical analysis of the expressive power of implicit neural models, offering how iterative fixed-point dynamics influence expressive capacity.", "weaknesses": "- The paper’s framing of \"implicit models\" is confusing and deviates from the standard definition in the community, where implicit models such as Deep Equilibrium Models find an **equilibrium point** via a numerical solver like root-finding techniques rather than explicitly unrolling the recurrence; here, the authors effectively treat implicit models as **recurrent networks** and study the number of unrolled iterations, which misrepresents the core concept, and the term \"test-time computation\" is also confusing in this context.\n\n- The theoretical analysis relies on well-known tools such as function composition, contraction mappings, and ODE formulations, offering limited novelty or new insights beyond existing expressivity analyses.\n\n- The experiments focus on small toy problems and shallow MLP-like architectures without evaluation on modern large-scale tasks or established implicit models; moreover, there is no discussion of how the proposed analysis or conclusions scale with model size, data, or computation, which is central to understanding expressivity and scaling laws in contemporary deep learning.\n\n- The paper overlooks key studies on implicit and equilibrium models, including those addressing training convergence, generalization, and the theoretical connections between implicit models, explicit deep networks, and kernel methods, which weakens its positioning within the broader literature and limits its conceptual impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtoIILLApB", "reviewer": "Reviewer_La71", "summary": "This paper adds to the understanding of how expressive implicit models can be. \n\nImplicit models are an interesting vein for adaptive test-time computation for adaptive model quality. Older related work in adaptive test time compute has focused on using more/less features (see e.g. Parrish et al. JMLR 2013) or more/less base models in an ensemble (see e.g. Wang 2021 Quit When You Can), this approach is a bit more like the strategy of only using as many base models as you have time for from an ensemble, but with the nice difference that you just need to re-use the same model over and over.", "questions": "Have you talked to pure mathematicians about Theorems 2.4 and 2.5? Does the ML angle really bring a novelty to this problem that means mathematicians who study similar compositions wouldn’t have already studied this exact question and you could just be citing their work?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper argues that implicit models are even more interesting because they can more easily express certain kinds of complexity than standard explicit models, especially certain kinds of nonlinearities where the iterative mapping G can be reasonably smooth but iterating on it leads to sharp behavior that may otherwise be difficult to model. \n\nThey conclude that their insights mean one popular strategy of implicit models is often over-regularized, and suggest using other regularization strategies instead.", "weaknesses": "Modeling Novelty: Bit weird to act like this is a totally new idea guys - using iterative models is an ancient strategy in signal processing (and image processing) and controls. Kalman filtering, iterative filtering, etc,  the idea of repeatedly applying the same model, and that model having free parameters, is not at all a new idea, especially in areas like image deblurring. \n\nTheory Novelty: Theorems 2.4 & 2.5 really seem like something that mathematicians who study fixed-point stuff would already know and have studied in the 70’s.   Does the ML angle really bring a novelty to this problem that means mathematicians who study similar compositions wouldn’t have already studied this exact question? Sadly I’m not a fixed-point math expert, but I’d like to hear from one before being willing to believe the presented theorem is at all novel. \n\nExperiments: 3 case studies were interesting but small and lacked enough comparisons that I was able to be convinced of anything. \n\nReminder to use {} in your bib tex for capitalization (e.g. Euclidean) and make sure the same conference has the same title throughout your references. \n\nOverall Contribution: even ignoring my kvetching about novelty, I didn't the contributions significant enough to merit inclusion at NeurIPS, but I could be swayed if other reviewers helped me see why their insights are more unexpected and interesting than my read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BrPp1wS3hR", "reviewer": "Reviewer_xk5a", "summary": "The paper defines a theory of expressivity for implicit models (also called deep equilibrium models or fixed point models). Doing so, the paper provides a theoretically sound contribution towards (and essentially a solution to) the open problem of understanding the richness of implicit neural networks. Implicit architectures have been the subject of much attention over the last several years, but a precise characterization of what can be represented by implicit models is absent in the literature.  The paper contains three case studies illustrating the new expressivity results and defining appropriate \"regular\" implicit models.", "questions": "Line 32: \"at inference time we apply\"--- but there might be better algorithms to apply than the simple Picard iteration, correct?\n\nLine 128: definition of regular implicit operator. Perhaps you could clarify whether the map has any joint Lipschitz properties.\n\nTheorem 2.5: \"must be locally Lipschitz\" -> \"is locally Lipschitz\".  (but I agree that this is simply a stilistic matter).\n\nLine 174: \"growth\" -> \"grows\"\n\nLine 304: \"Solving ... solves\" = typo\n\nLine 395: \"Programm\"\n\nLine 1157 \"postive\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The key result is that any locally Lipschitz map (on a bounded domain) can be represented as the fixed point of a \"regular\" implicit operator, and vice versa.  Here \"regular\" describes a map that is globally Lipschitz in both variables, but not uniformly so. This result means that a regular implicit model can express maps with singularities and unbounded locally-Lipschitz behavior.\n\nIn short, I find this result, stated in Theorems 2.4 (sufficiency) and 2.5 (necessity), to be remarkable and clearly deserving acceptance.\n\nThe presentation in Introduction and Main result (sections 1 and 2) is lucid.\n\nIn each of the case studies (image reconstruction, scientific computing, and operations research), (i) the target map is locally Lipschitz (ii) an implicit model is proposed that, after standard training, is verified to be regular, and (iii) indeed as the interation count increases the implicit model realized increasingly accurate maps.", "weaknesses": "The abstract sats \"these compact models can match or even exceed larger explicit networks\" --- it is unclear what \"exceed\" means. In what sense? Accuracy? Memory or computational efficiency?  (the same confusing language at line 42)\n\nThe document should address the following topic: Do there exist explicit/feedforward networks  capable of expressing locally Lipschitz target maps?  I feel such a comparison should be prominently addressed in the introduction.\n\nCould sparse reconstruction problems (as in compressive sensing) be a special case of the image reconstruction approach in case study 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOlvPfXezg", "reviewer": "Reviewer_KADN", "summary": "This work is concerned with trying to explain the viability of equilibrium models. It is shown using Banach's fixed point theorem that using a regular map can approximate a function class that is only locally Lipschitz, where regularity is defined by contractiveness and global Lipschitz property. The more the map is applied the closer the approximation, thus scaling with test-time compute. This is illustrated with a toy example and further validated by case studies in different domains as image reconstruction, scientific computing, and operational research. The result sheds light on how implicit models can outperform explicit models by finding that implicit models only need to approximate a more regular function, whereas an explicit network would need to be able to approximate the actual function.", "questions": "See weaknesses as well.\n\nIs the comparison in Figure 5 fair, as the two models have the same amount of parameters, and the implicit model is allowed to reiterate effectively using more compute?\n\nIs the finding of the ability of training implicit GCNs with more overparameterization a new finding or is it more a confirmation of previous work?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study of the expressivity of implicit models is well motivated. The work presents a clear advantage of equilibrium models in this context. The exposition of work is good with an illustrative toy example and various case studies, especially Fig1 helps to substantiate the theoretical claims made. The argument that we need to be able to approximate locally Lipschitz functions provides also a new generalization perspective.", "weaknesses": "1) The theory is an application of a well known result: Banach's fixed point theorem. The theory could be strengthened by providing a rate of minimal/maximal possible growth for the Lipschitz constant. For example Figure 4a seems to suggest non-monotone behavior of the Lipschitz constant growth. Providing a bound could help with heuristics of how many iterations we have to apply in practice.\n\n2) Even tough the work validates their finding in multiple applications they are all from areas in applied mathematics. The work could be strengthened by showing results on a language task or a more open ended task.\n\n3) The illustrations of the implicit and non-implicit FNO in Figure 5 seem to be not very different can this be highlighted better? The only visual difference is the black region in the bottom right corner. In addition, the channel figures might not convey much information. Perhaps removing them and then highlighting the difference more could benefit the exposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)