Title: Approximation theory for 1-Lipschitz ResNets
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lipschitz neural networks

Universal approximation

Residual neural networks

Stone–Weierstrass theorem
Abstract: $1$-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on $1$-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone–Weierstrass Theorem, we first show that these $1$-Lipschitz ResNets are dense in the set of scalar $1$-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine $1$-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for $1$-Lipschitz ResNets, laying a rigorous foundation for their practical use.


Peer reviews:
comments: SUMMARY:
In this paper, the authors study the approximation properties of 1-Lipschitz ResNets. The main results show that the class of 1-Lipschitz ResNests is the universal approximator of the 1-Lipschitz functions.

STRENGTHS AND WEAKNESSES:
# Strength
The paper is describes the problem clearly and the claims are easy to understand. The results presented in the paper seems to be useful as it theoretically explains the usefulness of ResNets. In my opinion the paper answers an important and interesting open problem in this area.

# Weakness

Although the main results are strong, I pose a couple of important questions below:

- Both the main results do not mention the number of Layers $L$ needed to get a desired level of approximations. Typically, in applications, we have a very high data dimension $d$. If $L$ depends linearly on $d$, then it would be very hard to train the model and would seriously hinder the implementability.

- Is it possible to quantify the approximation error in terms of the data dimension $d$? This has a practical importance as good approximation properties will help to implelemnt comparatively less complex models that are easier to train.

QUESTIONS:
Please see the above comments.

LIMITATIONS:
Please see the above comments.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-13

comments: SUMMARY:
This paper provides an approximation theory for 1-Lipschitz ResNets. The authors prove that 1-Lipschitz ResNets are dense in 1-Lipschitz continuous functions on compact domains, in two settings: 1) unbounded width and depth; 2) fixed width and unbounded depth.

STRENGTHS AND WEAKNESSES:
__Strengths:__

1. This paper is well-written and is easy to follow.

2. This paper provides an approximation theory for 1-Lipschitz ResNets, which is an important theoretical foundation. The authors establish the theory in two settings: 1) unbounded width and depth; 2) fixed width and unbounded depth.

__Weaknesses:__

1. Though the theoretical analysis is very interesting, most techniques (such as the Restricted Stone-Weierstrass) are already known in the literature, which makes me a bit worried about the novelty.

2. The analysis highly depends on ReLU and is not general enough.

3. Though the authors mention the implementability in Section 5, there are no numerical experiments. This might be fine given the theoretical focus of the paper.

QUESTIONS:
No.

LIMITATIONS:
Limitations are clearly discussed in Section 5.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies approximation capabilities of 1-Lipschitz ResNets (residual networks with a Lipschitz constant c < 1), which are commonly used in inverse problems, generative modeling, and adversarial defense. The main contribution of the paper is the first proof that such models can approximate _any_ 1-Lipschitz function, at least when considering negative gradient steps and interleaving residual layers with affine transformations. Authors first show that this holds under infinite width/depth, but proceed to show that this still holds at a fixed, finite layer width.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Significant and original theoretical result: the first proof of universality for 1-Lipschitz ResNets, albeit with some deviations from the commonly used architecture (adding affine layers).\n+ The proof and prior work are introduced with great clarity, guiding the reader through all the steps involved, their motivation, and how they relate to literature.\n+ High-quality manuscript: precise and clear writing, good formatting, consistent mathematical notation.\n+ In-depth discussion of the limitations of the proof and their implications.\n\nWeaknesses:\n- Authors have to add affine transformations to the ResNets to show universality. While arguably a minor addition, this might have implications in practice, e.g. for optimization. An important benefit of residual networks is better gradient propagation, which might suffer due to the addition of affine layers, especially for deeper nets. Which leads to the next weakness...\n- No numerical experiments. While the main contribution of the paper are the universality proofs, authors have to make adjustments to standard architectures to complete the proofs. The contribution would be stronger if authors provided some numerical results to confirm these additions are indeed practical, and don't impact the performance of these models on real problems.\n\nQUESTIONS:\n1. On lines 151-152 authors suggest that \"the lack of explicit constraint over $\\hat{Q}$ leads to problems when implementing these networks\". What are these problems? Isn't a lack of constraints easier to implement with gradient-based optimization, i.e. we don't have to resort to projected SGD or other methods?\n2. Have authors studied other common activation functions (sigmoid, tanh, etc.), to understand if they have the properties stated on lines 321-322 that would make the proof applicable?\n\nLIMITATIONS:\nI suggest authors extend their discussion on the potential practical consequences of adding affine layers to a residual architecture.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the approximation capabilities of 1-Lipschitz Residual Neural Networks (ResNets) based on explicit Euler steps of negative gradient flows. The authors provide two main theoretical contributions: (1) They prove that 1-Lipschitz ResNets with unbounded width and depth are dense in the set of scalar 1-Lipschitz functions on compact domains, and (2) They show that the same density property holds when the hidden width is fixed by introducing constrained linear maps between residual blocks. The proofs rely on the Restricted Stone-Weierstrass Theorem and constructive arguments showing that these networks can represent all piecewise affine 1-Lipschitz functions.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Strong theoretical results: The paper provides an universal approximation theory for 1-Lipschitz ResNets.\n\n2. Rigorous mathematical analysis: The dual proof strategies (Restricted Stone-Weierstrass and piecewise affine construction) provide comprehensive theoretical coverage and different perspectives on the problem.\n\n\n## Weaknesses\n\n1. Lack of experimental validation\nThe most significant weakness is the **complete absence of any experimental results**. For a neural network paper, this is a critical omission. The authors provide no evidence that:\n- Their theoretical constructions are computationally feasible\n- The proposed architectures perform competitively with existing methods\n- The universal approximation property translates to practical benefits\n\n2. Questionable practical implementability\nWhile the authors claim implementability, several concerns arise:\n- The regularization approach for unconstrained Q in G_{d,ρ}(X,R) using terms like Σ(ReLU(||∇_x N_θ(x_i)||_2 - 1))^2 is **not friendly for unconstrained optimization of deep neural networks**\n- Spectral norm constraints require expensive power method computations during training\n- Row-wise ℓ_2 norm constraints add significant computational overhead\n- No analysis of computational complexity or training stability\n\nHowever, these operation are not friendly for the optimization of deep neural network \n\n### 3. Limited scope and extensions\n- Results are restricted to scalar functions (c=1 case), with vector-valued extensions only briefly mentioned\n- Heavy reliance on ReLU activation limits generalizability, practically, we use silu, swiglu and etc.\n\n\n\n## Minor Issues\n\nI have one suggestion. Adopt standard mathematical typography conventions:\n- Use **bold** for vectors (e.g., **x**, **v**)\n- Use regular font for scalars (e.g., L, h, d)  \n- Use **uppercase bold** for matrices (e.g., **W**, **Q**, **A**)\nThis would improve readability and help readers quickly distinguish between different mathematical objects.\n\nQUESTIONS:\nThis paper makes solid theoretical contributions to understanding 1-Lipschitz ResNets, but falls short of demonstrating practical value. The theoretical results are rigorous and novel, but the complete absence of experimental validation raises serious questions about the practical relevance of these findings.\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the approximation capabilities of 1-Lipschitz ResNets, with ReLU activation function. It proves that 1-Lipschitz ResNets with unbounded width and length can approximate any 1-Lipschitz scalar function. The key to prove the results is to construct a 1-Lipschitz ResNets layer, which the paper leverages the existing results (called the explicit Euler step for the negative gradient flow differential equation; see Proposition 2.1 in line 125 and the remarks right after). The paper further proves that the same approximation capability holds when the width is fixed (specifically, width length can be fixed to be no less than d+3 where d is the input dimension) and only the depth is unbounded. The proof builds upon an understanding that any piecewise affine 1-Lipschitz scalar function can be presented by a fixed width 1-Lipschitz ResNets; the result is interesting. As a theory paper, there is no numerical experiments to validate or show the usefulness of the results under practical settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper appears to address a (less-motivated) research gap that 1-Lipschitz ResNets can approximate any 1-Lipschitz scalar function arbitrarily well. \n- The authors are knowledgeable about the relevant techniques and leverage them towards proving the results in this paper.  \n- The paper is relatively easy to follow. \n\nWeaknesses:\n\n- The motivation can be more concrete and stronger. The paper only briefly states the importance 1-Lipschitz NN without giving concrete evidence. Same for the claim in the introduction that “The commonly adopted strategies to constrain a network’s Lipschitz constant tend to reduce its expressiveness …”. Concrete evidence would strengthen the motivation, especially for laymen.\n- The challenges in proving the main results and their significance, that 1-Lipschitz ResNets with unbounded width and depth (or bounded width and unbounded depth) can approximate any 1-Lipschitz scalar function, should be discussed. It will be great if the authors can highlight the significance of such results in a broader context (beyond that it addresses a weekly-motivated research gap). \n\nMinor comments: \n\n- Why presenting Theorem 3.1, if the results in Theorem 4.1 already covers those in Theorem 3.1?  (Thus, one can present Theorem 4.1 directly, which seems to involve more new developments in this paper anyway.)\n\nQUESTIONS:\n1. Perhaps the authors could give more concrete motivation of why focusing on 1-Lipschitz NN and why addressing the research gap, i.e., proving the 1-Lipschitz ResNets can approximate any 1-Lipschitz function is important. For example, why just ResNets but not other NN architectures like GNN, Transformers, etc? Can the techniques for ResNets be applicable to GNN /Transformers?\n2. Perhaps the authors could highlight the novelty of the techniques used in the proof and the significance of the results (by for example putting them with respect to the existing universal approximation results for ResNets).\n\nI am ready to increase my score if the above issues are well addressed.\n\nLIMITATIONS:\nThe paper presents discussions on some proof techniques, extension of the results, and implementation of the proposed ResNets architecture in Sec. 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}