Title: Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach
Venue: NeurIPS 2025
Accepted: yes
Keywords: Universal Approimation

continuous functions

deep narrow MLPs

deep learning
Abstract: Recently, there has been a growing focus on determining the minimum width requirements for achieving the universal approximation property in deep, narrow Multi-Layer Perceptrons (MLPs).
Among these challenges, one particularly challenging task is approximating a continuous function under the uniform norm, as indicated by the significant disparity between its lower and upper bounds.
To address this problem, we propose a framework that simplifies finding the minimum width for deep, narrow MLPs into determining a purely geometrical function denoted as $w(d_x, d_y)$. 
This function relies solely on the input and output dimensions, represented as $d_x$ and $d_y$, respectively.
To achieve this, we first demonstrate that deep, narrow MLPs, when provided with a small additional width, can approximate any $C^2$-diffeomorphism.
Subsequently, using this result, we prove that $w(d_x, d_y)$ equates to the optimal minimum width required for deep, narrow MLPs to achieve universality.
By employing the aforementioned framework and the Whitney embedding theorem, we provide an upper bound for the minimum width, given by $\operatorname{max}(2d_x+1, d_y) + \alpha(\sigma)$, where $0 \leq \alpha(\sigma) \leq 2$ represents a constant depending explicitly on the activation function.
Furthermore, we provide novel optimal values for the minimum width in several settings, including $w(2,2)=w(2,3) = 4$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works studies the optimal network width, in a minimax sense, for deep but narrow MLPs under the uniform norm, with a primary focus on the regime when the output dimension is (around or at least twice) larger than the input dimension. Toward understanding of the optimal width, the authors draw interesting connections to a geometrical function $w(n,m)$ that describes the minimum dimension needed for restoring a continuous function from a `lifting' embedding. They provide (as the main result) upper and lower bounds on the optimal width, for different classes of activations, using this quantity $w(n,m)$. Then the authors provide a few bounds on this proposed quantity and translate them into the bounds for the MLP optimal widths using the main result.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed bottleneck-like quantity $w(n,m)$ is interesting and insightful, and it closely connects to the minimum UAP width of interest.\n- The paper is nicely written and well explained.\n- The results give a nice addition to this line of research on deep shallow MLPs.\n- The main results are stated in terms of $w(n,m)$ and are general, while later results and remarks in Section 4.3 and 4.4 find nice connections to existing bounds on the minimum width under various settings.\n\nWeaknesses:\n- In Eq.(8) it uses that the projection and the inclusion (or essentially the identity) can be represented by deep narrow MLPs. Is this true for general activations except for ReLU or Leaky-ReLU? Is Condition 1 necessary and/or sufficient? This may deserve an elaboration in the text if I am not missing something trivial. This question also relates to Line 183 in the proof of Theorem 4.4.\n- (minor) The authors should mention that starting from Eq.(8) it assumes $n \\ge d_x, d_y$.\n\nTypos:\n- Line 67 Related Works\n- Eq.(31) should be $\\sigma$ instead of Leaky-ReLU\n- $O$ is not explicitly defined in Definition 4.13\n- $\\ge w(2,2)$ instead of = in Line 300\n\nQUESTIONS:\n- Please see weaknesses.\n- Can the authors refer to the specific place of misconception and its usage in the proof as mentioned in Remark 4.17?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new framework for developing novel upper and lower bounds on the minimum width for deep MLPs to universally approximating continuous functions under the uniform norm. The paper develops this framework by first proving that a deep narrow MLP can approximate any $C^2$-diffeomorphism and then utilizing that result to show how any continuous function can be approximated by composing linear transformations and diffeomorphisms. Using this framework they provide a novel upper bound on the minimum width of $max(2d_x + 1, d_y) + \\alpha(\\sigma)$. Their results hold for a variety of activation functions including the Leaky-ReLU and ReLU.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- To the best of my knowledge the diffeomorphism approach is a novel framework and this approach may pave the way to apply these tools from geometric topology to other deep learning theory problems.\n- The paper contributes to a clear gap in the literature on universal approximation with deep narrow MLPs. In particular, using their framework they demonstrate that a width of max(2d_x + 1, d_y) always suffices for UAP of deep (narrow) MLPs. This improves upon earlier works which placed the upper bound between max(d_x + 1, d_y) and d_x+d_y.\n- The paper \n- While the paper makes use of very technical math the contributions of the paper are clear and the high level proof strategy is easy to follow.\n\n## Weaknesses:\n- While I understand that this is a theoretical contribution the paper has a very high technical barrier making the proofs inaccessible to most members of the Neurips community. For instance, the term \"diffeomorphism\" is never defined in the paper. \n- The optimal results are for very specific choices of d_x and d_y. It is unclear whether this framework can be useful for obtaining optimal minimum widths for arbitrary d_x and d_y.\n\nQUESTIONS:\n- How is Definition 3.1 different from saying that A is dense in B?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper investigates the minimum width required for deep, narrow Multi-Layer Perceptrons (MLPs) to achieve the universal approximation property (UAP) under the uniform norm. The authors propose a framework that reduces the problem of determining the minimum width to a purely geometric function w(d_x,d_y), dependent solely on the input and output dimensions d_x and d_y​.  

The work bridges gaps in existing literature by providing tighter bounds and exact values for the minimum width, particularly in cases where the output dimension is roughly twice the input dimension.

STRENGTHS AND WEAKNESSES:
Strengths:

1.The paper is technically rigorous, with well-constructed proofs and a clear methodology. The use of geometric and topological tools to derive bounds is innovative and solidifies the theoretical foundations of the work.

2.The authors provide intuitive explanations for their results, such as the role of diffeomorphisms in approximation, which aids understanding.

3.The introduction of the geometric function and the framework for analyzing minimum width through diffeomorphisms are novel contributions.

Weaknesses:

1.The paper focuses on the uniform norm, which may limit its applicability to other types of norms commonly used in machine learning.

2.The assumptions made about the activation functions (e.g., C^1 near a point with non-zero derivative) may not hold for all types of activation functions used in practice.

3.The paper does not provide extensive empirical validation of the theoretical results, which could be beneficial for demonstrating the practical implications of the findings.

QUESTIONS:
see weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Emergency Review\n\n### Summary\n\nThe paper establishes the **minimum hidden-layer width** $w(n,m,\\sigma)$ required for a deep, narrow MLP with a given activation function $\\sigma$ to be a universal approximator on $C(\\mathbb{R}^n, \\mathbb{R}^m)$. The main result is that:\n\n$$\nw(n, m, \\sigma) = w(n, m) + \\alpha(\\sigma),\n$$\n\nwhere $w(n, m) = max(2n+1, m)$ is a topological term and $\\alpha(\\sigma) \\in \\{0,1,2\\}$ depends on the smoothness of the activation function.\n\nThe authors prove that this width is both **sufficient** and **necessary**. The proof is constructive for Leaky ReLU networks, involving approximations of diffeomorphisms via affine coupling flows and known geometric theorems. Extensions to ReLU and $C^1$ activations are made using results by Hanin (2019) and Kidger–Lyons (COLT 2020). Lower bounds are proven using tools from differential topology, and a detailed topological construction is provided for the $n = m = 2$ case using winding numbers and Hurwicz theorem.\n\nSTRENGTHS AND WEAKNESSES:\n### Pros\n\n- Closes a known theoretical gap by identifying the minimum width required for universal approximation with deep, narrow MLPs.\n- Covers a broad class of activation functions, including ReLU and other $C^1$ activations.\n- The upper bound is constructive and leverages elegant geometric and topological tools.\n- The lower bounds use rich machinery (immersion theorems, algebraic topology), giving strong evidence of necessity.\n- Clear modular structure in the proof pipeline (diffeomorphisms $\\rightarrow$ composition $\\rightarrow$ universal approximation).\n\n### Cons\n\n- The key assumptions are tailored to cases where the output dimension is roughly twice the input dimension (e.g., $m = 2n - 1$); this limits generality.\n- The exposition could benefit from more explicit motivation for the specific techniques used (e.g., affine coupling flows) and a better connection to the broader literature on UAP.\n- The topological lower bounds, especially in Section 4.5, are difficult to follow and would benefit from an appendix with extended intuition and proof sketches.\n- The role of $\\alpha(\\sigma)$ as a correction term is technically derived but lacks a strong intuitive justification in the main text.\n- The manuscript does not deeply discuss recently concurrent or slightly older related works on minimal width UAP.\n\n\n\n---\n\n### Overall Assessment\n\nThis is a strong theoretical paper that makes **meaningful progress on a long-standing question** regarding the width needed for universal approximation by deep narrow networks. The constructive nature of the proof and use of geometry/topology is elegant and technically non-trivial. However, the **exposition could improve**, particularly in motivating the approach, situating it within the broader UAP literature, and expanding on difficult sections.\n\n### Overall Recommendation\n\n**Score**:  (Weak Accept)  \n**Confidence**: 4 (Expert in this area)  \n**Rationale**: The result is solid and interesting, though the work is **incremental** relative to other recent contributions. A stronger contextualization and more reader-friendly explanations would elevate the impact.\n\nQUESTIONS:\n### Questions for the Authors\n\n1. Could you clarify **why affine coupling flows** and **single-coordinate diffeomorphisms** are particularly suited for universal approximation proofs? What’s the intuition behind choosing these building blocks (as also used in Teshima et al., NeurIPS'20)?\n\n2. In what technical sense do your **constructions depart from Teshima et al.**? Do you rely on fundamentally different structural decompositions or just a different universal target?\n\n3. The overhead term $\\alpha(\\sigma)$ appears due to Hanin (2019) and Kidger–Lyons (2020). Are these $+1$ (ReLU) and $+2$ ($C^1$) corrections **provably unavoidable**, or could improved constructions close this gap?\n\n4. Could the results **extend to architectures** that include skip connections, convolutional layers, or residual blocks? Do you expect the minimum width bound to improve or worsen?\n\n5. Could you provide intuition for what the function $w(n,m,\\sigma)$ captures beyond a topological obstruction? Does it have geometric interpretation in network function space?\n\n6. Are there technical obstacles to extending the **lower bound arguments** to the case of odd $k$? Could methods from obstruction theory or embeddings help?\n\n\n### Suggestions for Related Work to Cite\n\nThe following works seem highly relevant but are not cited or discussed:\n- *ReLU Network with Width Can Achieve Optimal Approximation Rate* (ICML 2024)\n- *New Advances in Universal Approximation with Neural Networks of Minimal Width* (arXiv 2024)\n- *Minimal Width for Universal Property of Deep RNN* (JMLR 2023)\n- *Approximating Continuous Functions by ReLU Nets of Minimal Width* (Yarotsky, 2017)\n- *Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks* (arXiv 2025)\n\nIncluding a discussion of how your results compare with these would clarify your contribution.\n\n### Typos and Clarity\n\n- Line 383, Eq. 44 (Definition A.3): Does $\\tilde{\\tau} \\in C(\\mathbb{R}^d, \\mathbb{R})$ ensure invertibility?\n- Line 232: \"than that both\" $\\rightarrow$ \"than both\"\n- Line 242: \"To provided\" $\\rightarrow$ \"To provide\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-10"}

comments: SUMMARY:
This paper establishes new upper and lower bounds on the minimum width required for deep, narrow MLPs to achieve universal approximation within continuous function spaces. The new upper bounds now depend on $\max(2d_x + 1, d_y)$, sharpening earlier results. The argument is built on diffeomorphism-based constructions, bridging geometric topology and neural approximation theory.

STRENGTHS AND WEAKNESSES:
## Strengths

- The new upper bound improves prior results: it now scales with $\max(2d_x + 1, d_y)$ rather than $d_x + d_y$.
- The proof is built on diffeomorphism theory, which is original (to my knowledge) in the UAP study and provides new insights.
* The technical development seems solid. I did not verify every detail, but I checked the main ideas and several key lemmas.

Overall, I am impressed by the work.

## Weaknesses / Questions

1. Lines 153–154 introduce the notation $w(m,n)$ for the first time, and seem to define it as the “required width for approximation” by neural networks. However, Eq. (12) formally defines $w(m,n)$ as the minimum width needed to approximate arbitrary continuous functions using diffeomorphisms. This mismatch is confusing. Please clarify.

2. In Lemma 4.3, does the embedding $f$ need to be piecewise linear, or is smoothness sufficient? Please state explicitly what regularity is required.

3. Lemma 4.1 is crucial. Once it is established, the main result follows fairly directly from known results in topological algebra (if I understood correctly). However, its full proof is scattered throughout the appendix, which makes it difficult to follow. Could you provide a more organized proof outline or a concise roadmap in the main text?

QUESTIONS:
Please see "Weakness/Questions"

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-23