Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work tightens the upper bound from Park et al. (2021) while also proving a corresponding lower bound and prove that both $C-$UAP and $L^p-$UAP for functions on compact domains share a common minimum width $w_{min} = \max(d_x, d_u)$. In fact, they use some tools from Neural ODEs to show that the critical width for $L^p-$UAP is achieved by leaky-ReLU networks.
strength_and_weaknesses: **Strengths:**
- They prove matching upper and lower bounds, tightening the bound from Theorem 4 in Park et al. (2021) while also closing the problem of $L^p-$UAP.
- The paper is well written with clear proofs.

**Weaknesses:**
1. I think Table 1 should also include Theorem 4 from Park et al. (2021) which shows that $w_{min} \leq \max(d_x+2, d_y+1)$ for continuous non-poly activations. If I understand correctly, the main contribution of this paper is tightening that bound and proving a corresponding lower bound which Park et al., were lacking. Please correct me if I am wrong.
2. The definition of UOE functions should be stated (atleast intuitively) much earlier in the manuscript as it is mentioned several times.
3. I would rephrase Lemma 11 as a corollary since it follows directly from Li et al., and Duan et al.'s results. That said, it is a nice connection that proves this result quite easily. 
clarity,_quality,_novelty_and_reproducibility: **Clarity, Quality, Novelty:**

The paper is well written and the proofs are easy to understand. I think it is a good contribution that improves the results from past work.
summary_of_the_review: I have a few minor concerns and clarifications as mentioned above. If they are resolved, I would recommend acceptance. I think it is a good contribution to a line of work which has a rich history.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper builds on certain recent results on Neural-ODE way of approximating functions and gets sharp estimates on the width required for feed-forward nets to approximate continuous and Lp spaces over compact domains. 
strength_and_weaknesses: The two highlights of their results is that (a) they are able to get a sharp estimate of the width requirement for space of continuous functions mapping between two unit cubes in different dimensions.  And (b) they have a very interesting approach to these width estimation problems via recent results in Neural-ODE. 


clarity,_quality,_novelty_and_reproducibility: One of the key confusions I have here is about the rationale of proving theorems for the unit cube but claiming that the results hold for any compact domain. Why are the width requirements scale invariant? This claim seems to have been invoked many times but never really explained. 

The readability of the paper is quite questionable. For eg. (a) the authors are assuming familiarity with recent results like the ``encoder-memorizer-decoder approach" and hence are not seeming to define any of those terms. Very likely that only a small part of the audience will know of these immediately. This makes the paper feel quite incomplete. Maybe they could have given short descriptions of these in the appendix? (b) The $T_k$ matrices in equation $5$ are quite unclear. What is their definition? Equation 6 seems to give a definition but I am finding it quite hard to parse - where does this come from? 
summary_of_the_review: I think the paper needs some significant improvement in the presentation clarifying the issues raised above and my rating essentially reflects that.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The script studies the minimum width of neural networks to achieve universal approximation. Similar topics are also extensively studied in Park et al. 2022.

strength_and_weaknesses: The presentation needs to be improved. For instance, many terminologies such as C-UAP are not defined.
clarity,_quality,_novelty_and_reproducibility: See my comments below. 
summary_of_the_review: 1. I doubt the significance of the script compared with the existing work Park et al. 2022. I compare the results as follows.

**As for the lower bound:** the script provides a lower bound for arbitrary activation, which extends the results in Park et al which focuses on ReLU. 

Park et al. : w_min >= max (d_x +1 , d_y), ReLU

This script: w_min >= max (d_x , d_y), arbitrary activation


**As for the upper bound:** both the script and Park et al. construct neural networks to match the lower bound. 

Park et al. : w_min < = max (d_x +2 , d_y+1 ), using general activation

This script: w_min < = max (d_x , d_y), using Leaky-ReLU+ ABS. 


For both the upper and lower bound, the script improves the results in Park et al. only by constant 1 or 2. I doubt the significance of such improvement, please clarify the importance if there is any.



2. The most important theoretical result is Theorem 2. However, the relevant analysis (the neuron ODE part) is mainly based on  Li et al. 22 and Duan et al. 22, please clarify the novelty if there is any. 


3. The definition of C-UAP and Lp_UAP is missing


++++++++++++++++++ **POST REBUTTAL** ++++++++++++++++++++++

Thanks for the reply from the authors. Regarding my question 1, I am still not convinced about the importance of this result compared with the upper and lower bound of Parker et al.. I will keep my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the universal approximation problem of functions using neural networks. They showed that, the minimum width to approximate any functions in $L^p(\mathcal K, \mathbb{R}^{d_y})$ and $C(\mathcal K, \mathbb{R}^{d_y})$ is at least $w^*_{\min} = \max (d_x, d_y)$ for any activation functions. Furthermore, they showed that for some specific activation functions, such as Leaky-ReLU or ReLU+FLOOR, the minimal width can be achieved. 
strength_and_weaknesses: The related works are adequately cited. The main results in this paper will certainly help us have a better understating of the universal approximation property of deep neural networks from a theoretical way. I have checked the technique parts and found that the proofs are solid. The main result, which states that for any activation functions, the minimum width of a DNN to approximate any functions in $L^p(\mathcal K, \mathbb{R}^{d_y})$ and $C(\mathcal K, \mathbb{R}^{d_y})$ is at least $\max (d_x, d_y)$, is an unexpected and elegant result, which is a non-trivial extension of previous results in this field. It would be more interesting if the authors could study the exact minimum width for more activation functions and more architectures used in practice. 
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is clearly written and well organized. I find it easy to follow.
Quality: This paper is technically sound.
Novelty: The novelty of this paper is high.

summary_of_the_review: The main result, which states that for any activation functions, the minimum width of a DNN to approximate any functions in $L^p(\mathcal K, \mathbb{R}^{d_y})$ and $C(\mathcal K, \mathbb{R}^{d_y})$ is at least $\max (d_x, d_y)$, is an unexpected and elegant result, which is a non-trivial extension of previous results in this field.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work studies the universal approximation by finite width neural networks. It shows with max(d_x, d_y) neurons, a network can approximate any function in L^p(K, R^{d_y}) and C(K, R^{d_y}). Such minimum width can be achieved for some avtivation functions like Leaky ReLU. It extends the results on Park et al. 2022 by considering more general activation functions. This extension is nontrivial and may of interest to the community.  I think the theoretical contribution is enough for the acceptance and I recommend accept. 
justification_for_why_not_higher_score: It's a extension of the existence theorem, not a newly proposed result.
justification_for_why_not_lower_score: This extension is nontrivial and may of interest to the community.
Decision: Accept: poster