Title: On Robust Concepts and Small Neural Nets
Authors: Amit Deshpande
Sushrut Karmalkar
Venue: ICLR 2017
Keywords: Theory

Abstract: The universal approximation theorem for neural networks says that any reasonable function is well-approximated by a two-layer neural network with sigmoid gates but it does not provide good bounds on the number of hidden-layer nodes or the weights. However, robust concepts often have small neural networks in practice. We show an efficient analog of the universal approximation theorem on the boolean hypercube in this context.

We prove that any noise-stable boolean function on n boolean-valued input variables can be well-approximated by a two-layer linear threshold circuit with a small number of hidden-layer nodes and small weights, that depend only on the noise-stability and approximation parameters, and are independent of n. We also give a polynomial time learning algorithm that outputs a small two-layer linear threshold circuit that approximates such a given function. We also show weaker generalizations of this to noise-stable polynomial threshold functions and noise-stable boolean functions in general.

TL;DR: an efficient analog of the universal approximation theorem for neural networks over the boolean hypercube

Review discussion:
The constants appear to grow with smaller \delta and \epsilon.
Are the presented results not contradictory when smaller sensitivity to noise goes with larger number of units?
Are the presented results not contradictory when the same sensitivity at larger noise goes with a larger number of units?
How different is Theorem 3 from what could be obtained using Theorem 1 alone and not Theorem 2?
In Bourgain's theorem, the same \delta appears as the noise sensitivity parameter and the closeness of approximation. It is difficult to decouple both. Our theorem does give a statement like "more robust concepts have smaller neural nets". Consider the closeness of approximation as a fixed constant, say, 0.01.
That is, we are looking for a neural net that approximates the given concept on 1% of the inputs. For (\epsilon, 0.01)-noise stable concepts, the more robust concepts are the ones that are noise stable even under large \epsilon-perturbations. Our theorem implies that these concepts are approximated by a small poly(1/\epsilon)-sized neural net on 99% of the inputs. Theorem 2 gives a better bound than Theorem 1 where the constants are polynomial in 1/\epsilon and 1/\delta.
You have talked about small neural networks but there is an exponential dependence on 1/eps . How is this more interesting (or tighter) than the dependence on the dimension? Is it possible to combine this with the results that only depends on the dimension to get something tighter?
Theorem 2 gives a tighter result than Theorem 1, where the neural net size is polynomial in 1/\epsilon and 1/\delta. Our bounds depend on the noise-stability of concepts and are independent of the dimension, which is very useful in an infinite-dimensional kernel setting.
The results of the paper seem to be a straightforward application of results of previous work.
Besides being able to connect results from different domains (the Junta problem, circuit complexity) to the two-layer neural net, can the authors justify their own contributions in their analysis?
Yes, our results are technically simple, and obtained by combining Bourgain's junta theorem and size-weight-depth trade-offs from circuit complexity in a fairly straightforward manner. To our knowledge, this observation is new. The main contribution of our paper is in making this connection, trying to formulate an efficient universal approximation theorem for neural nets, and exploring its limitations.
The approximation capabilities of neural networks have been studied before for approximating different classes of functions. The goal of this paper is to provide an analog of the approximation theorem for the class of noise-stable functions. The class of functions that are noise-stable and their output does not significantly depend on an individual input seems an interesting class and therefore I find the problem definition interesting.  The paper is well-written and it is easy to follow the proofs and arguments.
I have two major comments:
1- Presentation: The way I understand this arguments is that the noise-stability measures the "true" dimensionality of the data based on the dependence of the function on different dimensions.
Therefore, it is possible to restate and prove an analog to the approximation theorems based on "true" dimensionality of data. It is also unclear when the stability based bounds are tighter than dimension based bounds as both of them grow exponentially. I find these discussions interesting but unfortunately, the authors present the result as some bound that does not depend on the dimension and a constant (!??) that grows exponentially with (1/eps). This is not entirely the right picture because the epsilon in the stability could itself depend on the dimension. I believe in most problems (1/epsilon) grows with the dimension.
2- Contribution: Even though the connection is new and interesting, the contribution of the paper is not significant enough. The presented results are direct applications of previous works and most of the lemmas in the paper are restating the known results. I believe more discussions and results need to be added to make this a complete work.
Thank you for your comments and suggestions for improvement. Our paper does mention interesting classes of (\epsilon, \delta)-noise-stable functions where 1/\epsilon and 1/\delta are constants INDEPENDENT of the dimension, e.g., constant-margin halfspaces and submodular functions. Please contrast this with arbitrary halfspaces of the d-dimensional hypercube where margin can be as small as exp(-d \log d) in the worst case. In this sense, noise-stability as the "true" dimension can be much smaller than the actual dimension.
We do mention that our results are a direct combination of known results (a) Bourgain's junta theorem and its more efficient version for halfspaces and (b) size-depth-weight trade-offs for linear threshold circuits. We did look for more general or surprising results beyond these. The definition of robust concepts can be generalized as those having a small probability of label-flipping (from +ve to -ve or vice versa) when a small random gaussian perturbation is applied to a data point picked from an underlying distribution.
However, proving similar results would require very different ideas. We have a detailed discussion of the obstacles to improvement in Subsection 2.1.
This work finds a connection between Bourgain's junta problem, the existing results in circuit complexity, and the approximation of a boolean function using two-layer neural net. I think that finding connections between different fields and applying the insights gained is a valid contribution.
For this reason, I recommend acceptance.
But my current major concern is that this work is only constrained to the domain of boolean hypercube, which is far from what is done in practice (continuous domain).
Indeed, the authors could argue that understanding the former is a first step, but if the connection is only suitable for this case and not adaptable to more general scenarios, then it probably would have limited interest.
Thank you for your comments and suggestions. We have a detailed discussion in Subsection 2.1 on obstacles to extending our approach and ideas to a more general setting. The definition of robust concepts can be generalized as those having a small probability of label-flipping (from +ve to -ve or vice versa) when a small random gaussian perturbation is applied to a data point picked from an underlying distribution.
However, proving similar results would require very different ideas than what works for the Boolean hypercube. We are interested in this question, and we do have a detailed discussion of the obstacles to improvement in Subsection 2.1.
SUMMARY
This paper presents a study of the number of hidden units and training examples needed to learn functions from a particular class.
This class is defined as those Boolean functions with an upper bound on the variability of the outputs.
PROS
The paper promotes interesting results from the theoretical computer science community to investigate the efficiency of representation of functions with limited variability in terms of shallow feedforward networks with linear threshold units.
CONS
The analysis is limited to shallow networks. The analysis is based on piecing together interesting results, however without contributing significant innovations.
The presentation of the main results and conclusions is somewhat obscure, as the therein appearing terms/constants do not express a clear relation between increased robustness and decreasing number of required hidden units.
COMMENTS
- In the abstract one reads ``The universal approximation theorem for neural networks says that any reasonable function is well-approximated by a two-layer neural network with sigmoid gates but it does not provide good bounds on the number of hidden-layer nodes or the weights.''
In page 1 the paper points the reader to a review article. It could be a good idea to include also more recent references.
Given the motivation presented in the abstract of the paper it would be a good idea to also comment of works discussing the classes of Boolean functions representable by linear threshold networks.
For instance the paper [Hyperplane Arrangements Separating Arbitrary Vertex Classes in n-Cubes. Wenzel, Ay, Paseman] discusses various classes of functions that can be represented by shallow linear threshold networks and provides upper and lower bounds on the number of hidden units needed for representing various types of Boolean functions.
In particular that paper also provides lower bounds on the number of hidden units needed to define a universal approximator.
- It certainly would be a good idea to discuss the results on the learning complexity in terms of measures such as the VC-dimension.
- Thank you for the explanations regarding the constants.
So if the noise sensitivity is kept constant, larger values of epsilon are associated with a smaller value of delta and of 1/epsilon.
Nonetheless, the description in Theorem 2 is in terms of poly(1/epsilon, 1/delta), which still could increase?
Also, in Lemma 1 reducing the sensitivity at a constant noise increases the bound on k?
- The fact that the descriptions are independent of n seems to be related to the definition of the noise sensitivity as an expectation over all inputs. This certainly deserves more discussion. One good start could be to discuss examples of functions with an upper bound on the noise sensitivity (aside from the linear threshold functions discussed in Lemma 2).
Also, reverse statements to Lemma 1 would be interesting, describing the noise sensitivity of juntas specifically, even if only as simple examples.
- On page 3 ``...variables is polynomial in the noise-sensitivity parameters'' should be inverse of?
MINOR COMMENTS
On page 5 Proposition 1 should be Lemma 1?
Thank you for your comments and suggestions. We will add references such as Wenzel et al. on hyperplane arrangements and their comparison or discussion.
We do mention interesting classes of (\epsilon, \delta)-noise-stable functions where 1/\epsilon and 1/\delta are constants INDEPENDENT of the dimension, e.g., constant-margin halfspaces and submodular functions. Please contrast this with arbitrary halfspaces of the d-dimensional hypercube where margin can be as small as exp(-d \log d) in the worst case. In this sense, noise-stability as the "true" dimension can be much smaller than the actual dimension. As you suggested, we will add a few examples of simple noise-stable functions to improve readability.
The confusion about the constants arises from Bourgain's junta theorem where the same \delta appears as closeness of approximation as well as in the size of the junta (as 1/\delta). We have not been able to decouple these two, and as far as we know, any such resolution would be of independent interest in the analysis of Boolean function.

Meta-review:
role: PC
data: comment: The paper makes a solid technical contribution in establishing a universal approximation theorem for the boolean hypercube. They characterize the class of boolean functions that can be efficiently approximated by a two-layer network.
 
 We like the idea of noise stability, and it could explain why in practice, perturbation techniques such as dropout are effective. Moreover, humans can identify images, despite corruptions, and hence, intuitively the concepts we aim to learn should be robust.
 
 However, the framework of the paper deviated from the networks and data structures that are the norm in practice. In practice, we rarely have boolean functions. And it is well known that boolean functions can behave quite differently from continuous functions. 
 
 We recommend that the authors widen the scope of their work, and attempt to connect their findings to practical networks and functions. Moreover, we recommend that they do a more thorough literature survey. For instance, the idea of robust concepts has appeared before
 http://www.pnas.org/content/113/48/E7655.full
 And nice connections to elastic learning have been made in the above paper.
 
 We recommend a workshop presentation since this will enable more interaction with people in the area, and people who are working on practical networks.

Decision: Invite to Workshop Track