Title: Nonparametric Neural Networks
Authors: George Philipp
Jaime G. Carbonell
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch. In this paper, we address the problem of automatically finding a good network size during a single training cycle. We introduce {\it nonparametric neural networks}, a non-probabilistic framework for conducting optimization over all possible network sizes and prove its soundness when network growth is limited via an $\ell_p$ penalty. We train networks under this framework by continuously adding new units while eliminating redundant units via an $\ell_2$ penalty. We employ a novel optimization algorithm, which we term ``Adaptive Radial-Angular Gradient Descent'' or {\it AdaRad}, and obtain promising results.

TL;DR: We automatically set the size of an MLP by adding and removing units during training as appropriate.

Review discussion:
It is not so clear to me when and why you would be able to remove units.  It seems that units can be removed only when there are all zero weights either into or out of the unit.  Why would this occur during training?  Once units have non-zero weights, do they ever evolve to have zero weights?  Is this caused by regularization on the weights?  If so, why not start with a large number of units, and rely on this pruning to reduce the number of units, rather than starting with a small number of units?
Dear Reviewer,
Thank you for your comment. Units can be removed when either all incoming weights (fan-in) or all outgoing weights (fan-out) are zero, because this causes them to have no impact on the output of the neural network.
The fan-in and fan-out regularizer, which we employ in the paper, achieve this. As an example, take the fan-in regularizer. It applies a penalty to the fan-in which causes it to be shrunk to zero during training. This is the same effect that is utilized in many sparse models, such as Lasso or Sparse Coding. The l2 group penalty that we apply has the specific effect of setting all weights in the fan-in to zero simultaneously. This is the point at which the unit can be removed.
For a fan-in not to shrink to zero, the effect of the regularizer has to be counterbalanced by the gradient of the loss on the dataset.
Hence, the criterion for a unit to stay "alive" is that the gradient of the loss on the dataset with respect to its incoming weights is of a certain size.
However, this implies that the unit has a certain "importance" in computing the output of the network, which is precisely the criterion we want to use for keeping a unit alive.
The goal of this paper is to eliminate the need for choosing the size of each layer manually. The strategy of starting with many units and removing some via regularizers does not eliminate that need, as one still has to choose the number of units with which to begin training. As far as I can tell, the pruning methodology is generally applied to well-known architectures such as AlexNet, VGG etc. on well-known datasets such as ImageNet. Those papers rely on the availability of these well-tuned architectures as an initial point from which to start the pruning process. The situation we are interested in is when such an architecture is not available, as is the case in many non-vision, non-NLP tasks. Our goal is to contribute to the development of out-of-the-box methods that do not require the user to specify any manual inputs, or at least as few manual inputs as possible.
Hence, we start with a generic small number of neurons (10) and add 1 unit per layer per epoch.
The only way to eliminate the need for adding units while not requiring the specification of an initial number of units is to start with an enormous number of units, a number potentially much larger than the best possible size. I don't think it is obvious that this would work in practice, however.
After all, larger networks tend to be harder to optimize. I agree that the strategy of starting with a generic enormous network warrants investigation. We chose to investigate the "start small, add units" case instead as we believe it may be more practical, for obvious computational reasons.
I look forward to further comments.
George
Dear Reviewers,
In response to the first question posed to me, I revised the paper to include a discussion of pruning, and added several references. I would like to thank reviewer 2 for his / her feedback. I was not fully aware of how prominent this topic had become in the deep learning community recently as well as the increased use of l2 and l1 regularization.
The main changes are:
- abstract
- introduction
- section 3 before 3.1
- further background (first half)
- conclusion
Plus, I improved the grammar / wording in a few places.
I look forward to further comments.
Best,
George
This paper addresses the problem of allowing networks to change the number of units that are used during training.  This is done in a simple but elegant and well-motivated way.  Units with zero input or output weights are added or removed during training, while a group sparsity norm for regularization is used to encourage unit weights to go to zero.  The main theoretical contribution is to show that with proper regularization, the loss is minimized by a network with a finite number of units.
In practice, this result does not guarantee that the resulting network will not over- or under-fit the training data, but some initial experiments show that this does not seem to be the case.
One potential advantage of approaches that learn the number of units to use in a network is to ease the burden of tuning hyperparameters.   One disadvantage of this approach (and maybe any such approach) is that it does not really solve this problem.  The network still has several hyperparameters that implicitly control the number of units that will emerge, including parameters that control how often new units are added and how rapidly weights may decay to zero.  It is not clear whether these hyperparameters will be easier or harder to tune than the ones in standard approaches.  In fairness, the authors do not claim that they have made training easier, but it is a little disappointing that this does not seem to be the case.
The authors do emphasize that they are able to train networks that use fewer units to achieve comparable performance to networks trained parametrically.  This is potentially important, because smaller networks can reduce run-time at testing, and power consumption and memory footprint, which is important on mobile devices in particular.
However, the authors do not compare experimentally to existing approaches that attempt to reduce the size of parametrically trained networks (eg., by pruning trained networks) so it is not clear whether this approach is really competitive with the best current approaches to reducing the size of trained networks.
Another potential disadvantage of the proposed approach is that the same hyperparameters control both the number of units that will appear in the network and the training time.
Therefore, training might potentially be much slower for this approach than for a parametric approach with fixed hyperparameters.
In practice, many parametric approaches require methods like grid search to choose hyperparameters, which can be very slow, but in many other cases experience with similar problems can make the choice of hyperparameters relatively easy.  This means that the cost of grid search is not always paid, but the slowness of the authors’ approach may be endemic.  The authors do not discuss how this issue will scale as much larger networks are trained.  It is a concern that this approach may not be practical for large-scale networks, because training will be very slow.
In general, the experiments are helpful and encouraging, but not comprehensive or totally convincing.  I would want to see experiments on much larger problems before I was convinced that this approach can really be practical or widely useful.
Overall, I found this to be an interesting and clearly written paper that makes a potentially useful point.  The overall vision of building networks that can grow and adapt through life-long learning is inspiring, and this type of work might be needed to realize such a vision.
But the current results remain pretty speculative.
Dear Reviewer,
Thank you for your time and effort in writing this review. Before I respond to each paragraph of the review, I am would love to ask you if there are any specific experiments that you would like to see done. This would greatly help me improve the paper. Thank you very much.
My main response is that most of your criticisms apply to many or all hyperparameter optimization papers published at top ML conferences in recent years and are indeed intrinsic to the task of hyperparameter optimization. I don’t think it makes sense to punish this paper for the difficulties that make hyperparameter optimization a hard problem and, in my opinion, and interesting problem to study in the first place.
On to the paragraphs:
"In practice, this result does not guarantee that the resulting network will not over- or under-fit the training data, but some initial experiments show that this does not seem to be the case."
Guaranteeing that a model does not overfit or underfit data is a very difficult problem in general. I am not aware of any hyperparameter optimization method that achieves this without an expensive validation procedure. Note though that our model can guarantee overfitting by setting lambda to zero and guarantee underfitting by setting lambda to infinity.
Hence it is reasonable to expect that there exists some value in between these two extremes that balances capacity and generalization.
"One potential advantage of approaches that learn the number of units to use in a network is to ease the burden of tuning hyperparameters.   One disadvantage of this approach (and maybe any such approach) is that it does not really solve this problem.  The network still has several hyperparameters that implicitly control the number of units that will emerge, including parameters that control how often new units are added and how rapidly weights may decay to zero.  It is not clear whether these hyperparameters will be easier or harder to tune than the ones in standard approaches.  In fairness, the authors do not claim that they have made training easier, but it is a little disappointing that this does not seem to be the case."
Every single hyperparameter optimization algorithm for neural networks has itself hyperparameters. It lies in the nature of hyperparameters that all algorithms that are not extremely simple have them. Since setting hyperparameters of deep networks is a complex problem, algorithms created to address this problem are themselves complex and thus have hyperparameters. Consider the papers:
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In NIPS, 2016.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In ICML, 2015.
Both of those papers describe extremely complex hyperpareter optimization schemes that introduce many more and more complex hyperameters than our method. If anything, the fact that we add so few hyperparameters compared to other papers is an advantage for us.
The goal of automated hyperparameter optimization is generally to replace "hard hyperparameters" with "easy hyperparameters". Hard hyperparameters are those to which the algorithm is sensitive and which need to be set to different values for different datasets. Easy hyperparameters are those to which the algorithm is not sensitive or hyperparameters for which a single value can be chosen for all datasets or which can be adaptively chosen via a rule of thumb.
The standard methodology for hyperparameter optimization papers is to (i) identify hard hyperparameters, (ii) introduce an algorithm with new hyperparameters that automatically chooses the original hyperparameters and (iii) argue that the new hyperparameters are easy hyperparameters.
For example, the two papers cited above use this methodology and argue that their plethora of newly introduced hyperparameters are easy.
We follow this same 3-step methodology in our paper.
First, we identify hard hyperparameters (size of each layer).
Then, we introduce an algorithm with three additional hyperparameters:
(1) unit addition rate
(2) radial step size
(3) regularization hyperparameter
In our experiments, we choose hyperparameters (1) and (2) uniformly across datasets and in advance. We set the unit addition rate to one and the radial step size to one over fifty.
While this does not prove that those values are good for all datasets, it provides significant evidence that those hyperparameters are indeed easy, at least for small-to-medium datasets.
Hyperparameter (3) is hard.
However, note that we have replaced one hard parameter per layer (size) with one hard parameter for the entire network (lambda).
Secondly, note that even in parametric training the regularization hyperparameter exists, hence it is debatable whether we even added it in the first place.
Thirdly, eliminating the regularization hyperparameter altogether would be tantamount to eliminating the possibility of over- and underfitting which, as I pointed out above, is an unrealistic expectation.
So, we introduce two hyperparameters that we were able to choose "blind" in our experiments (no pre-tuning) and one hyperparameter that is not really new to begin with. The two papers I cited above don’t get anywhere close to this. They simply state that "after experiments, we found a way of setting the new hyperparameters that worked for us". If anything, the criterion of additional hyperparameters is a strength of our paper, not a weakness.
In summary, we very much DO claim to make training easier. Our solution is: instead of tuning layer size and lambda, tune only lambda while setting unit addition rate to one and radial step size to one over fifty.
"The authors do emphasize that they are able to train networks that use fewer units to achieve comparable performance to networks trained parametrically.  This is potentially important, because smaller networks can reduce run-time at testing, and power consumption and memory footprint, which is important on mobile devices in particular.
However, the authors do not compare experimentally to existing approaches that attempt to reduce the size of parametrically trained networks (eg., by pruning trained networks) so it is not clear whether this approach is really competitive with the best current approaches to reducing the size of trained networks."
Based on your review, I get a sense that you see this paper as „pruning but with unit addition“. This paper is NOT comparable to pruning papers, because pruning uses a good network size as prior information to start training, whereas we do not. I cannot overemphasize the importance of this difference.
Pruning papers invariably use the following strategy:
-	Train a network we know works for a given task
-	Remove some edges / neurons that seem unimportant based on rules of thumb that are often as simple as „remove edges with low absolute weights“
Compare this to the recent paper by Springenberg et. al. (see full citation above)
This paper performs hyperparameter optimization for neural networks. To do this, it performs global model-based optimization via a bayesian neural network that is significantly more complicated than even the original network that is being tuned and introduce a ton of new hyperparameters.
So the difference between knowing a good starting network size and not knowing it is the difference between „remove some edges with small weights“ and „run the network hundreds of times guided by a Bayesian neural network“. If you don’t appreciate this difference, it is impossible for you to review our paper fairly. The reason we don’t compare our method to pruning methods is because they address different problems. If we were to compare to pruning, we would have to give pruning a starting network size. If we did, our method, like all hyperparameter optimization methods, could not hope to outperform pruning, or even to achieve comparable performance.
"In practice, many parametric approaches require methods like grid search to choose hyperparameters, which can be very slow, but in many other cases experience with similar problems can make the choice of hyperparameters relatively easy.  This means that the cost of grid search is not always paid,"
We specifically state that our method is most useful when there is no prior knowledge regarding what a good network looks like. This is the case for the vast majority of datasets in the world. There are many more datasets in the world than experiments published in ML conferences. It is hardly a significant weakness that we only cater to this vast majority.
"but the slowness of the authors’ approach may be endemic.  The authors do not discuss how this issue will scale as much larger networks are trained.  It is a concern that this approach may not be practical for large-scale networks, because training will be very slow."
I agree that results on large datasets would make the paper stronger.
However, again, not all datasets in the world are large. A method that is simple, innovative and powerful for small to medium-sized datasets may still be interesting. I don't think that a machine learning method must necessary be applicable to ImageNet to be scientifically interesting or useful in practice, or to warrant publication.
Also, note that our methodology for evaluating our algorithm required around 1000 training runs per dataset. The requirements for expensive validation procedures is also intrinsic to hyperparameter optimization. Consider for example, in addition to Springenberg et al, the following papers:
Jelena Luketina, Mathias Berglund, Klaus Greff, and Raiko Tapani. Scalable gradient-based tuning of continuous regularization hyperparameters. In ICML, 2016.
Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015.
All of these hyperparameter optimization papers were published at top conferences, but did not apply their algorithm to large-scale datasets. All of them used many training runs to validate their methods.
Thanks again,
George
Dear Reviewer 2,
Thank you again for your review. Did you have a chance to look at my response? It would be great to hear your thoughts on it.
Best,
George
The authors are certainly correct that we cannot expect research to eliminate the need for hyperparameters.  It is a perfectly reasonable goal to seek hyperparameters that are more intuitive or easier to set.  The paper does not offer any real evidence that this is the case in the proposed method.  There is no theoretical result on this point.  Experiments are performed on MNIST, which is a small and easy dataset and on two small toy datasets.  It is not clear how these results would generalize to more complex, real-world datasets.
The authors do not address the question of whether their approach is too slow to apply to large-scale problems.  It would be good to know how large are the problems that they can reasonably hope to address, and to see examples of their system run on real-world datasets that are typical of the hardest problems that they expect to be able to handle.  I realize that it may not be practical to run exhaustive experiments on larger datasets.
But the paper would benefit from some discussion of the issues that will arise with such datasets, and some sample runs on them.
I raised the question of comparison to pruning methods because I thought that one potential contribution of this work was to train networks that would be smaller than those created by traditional means.  If this is not a contribution of this work, this comparison is less important.  The contribution of the paper then is more limited than I had initially thought.
“If you don’t appreciate this difference, it is impossible for you to review our paper fairly.”
I will certainly do my best to review your paper fairly.  Let us try to maintain a civil tone to this discussion.
Dear Reviewer,
Thank you for your response. I have just added results on a big dataset (> 1m datapoints). I apologize for the delay, which was caused by various technical issues.
To make things as big and as real-world as possible, I used the biggest dataset on openml.org that seemed suitable (some prior published results + no gross label imbalance + no prior result better than 95%). I ran four-layer nonparametric networks on that dataset, with good results.
Note that while both nonparametric and parametric networks took a lot longer to train on the big dataset, the time it took the nonparametric network to add the necessary units was actually less of a factor overall, precisely because of the increased training time. By design, the time it took to add the units was fairly invariant to dataset size.
Please see the newly added section 4.3 in the paper.
"I will certainly do my best to review your paper fairly.  Let us try to maintain a civil tone to this discussion."
I apologize for any offence caused.
George
I agree with reviewer 2 on the interesting part of the paper. The idea of removing or adding units is definitely an interesting direction, that will make a model grow or shrink along the lines required by the problem and the data, not the user prior knowledge.
The authors offer an interesting theoretical result that proves that under fan out or fan in regularization the optimum of the error function is achieved for finite number of parameters - so the net does not grow indefinitely, until it over-fits perfectly the data.
That reminds me of more traditional approaches such as Lasso or Elastic Net, in which the regularization produces sparse weights. I would  have like more intuition to be given for this theorem. It is a nice result, somewhat expected (at last for me it is intuitive) and I would have liked such intuition to be given some space in the paper.
For example, less discussion of prior work (that is nice too, but not as important as discussing and studying the main result of the paper) could make more room for addressing the theoretical results. Please also see below (point 2) for some suggestions.
I have a few other comments to make:
1. An interesting experiment would be to show that a model such as yours, where the nodes (neurons) are added or removed automatically can outperform a net with the same number of nodes (at the end, after complete learning), in which the size and number of nodes per layer are fixed from the start. This would prove the efficiency of the idea. This is where your method is interesting: do you save nodes that are not needed and replace them with nodes that are needed? Do you optimize performance vs. memory?
I understand that experiments along this line are given in Figure 2, with mixed results. The Figure i must say, is not very clear, but it is possible to interpret under careful inspection.
In some the non-parametric nets are doing better and others are doing worse than the parametric ones. Even in such case i could see the usefulness of the method as it helps discovering the structure.
What i don't fully understand is why they can do better sometimes than the end net which could be trained from scratch: why is the nonparametric version of learning better than the parametric version, when the final net is known in advance? Could you give more insight?
2. Can you better discuss the meaning and implications of Theorem 1. I feel this theorem is just put there with no proper discussion. Beyond the proofs, from the Appendix, what is the key insight of the Theorem? What does it say, in plain English? To me, the conclusion seems almost natural and obvious. Is there some powerful insight?
As i have mentioned previously, i feel this theoretical result deserves more space, with even more experiments to back it up.
For example, can regularizer parameter lambda  be predicted given the data - is there a property in the data that can help guessing the right lambda? My feeling is that lambda is the key factor for determining the final net structure. Is this true?
How much does the structure of the final net depend on the initialization? Do you get different nets if you start from different random weights? How different are they?
What happens when fan in and fan out regularizers are combined? Do you still have the same theoretical result?
I have a few additional questions:
1. Why do you say that adding zero units changes the regularizer value?
For example, does L2 norm change if you add zero values?
2. Zero units are defined as having either the fan in or the fan out weights being zero. I think that what you meant is that both fan in and fan out weights are zero, otherwise you cannot remove the unit and keep the same output f. This should be clarified better I think.
I changed my rating to 7, while hoping that the authors will address my comments above.
Dear Reviewer,
I am dismayed that you believe that our paper only warrants 2 short paragraphs of review.
While our paper isn't perfect, I think it deserves more effort. The review also did not justify the rating you gave the paper. Could you clarify why you chose the rating you did?
Nevertheless, I will look at each paragraph.
"An interesting experiment would be to show that a model such as yours, where
the nodes (neurons) are added or removed automatically can outperform a net with the same numb
er of nodes (at the end, after complete learning), in which the size and number of nodes per layer are fixed from the start. This would prove the efficiency of the idea. This is where your method is interesting: do you save nodes that are not needed and replace them with nodes that are needed? Do you optimize performance vs. memory?"
I am not quite sure what you mean here. If you are asking me to compare the performance of
our method against training networks with a fixed number of neurons throughout the entire training process, we do precisely this in section 4.1 / Figure 2. If you are asking me to compare our method against starting with a large network and then eliminating neurons but not adding them, I would agree that such a comparison is interesting. I guess one could incorporate the l2 regularization hyperparameter into the random search process and allow it to eliminate neurons. I will make sure to include this in the next version as an additional experiment.
"Can you better discuss the meaning and implications of Theorem 1. I feel this theorem is just put there with no proper discussion. Beyond the proofs, from the Appendix, what is the key insight of the Theorem? What does it say, in plain English? To me, the conclusion seems almost natural and obvious. Is there some powerful insight?"
Please refer to the fourth paragraph of section 2, which is:
***In the nonparametric setting, because the existence of a global minimum is not guaranteed, we may be able to reduce the error further and further by using larger and larger networks. This would be problematic, because as networks become better and better with regards to the objective, they would become more and more undesirable in practice.***
The theorem simply states that this does not occur.
While this result is important, it is not spectacular. It is more of a bookkeeping result, a confirmation of a basic property of our algorithm.
Hence, the proof is also in the appendix. If you were looking for some deep insight, I don't think there is any to find.
The theorem may seem "natural and obvious" to you, but the complexity of the proof shows that it clearly isn't. Are you aware of a shorter and simpler proof?
George
Dear author,
I have reviewed your comment and updated my rating. It helped me better understand the results in Figure 2. I still have some questions related to the two points I made. I hope they will help you improve your paper. It is good work, i feel you should do your best to present the value of the contribution, regarding the structure of the network learned and reasons behind its good or less favorable performance (when it is the case), as well as the importance of the regularizer for the final net learned.
In particular, have you tried learning the number of layers also? What happens in practice if you add new layers? Is there a practical way to merge two layers and thus reduce the depth of the net.
For example if a layer gets mostly zero units, it could probably be removed. Does that ever happen in practice?
Thank you!
Dear Reviewer,
Thank you for your review.
"The authors offer an interesting theoretical result that proves that under fan out or fan in regularization the optimum of the error function is achieved for finite number of parameters - so the net does not grow indefinitely, until it over-fits perfectly the data.
That reminds me of more traditional approaches such as Lasso or Elastic Net, in which the regularization produces sparse weights. I would  have like more intuition to be given for this theorem. It is a nice result, somewhat expected (at last for me it is intuitive) and I would have liked such intuition to be given some space in the paper.
For example, less discussion of prior work (that is nice too, but not as important as discussing and studying the main result of the paper) could make more room for addressing the theoretical results."
I do agree that the result is intuitive in the sense that "punishing additional units keeps the number of units in check" is an intuitive statement. I am not entirely sure what additional discussion I should give for the result. The "practical eivdence" for the theorem is that during training, we always observed stability in the number of units when training for a long time. We discuss this in section 4.2 / Figure 3(c).
"What i don't fully understand is why they can do better sometimes than the end net which could be trained from scratch: why is the nonparametric version of learning better than the parametric version, when the final net is known in advance? Could you give more insight?"
Thank you for this comment. The reason that nonparametric training can do better for the same size is that the regularization term ensures that features learnt are meaningful (in the same way Lasso often given better results than vanilla linear regression) while node addition ensures that eliminated nodes are replaced with new nodes that have another chance at learning something meaningful, thus potentially capturing many meaningful features while getting rid of marginal ones. I will add this information to the final paper version. For reasons why nonparametric training might do worse, see my response to reviewer 1.
"As i have mentioned previously, i feel this theoretical result deserves more space, with even more experiments to back it up.
For example, can regularizer parameter lambda  be predicted given the data - is there a property in the data that can help guessing the right lambda? My feeling is that lambda is the key factor for determining the final net structure. Is this true?"
You are right. Lambda determines how many nodes are present in the final network. A large lambda means that only the most meaningful features are kept whereas a small lambda means less meaningful features are kept as well. Ideally, we would pick a value of lambda that leads to neither overfitting or underfitting, a problem that also appears in e.g. Lasso. Preventing over- and underfitting automatically is a hard problem in general and this is future work for us. One popular strategy for this is annealing: Start training with a large value of lambda and gradually reduce it during training. This leads to a succession of solutions that are obtained in rapid succession without restarting training. We'll see how / whether this can be applied to neural networks.
"How much does the structure of the final net depend on the initialization? Do you get different nets if you start from different random weights? How different are they?"
The horizontal and vertical bars in Figure 2 show the range of results over 10 different random initializations.
"What happens when fan in and fan out regularizers are combined? Do you still have the same theoretical result?"
Yes, any linear combination of such regularizers works.
"Why do you say that adding zero units changes the regularizer value?
For example, does L2 norm change if you add zero values?"
Zero units can have a non-zero fan-in if it has a zero fan-out and it can have a non-zero fan-out if it has a zero fan-in.
Hence, removing a zero unit can remove non-zero weights and thus alter the value of Omega.
Similarly, adding a zero unit can add non-zero weights and thus alter the value of Omega.
"Zero units are defined as having either the fan in or the fan out weights being zero. I think that what you meant is that both fan in and fan out weights are zero, otherwise you cannot remove the unit and keep the same output f. This should be clarified better I think."
A zero unit indeed has a zero fan-in OR a zero fan-out, but not necessarily both. Adding or removing zero units does not change the value of f. To see this, consider the feedforward evaluation of a neural network.
If any unit has a zero fan-in, its activation value pre-nonlinearity is always zero. In our paper, we force non-linearities sigma to have the property sigma(0) = 0.
Therefore, if a unit has a zero fan-in, its post-nonlinearity activiation is also zero.
Therefore, it does not contribute to the activations of units in the next layer, even if the fan-out is non-zero.
Hence, removing that unit would not alter the activations in the next layer, and hence it would not alter activations in the layer after that etc. Since f is computed from the activations in the highest layer, f is unaltered as well.
Conversely, if any unit has a zero fan-out, no matter what its activation values, it does not contribute to activations in the layer above, and hence it does not contribute to the activations in the layer above that and hence, removing that unit leaves f unaltered.
Therefore, these zero units can be removed without altering f and, similarly, can be added without altering f.
"In particular, have you tried learning the number of layers also?"
We plan to work on this.
"Is there a practical way to merge two layers and thus reduce the depth of the net.
For example if a layer gets mostly zero units, it could probably be removed."
We haven't worked on adding / removing layers yet.
Of course, there is no way to eliminate a layer from a neural network without altering f.
Also, we know that deep networks often work better than shallow networks. In a very deep architecture such as ResNet, it can happen that successive layers have limited interaction and are thus potentially parallelizable, i.e. reducable to fewer layers.
For more information on this see:
Residual Networks Behave Like Ensembles of Relatively Shallow Networks, Veit et al, NIPS 2016
My tentative intuition is that the fan-in regularizer would actually be fairly good at performing this reduction, though I don't think I can really argue this yet. We'll see once experiments come in.
I'm sorry I was not able to give complete answers to many of your questions. Thank you very much for asking them though, they will guide my research going forward.
George
This paper proposes a nonparametric neural network model, which automatically learns the size of the model during the training process. The key idea is to randomly add zero units and use sparse regularizer to automatically null out the weights that are irrelevant. The idea sounds to be a random search approach over discrete space with the help of sparse regularization to eliminate useless units. This is an important problem and the paper gives interesting results. My main comments are listed below:
What is the additional computation complexity of the algorithm? The decomposition of each fan-in weights into a parallel component and an orthogonal component and the transformation into radial-angular coordinates may require a lot of extra computation time. The authors may need to discuss the extra amount of operations relative to the parametric neural network.
Furthermore, it would be useful to show some running time experiments.
It is observed that nonparametric networks return small networks on the convex dataset so that it is inferior to parametric networks. Any insight on this?
Dear Reviewer,
Thank you for the review.
The additional complexity of AdaRad and unit addition / deletion over parametric SGD is cp per minibatch, where c is a constant and p is the number of parameters, assuming a dense weight matrix. The relevance of this overhead depends thus on the size of the minibatch. I will add this comment and some runtimes to the final version of the paper.
"It is observed that nonparametric networks return small networks on the convex dataset so that it is inferior to parametric networks. Any insight on this?"
We're working on this ATM ... My current guess would be that we need to figure out how to train new units even faster, i.e. 50 iterations is still not enough to initialize new units. (Of course, I'm reluctant to reduce the radial step size to keep the overhead of useless units during training bearable.)
Other things we are looking at are: bad co-adaptations of units early in training (also referred to as "stuck in bad local minima") which we try to combat with noise / regularization or "shocking" the parameters from time to time as in "DSD: Dense-Sparse-Dense Training for Deep Neural Networks", submitted to this very conference.
Thanks,
George
Dear Reviewer 2,
Thank you again for your review. Did you have a chance to look at my response? It would be great to hear your thoughts on it.
Best,
George
Dear all,
Below are some more responses to two points raised. Both responses are reflected in a new version of the paper I just uploaded.
The following has changed in the paper:
- section 3.1 (self-similar nonlinearities) is new
- the final paragraph of section 3 is new
- section 7.2 (proof of new proposition) is new
The paper is now longer than 8 pages. Having looked at many other submitted papers, it appears that the 8 page requirement is not very serious. If the area chair would prefer an 8 page paper, I can move some more stuff to the appendix.
** Reducing the number of hyperparameters **
Two questions that came up throughout the reviews is whether our method reduces the number of hyperparameters and whether there is an automatic way to set lambda.
One advantage of nonparametric networks is that instead of having one hyperparameter per layer (size) there is one hyperparameter for the entire network (lambda) that controls size. It is worth pointing out that this reduction in complexity is not arbitrary.
In fact, one can prove that in a ReLU network, one regularization parameter lambda captures all the complexity of having one regularization parameter per layer because we could replace all of these regularization parameters with their geometric mean without changing the objective. (See the newly included-in-the-paper proposition 1.) Hence, nonparametric networks apportion regularization to each layer automatically.
While we don't (yet) have a great way of efficiently picking the single remaining lambda, this reduction in complexity certainly contributes to reducing hyperparameter complexity.
** Computational cost of AdaRad (in response to reviewer 1) **
(From the paper:) Using AdaRad over SGD incurs additional computational cost.
However, that cost scales more gracefully than the cost of, for example, RMSprop. AdaRad normalizes each fan-in instead of each individual weight, so many of its operations scale only with the number of units and not with the number of weights in the network. In Table \ref{costTable}, we compare the costs of SGD, AdaRad and RMSprop.
Further, RMSprop has a larger memory footprint than AdaRad. It requires a cache of size equal to the number of weights, whereas AdaRad only requires 2 caches of size equal to the number of neurons.
Costs (per minibatch and weight)
SGD, no $\ell_2$ shrinkage: 1 multiplication
SGD with $\ell_2$ shrinkage: 3 multiplications
AdaRad, no $\ell_2$ shrinkage: 4 multiplications
AdaRad with $\ell_2$ shrinkage: 4 multiplications
RMSprop, no $\ell_2$ shrinkage: 4 multiplications, 1 division, 1 square root
RMSprop with $\ell_2$ shrinkage: 6 multiplications, 1 division, 1 square root
Best,
George
I just added a new version of the paper with experiments on a large dataset (> 1m datapoints, http://www.openml.org/d/354) using a 4-hidden-layer network, in response to Reviewer 2. I apologize for the lateness of this change, which was mainly due to my current lack of access to appropriate computational resources. I added new sections 4.3 and 7.4.2 and left the rest of the paper essentially unchanged, though I did fix any typos or language impreciseness I spotted throughout the paper.
Note that while both nonparametric and parametric networks took a lot longer to train on the big dataset, the time it took the nonparametric network to add the necessary units was actually less of a factor overall, precisely because of the increased training time. By design, the time it took to add the units was fairly invariant to dataset size.

Meta-review:
role: PC
data: comment: The paper presents a clean framework for optimizing for the network size during the training cycle. While the complexity of each iteration is increased, they argue that overall, the cost is significantly reduced since we do not need to train networks of varying sizes and cross-validate across them. The reviewers recommend acceptance of the paper and I am in agreement with them.

Decision: Accept (Poster)