Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes an approach to learning the non-linear activation function in deep neural nets.  This is achieved by representing the activation function in a basis of non-linear functions and learning the coefficients.  Authors use Fourier basis in the paper.  A theoretical analysis of the proposed approach is also presented, using algorithmic stability arguments, to demonstrate good generalization behavior (vanishing generalization error with large data sets) of networks with learnt non-linearities.

The main question I have about this paper is that writing a non-linear activation function as a linear or affine combination of other non-linear basis functions is equivalent to making a larger network whose nodes have the basis functions as non-linearities and whose weights have certain constraints on them.  Thus, the value of the proposed approach of learning non-linearities over optimizing network capacity for a given task (with fixed non-linearities) is not clear to me.  Or could it be argued that the constrained implied by learnt non-linearity approach are somehow good thing to do?

Another question - In the two stage training process for CNNs, when ReLU activation is replaced by NPFC(L,T), is the NPFC(L,T) activation initialized to approximate ReLU, or is it initialized using random coefficients?

Few minor corrections/questions:
- Pg 2. “ … the interval [-L+T, L+T] …” should be “ … the interval [-L+T, L-T] … “ ?
- Pg 2., Equation for f(x), should it be “ (-L+T) i \pi x / L “ in both sin and cos terms, or without “ x “ ?
- Theorem 4.2 “ … some algorithm \eps-uniformly stable …” remove the word “algorithm”
- Theorem 4.5.  SGM undefined
review_title: null

final_score: rating: null
review_content: This is perhaps more of a discussion than a question.  It seems like writing a non-linearity as a linear or affine combination of a set of non-linear basis functions would be equivalent to a network of larger capacity with nodes with those basis functions as non-linearities and constraints on the weight matrices.  Thus could one argue that the gains observed in the paper are perhaps coming from this capacity increase?  And if that’s the case, this may be more of a choosing the appropriate capacity question (with any fixed non-linearity) and less so of learning non-linearities?  Would greatly appreciate your thoughts on this.
review_title: learnt non-linearities vs network capacity

final_score: rating: 7: Good paper, accept
review_content: This paper provides a principled framework for nonparametrically learning activation
functions in deep neural networks. A theoretical justification for authors' choice of 
nonparametric activation functions is given. 
Theoretical results are satisfactory but I particularly like the experimental setup
where their methods are tested on image
recognition datasets and achieve up to a 15% relative increase in test performance
compared to the baseline.
Well-written paper and novel theoretical techniques. 
The intuition behind the proof of Theorem 4.7 can be given in a little bit more clear way in the main body of the paper, but the
Appendix clarifies everything.

review_title: strong submission, interesting from both theoretical and experimental point of view

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper introduces a parametric class for non linearities used in neural networks. The paper suggests two stage optimization to learn the weights of the network, and the non linearity weights.

significance:

The paper introduces a nice idea, and present nice experimental results. however  I find the theoretical analysis not very informative, and  distractive from the main central idea of the paper. 
 
A more thorough experimentation with the idea using different basis and comparing it to wider networks (equivalent to the number of cosine basis used in the leaned one ) would help more supporting results in the paper. 


Comments: 

- Are the weights of the non -linearity learned shared across all units in all layers ? or each unit has it is own non linearity?

- If all weights are tied across units and layers. One question that would be interesting to study , if there is an optimal non linearity. 

- How different is the non linearity learned if the hidden units are normalized or un-normalized.  In other words how does the non linearity change if you use or don't use batch normalization? 

- Does normalization affect  the conclusion that polynomial basis fail? 






review_title: Good paper, room for more  empirical study

final_score: rating: null
review_content: Can you extend Theorem 4.7 for settings other than SGD with a max-norm constraint ?
review_title: Theorem 4.7

final_score: rating: null
review_content: The paper suggests a two stage learning of weights and non linearities in neural networks.

- the network is trained with RELU , then it is initialized with tanh Fourier series coefficients. this seems inconsistent. why not train with tanh first? 

- It is not intuitively clear what is the effect of learning the non linearity. I suggest to do a 2D data of 2 classes (a ring data for instance ) and to train a a 2 layer network (Linear -> Relu -> linear-> softmax), and then one with a learned non linearity in the fourier basis , and another with tanh ,and plot the decision function surface  to see how it changes with the non linearity choice. 

 
-   Seems the non linearity is not moving too much from a tanh. maybe there is a capacity problem in the Fourier expansion. Have you tried using an over-complete basis expansion and regularize with l_1 the coefficient a_i? Maybe this will help also with other basis such polynomials that did not work?


review_title: Effect of non linearity and basis expansion choice

Meta-review: The authors propose a nonparametric regression approach to learn the activation functions in deep neural networks. The proposed theoretical analysis, based on stability arguments, is quite interesting. Experiments on MNIST and CIFAR-10 illustrate the potential of the approach. 
 
 Reviewers were somewhat positive, but preliminary empirical evidence on small datasets makes this contribution better suited for the workshop track.
Decision: Invite to Workshop Track