Task: Language Identification
Input (Spanish, English): This paper describes an approach to learning the non-linear activation function in deep neural nets.  This is achieved by representing the activation function in a basis of non-linear functions and learning the coefficients.  Authors use Fourier basis in the paper.  A theoretical analysis of the proposed approach is also presented, using algorithmic stability arguments, to demonstrate good generalization behavior (vanishing generalization error with large data sets) of networks with learnt non-linearities.  The main question I have about this paper is that writing a non-linear activation function as a linear or affine combination of other non-linear basis functions is equivalent to making a larger network whose nodes have the basis functions as non-linearities and whose weights have certain constraints on them.  Thus, the value of the proposed approach of learning non-linearities over optimizing network capacity for a given task (with fixed non-linearities) is not clear to me.  Or could it be argued that the constrained implied by learnt non-linearity approach are somehow good thing to do?  Another question - In the two stage training process for CNNs, when ReLU activation is replaced by NPFC(L,T), is the NPFC(L,T) activation initialized to approximate ReLU, or is it initialized using random coefficients?  Few minor corrections/questions: - Pg 2. “ … the interval [-L+T, L+T] …” should be “ … the interval [-L+T, L-T] … “ ? - Pg 2., Equation for f(x), should it be “ (-L+T) i \pi x / L “ in both sin and cos terms, or without “ x “ ? - Theorem 4.2 “ … some algorithm \eps-uniformly stable …” remove the word “algorithm” - Theorem 4.5.  SGM undefined
Output (English): en