Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors a new type of polynomial neural networks LPNN that can have an arbitrary polynomial order. The network has a feedforward structure which provides a good control of its polynomial order. Empirical study shows that deep LPNN models achieve good performances in regression and classification tasks. In general, the paper is clearly written by addressing an interesting problem but I still have several concerns. 
1.	The authors are expected to analyze the time complexity in terms of both theoretical and experimental analysis since the time cost the one of the major limitation of PNN. 
2.	The experimental section is rather weak since the authors only report results in some simple data. The authors are expected to report more complex tasks to show its effectivenss. 
3.	It would be interesting if the authors could show whether the algorithm can integrate with other structure such as convolutional operator to cope with the image classification, and other relevant tasks.   

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes Ladder Polynomial Neural Networks (LPNNs) that use a new type of activation primitive -- a product activation -- in a feed-forward architecture. Unlike other polynomial architectures that grow in the order exponentially with network depth, the proposed approach gives explicit control over the order and smoothness of the network output and enables training with standard techniques.	
				
The proposed architecture is closely related to a decomposition of a k’th order multivariate polynomial function
[T, x^{\otimes k}] = \lambda^\top (A x \odot A x \odot …  \odot A x)	=  \lambda^\top (A x)^{\odot k}
where T is a symmetric tensor of polynomial coefficients and [\cdot,\cdot] denotes contraction. This is a shallow (one layer architecture) and sometimes referred as a Waring decomposition.

In this paper, the authors propose a specific chain factorization of the polynomial (Eq 5 in the paper), where they write the factors recursively, that they name as a ladder polynomial neural network. 

h^\ell = (W_\ell h^{\ell-1} \odot V^{\ell} x)

The ladder architecture is very closely related to tensor trains (https://epubs.siam.org/doi/10.1137/090752286). I found it surprising and somewhat alarming that this literature is not being cited as these methods are also quite well known in deep learning.

I like the smoothness analysis of section 3.1 -- the proof is quite easy to follow and direct. I would be quite surprised if this result would not be known in the literature in some other form but I don’t recall seeing it. On the other hand it seems to be inevitably very loose for a deep ladder network unless the network models the zero function. It would have been a valuable addition to the experimental section, if this bound would have been illustrated numerically on synthetic examples.

In 3.2, The authors say that the objective is multiconvex -- I would argue that it is multilinear (apart from the regularization term, that is later introduces). The observation in 3.3, that batch-normalization or dropout can be used for this model is perhaps tangential to the main argument. These is investigated in the experimental section but I don’t see a clear conclusion. The section in 3.4 must include links to tensor decompositions beyond factorization machines.
		
Overall, I think the paper has some merit and could be interesting for some readers, despite the fact that the contribution is not very original and the treatment could be improved in many ways. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work introduces a new polynomial feed-forward neural network called Ladder Polynomial Neural Network (LPNN). Theoretical results show that LPNNs generalize vanilla PNNs and FMs. In the experimental analyses, LPNNs perform similar to the vanilla FMs and PNNs, as well.

- In the statement “V has a shape of (d, d0) if u has d entries”, what do you mean by shape, more precisely?

- x is a feature vector fed to a neural network as an input. I assume that it is given in the input layer l=0, according to the notation. Then, should x^l be used instead of x in equations (4), (5), (6), (9), and the other corresponding statements? Otherwise, is V^l applied to the input vector x at each layer l?

- What do \| \|^2, \| \|^l, \| \|^{l+1} denote?

- How do you define the norm \| \| for matrices, more precisely?

- Please provide standard deviation/variance of classification error of different models in Table 2.

- Please clarify novelty and superiority of the proposed LPNNs compared to the vanilla and state-of-the-art methods in theory and practice. For this purpose, I suggest to further analyze and compare convergence and generalization properties of LPNNs with the sota in theory and practice.

After the rebuttal:

Authors responded some of my questions. However, I still consider that the contribution of the paper is limited, and should be improved for a clear acceptance. Therefore, I keep my rating.
review_title: Official Blind Review #3

Meta-review: This paper proposes a new type of Polynomial NN called Ladder Polynomial NN (LPNN) which is easy to train with general optimization algorithms and can be combined with techniques like batch normalization and dropout.  Experiments show it works better than FMs with simple classification and regression tasks, but no experiments are done in more complex tasks. All reviewers agree the paper addresses an interesting question and makes some progress but the contribution is limited and there are still many ways to improve.
Decision: Reject