Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes how to use a tensor factorization method called Tensor Train for modeling the interactions between features for supervised classification tasks. Tensor Train approximates tensors of any dimensions using low rank products of matrices. The rank is used as a parameter for controlling the complexity of the approximation. Experiments are performed on different datasets for binary classification problems.

The core of the paper consists in demonstrating how the TT formalism developed by one of the authors could be adapted for modeling interactions between features. Another contribution is a gradient algorithm that exploits the geometrical structure of the factorization. These ideas are probably new in Machine learning. The algorithm itself is of reasonable complexity for the inference and could probably be adapted to large size problems although this is not the case for the experiments here.

The experimental section is not well structured, it is incomplete and could be improved. We miss a description of the datasets characteristics. The performance on the different datasets are not provided. Each dataset has been used for illustrating one aspect of the model, but you could also provide classification performance and a comparison with baselines for all the experiments. The experiments on the 2 UCI datasets show optimization performance on the training set, you could provide the same curves on test sets to show how the algorithm generalizes. The comparison to other approaches (section 8.4) is only performed on artificial data, which are designed with interacting features and are not representative of diverse situations. The same holds for the role of Dropout. The comparison on the Movielens dataset is incomplete. Besides, all the tests are performed on small size problems.

Overall, there are original contributions which could be worth a publication. The experiments are incomplete and not conclusive. A more detailed comparison with competing methods, like Factorization Machines, could also improve the paper. 

review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents an application of a tensor factorization to linear models, which allows to consider higher-order interactions between variables in classification (and regression) problems, and that maintains computational feasibility, being linear in the dimension. The factorization employed is based on the TT format, first proposed by Oseledests (2011). The authors also propose the adoption of a Riemannian optimization scheme to explicit consider the geometry of the tensor manifold, and thus speed up convergence.

The paper in general is well written, it presents an interesting application of the TT tensor format for linear models (together with an application of Riemannian optimization), which in my opinion is quite interesting since it has a wide range of possible applications in different algorithms in machine learning.

On the other side, I have some concerns are about the experimental part, which I consider not at the level of the rest of the paper, for instance in terms of number of experiments on real datasets, role of dropout in real datasets, comparison with other algorithms on real datasets. Moreover the authors do not take into account explicitly the problem of the choice of the rank to be used in the experiments. In general the experimental section seems a collection of preliminary experiments where different aspects have been tested by not in a organic way.

I think the paper is close to a weak acceptance / weak rejection, I don't rate it as a full acceptance paper, mainly due to the non-satisfactory experiment setting. In case of extra experiments confirming the goodness of the approach, I believe the paper could have much better scores.

Some minor comments:
-formula 2: Obvious comment: learning the parameters of the model in (1) can be done as in (2), but also in other ways, depending on the approach you are using.
-the fact that the rank is bounded by 2r, before formula 9, is explained in Lubich et al., 2015?
-after formula 10: why the N projections in total they cost O(dr^2(r+N)), it should be O(Ndr^2(r+1)), no? since each of the elements of the summation has rank 1, and the cost for each of them is O(dr^2(r+TT_rank(Z)^2)), where TT-rank(Z)=1. Am I wrong?
-section 6.2: can you explain why the random initialization freezes the convergence? This seems interesting but not motivated. Any guess?
-section 6.3: you adopt dropout: can you comment in particular on the advantages it gives in the context of the exponential machines? did you use it on real datasets?
-how do you choose r_0 in you experiments? with a validation set?
-in section 7: why you don't have x_1 x_2 among the variables?
-section 8: there is a typo in "experiments"
-section 8.1: "We simplicity, we binarized" I think there's a problem with the English language in this sentence
-section 8.3: "we report that dropout helps".. this is quite general statement, only tested on a synthetic dataset
-section 8.5: can you provide more results for this dataset, for instance in terms of training and inference time? or test wrt other algorithms?
review_title: Nice idea, but experiments are very preliminary

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a polynomial linear model for supervised classification tasks. The model is based on a combination of the Tensor Train (TT) tensor decomposition method and a form of stochastic Riemannian  optimization. A few empirical experiments are performed that demonstrate the good performance of the proposed model relative to appropriate baselines.

From a theoretical standpoint, I think the approach is interesting and elegant. The main machinery underlying this work are the TT decomposition and the geometric structure of the manifold of tensors with fixed TT-rank, which have been established in prior work. The novelty of this paper is in the combination of this machinery to form an efficient polynomial linear model. As such, I would have hoped that the paper mainly focused on the efficacy of this combination and how it is superior to obvious alternatives. For example, I would have really appreciated seeing how FMs performed when optimized over the manifold of positive definite matrices, as another reviewer mentioned. Instead, there is a bit too much effort devoted to explaining prior work.

I think the empirical analysis could be substantially improved. I am particularly puzzled by the significant performance boost obtained from initializing with the ordinary logistic regression solution. I would have liked some further analysis of this effect, especially whether or not it is possible to obtain a similar performance boost with other models. Regarding the synthetic data, I think an important baseline would be against a vanilla feed forward neural network, which would help readers understand how complicated the interactions are and how difficult the dataset is to model. I agree with the previous reviewer regarding a variety of other possible improvements to the experimental section.

A few typos: 'Bernoulli distrbution', 'reproduce the experiemnts', 'generilize better'.

Overall, I am on the fence regarding this paper. The main idea is quite good, but insufficient attention was devoted to analyzing the aspects of the model that make it interesting and novel.
review_title: Interesting idea, analysis could be improved

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use the tensor train (TT) decomposition to represent the full polynomial linear model. The TT form can reduce the computation complexity in both of inference and model training. A stochastic gradient over a Riemann Manifold has been proposed to solve the TT based formulation. The empirical experiments validate the proposed method.

The proposed approach is very interesting and novel for me. I would like to vote acceptance on this paper. My only suggestion is to include the computational complexity per iteration.

review_title: A good paper 

final_score: rating: null
review_content: Is the tensor decomposition in figure 1 Tucker decomposition or Paraface decomposition?
review_title: tensor decomposition

Meta-review: Modeling nonlinear interactions between variables by imposing Tensor-train structure on parameter matrices is certainly an elegant and novel idea. All reviewers acknowledge this to be the case. But they are also in agreement that the experimental section of this paper is somewhat weak relative to the rest of the paper, making this paper borderline. A revised version of this paper that takes reviewer feedback into account is invited to the workshop track.
Decision: Invite to Workshop Track