Title: Exponential Machines
Authors: Alexander Novikov
Mikhail Trofimov
Ivan Oseledets
Venue: ICLR 2017
Keywords: Supervised Learning
Optimization

Abstract: Modeling interactions between features improves the performance of machine learning solutions in many domains (e.g. recommender systems or sentiment analysis). In this paper, we introduce Exponential Machines (ExM), a predictor that models all interactions of every order. The key idea is to represent an exponentially large tensor of parameters in a factorized format called Tensor Train (TT). The Tensor Train format regularizes the model and lets you control the number of underlying parameters. To train the model, we develop a stochastic Riemannian optimization procedure, which allows us to fit tensors with 2^160 entries. We show that the model achieves state-of-the-art performance on synthetic data with high-order interactions and that it works on par with high-order factorization machines on a recommender system dataset MovieLens 100K.

TL;DR: A supervised machine learning algorithm with a polynomial decision function (like SVM with a polynomial kernel) that models exponentially many polynomial terms by factorizing the tensor of the parameters.

Review discussion:
Is the tensor decomposition in figure 1 Tucker decomposition or Paraface decomposition?
Neither of both; it is a tensor-train decomposition.  You can read more about it here: http://epubs.siam.org/doi/abs/10.1137/090752286
Shortly, it can be computed using SVD (PARAFAC can not), and does not have intrinsic curse of dimensionality (as Tucker) for really high dimensions.
Yesterday I updated the pdf. The only thing that changed is that I added a paragraph at the end of the related works section to credit the papers suggested by the comment below.
The paper describes how to use a tensor factorization method called Tensor Train for modeling the interactions between features for supervised classification tasks. Tensor Train approximates tensors of any dimensions using low rank products of matrices. The rank is used as a parameter for controlling the complexity of the approximation. Experiments are performed on different datasets for binary classification problems.
The core of the paper consists in demonstrating how the TT formalism developed by one of the authors could be adapted for modeling interactions between features.
Another contribution is a gradient algorithm that exploits the geometrical structure of the factorization. These ideas are probably new in Machine learning. The algorithm itself is of reasonable complexity for the inference and could probably be adapted to large size problems although this is not the case for the experiments here.
The experimental section is not well structured, it is incomplete and could be improved. We miss a description of the datasets characteristics. The performance on the different datasets are not provided. Each dataset has been used for illustrating one aspect of the model, but you could also provide classification performance and a comparison with baselines for all the experiments. The experiments on the 2 UCI datasets show optimization performance on the training set, you could provide the same curves on test sets to show how the algorithm generalizes. The comparison to other approaches (section 8.4) is only performed on artificial data, which are designed with interacting features and are not representative of diverse situations. The same holds for the role of Dropout. The comparison on the Movielens dataset is incomplete.
Besides, all the tests are performed on small size problems.
Overall, there are original contributions which could be worth a publication. The experiments are incomplete and not conclusive. A more detailed comparison with competing methods, like Factorization Machines, could also improve the paper.
Thank you for your time and thorough evaluation of our work.
Note that it’s fairly easy to make a fast inference algorithm for sparse data by building a segment tree on the TT-cores (but we believe it is out of the scope of this already pretty packed paper) and one can readily adapt the SGD learning algorithm to it.
However, we don’t know if it’s possible to adapt the Riemannian approach to the sparse setting.
We agree with your (and some other reviewer’s) criticism of the experimental evaluation and significantly revised the experiments section.
1) We added a more detailed description of the datasets and preprocessing steps
2) We added the test set performance for all the datasets
3) We did our best to evaluate each of the characteristics of the proposed methods on each of the datasets.
4) We found that after a slight change to the learning procedure dropout becomes unnecessary, so we decided to remove it whatsoever from all the experiments (see the description of the updates made to the paper).
This paper proposes to use the tensor train (TT) decomposition to represent the full polynomial linear model. The TT form can reduce the computation complexity in both of inference and model training. A stochastic gradient over a Riemann Manifold has been proposed to solve the TT based formulation. The empirical experiments validate the proposed method.
The proposed approach is very interesting and novel for me. I would like to vote acceptance on this paper. My only suggestion is to include the computational complexity per iteration.
Thank you for your review, I’m glad you liked our work.
The computational complexity is indeed important, we updated the paper to add it (it’s O(d r^2 (r + M)) per step of the Riemannian optimization where d is the number of features, r is the TT-rank, and M is the size of the mini-batch).
This paper introduces a polynomial linear model for supervised classification tasks. The model is based on a combination of the Tensor Train (TT) tensor decomposition method and a form of stochastic Riemannian  optimization. A few empirical experiments are performed that demonstrate the good performance of the proposed model relative to appropriate baselines.
From a theoretical standpoint, I think the approach is interesting and elegant. The main machinery underlying this work are the TT decomposition and the geometric structure of the manifold of tensors with fixed TT-rank, which have been established in prior work. The novelty of this paper is in the combination of this machinery to form an efficient polynomial linear model.
As such, I would have hoped that the paper mainly focused on the efficacy of this combination and how it is superior to obvious alternatives.
For example, I would have really appreciated seeing how FMs performed when optimized over the manifold of positive definite matrices, as another reviewer mentioned.
Instead, there is a bit too much effort devoted to explaining prior work.
I think the empirical analysis could be substantially improved. I am particularly puzzled by the significant performance boost obtained from initializing with the ordinary logistic regression solution. I would have liked some further analysis of this effect, especially whether or not it is possible to obtain a similar performance boost with other models.
Regarding the synthetic data, I think an important baseline would be against a vanilla feed forward neural network, which would help readers understand how complicated the interactions are and how difficult the dataset is to model. I agree with the previous reviewer regarding a variety of other possible improvements to the experimental section.
A few typos: 'Bernoulli distrbution', 'reproduce the experiemnts', 'generilize better'.
Overall, I am on the fence regarding this paper. The main idea is quite good, but insufficient attention was devoted to analyzing the aspects of the model that make it interesting and novel.
We would like to thank the reviewer for a thorough review and valuable suggestions.
Regarding the comparison with FMs optimized over the manifold of positive definite matrices, we believe this comparison is out of the scope of our work since we mainly aimed at modeling high-order interactions, while the Riemannian optimization is possible only for the second-order FMs.
You are very right to question the random initialization results. We further investigated this effect and found out that it is not that important to initialize the learning from the linear solution built for the dataset at hand, but we may as well generate a random linear model and initialize the learning from it.
So there is nothing special about the initialization we had chosen, but it’s important to initialize the model from a proper distribution.
Two reasons that may cause this effect:
a) The Tensor Train is a product of a large number of factors (especially in the UCI experiment with 160 TT-cores), and it may raise the problem of vanishing and exploding gradients. It may be interesting to explore how our proposed initialization (a random linear model) is connected to the Xavier initialization from the Neural Networks community.
b) If we initialize the model to put too much weight on a large number of high-order interactions, it may be hard to recover by gradient optimization: each gradient step would largely affect the high-order interactions and slightly affect the low-order interactions.
On the other hand, it seems important to firstly approximate the linear part of the model, and only then try to model the residuals with interactions of a higher order.
As you suggested, we tried a vanilla feedforward neural network baseline on the synthetic dataset and ran into severe overfitting: it’s easy to optimize the training loss to zero, while the test set performance was 0.5 AUC in all our experiments.
> A few typos: 'Bernoulli distrbution', 'reproduce the experiemnts', 'generilize better'.
Thank you for pointing out the typos, we fixed them in the new version of the paper.
The paper presents an application of a tensor factorization to linear models, which allows to consider higher-order interactions between variables in classification (and regression) problems, and that maintains computational feasibility, being linear in the dimension. The factorization employed is based on the TT format, first proposed by Oseledests (2011). The authors also propose the adoption of a Riemannian optimization scheme to explicit consider the geometry of the tensor manifold, and thus speed up convergence.
The paper in general is well written, it presents an interesting application of the TT tensor format for linear models (together with an application of Riemannian optimization), which in my opinion is quite interesting since it has a wide range of possible applications in different algorithms in machine learning.
On the other side, I have some concerns are about the experimental part, which I consider not at the level of the rest of the paper, for instance in terms of number of experiments on real datasets, role of dropout in real datasets, comparison with other algorithms on real datasets.
Moreover the authors do not take into account explicitly the problem of the choice of the rank to be used in the experiments. In general the experimental section seems a collection of preliminary experiments where different aspects have been tested by not in a organic way.
I think the paper is close to a weak acceptance / weak rejection, I don't rate it as a full acceptance paper, mainly due to the non-satisfactory experiment setting.
In case of extra experiments confirming the goodness of the approach, I believe the paper could have much better scores.
Some minor comments:
-formula 2: Obvious comment: learning the parameters of the model in (1) can be done as in (2), but also in other ways, depending on the approach you are using.
-the fact that the rank is bounded by 2r, before formula 9, is explained in Lubich et al., 2015?
-after formula 10: why the N projections in total they cost O(dr^2(r+N)), it should be O(Ndr^2(r+1)), no? since each of the elements of the summation has rank 1, and the cost for each of them is O(dr^2(r+TT_rank(Z)^2)), where TT-rank(Z)=1. Am I wrong?
-section 6.2: can you explain why the random initialization freezes the convergence? This seems interesting but not motivated. Any guess?
-section 6.3: you adopt dropout: can you comment in particular on the advantages it gives in the context of the exponential machines? did you use it on real datasets?
-how do you choose r_0 in you experiments? with a validation set?
-in section 7: why you don't have x_1 x_2 among the variables?
-section 8: there is a typo in "experiments"
-section 8.1: "We simplicity, we binarized" I think there's a problem with the English language in this sentence
-section 8.3: "we report that dropout helps".. this is quite general statement, only tested on a synthetic dataset
-section 8.5: can you provide more results for this dataset, for instance in terms of training and inference time? or test wrt other algorithms?
Thank you for this very thorough review and for the numerous comments, suggestions, and insightful questions. We’ve updated the paper to address them.
Regarding the experimental evaluation, we added more baselines, restructured the text and added experiments to assess different aspects of the model on all the datasets considered, and investigated the influence of the rank on the proposed model performance.
> -formula 2: Obvious comment: learning the parameters of the model in (1) can be done as in (2), but also in other ways, depending on the approach you are using.
Thanks, we clarified this part.
> -the fact that the rank is bounded by 2r, before formula 9, is explained in Lubich et al., 2015?
That’s right, we clarified this in the new version.
>-after formula 10: why the N projections in total they cost O(dr^2(r+N)), it should be O(Ndr^2(r+1)), no? since each of the elements of the summation has rank 1, and the cost for each of them is O(dr^2(r+TT_rank(Z)^2)), where TT-rank(Z)=1. Am I wrong?
Good point, I haven’t explained it properly in the original submission (fixed in the new version).
The reason why it’s O(d r^2 (r+N)) is because the projection consists of two parts: 1) preprocessing, which cost O(d r^3); 2) and the actual projection, which cost O(d r^2 TT_rank(Z)^2).
Since we are projecting all the gradients on the same tangent space defined by W, we have to do the preprocessing step just once.
>-section 6.2: can you explain why the random initialization freezes the convergence? This seems interesting but not motivated. Any guess?
That’s an excellent question, and we believe that we found an answer (and added it to the new version of the paper). We observed that it is not that important to initialize the learning from the linear solution built for the dataset at hand, but we may as well generate a random linear model and initialize the learning from it.
So there is nothing special about the initialization we had chosen, but it’s important to initialize the model from a proper distribution.
Two reasons that may cause this effect:
a) The Tensor Train is a product of a large number of factors (especially in the UCI experiment with 160 TT-cores), and it may raise the problem of vanishing and exploding gradients. It may be interesting to explore how the initialization from a random linear model is connected to the Xavier initialization from the Neural Networks community.
b) If we initialize the model to put too much weight on a large number of high-order interactions, it may be hard to recover by gradient optimization: each gradient step would largely affect the high-order interactions and slightly affect the low-order interactions.
On the other hand, it seems important to firstly approximate the linear part of the model, and only then try to model the residuals with interactions of a higher order.
> -section 6.3: you adopt dropout: can you comment in particular on the advantages it gives in the context of the exponential machines? did you use it on real datasets?
> -section 8.3: "we report that dropout helps".. this is quite general statement, only tested on a synthetic dataset
We found that after a slight change to the learning procedure dropout becomes unnecessary, so we decided to remove it whatsoever from all the experiments (see the description of the updates made to the paper).
> -how do you choose r_0 in you experiments? with a validation set?
We tried a few reasonable options by hand and chose the best one according to the test error.
I agree, using the validation set would be fairer, but the overfitting is unlikely to occur in this situations because of the small number of variants we tried and because of the robustness of the model to the choice of the TT-rank (see the experiment we added to investigate this).
> -in section 7: why you don't have x_1 x_2 among the variables?
Actually, we have all terms from the previous model (\hat{y}(x)) plus some new terms which involve logarithms. Thank you for spotting this clarity issue, we made this point more clear in the new version.
> -section 8: there is a typo in "experiments"
> -section 8.1: "We simplicity, we binarized" I think there's a problem with the English language in this sentence
Thanks, we fixed these typos.
> -section 8.5: can you provide more results for this dataset, for instance in terms of training and inference time? or test wrt other algorithms?
We added the training and inference time and also the performance of the logistic regression.
We would like to thank all the reviewers and commenters for their time, feedback, and ultimately for making our paper better.
We updated the paper to address the questions raised in the reviews.
We found out that the backtracking algorithm is beneficial only for the small-scale setting when the mini-batch size is of the order of the full dataset (e.g. UCI experiments).
Otherwise, backtracking is (locally) tuning the learning rate too well and overfits to each mini-batch, which stagnates the convergence process.
For this reason, we removed the backtracking from the paper text and from all the experiments.
We also found out that after removing the backtracking, the dropout becomes unnecessary, and we removed it as well.
Other changes:
1) We added the complexity of each step of the Riemannian gradient descent.
2) Investigated the effect of the proper initialization on the model and proposed a random initialization that works on par with the initialization from the solution of the linear model.
3) Added the validation loss for the comparison of optimizers on the UCI datasets.
4) Added the convergence plots to compare Riemannian optimization vs SGD baseline on the synthetic dataset.
5) Added 3 suggested papers to the related work section.
6) Plotted the TT-rank vs. accuracy graph on the MovieLens 100K dataset.
7) Restructured the experiments section to make it easier to assess different aspects of the model on different datasets.
8) Added a feedforward neural network baseline for the synthetic dataset.
9) Fixed typos and made a few clarifications.

Meta-review:
role: PC
data: comment: Modeling nonlinear interactions between variables by imposing Tensor-train structure on parameter matrices is certainly an elegant and novel idea. All reviewers acknowledge this to be the case. But they are also in agreement that the experimental section of this paper is somewhat weak relative to the rest of the paper, making this paper borderline. A revised version of this paper that takes reviewer feedback into account is invited to the workshop track.

Decision: Invite to Workshop Track