Title: Tensorial Mixture Models
Authors: Or Sharir
Ronen Tamari
Nadav Cohen
Amnon Shashua
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning
Unsupervised Learning

Abstract: We introduce a generative model, we call Tensorial Mixture Models (TMMs) based on mixtures of basic
component distributions over local structures (e.g. patches in an image) where the dependencies between
the local-structures are represented by a "priors tensor" holding the prior probabilities of assigning a
component distribution to each local-structure.

In their general form, TMMs are intractable as the priors tensor is typically of exponential size. However,
when the priors tensor is decomposed it gives rise to an arithmetic circuit which in turn transforms the
TMM into a Convolutional Arithmetic Circuit (ConvAC). A ConvAC corresponds to a shallow (single hidden layer)
network when the priors tensor is decomposed by a CP (sum of rank-1) approach and corresponds to a
deep network when the decomposition follows the Hierarchical Tucker (HT) model.

The ConvAC representation of a TMM possesses several attractive properties. First, the inference is tractable
and is implemented by a forward pass through a deep network. Second, the architectural design of the model
follows the deep networks community design, i.e.,  the structure of TMMs is determined by just two easily
understood factors: size of pooling windows and number of channels. Finally, we demonstrate the effectiveness
of our model when tackling the problem of classification with missing data, leveraging TMMs unique ability of
tractable marginalization which leads to optimal classifiers regardless of the missingness distribution.

TL;DR: A generative model realized through convolutional networks, which has the unique property of having both tractable inference and marginalization, showing state-of-the-art results on classification with missing data.

Review discussion:
The authors begin by representing a distribution using an arbitrary tensor with an exponential number of parameters.  Do these parameters have an intuitive meaning, eg., in terms of the joint distribution of different variables?
Then they say that they can restrict themselves to decomposable tensors that do not have an exponential number of parameters, without any loss of expressive power.  How is this possible?  Is the original representation redundant?  Does the decomposable representation have the ability to represent all distributions captured by the original tensor representation only when they grow to have an exponential number of parameters?  It would be helpful if the authors could provide any concrete intuitions about what is actually represented by the individual components of these representations.
Thank you for your questions. Our answers below:
> Do these parameters have an intuitive meaning, eg., in terms of the joint distribution of different variables?
As stated in the article, each entry of the coefficients tensor corresponds to the joint distribution P(d1,…,dN) of assigning components d1,…,dN to their respective local structures x1,…,xN. More intuitively, if the distribution of each mixing component is sufficiently concentrated, it could be viewed as a discretization step, and thus for each X there is a specific assignment d1,…,dN describing it (up to small variations), which means P(X)≈P(d1,…,dN) and the coefficient tensor act as an exponentially large truth table holding the probability of each possible X. A rigorous treatment of this argument can be found in appendix C3 of Cohen et al. (2016a).
> Then they say that they can restrict themselves to decomposable tensors that do not have an exponential number of parameters, without any loss of expressive power.  How is this possible?  Is the original representation redundant?  Does the decomposable representation have the ability to represent all distributions captured by the original tensor representation only when they grow to have an exponential number of parameters?
Tensor decompositions can represent all possible tensors only if allowed to grow to an exponential size, and only then can our model represent all possible distributions.
Indeed, no imaginable model could be polynomial in both time and space and still represent all possible distributions. We will update our article to reflect this point more clearly.
Though our model, as any model, cannot represent all distributions efficiently, we both analyze its expressive capacity theoretically, as well as cite prior work (Cohen et al., 2016b) which suggest it can efficiently represent the type of distributions commonly found in natural data (images, audio, etc.).
Thus, though restricting our model to polynomial setting does have diminish expressivity in general, in most common settings it should not pose significant limitations.
>  It would be helpful if the authors could provide any concrete intuitions about what is actually represented by the individual components of these representations.
In an effort to reduce the total number of pages, we chose to edit out the segments detailing exactly what is represented by the individual components. In our long-form version of the article (available on arXiv), we explain in finer detail the graphical model representation of our model, which directly connect each individual component to a matching latent variable.
Specifically, each individual component describes a conditional categorical distribution of a latent variable, which together form a Latent Tree Model, for graphical representation, see figure 3 and 5 in our long-form version.
References:
* Cohen et al., 2016a — On the Expressive Power of Deep Learning: A Tensor Analysis, COLT 2016.
* Cohen et al., 2016b — Inductive Bias of Deep Convolutional Networks through Pooling Geometry, arXiv pre-print, 2016.
* Long-form version on arXiv — https://arxiv.org/abs/1610.04167
The reviewer is pretty familiar with CP and Tucker decomposition, but is having a hard time interpreting the decoding algorithm of an arbitrary tensor decomposition by a ConvAC. The convolution part is quite confusing, as the generative model depicted in equation (2) is nothing related to convolution. In general, it would be useful if the authors write out the exact equation of the generative model (2) in the tensor form. It will also be useful for me to understand better if the authors relate \mathcal{A}_{d_1,\ldots,d_N} with \delta_i mathematically.  BTW, the definition of \delta_i in the preliminary section is not clear. What is j? What is the dimension of \delta? What does the \mathbf{1} mean?
In figure 2, I am confused about the notations, which are not introduced: e.g., r_0, r_{L-1}, is that the dimension? In equation 3, \Phi is a layer in Figure 2? Can you define it concretely? I couldn't follow Figure 2 as the notation definitions are missing.
That being said, I have a couple of high level questions. The authors use the non-negative tensor factorization. As far as I know, there is no theoretical guarantees on convergence as the objective function is non-convex.
I like section 5, where advantage of generative models over discriminative models is discussion when doing classification with missing values.
But I am again confused by the notations used in 5.1. What is V?  Why the rep(i,d) would be 1 when x_i is missing?
I haven't had the time to read the appendix yet, but maybe many of the missing details are in the appendix. I would appreciate it if the author point out where in the appendix I could find the answers.
Thank you for your questions and for highlighting the parts which could cause confusion in understanding our work. As a general note, in our effort to submit a shorter version of our article to ICLR, many of the technical details were moved to the appendices (specifically appendix A), which were replaced by more abstract descriptions of tensor decompositions and how they are applied to our model. Our long-form version of the article (available on arXiv: https://arxiv.org/abs/1610.04167) does address most of your questions in the body of the article, which is the version we intend to present if accepted. We will also make sure to improve the notations used in the places you have pointed out to be unclear.
To answer your specific questions:
1. The convolutional structure comes not from equation 2, but from the tensor decompositions themselves as explained in appendix A. More specifically, if \mathcal{A} is an N order tensor of dimension M in each mode, then the common CP and HT representations can be viewed as an arithmetic circuit \Phi, comprising product nodes and weighted sum nodes, which gets as input indicator vectors \delta^{(1)},…,\delta^{(N)} \in \R^M, s. t. \delta^{(i)}_j = 1 if d_i = j and otherwise 0, and the output of the circuit holds \mathcal{A}_{d_1,\ldots,d_N} = \Phi(\delta^{(1)},\ldots,\delta^{(N)}).
The exact structure of the circuit is derived from the CP or HT representations, which has a convolutional like structure. This specific form is displayed in equation 9 at appendix A. Returning to equation 2,  using \Phi(\delta^{(1)},\ldots,\delta^{(N)}) to denote the entries of the tensor, and rearranging the terms, results in equation 3 in the article. A more explicit representation can be found in equation 6 of our long-form version.
2. r_0,…,r_{L-1} are simply the number of output channels of each layer. They are also the ranks of the HT-decompositions, or in the case of the CP decomposition, where L=1, then r_0 is the CP rank. This is explained in more detail in appendix A, and specifically equation 8. \Phi is the complete arithmetic circuit computing the entries of the priors tensor according to either the CP or HT decompositions as explained in the text and in our answer to 1 above. We define it in general terms in the preliminaries, and in specific terms in appendix A (e.g. equation 6 for the CP decomposition).
3.
Regarding convergence of the non-negative tensor factorization, while it is indeed true there are no guaranties, the same is true for general tensor factorizations which are also non-convex, and not just for the non-negative case. This is unlike the matrix case where non-negativity is a critical issue as it makes something previously solvable (through spectral methods) unsolvable. The common algorithms we are aware of offer only weak guaranties in the general case (e.g. quasi-optimality for Tucker through HOSVD), and strong guaranties only for special cases (e.g. symmetric orthogonal decompositions).
Additionally, non-convexity is also a property of neural networks, and that is what motivated us to use the same common algorithms which work surprisingly well in practice. It is also worth mentioning that though we use non-negative tensor representations, we do not actually perform tensor factorization directly, as that is not the objective of our model and we have no access to the “true” tensor \mathcal{A}. We instead perform optimization through gradient ascent on the Maximum Likelihood objective, which indirectly results in factorizing the implicit tensor \mathcal{A} (assuming sufficient number of i.i.d. examples, and that there exist such \mathcal{A} fitting the data distribution).
4. In section 5.1 we took an arbitrary subset of the variables x_{i_1}, …, x_{i_V}, where V is an arbitrary number. We then wish to marginalize over the other N-V variables, which due to the separable structure of equation 6, is equivalent to simply integrating over P(x_i|d_i;\theta_{d_i}) for i not in {i_1,\ldots,i_V}. Since integrating over P(x|d, \theta) is always equal to 1, then all we need to compute the marginalized likelihood is to use rep(i,d) = 1 if x_i is missing, and otherwise rep(i,d) = P(x_i|d;\Theta) as usual.
We wish to thank you again for your specific questions and for pointing out the places where our notations and explanations could be improved.
1) The authors rely on replacing the tensor with its factorization which takes them from non-tractable to tractable setting, but they do fail to mention what are the conditions and convergence guarantees, especially that they use non-negative CP for which there exist no convergence guarantees. As a response to authors earlier comment in this thread, For CP decomposition in general, there does exist an algorithm with guarantees to converge to global optima (Anandkumar et al 2014). This is the case for Tucker decomposition with some assumptions too (Anandkumar et al 2015). My question is, what are the conditions for identifiability of your model and the convergence guarantees? For what class of tensors will this replacement make sense?
2) The paper talks about expressive power of the model referring to earlier works of Cohen et. al. What are the learning guarantees? what is the approximation error?
3) On MNIST dataset the state-of-the-art unsupervised CNN models achieve above 99% accuracy, http://yann.lecun.com/exdb/mnist/ , why don't the authors compare to state-of-the-art?
4) The authors mentioned in many places that the analysis in this paper is not confined to images and it works for many other applications. Given the fact that CNNs are greatly successful in images and not in text or speech (to the best of my knowledge), I was hoping they could clarify what other classes of data this method is expected to be successful in?
Thank for your questions and for your response to our previous comments. Our answers bellow:
1) We wish to clarify our earlier comments. We meant that to the best of our knowledge, there are no known guaranties for low-rank tensor approximations for general d-order tensors, i.e.
when the rank (or ranks in case of HT) of the decomposition is assumed to be significantly lower than the true rank, which for a generic tensor is exponential in its order and thus typically the case. We additionally wish to emphasize that our work is not about tensor factorization. We use tensors as a framework to express and analyze a specific family of arithmetic circuits, which correspond to tensor factorizations. As in any work on deep networks, it is all about gradient based learning. If you can calculate the gradient, then you use SGD until you stop. Today this constitutes a very large family of algorithms and in our work we connect this family to a tractable, both with regard to inference and marginals, generative model.
Therefore, the topic of convergence is outside the scope of our work, and instead rely on the vast empirical evidence, which suggests that in practice convergence is not a significant obstacle in learning deep networks.
2) As stated and proven in appendix B in our article, the model as defined can approximate to any degree any distribution given sufficient resources, which in its network form translates to sufficient number of channels in each layer. Once again we emphasize that this analysis is about what our model can theoretically express, and not what it is guaranteed to converged to, which in practice is typically not an obstacle. With regards to generalization, though a complete analysis is beyond the scope of this paper, there are early results suggesting it is especially suitable to data exhibiting strong local correlations and weak global correlations, as typically found in images and sounds (see Cohen et al. 2016b).
3) These results are only valid on the fully-observed dataset, while we are interested in the setting where parts of the pixels are missing. Considering the sharp decrease in performance of ConvNets as more pixels goes missing, any effect an extra ~0.5% could have made on the results of our experiments are insignificant.
4) There are actually many studies showing the success of CNN models on both text (e.g. Zhang et al, 2015) and speech (e.g. Abdel-Hamid et al., 2014). Very recently, generative models based on CNNs were able to beat the state-of-the-art on the text to speech task (van den Oord et al., 2016). In general, we expect our model to work in most cases where CNN works well, which is typically when the data exhibit strong local correlations and weaker global correlations. In Cohen et al. 2016b, it is shown that such correlations are the result of the pooling geometry, and thus our model could also be adapted by changing the pooling geometry to suit other types of data beyond those typically considered for CNNs.
Please let us know if the above has addressed your question.  Thank you!
References:
Cohen et al. 2016b — Inductive Bias of Deep Convolutional Networks through Pooling Geometry, arXiv pre-print, 2016.
Abdel-Hamid et al. 2014 — Convolutional Neural Networks for Speech Recognition
Zhang et al. 2015 — Character-level Convolutional Networks for Text Classification
van den Oord et al. 2016 — WaveNet: A Generative Model for Raw Audio
The paper provides an interesting use of generative models to address the classification with missing data problem. The tensorial mixture models proposed take into account the general problem of dependent samples. This is an nice extension of current mixture models where samples are usually considered as independent.
Indeed the TMM model is reduced to the conventional latent variable models.
As much as I love the ideas behind the paper, I feel pitiful about the sloppiness of the presentation (such as missing notations) and flaws in the technical derivations.  Before going into the technical details, my high level concerns are as follows:
(1) The joint density over all samples is modeled as a tensorial mixture generative model. The interpretation of the CP decomposition or HT decomposition on the prior density tensor is not clear. The authors have an interpretation of TMM as product of mixture models when samples are independent, however their interpretation seems flawed to me, and I will elaborate on this in the detailed technical comments below.
(2) The authors employ convolution operators to compute an inner product. It is realizable by zero padding, but the invariance structure, which is the advantage of CNN compared to feed-forward neural network, will be lost.
However, I am not sure how much this would affect the performance in practice.
(3) The author could comment in the paper a little bit on the sample complexity of this method given the complexity of the model.
Because I liked the ideas of the paper so much, and the ICLR paper submitted didn't present the technical details well due to sloppiness of notations, so I read the technical details in the arXiv version the authors pointed out. There are a few technical typos that I would like to point out (my reference to equations are to the ones in the arXiv paper).
(1) The generative model as in figure (5) is flawed. P(x_i|d_i;\theta_{d_i}) are vectors of length s, there the product of vectors is not well defined. It is obvious that the dimensions of the terms between two sides of the equation are not equal.
In fact, this should be a tucker decomposition instead of multiplication. It should be P(X) = \sum_{d1,\ldots,d_N} P(d_1,\ldots,d_N) (P(x_1|d_1;theta_{d_1},P(x_2|d_2;theta_{d_2},\ldots,P(x_N|d_N;theta_{d_N}), which means a sum of multi-linear operation on tensor P(d_1,\ldots,d_N), and each mode is projected onto P(x_i|d_i;theta_{d_i}.
(2) I suspect the special case for diagonal Gaussian Mixture Models has some typos as I couldn't derive the third last equation on page 6.
But it might be just I didn't understand this example.
(3) The claim that TMM reduces to product of mixture model is not accurate. The first equation on page 7 is only right when "sum of product" operation is equal to "product of sum" operation.
Similarly, in equation (6), the second equality doesn't hold unless in some special cases.
However, this is not true. This might be just a typo, but it is good if the authors could fix this. I also suspect that if the authors correct this typo,the performance on MNIST might be improved.
Overall, I like the ideas behind this paper very much. I suggest the authors fix the technical typos if the paper is accepted.
We thank the reviewer for the time and the detailed remarks. We first want to emphasize that we acknowledge that our ICLR version is less elaborate than our long-form arXiv version, and that this may have led to the misunderstandings we address below.
1) Regarding your comments on equation 5: P(x_i|d_i;\theta_{d_i}) are scalars and not vectors, and thus the product over these scalars is defined as usual. More generally, equation 5 should be taken as a composition of functions for an arbitrary X=x_1,…,x_N. The reviewer is correct to notice the similarity to the Tucker Decomposition, however, it is not a tucker decomposition of an algebraic tensor in the tensor product space of R^s vector spaces as the reviewer suggests, but a Tucker decomposition of a topological tensor over the tensor product of L^1(R^s) Banach spaces, where the tensor product of functions f and g is equal to the product function (f*g)(x) = f(x) * g(x). We chose not to discuss this perspective of equation 5 in the paper itself, as it only complicates the introduction of the model in an already highly technical paper.
2) Regarding the special case of diagonal Gaussian Mixture Models: We simply employ the fact that the product of diagonal Gaussian PDFs is itself a diagonal Gaussian PDF, where its mean is the concatenation of the means in the product, and the diagonal of the covariance matrix is the concatenation of the diagonals in the product. This can be directly proven by explicitly writing the product of these PDFs as a product of exponential functions. We will make sure to clarify this step in the final version of the article.
3) Regarding our claim that TMM reduces to product of mixture models: It is important to emphasize that we *do not* claim that a “sum of product” is equal to “product of sum”. What we do claim is that multiple sums, each with its respective running index d_i, over an expression which can be factorized to a product of separable expressions, each depending only on a single index d_i, can be rearranged to form a product of sums. This claim is always true as long as the expression over which we sum is indeed factorable as described above, which is the case when we assume the priors tensor from equation 5 holds P(d_1,…,d_N) = \prod_{i=1}^N P(d_i).
4) Regarding equation 6: there is indeed a minor typo in the second equality, where P(x_i|d_i;\theta_{d_i}) should be P(x_i|d_i = d;\theta_d). Beyond this minor typo, the equality itself is correct, as the transition from multiple sums to product of sum satisfy the same condition as the one mentioned above in point (3).
5) Regarding the loss of invariance structure of typical CNNs: This invariance can be mostly attributed to the use of parameter sharing, which we do discuss, although briefly, in the article. Do note that complex parameter sharing schemes could be devised to encourage different types of invariances. We also give more details on the exact parameter sharing scheme we use in appendix  F, which describes the models used in our experiments. There we use shared parameters in the first two layers, where invariance matter most, while using unshared parameters scheme in the rest of the network, analogue to the use of fully-connected layers in standard ConvNets.
We hope the above addresses the concerns of the reviewer, and would be happy to answer any further questions the reviewer might have.
This paper uses Tensors to build generative models.  The main idea is to divide the input into regions represented with mixture models, and represent the joint distribution of the mixture components with a tensor.
Then, by restricting themselves to tensors that have an efficient decomposition, they train convolutional arithmetic circuits to generate the probability of the input and class label, providing a generative model of the input and labels.
This approach seems quite elegant.  It is not completely clear to me how the authors choose the specific architecture for their model, and how these choices relate to the class of joint distributions that they can represent, but even if these choices are somewhat heuristic, the overall framework provides a nice way of controlling the generality of the distributions that are represented.
The experiments are on simple, synthetic examples of missing data.  This is somewhat of a limitation, and the paper would be more convincing if it could include experiments on a real-world problem that contained missing data.  One issue here is that it must be known which elements of the input are missing, which somewhat limits applicability.  Could experiments be run on problems relating to the Netflix challenge, which is the classic example of a prediction problem with missing data?
In spite of these limitations, the experiments provide appropriate comparisons to prior work, and form a reasonable initial evaluation.
I was a little confused about how the input of missing data is handled experimentally.  From the introductory discussion my impression was that the generative model was built over region patches in the image.  This led me to believe that they would marginalize over missing regions.
However, when the missing data consists of IID randomly missing pixels, it seems that every region will be missing some information.  Why is it appropriate to marginalize over missing pixels?
Specifically, $x_i$ in Equation 6 represents a local region, and the ensuing discussion shows how to marginalize over missing regions.  How is this done when only a subset of a region is missing?  It also seems like the summation in the equation following Equation 6 could be quite large.  What is the run time of this?
The paper is also a bit schizophrenic about the extent to which the results are applicable beyond images.  The motivation for the probabilistic model is mostly in terms of images.
But in the experiments, the authors state that they do not use state-of-the-art inpainting algorithms because their method is not limited to images and they want to compare to methods that are restricted to images.  This would be more convincing if there were experiments outside the image domain.
It was also not clear to me how, if at all, the proposed network makes use of translation invariance.  It is widely assumed that much of the success of CNNs comes from their encoding of translation invariance through weight sharing.   Is such invariance built into the authors’ network?  If not, why would we expect it to work well in challenging image domains?
As a minor point, the paper is not carefully proofread.  To just give a few examples from the first page or so:
“significantly lesser” -> “significantly less”
“the the”
“provenly” -> provably
We thank the reviewer for the time and detailed remarks. Our response follows:
1) Regarding our experiments: when it comes to the missing data problem, it could be divided into two parts: (1) learning from data containing missing data, (2) predicting based on data containing missing values. There are many studies that have focused on the first part, and while our model is well suited to solve this task through the marginalized Maximum Likelihood objective, this is not where its advantages lies. Our advantage is when we wish to predict based on data containing missing values, and the missingness distribution is not guaranteed to be the same as in the training set (e.g. due to sensors malfunction, or unexpected / rare behavior not modeled by the limited training set). This is why we chose to simulate different missingness distribution to highlight this advantage.
2) Regarding the requirement to know which elements are missing:
In this paper we focused on the case where data is strictly missing, where we know when a value is missing or not, a case which is quite common in many circumstances. We do not discuss the case where data is corrupted due to noise, however, it is easy to prove similar guaranties for the noisy case as well. The limitation of using our model for classification under noise is that we have to know something regarding the noise mechanism, and in the extreme case where we know exactly the noise distribution, we get similar result of optimal classification. This is a topic which we might discuss more in a future work.
3) Regarding how missing data is handled: In the paper we discussed a simplified scenario, where only complete regions are missing.
However, it is just as easy to marginalize over parts of region, where rep(i,d) equals the marginalized probability over the parts of the that region. If the whole region is missing, this results to 1, as in our example.
4) Regarding the run time of marginalization: the run-time is exactly the same as prediction without marginalization. Eq. 6 after marginalization is realized by the same ConvAC network of figure 3, where the representation layer is adjusted to handle the missing data according to (3).
In practice, our model is implemented to use GPUs as part of Caffe’s neural network framework (our code is available on GitHub), and though there is room for farther optimization of our code, it can handle missing data in real-time.
5) Regarding the applicability of our model to other domains: we conjecture that our model should work just as well on other domains exhibiting strong local correlations and weak global correlations as in images, which means it should probably work well on audio, video and text. We agree that this point would have been more convincing had we run experiments on other domains as well, however, limited resources and time led us to focus on the domain of images, which we are more acquainted with. If time will allow, we will gladly add an experiment on audio domain as well.
6) Regarding translation invariance: we do briefly discuss both the shared and the unshared cases, and expand on it more in appendix F, which explain the details of the models used in our experiments. There we explain that in the first two layers we are using weight sharing, where translation invariance matter most, and in the rest of the layers we use unshared parameters, which is analogue to the use of fully connected layers in standard ConvNets, which also do not employ parameter sharing. We have experimented with many other configurations of shared and unshared layers, and have found that the above function best on the datasets we have tested.
Thank you once more for taking the time to read and review our article, and we would be happy to answer any followup questions.
This paper proposes a generative model for mixtures of basic local structures where the dependency between local structures is a tensor. They use tensor decomposition and the result of their earlier paper on expressive power of CNNs along with hierarchical Tucker to provide an inference mechanism.
However, this is conditioned on the existence of decomposition. The authors do not discuss how applicable their method is for a general case, what is the subspace where this decomposition exists/is efficient/has low approximation error. Their answer to this question is that in deep learning era these theoretical analysis is not needed.
While this claim is subjective, I need to emphasize that the paper does not clarify this claim and does not mention the restrictions.
Hence, from theoretical perspective, the paper has flaws and the claims are not justified completely. Some claims cannot be justified with the  current results in tensor literature as the authors also mentioned in the discussions.
Therefore, they should have corrected their claims in the paper and made the clarifications that this approach is restricted to a clear subclass of tensors.
If we ignore the theoretical aspect and only consider the paper from empirical perspective, the experiments the appear in the paper are not enough to accept the paper. MNIST and CIFAR-10 are very simple baselines and more extensive experiments are required.
Also, the experiments for missing data are not covering real cases and are too synthetic.
Also, the paper lacks the extension beyond images. Since the authors repeatedly mention that their approach goes beyond images, and since the theory part is not complete, those experiments are essential for acceptance of this paper.
We thank the reviewer for reading our paper and taking the time reviewing it. Our response follows:
Regarding our experiments, it is important to stress that beyond the usual hurdles of implementing and training a completely new computational model, we also had to benchmark several other methods on the same task, tremendously increasing the resources required to add each new dataset to our experiments.
Unlike the case of standard classification, for which many existing results are available, the task of classification with missing data lack current results for comparison, and specifically when we wish to examine the case where the missingness distribution of the test set is different from the training set.
Nevertheless, we will consider adding another experiment on a different domain.
In regard to the reviewer’s comments on theoretical justifications of our model: we begin by clarifying that we did not wish to imply that theoretical questions, such as the ones the reviewer mentioned (e.g. convergence guarantees), are unimportant. What we do wish to convey is that these are generally very hard open questions in the field of deep learning, which are outside the scope of our work, and from a practical perspective, typically do not pose significant obstacles to learnability.
Despite the reviewer’s comments, we do allocate a significant amount of space to the topic of expressivity of our model, as seen throughout the body of the paper, and specifically in appendices B, C and D of our ICLR submission.
Additionally, the reviewer suggests our paper contains theoretical flaws and unjustified claims, however, does not support these remarks with concrete instances where this is actually the case. Considering that our submission is part of a larger line of works relating tensor decompositions to deep networks, which had already been heavily scrutinized before they were published at reputable venues, such as COLT and ICML, we expect any claims of theoretical flaws of our model to be argued in a more specific manner.
We wish to thank the reviewer again, and will be happy to answer any followup questions to our response.

Meta-review:
role: PC
data: comment: The authors have recently made several connections between deep learning and tensor algebra. While their earlier works dealt with supervised learning, the current work analyzes generative models through the lens of tensor algebra. 
 The authors show propose a tensorial mixture model over local structures where the mixture components are expressed as tensor decompositions. They show that hierarchical tensor decomposition is exponentially more expressive compared to the shallow models. 
 The paper makes original contributions in terms of establishing expressivity of deep generative models. The connections with tensor algebra could lead to further innovations, e.g. in training algorithms.
 However, the paper can be improved in two aspects: 
 (1) It will be nice if the authors make a connection between the algebraic view presented here with the geometric view presented by: https://arxiv.org/abs/1606.05340
 (2) It is also desirable if more extensive experiments are performed.  
 Given the improvements outlined for the paper, it does not meet the bar for acceptance at ICLR. 

Decision: Reject