Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper combines the tensor contraction method and the tensor regression method and applies them to CNN. This paper is well written and easy to read. 

However, I cannot find a strong or unique contribution from this paper. Both of the methods (tensor contraction and tensor decomposition) are well developed in the existing studies, and combining these ideas does not seem non-trivial.

--Main question

Why authors focus on the combination of the methods? Both of the two methods can perform independently. Is there a special synergy effect?

--Minor question

The performance of the tensor contraction method depends on a size of tensors. Is there any effective way to determine the size of tensors?
review_title: Contribution seems less.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper incorporates tensor decomposition and tensor regression into CNN by replacing its flattening operations and fully-connected layers with a new tensor regression layer. 

Pros:

The low-rank representation of tensors is able to reduce the model complexity in the original CNN without sacrificing much prediction accuracy. This is promising as it enables the implementation of complex deep learning algorithms on mobile devices due to its huge space saving performance.  Overall, this paper is easy to follow. 

Cons: 

Q1: Can the authors discuss the computational time of the proposed tensor regression layers and compare it to that of the baseline CNN? The tensor regression layer is computationally more expensive than the flattening operations in original CNN. Usually, it also involves expensive model selection procedure to choose the tuning parameters (N+1 ranks and a L2 norm sparsity parameter). In the experiments, the authors simply tried a few ranks without serious tuning. 

Q2: The authors reported the space saving in Table 1 but not in Table 2. Since spacing saving is a major contribution of the proposed method, can authors add the space saving percentage in Table 2?

Q3: There are a few typos in the current paper. I would suggest the authors to take a careful proofreading. For example,

(1) In the “Related work“ paragraph on page 2, “Lebedev et al. (2014) proposes…” should be “Lebedev et al. (2014) propose…”. Many other references have the same issue. 

(2) In Figure 1, the letter $X$ should be $\tilde{\cal X}$.

(3) In expression (5) on page 3, the core tensor is denoted by $\tilde{\cal G}$. Is this the same as $\tilde{\cal X}^{‘}$ in Figure 1?

(4) In expression (5) on page 3, the core tensor $\tilde{\cal G}$ is of dimension $(D_0, R_1, \ldots, R_N)$. However, in expression (8) on page 5, $\tilde{\cal G}$ is of dimension $(R_0, R_1, \ldots, R_N, R_{N+1})$.

(5) Use \cite{} and \citep{} correctly. For example, in the “Related work“ paragraph on page 2,

“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models Guo et al. (2012); Rabusseau & Kadri (2016); Zhou et al. (2013); Yu & Liu (2016).”

should be

“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models (Guo et al., 2012; Rabusseau & Kadri, 2016; Zhou et al., 2013; Yu & Liu, 2016).”









review_title: see detailed review below

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, new layer architectures of neural networks using a low-rank representation of tensors are proposed. The main idea is assuming Tucker-type low-rank assumption for both a weight and an input. The performance is evaluated with toy data and Imagenet.

[Clarity]
The paper is well written and easy to follow.

[Originality]
I mainly concern about the originality. Applying low-rank tensor decomposition in a network architecture has a lot of past studies and I feel this paper fails to clarify what is really distinguished from the other studies. For example, I found at least two papers [1,2] that are relevant. ([2] appears at the reference but it is not referred to.) How is the proposed method different from them?

Also, the "end-to-end" feature is repeatedly emphasized in the paper, but I don't understand its benefit. 

[1] Tai, Cheng, et al. "Convolutional neural networks with low-rank regularization." arXiv preprint arXiv:1511.06067 (2015).
[2] Lebedev, Vadim, et al. "Speeding-up convolutional neural networks using fine-tuned cp-decomposition." arXiv preprint arXiv:1412.6553 (2014).

[Significance]
In the experiments, the proposed method is compared with the vanilla model (i.e., the model having no low-rank structure) but with no other baseline using different compression techniques such as Novikov et al., 2015. So I cannot judge whether this method is better in terms of compression-accuracy tradeoff.


Pros:
- The proposed model (layer architecture) is simple and easy to implement

Cons:
- The novelty is low
- No competitive baseline in experiments

review_title: Interesting but contributions are not enough

Meta-review: This paper proposes methods for replacing parts of neural networks with tensors, the values of which are efficiently estimated through factorisation methods. The paper is well written and clear, but the two main objections from reviewers surround the novelty and evaluation of the method proposed. I am conscious that the authors have responded to reviewers on the topic of novelty, but the case could be made more strongly in the paper, perhaps by showing significant improvements over alternatives. The evaluation was considered weak by reviewers, in particular due to the lack of comparable baselines.

Interesting work, but I'm afraid on the basis of the reviews, I must recommend rejection.
Decision: Reject