Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: * Summary:
The paper introduces a novel tensor decomposition that is reminiscent of canonical decomposition (CP) with low-rank factors, based on the observation that the core tensor in Tucker decomposition can be decomposed, resulting in a model interpolating between CP and Tucker. The authors argue that a straight application of AdaGrad on this decomposition is inadequate, and propose Ada^{imp} algorithm that enforces rotation invariance of the gradient update. The new decomposition is applied to ComplEx model (called PComplEx) that demonstrates better performance than the baseline.

* Comments:
Although the approach is well motivated, the paper has many ambiguities that need to better clarification.
1. Tucker decomposition results in lower dimension factors, "d" in the paper. So the resulting core tensor is of size (d \times d \times d). However, this core tensor is further decomposed with a rank-D CP as shown in Section 3, where D >= d. Basically, first the original tensor is factored into lower rank d, and the core tensor is then expanded into rank D >= d. The reader did not understand what is the justification for this approach? Please provide further explanation on this part.
2. The confusion of P_2 and P_3 terms in the paper. At the beginning of Section 3, P_2 is assumed to be identity through out the paper. But P_2 is mentioned to have specific attributes in other parts of the paper, such as in the second paragraph from the bottom of page 4, the first paragraph and first equation on page 5. And P_2 does not appear in AdaGrad algorithm.
3. The experiment is lacking. First, the paper does not explain the meaning of evaluation metrics. Second, the authors do not provide an insight, why PComplEx is better than the ComplEx baseline on SVO dataset, but performs similarly on other datasets. Which factors lead to such improvement?
4. The comparison to other state-of-the-arts is inadequate, each compared method only has one or few configurations in terms of number of parameters.

Overall the proposed decomposition method might have significant contribution to research progress in this field, but the paper fails to convince the reader of its significance. I feel the paper should be overhauled.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors present a new way of decomposing 3-order tensors which uses interpolation
between the Tucker and CP decompositions, called CPT. The main idea is to present the components of the CP model
with an additional low-rank structure.
The authors also provide a new optimization algorithm called ADA-imp, for learning this decomposition,
which is a variant of Adagrad adapted to their settings. 
The paper is overall interesting, clearly written and well-motivated. 
The mathematical derivations are, as far as I could follow, correct and non-trivial.  (I did not read all the details in the Appendix). 
The authors also show favorable experimental results on two knowledge-base datasets, with improved loss vs. #parameter used tradeoff.
A few unclear issues and suggestions for improvements are below.

The authors present the problem as completion of a binary 3-order tensor, i.e. predicting for triplets (subject, predicate, ?) if '?' refers to 0 or 1.
But they also write 'we formulate this problem as a multi-class classification problem, where the classes are the entities of the knowledge base'  - so this is not a binary problem? does this mean there is some structure that must be present in the tensor? (e.g. there is exactly one '1' in each column of length N? This should be clarified. 

It would be good to make the description of Algorithms 1 and 2 more precise and detailed. 
For example, the operation/algorithm AdaGrad(\eta;w_k; g_k;G_k) is not defined. AdaGrad is described in the Appendix but it is hard to match it to get the precise operation used in Algorithm 1. 
Algorithm 1 shows one step of PComplEx, and it would be good to add the entire PComplEx algorithm, with input,output&parameters. 

The authors present their method in the context of knowledge base completion, thus for tensors of order 3, but it is not clear if any of the components they proposed indeed specialized for this problem, or is it a contribution to general tensor decomposition. Some remarks regarding the (in?)applicability of the method more generally would be helpful. 

Figure 3 describing the experimental results should be explained better. There are few methods shown only in some of the graphs and only for some parameter values - why?
The complexity measure 'parameters-per-entity' should be clearly defined (I didn't find it in the text). Similarly, the performance measures 'mean reciprocal rank' and 'hits at 5%' 
should be defined in terms of the tensor. 
The authors should also add running times of the different experiments and methods. 


Minor:
--------
In the main paper, the authors define an (N,L,N) tensor, but in the appendix Section 9.9 they list N and P. Does P refer to L here? 

The authors mention a few times usage of 'deep-learning techniques' - but I believe that in at least some of the contexts, they refer to optimization methods which are typically used in deep learning, and  are applied here to train other models presented in the text, and not to the usage of actual deep learning architectures - this is confusing and should be clarified. 

Page 7, top: what are the matrices M^(1), M^(2), M^(3)? they seem to be different for different decompositions  


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, a tensor decomposition method is studied for link prediction problems. The model is based on Tucker decomposition but the core tensor is decomposed as CP decomposition so that it can be seen as an interpolation between Tucker and CP. The performance is evaluated with several NLP data sets (e.g., subject-verb-object triplets). 

Although the entire idea is interesting, the current form of the paper is not sufficient for acceptance. The main reasons are (A) the proposed model is not completely novel and (B) the empirical results are not significant. 

(A) The idea of combining CP and Tucker is not new. For example, Tomioka et al. (2010; Section 3.4) considered the Tucker-CP patterns (CP decomposition of the Tucker core). Although they used the Tucker-CP model to improve the interpretability rather than link prediction, the paper needs to make some attribution to the prior work. 

(B) By looking Figure 3, the proposed method, PComplEx, is not significantly better than the existing methods such as ComplEx. Except SVO data, PComplEx and ComplEx share almost the same performance curve. Also, other existing methods such as TuckER and MurP are evaluated only in a few points while (P)ComplEx is evaluated in many points. I feel this is unfair.

Tomioka, R., Hayashi, K., & Kashima, H. (2010). Estimation of low-rank tensors via convex optimization. arXiv preprint arXiv:1010.0789.
review_title: Official Blind Review #2

Meta-review: The paper proposes a tensor decomposition method that interpolates between Tucker and CP decompositions. The authors also propose an optimization algorithms (AdaImp) and argue that it has superior performance against AdaGrad in this tesnor decomposition task. The approach is evaluated on some NLP tasks.
The reviewers raised some concerns related to clarity, novelty, and strength of experiments. As part of addressing reviewers concerns, the authors reported their own results on MurP and Tucker (instead of quoting results from reference papers). While the reviewers greatly appreciated these experiments as well as authors' response to their questions and feedback, the concerns largely remained unresolved. In particular, R2 found the gain achieved by AdaImp not significantly large compared to Adagrad. In addition, R2 found very limited evaluation on how AdaImp outperforms Adagrad (thus little evidence to support that claim). Finally, AdaImp lacks any theoretical analysis (unlike Adagrad).
Decision: Reject