Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 
In this paper, an extension of nonnegative CP decomposition called hierarchical nonnegative CP decomposition (HNCPD) is proposed. This method is designed to capture the hierarchical structure in e.g., topic modeling. Also, an optimization method called neural NCPD is proposed. Several decomposition results of synthetic data, video data, and Twitter data are presented.

Strong points:
- S1. A new algorithm is proposed.
- S2. Experimental results with real data are reported.

Weak points:
- W1. The technical contributions are small.
- W2. The motivation of HNCPD is unclear.
- W3. The paper is not clearly written.

My recommendation: 
Although this paper contains some interesting ideas, I feel the overall quality is not high enough to be accepted in the ICLR community. Here I would like to elaborate on the reasons. 

W1. The formulation of HNCPD is not new. Cichocki et al. (2007) proposed a more general form of HNCPD; see Eq. (13) of the following paper. So the technical contribution of this paper is summarized to the development of neural NCPD, but it is a direct extension of Gao et al. (2019). 
````
Cichocki A., Zdunek R., Amari S. (2007) Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization. In: Davies M.E., James C.J., Abdallah S.A., Plumbley M.D. (eds) Independent Component Analysis and Signal Separation. ICA 2007. Lecture Notes in Computer Science
```````
W2. It is not clearly explained how HNCPD is beneficial to real-world applications. For example, the pictures in e.g. Fig. 4 seem to tell us some interesting information, but how can we use that? In topic modeling, what kind of knowledge/insights can we get from the results? And also, how can we quantify the benefits?

W3. Some part is not easy to read. 
- In Section 2, $r$ is defined as the number of super topics. However, in Table 1, $r$ seems to be used as the number of subtopics, which is previously defined as $r_0, r_1, r_2$. A similar inconsistency is observed in Fig4, Fig6, etc.
- The interpretation of experimental results is unclear. In Section 3.2, it is claimed that HNCPD is better than NMF because "the chromatic NMFs obscure much of the chromatic interaction". Here, the true structure of chromatic interaction is hidden and we cannot observe it. So in what sense can we show the superiority of HNCPD?
review_title: Good work but the quality is not enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY:

This paper presents a hierarchical nonnegative CP tensor decomposition method. It also proposes a training method that leverages forward and backward propagation. The method is tested on both synthetic and real datasets. These experiments illustrate how the method can be used to discover topics and how they vary over time. The hierarchical nature of the method makes it possible to group the topics into supertopics in multiple steps.

I thinks the paper is well-written for the most part. The topic is interesting and should be of interest to the ICLR community. It seems like the authors combine existing ideas (hierarchical and neural NMF, NCPD) into a new method. While this may be somewhat incremental, it seems like the authors are doing something that hasn't been done before. My main issue with the paper is that there are a few portions that are currently difficult to follow. In particular, in the discussion on approximation in Section 2.1, it is not clear if this idea is used in the implementation, which might make it difficult to replicate the results. Fixing this would help improve the quality of the paper. I provide more details on which parts are unclear below.


ADVANTAGES:

- Introduction provides a clear overview of what has been done before. It is easy to see where the present paper fits in with the existing literature. It also does a good job of connecting to applications.

- The experiments in Sections 3.2 and 3.3 are convincing and nicely explained, especially the illustrations of how topics vary over time. The last sentence in Section 3.3 help explain the benefit of a hierarchical method.

- Well-written for the most part. 

- Should be of interest to ICLR community.


CONCERNS/QUESTIONS:

- In the last sentence on page 2, it is not clear which loss function is used for Neural NMF when doing the backpropagation. Is it the expression in the Frobenius norm in the paragraph titled "Hierarchical NMF (HNMF)" above?

- In Section 2.1, I follow the discussion up to Equation (6). But the rest of the first paragraph is difficult to follow. For example, it's not clear to me how the columns of the hierarchical NMF factors are used to form NCPDs of ranks $r^{(0)}$, $r^{(1)}$, ..., $r^{(L-2)}$. The subsequent discussion about the dependencies and indices is also unclear. Perhaps this discussion could be made more clear with the help of an example?

- In Section 2.1, I think the second paragraph is clear until Equation (7), but the rest of the paragraph is difficult to follow. The discussion about approximating the relationship between the columns of $A_i^{(0)}$ and $A_k^{(0)}$ is unclear. What do you mean by "relationship"? In what sense is $(W_i)_{p_1, p_2}$ approximating it? It is not clear why the definition in Equation (9) is chosen. Is the idea that $[[ \tilde{A}_1^{(0)}, \ldots, \tilde{A}_k^{(0)} ]] \approx [[ \tilde{X}_1, \ldots, \tilde{X}_k$ ]]? In the experiments, do you use this approximation, or do you use the NCPD combined with HNMF for each factor matrix as discussed in the beginning of Section 2.1? 

- In the last sentence of Section 2.1, you mention that "Neural NCPD allows factor matrices for all other modes to influence the factorization of a given mode." It makes sense that this is the case when all parameters, including the initial NCPD, are trained together via backpropagation. But does this still remain true when the initial NCPD is computed independently at first and then kept fixed throughout the fitting of the HNMFs, like in Algorithm 2? As far as I can tell, the approach in Algorithm 2 of keeping the initial NCPD fixed through the training is also what you do in the experiments.

- In the experiments, do you use a combination of the loss functions in Equations (11) and (13) (e.g., C+E), or just one of them? This is not clear from the discussion. Also, since the NCPD is computed by itself at the start of the algorithm and then kept fixed, isn't the term $|| X - [[ X_1, \ldots, X_k ]] ||_F$ in (13) a constant that could be ignored?

- For the discussion in Section 2.2 about the derivatives, it would be a good idea to let the reader know that there is a more in-depth explanation available in the appendix.

- It is not clear how the Standard NCPD in Section 3 is computed. Could you perhaps add an explanation in the appendix or point to a relevant reference?

- Below Figure 2, you say that $g \sim N(z; 0, \sigma^2)$. What does $z$ here mean?

- For the experiment in Section 3.2, did you try keeping each frame as a matrix in the tensor? In other words, did you try reshaping the tensor into a 4-dimensional tensor of size 37 $\times$ 3 $\times$ (number of x pixels) $\times$ (number of y pixels)? One motivation for using tensors is to avoid having to vectorize things like images, so it would be interesting to know if this also worked well. I don't think you need to change the current example, I'm just asking out of curiosity. 


MINOR CONCERNS/QUESTIONS:

- In Figure 1, the colors are a nice addition, but they're quite muted which makes it hard to see them. Could they be made brighter? Also, should the red line along the columns of the $S_2^{(0)}$ matrix instead be along the rows of the $A_2^{(0)}$ matrix?

- In Figure 1, in the caption, the tensor $X$ is not using the bold tensor notation.

- Below Equation (7), in the definition of $\alpha_{j_1, j_2, \ldots, j_k}$, should the sum go to $r$ instead of $r^{(0)}$? It looks like it should based on the derivation in the appendix.

- Equation (15) should end with a comma instead of a period since the sentence keeps going below.

- In the second sentence of Section 3.1, the word "size" appears twice in a row.

- At the bottom of page 7, should the tensor size be 8 x 100 x 12721 instead of 8 x 10 x 12721 since the number of tweets are capped at 100?

- In the text to the left of Figure 7, the words "the" appears twice in a row in two places.

- In the appendix, in the section "HNCPD expansion", in the 2nd sentence (starting with "We have that by definition..."), I think the square bracket "]" should be removed on the right hand side of the equation?

#######################

Update:

I thank the reviewers for their responses. I appreciate the effort they put into clarifying the paper. However, I still think Section 2.1 in particular is difficult to follow. I will therefore keep my original rating.
review_title: Interesting paper which could be improved if certain unclear parts are explained better

final_score: rating: 4: Ok but not good enough - rejection
review_content: Short summary:

The paper introduces a promising new method, hierarchical nonnegative CP decomposition (HNCPD), as well as a training method for the HNCPD, neural NCPD, for topic modeling problems. After presenting some fundamentals on the topic, as well as some related work, a detailed and formal mathematical description of the suggested method is given. Finally, results of experiments on three data sets (synthetic data, image data and text data) are utilized to evaluate the suggested methods.

Potential Improvements:
- Equation (4) introduces the forward propagation for a NNMF, which is crucial for the HNCPD and Neural NCPD. However, the method of determining 

$\argmin\ limits_{\boldsymbol{S}}$

 is not introduced. Further, it is not shown how the derivative of the argmin function with respect to 

$\boldsymbol{A}^{(l)}$

 is formed, which is later on used for the backpropagtion.

- The introduction mentions the importance of topic modeling given large amounts of data, as well as multi-modal tensor data. However, the present experiments only utilise rather small uni-modal data sets (for instance a selection of 37 forest images). It would therefore be of value to answer 1) how the experiments would scale to higher dimensions (for instance for images of the forest time laps) and 2) how the suggested approach would work on a multi-modal database.

- The analysis of experiments in section 3.2 and 3.3 is rather qualitatively than quantitatively. A quantitative measure like the reconstruction loss would most likely be of value.- Sections 3.2 and 3.3 also lack a comparison to other methods.

- The conclusions section could include a short summary of future work on this novel approach.


Minor Comments:
- The for-loop over the two variables i and l in Algorithm 1 could be expressed as two for-loops to be more precise.
- In section 3: presidential election --> United States presidential election
- In section 3: combines the the rank 8 --> combines the rank 8 
- In section 3: we display the the results --> we display the results

Overall Evaluation:
The paper is well and clearly written, the significance of this work, however, needs to be further proven by experiments. The originality seems moderate, as already existing concepts of neural NMF and Hierarchical NMF are applied to NCPD.
review_title: A short review for Neural Nonnegative CP Decomposition for Hierarchical Tensor Analysis

Meta-review: The paper presents a hierarchical version of NMF for the CP decomposition of tensors.

The idea is similar to Chinocki etal 2007 and extends Gao etal 2019, and in Chinocki was presented for the standard linear formulation with regularisation terms.  The extension here doesn't use the standard ALS algorithm but rather presents a neural network analogue, though the functions are still linear, its just that back-prop etc. are used for the computation.  The authors point out their formulation is a more flexible representation and optimisation (in response to AnonReviewer5), and thus represents an improvement.  While this is an interesting implementation, in NNs, the model is still fairly simple.  

Moreover the experimental results are restricted to a few data sets.  There are literally hundreds of NMF variants in publication and many different evaluations are done.  The experimental work here, while showcasing the work, is not extensive.  For instance, more empirical comparisons should have been made against prior hierarchical NMF on a battery of data.

So this is good, publishable work, and the authors have repaired many of the issues raised by the reviewers.  The work, however, is borderline in empirical work and the contribution is not strong.
Decision: Reject