Neural Nonnegative CP Decomposition for Hierarchical Tensor Analysis
Authors: Joshua Vendrow
Jamie Haddock
Deanna Needell
Venue:  (2021)
Keywords: nonnegative tensor decompositions
topic modeling
hierarchical model
CP decomposition
neural network
backpropagation

Abstract: There is a significant demand for topic modeling on large-scale data with complex multi-modal structure in applications such as multi-layer network analysis, temporal document classification, and video data analysis; frequently this multi-modal data has latent hierarchical structure. We propose a new hierarchical nonnegative CANDECOMP/PARAFAC (CP) decomposition (hierarchical NCPD) model and a training method, Neural NCPD, for performing hierarchical topic modeling on multi-modal tensor data. Neural NCPD utilizes a neural network architecture and backpropagation to mitigate error propagation through hierarchical NCPD.  

Under review as a conference paper at ICLR 2021
NEURAL NONNEGATIVE CP DECOMPOSITION FOR HIERARCHICAL TENSOR ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
There is a significant demand for topic modeling on large-scale data with complex multi-modal structure in applications such as multi-layer network analysis, tempo- ral document classification, and video data analysis; frequently this multi--odal data has latent hicrarchical structure. We propose CANDECOMP/PARAN AFAC (CP) decomposition (hierarchical NCPD) model and a training method, Neural NCPD, for performing hierarchical topic modeling on multi-modal tensor data. Neural NCPD utilizes neural network architecture and backpropagation mitigate error propagation through hierarchical NCPD.
hierarchical nonnegative
INTRODUCTION
The recent explosion in the collection and availability of data has led to an unprecedented demand for scalable data analysis techniques. Furthermore, data that has a multi-modal tensor format has become ubiquitous across numerous fields (Cichocki et al.] 2009). The need to reduce redundant dimensions (across modes) and to identify meaningful latent trends within data has rightly become an integral focus of research within signal processing and computer science. An important application of these reddutiionreducction techniques is topic modeling, the task of identifying latent topics and themes of dataset in an unsupervised or partially supervised approach. A popular topic modeling approach for matrix data is the dimeesioonreduction technique nonnegative matrix factorization (NMF) (Lee & Seung, 1999), which is generalized to multi-modal tensor data by the nonnegative CP decomposition (NCPD) (Carroll & Chang, 1970, Harshman et al. 1970). These models identify latent topics within the data; here the rank user-defined parameter that can be challenging select without priori knowledge or a heuristic selection procedure.
In topic modeling applications, one often additionally wishes to understand the hierarchical topic structure (i.c., how the topics naturally related and combine into supertopics). For matrices (tensors), a naive approach is to apply NMF (NCPD) first with rank r and then again with rank < and simply identify the j supertopics as linear (multilinear) combinations of the original subtopics. However, due to the nonconvexity of the NMF (NCPD) objective function, the supertopics identified in this way need not be linearly (multi-linearly) related to the subtopics. For this reason, hierarchical models which enforce these relationships between subtopics and supertopics have become a popular direction of research. challenge of these models is that the nonconvexity of the model at each level of hierarchy techniques for mitigating this cascade of error (Flenner & Hunter 2018 Trigeorgis et al. 2016
yield cascading error through the layers of models; several works have proposed
Le Roux et al. 2015 Sun et al. 2017 Gao et al. 2019).
In this work, we propose a hierarchical NCPD model and Neural NCPD, an algorithm for training this model which exploits backpropagation techniques to mitigate the effects of error introduced at earlier (subtopic) layers of hierarchy propagating downstream to later (supertopic) layers. This approach allows us to (1) explore the topics learned at different ranks simultaneously. and (2) illustrate the hierarchical relationship of topics learned at different tensor decomposition ranks.
Notation. We follow the notational conventions of Goodfellww et al. 2016);; e.g., tensor X, matrix X, vector æ, and (integer or real) scalar x. In models, we use variable (with superscripts denoting layer of hierarchical models) to denote model rank and when indexing through rank-one

---

Under review as a conference paper at ICLR 2021
components. In all tensor decomposition models, use k to denote the order (number of modes) of the tensor and use i when indexing through modes of the tensor. In all hierarchical models, we use L to denote the number of layers in the model and use to index layers. We let @ denote the vector outer product and adopt the CP decomposition notation
[X, X2, X =
Σ 2=1
j
2
3
8 ... & )
where 2F5") is the jth column of the ith factor matrix X; Kolda & Bader 2009
Contributions. Our main contributions are two-fold. First, we propose a novel hierarchical non- negative tensor decomposition model that we denote hierarchical NCPD (HNCPD). Our model treats all tensor modes alike and the output is not affected by the order of the modes in the tensor representation; this is property not shared by other hierarchical tensor decomposition models such as that of Cichooci et al. (2007a). Second, we propose an effective neural network -nspiierd training method that we call Neural NCPD. This method builds upon the Neural NMF method proposed in Gao et al. (2019), but is not direct extension; Neural NCPD consists of branch of Neural NMF for each tensor mode, but the backpropagation scheme must be adapted for factorization information flow between branches.
Organization. In the remainder of Section I we present related work on tensor decompositions and training methods. In Section 2] we present our main contributions, hierarchical NCPD and the Neural NCPD method. In Section 3 we test Neural NCPD on real and synthetic data, and offer some brief conclusions in Section
4 We include justification of several computational details of method and
further experimental results in Appendix A
1.1 RELATED WORK
In this section, we introduce NMF, hierarchical NMF, the Neural NMF method, and NCPD, and then summarize some relevant work.
Nonnegative Matrix Factorization (NMF). Given a nonnegative matrix X E R and desired dimension N, NMF seeks to decompose X into product of two low-dimensional nonnegative matrices; dictionary matrix A E puu and representation matrix S E R so that
X AS
Σ a, aa
(2)
where aj is a column (topic) of A and sy is row of S. Typically, is chosen such that < min(n1, ] to reduce the dimension of the original data matrix or reveal latent themes in the data. Each column of S provides the approximation of the respective column X the lower-dimensional space spanned by the columns of A. The nonnegativity of the NMF factor matrices yields clear interpretability: thus, NMF has found application in document clustering (Xu et al., 2003 Gaussier & Goutte 2005; Shahnaz et al.) 2006), and image processing and computer vision (Lee & Seung
1999 Guillamet & Vitrial 2002 Hoyer 2002), amongst others. Popular training methods include
multiplicative updates (Lee & Seung 1999: 2001 Lee et al.) 2009). projected gradient descent (Lin
2007 and alternating least-squares Kim et al. 2008 Kim & Park 2008).
Hierarchical NMF (HNMF). HNMF seeks to illuminate hicrarchical structure by recursively factor- izing the NMF S matrices; e.g.. (Cichocki et al. 2009). We first apply NMF with rank (0) and then apply NMF with rank (1) to the S matrix, collecting the y:(0) subtopics into (()) supertopics. HNMF with L layers approximately factors the data matrix as
X ~ (0) 5(0) = A (0) (
(1) (())
=
(0) ()... (C-1))
(3)
Here the (00 matrix represents how the subtopics at layer collect into the supertopics layer i 1. Note that as L increases, the error ||X - (0) (1) (C-1) g(4-1)) necessarily increases as error propagates with each step. As result, significant error introduced when large. Choosing (()), p(1). (C-1) in practice proves difficult the number of possibilities grow combinatorially.
2

---

Under review as a conference paper at ICLR 2021
Neural NMF (NNMF). In the previous work Ga et al. (2019), the authors developed an iterative algorithm for training HNMF that uses backpropagation techniques to mitigate cascading error through the layers. To form this hierarchical factorization, the Neural NMF algorithm uses a neural net architecture. Each layer l of the network has weight matrix A In the forward propagation step, the network accepts matrix ((--)), calculates the nonnegative least-squares solution
g( (A (2) )((-1))
arg min ||S $20
A SlP:
(4)
and sends the matrix ( to the next layer. In the backpropagation step, the algorithm calculates gradients and updates the weights of the network, which in this case are the A matrices.
Nonnegative CP Decomposition (NCPD). The NCPD generalizes NMF to higher-order tensors; specifically, given an order-k tensor X E of X seeks 11 E R200 X2 E
10 X.. Xny and a fixed integer r, the approximate NCPD
ng
X E R so that
20
X X2, .
The X; matrices will be referred as the NCPD factor matrices. A nonnegative approximation with fixed r is obtained by approximately minimizing the reconstruction error between X and the NCPD reconstruction. This decomposition has found numerous applications in the area of dynamic topic modeling where one seeks to discover topic emergence and evolution (Cichocki et al.) 2007b] Traoré et al. 2018: [aha & Sindhwani] 2012). Methods for training NMF models can often be generalized to NCPD; for example, multiplicative updates (Welling & Weber 200T) and alternating
least-squares (Kim al. 2014).
Other Related Work. Other works have sought to mitigate error propagation in HNMF models
with techniques inspired by neural networks (Trigeorgis et al.) 2016 Le Roux et al. 2015 Sun et al.
2017 Flenner & Hunter, 2018). Additionally, previous works have developed hierarchical tensor decomposition models and methods (Vasilescu & Kim 2019 Song et al. 2013 Grasedyck 2010). The model most similar to ours is that of Cichocki et al. (0077)), which we refer to as hierarchical nonnegative tensor factorization (HNTH). This model consists of a sequence of NCPDs, where a factor matrix for one mode is held constant, the remaining factor matrices produce the tensor which is decomposed at the second layer, and this decomposition is combined with the fixed matrix from the previous layer. We note that HNTF is dependent upon the ordering of the modes, and specifically which data mode appears first in the representation of the tensor. We refer to 'HNTF-i' as HNTF applied to the representation of the tensor where the modes are reordered with mode first.
2 OUR CONTRIBUTIONS
In this section, we present our two main contributions. We first describe the proposed hierarchical NCPD (HNCPD) model, and then propose a training method, Neural NCPD, for the model.
2.1 HIERARCHICAL NCPD (HNCPD)
Given an order-k tensor X E R X HNCPD consists of an initial rank-r NCPD layer with factor matrices X1, X .,, XR. each with columns, and an HNMF with ranks ,(()),r())) r(!), ( (CC-2) for
each of these factors matrices; that is, for each X, at layer f, we factorize X; as
X = 1
(0)
A (t-2) S( ((t-2)
(6)
where A ( has (4) columns; see Figure for a visualization. Thus, HNCPD consists of tensor approximations
x [A (0)
(E -
To access hierarchical structure between tensor topics each layer, need to utilize information in the s(()) matrices for all modes. To simplify this hierarchical structure, we develop an approximation scheme such that the hierarchical topic structure for all modes is given by single matrix.
3

---

Under review as a conference paper at ICLR 2021
For simplicity, we first consider the two layer case. We note that
[1 X2, X2,
Xal
E
(AA
Dis @ (AA2)... )ode .... (A4))..))
(8)
where ff...a.....
p=1 (5 (S. 12.0 (S pp: we justify this statement in Appendix We refer to decomposition summands in (8) where ji = j2 = js as vector outer products of same-index factor matrix topics, and all other summands vector outer products of different-index factor matrix topics. To identify clear hierarchy, we avoid these different-index column outer products.
The approximation scheme computes ma- trices A (0) whose columns visualize the desired r(()) NCPD topics along each mode while avoiding different-index col- umn outer products in the decomposition. We approximate the summation (8) by placing all summands that include column P2 of ( with single rank-one vector ):pp
(0)
X
X
$10)
(A
x
23
x;
outer product, (Al (0)) J.pp ) @ (A
Figure I: A visualization of two-layer HNCPD model. The colored edges of the order-three tensor, X, represent the three modes.
80 (A )).p. @ (AA ).pp. To minimize error introduced by this approximation, we transform factor matrices A
(o))
(0) for i + k to
A (0) by collecting into (A, (0) )app the ap-
proximate contribution of all columns of A, in vector outer products with (A )..p in (8 That is,
for S P1,P2 S (0) and S k let W, R xn
matrix with
(Wi)
Σ
and A A 00) W.
(9)
Furthermore, we can identify the topic hierarchy from the S matrix. We can generalize this process to later layers by noting that we can group the A, matrices together, so X; Ro (A (0) A(") (1) A (2) )(( Thus,
treat this approximation as above, replacing A (0) with the
product A (0) ( (0)
Like in HNMF, CITOrS in earlier layers can propagate through to later layers and produce highly suboptimal approximations. Challenges encountered during computation of HNMF exacerbated in an HNCPD model. For this reason, we exploit approaches developed for HNMF in Gao et al. (2019) in our training method Neural NCPD. Furthermore, the computation of HNMF factor matrices X; are independent from X, the factorizations are applied sequentially: Neural NCPD allows factor matrices in o for all other modes to influence the factorization of given mode.
2.2 NEURAL NCPD
Our iterative method consists of two subroutines, forward wwwaard-roopagation and a backpropagation. In Algorithms I and we display the pscudocode for proposed method. Following this learning process for the factor matrices in (6), we apply the approximation scheme described in Section [.] to the learned factor matrices to visualize the hierarchical structure of the computed HNCPD model.
Forward Propagation. The forward-propagation treats A matrices as neural network weights and uses A and previous layer output to compute
8
((A S (((-1))
(6)
(10)
where q is defined in equation 4 and S (
X, producing the matrices s ..... s( (C
for 1 < k. The function q( (40) ((())) nonnegative least-squares problem, be
4

---

Under review as a conference paper at ICLR 2021
calculated via any convex optimization solver, utilize inplementation of the Hanson-Lawson
algorithm (Lawson & Hanson 1995). Finally, we pass the A ( and s! matrices and X into a loss function, which differentiate and backpropagate.
Backpropagation. Our goal is to dif- ferentiate our cost function C' with respect the weights in each layer, the A (f) matrices and backpropa- gate. This algorithm accepts any first- order optimization method, denoted opt aptimiian zer (e.g., SGD (Robbins & Monro 195T). Adam (Kingma Ba 2014)), but projects the updated weight matrix into the positive orthant to main- tain nonnegativity.
Algorithm Forward Propagation
procedure FORWARDPROP (:) (A for = 1, do for = 0, _ do
k. )==0)==0)
S -- q( , ((1
see equation
Algorithm 2 Neural NCPD
For the NCPD task, the most natural loss function is the reconstruction loss,
Input: Tensor X R x Xnk cost C X1. X2..... X₃ f (X). initialize (4 for iterations 1..... T do
R.C-2 11=0,8=0
C= X X1, X2, XxIlle.
(11) In order to encourage optimal fit each layer, we also introduce loss function that we refer to as energy loss. First denote the approximation of X at layer of our network as
ForwardP rop ) (4 N,C--22 1=0,200) D Alg.
for = 1,..,,,. ,k, E = , - do
A + (optimizer( (A())
ac "aA"") b any first-order method
X:
|A| 0) A A
(t- ((--2)
A A !..A! (--)) S/(
(t-2)
A A
! ((-2) s((-2)
(1)
(
(12)
Then, we calculate energy loss as
L-2
X X2, Xxllle
X Xellr.
(13)
E=
The derivatives of AA, X) with respect A and X are derived and exploited to differentiate generic cost function for the hierarchical NMF model in Gao et al. (2019;; here we summarize these derivatives and illustrate how to combine them with simple multilinear algebra for HNCPD.
Gao et al. 2019) show that, if
ac
is the derivative of C' with respect to A holding the S
(4)
matrices constant, then
ac
aC
0A(1) (4)
=
A (41)
Σ
U
(14)
B5555-2 IS/ST
where (lll. relates C' to (() through S and (((,) defined column-wise (j), and depends
ac (s((f))
the derivative of C' with respect to s (ta)) holding s( (((2+ ((C-)) constant.
upon
The definition of U.! (f) given in Gao et al. (019) and utilizes, via the chain-rule, the partial
derivative of s - for all = [2.
Example. The derivative of the previously defined, or other differentiable cost functions can be calculated using these results of [ao et al. (2019) and some simple multi-linear algebra. As example, we directly compute the backpropagation step for the reconstruction loss function C' given in equation
Let X()) be the mode-i matricized version X, and define
H, X
X
X 1 X1,
(15)

---

Under review as a conference paper at ICLR 2021
where 3 denotes the Khatri-Ras product (see e.g., (Kolda & Bader 2009)). Then we have that
00
(A!
0 (2)
........--11)
(x(() 1
X.H, H: (A(mm) ([-2)s((-29)
2
0A!(1)
(16)
and
ac
A (0) A ......A
..) (X() X,H,-
(17)
a5!(0)
These derivatives are justified in Appendix A With equation 14 these derivatives are sufficient to calculate the partial derivative of C' with respect to any A matrix.
3 EXPERIMENTAL. RESULTS
We test Neural NCPD on three datasets: one synthetic, one video, and one collected from Twitter. The synthetic dataset is constructed as a simple block tensor with hierarchical structure. The Twitter dataset consists of tweets from political candidates during the 2016 United States presidential election (Littman al., 2016). We pull the video, time-lapse of forest over the span of one year, from (Solheim). We also compare Neural NCPD to Standard NCPD, in which we perform an independent NCPD decomposition at each rank, and to Standard HNCPD, which we perform NCPD first the full dataset, and apply HNMF to the fixed factor matrices; here we sequentially apply NMF to the factor matrices using multiplicative updates and do not update previous layer factorizations as in Neural NCPD. In all experiments, we use Tensorly (Kossaffi et al. (2118) for Standard NCPD calculations and initialize the NCPD layer of our hierarchical NCPD, and in Neural NCPD we do not backpropagate to this layer the initialization has usually found stationary point. We use Energy Loss (Eq. 12) for all experiments to encourage fit at every layer. Because we do not backpropagage to the initial factor matrices, the first term in (Eq. 12) is fixed. For the Twitter and video experiments
the approximation scheme of Section 2. to recover the relationship
between the columns of the A matrices and visualize the A! matrices.
3.1 EXPERIMENT ON SYNTHETIC DATA
We test the Neural NCPD algorithm first using a synthetic dataset. This dataset is a rank seven tensor of size 40 x 40 40 with positive noise added to each entry; we generate noise as = [g] where g (O, o) To generate this dataset, we begin with the all-zeros tensor and create three large nonoverlapping blocks with value I, and then overlay each block with either two or three additional blocks with value 3. We display this tensor with two levels of noise at the left of Figure 2 here we plot projections of all tensors (and all approximations) along the third mode; that is, we construct matrix with entries equal to the largest entries of the mode-three fibers (see e.g., (Kolda & Bader 2009) for relevant definitions). The projections on the remaining two modes are included in the Appendix A and are all similar to the third mode.
Table Relative reconstruction loss, Crei, on a synthetic dataset for Neural NCPD, Standard HNCPD, and HNTF with two different levels of noise. We list the loss of the approximation (1) 3. The results of HNTF are similar for all orderings of the modes, so we list only one.
o² = 0.05
o² = 0.5
Method
r= 7
(0) = 5
y()) = 3
== 7
(0))) 5
r(1) 3
Neural NCPD
0.091
0.229
0.467
0.390
0.438
0.490
Standard HNCPD
0.091
0.525
0.674
0.390
0.441
0.643
HNTF
0.091
0.234
0.539
0.390
0.450
0.578
We run Neural NCPD, Standard HNCPD, and Chichocki et al. on this synthetic dataset at two different levels of noise with three layers of ranks 5, and 3, and display the results in Figure 2
and Table 1 we present the relative reconstruction loss Ciu = X - [X,, X2, ..... X&Ill=/112 |||||| F. For each level of noise, display the rank approximation shared by all methods, and the rank

---

Under review as a conference paper at ICLR 2021
Neural NCPD
Standard HNCPD
.5 (0)
Sy (1)
Original
Rank
Rank
Rank
Rank
High
8 0.05
Low
Figure 2: Data tensor X with two levels of noise (left), ranks 7, 5, and 3 Neural NCPD and Standard HNCPD approximations of X (middle), transposed Neural NCPD S and s(1) matrices (right).
and rank approximations produced by Neural NCPD and Standard HNCPD. We also display the transposed Neural NCPD Sg and S/ matrices, which show how rank 7 topics collect into ranks and 3 topics. From Table we see that the loss for Neural NCPD is at or below that of Standard HNCPD and HNTF at each rank and level of noise.
3.2 TEMPORAL DOCUMENT ANALYSIS
We next apply Neural NCPD dataset of tweets from four Republican [R] and four Democratic
[D] 2016 presidential primary candidates, (1) Hillary Clinton [D], (2) Tim Kaine [D), (3) Martin
O' Malley [D). (4) Bernie Sanders [D), (5) Ted Cruz [R], (6) John Kasich [R]. (7) Marco Rubio [R].
and (8) Donald Trump [R]: this is constructed from a subset of the dataset of Littman al. (2016).
We seg-of-worde (12,721 words in corpus) representation of all tweets made by candidate
within bins of 30 days (from February to December 2016), and cap each of these groups at 100 tweets
to avoid oversampling from any candidate; resulting a tensor of size 10 x 12721.
Rank Topics
Rank 4 Topics
Rank Topics
Topic
Topie
Tople
Topic 4
Topic
Topic
Topic
trump
senate
martinomalley
berniesandors
marcorabio
trump
trmm
hillary
hillaryclinton
people
teammarco
hillary
hillary
donald
zika
realdonaldrump
bernie
vote
donald
vote
president
venezucla
campnigning
must
ilsen
people
peuple
timkaine
nicolasmaduro
mary land
change
click
donald
Topic
Topic 6
Topie
Topic 8
Topic 3
Tupic 4
Tople
tederuz
johnkusäch
marcorubio
crooked
tederuz
senate
tederruz
kasich
hillary
florida
craz
ted
olio
vote
thank
led
zika
ted
john
great
johnkusich kasich
venezuela
johnkasich
choosecruz
click
elinton
nicolasmaduro
kasich
Hlggh
teb
reb
hD
oftates
cane
May
Mai
Apr
Halley
Kains
Vay
Apr
teire
Pay
KW
Apr
Ju
Jun
Jan
Cnor
Criy
jal
Cur
Aug
Aug
kisich
Aug
Sea
Rub
3umz
Figure 3: A three-layer Neural NCPD on the Twitter dataset at ranks r 8, (0) = 4 and (0) 2. At each rank, we display the top keywords and topic heatmaps for candidate and temporal modes.
In Table 2 we display the relative reconstruction loss on the Twitter political dataset for all models. We see that Neural NCPD significantly outperforms Standard HNCPD, slightly outperforms Standard NCPD while offering hierarchical topic structure, and outperforms all HNTF-i, for which loss varies significantly based on the arrangement of the tensor. In Figure 3 we show the topic keywords and factor matrices of a rank 8, 4, and 2 hierarchical NCPD approximation computed by Neural NCPD. Note that in the rank candidates mode factor and keywords we see that nearly every topic is identified with single candidate. Topic two of the rank 8 approximation aligns with
7

---

Under review as a conference paper at ICLR 2021
Table 2: Relative reconstruction loss, Cral, on the Twitter political dataset for Neural NCPD, Standard NCPD, Standard HNCPD, and HNTF at ranks r = r()) 4, and r()) = 8. For HNTF we display the loss given the three possible arrangements of the tensor.
Hlih
Rank 4
Method
r 8
(())
((1)
Neural NCPD
0.834
0.883
0.918
Renk
Low
6
7
8
Standard NCPD
0.834
0.889
0.919
Rank 8 Topics
Standard HNCPD
0.834
0.931
0.950
HNTF-1
0.834
0.890
0.927
Figure 4: The s (top) and S (bottom) matrices produced by Neu- ral NCPD on the Twitter dataset.
HNTF-2
0.834
0.909
0.956
HNTF-3 0.834
0.895
0.942
political issues (the Zika virus and the Venezuelan government) rather than single candidate, and temporally most present in May to July 2016 (during the Zika outbreak and the Venezualan state
of emergency). Topics one and eight, corresponding to candidates Clinton and Trump, are most present in the months immediately leading up to the election. At rank we see that topics one and four inherited from the rank 8 approximation, topic two combines the rank 8 topics of candidates Trump and Clinton (final candidates), and topic 3 combined the topies of candidates Cruz and Kasich (Republi- cans). Meanwhile, the rank 2 NCPD topics are nearly identical to rank 4 NCPD topics two and three. We display HNTF for each ordering of the tensor modes in Appendix A
Rank Topics
Rank Topics
Topic
Topic
Topic
cnump
berniesanders
tramp
hillary
people
hillary
jolukasich
banic
vote
people
kasich
vote
doaald
Topic
Tupic
Topic
crooked
tederuz
tederuz
In Figure we display the 500) (top) and 5/²1) (bottom) matri- ces produced by Neural NCPD on the Twitter dataset, which illustrate how topics collect at each rank. We see topics 5 and 6 from the rank 8 factorization combine to form topic 3 at rank 4 and topic 2 at rank 2. This is expected because both topics in- clude keywords from Cruz and Kasich, who had high presence in topies 5 and 6 respectively in the rank 8 factorization.
thank
ted
red
marcomabia
internet
johnkasich
great
chooseenuz
kasich
Chion
Clroon
Kanw
Kaiw
Mis
May
Jm
Cup
Chuc
12sch
56c
Keach
D-1
Nov
In Figure we display the results of performing separate NCPD decomposition of ranks 4 and on the Twitter dataset. We that the results are similar those of Neural NCPD, but these independent decompositions lack the clear hierarchical structure provided by Neural NCPD. Note that while the topics corresponding Kasich and Clinton combine the rank NCPD, these candidates present in different topics in the rank 2 NCPD; Neural NCPD prevents this breach of hierarchy.
Figure 5: Standard ranks 4 and NCPD of the Twitter dataset. At each rank, display the top five keywords and candidate and tempo- ral mode heatmaps.
3.3 VIDEO DATA ANALYSIS
We next apply Neural NCPD video data constructed from year-long time-lapse video of a forest; see Figure 6 for a selection of frames and Figure IT in Appendix A for more details. We extract 37 frames and flatten each frame (RGB image) into a single matrix, to form a tensor of size
37 3 57600; here the first mode represents frames (temporal mode), the second colors (chromatic mode), and the third pixels (spatial mode).
Frame
Frame
Frame 13
Frame
Frame
Frame 31
Frame
Figure 6: We display seven of 37 extracted frames from year-long time-lapse video of forest.
8

---

Under review as a conference paper at ICLR 2021
In Figure 7 show the three-layer Neural NCPD decompositions of the video tensor with ranks 8, 6, and 3. For each rank, we plot the topics in the spatial (left), temporal (top right), and chromatic (bottom right) modes. We note that many of the identified topies represent visual seasonal changes. Topic six of the rank 8 decomposition represents the green and leafy late-summer to early fall. Topic one of the rank 6 decomposition represents the winter sky and leafless trees. Topic three of the rank 3 decomposition represents the summer and fall sky and tree leaves.
Rank 8 Topics
Tapi
Ttpic
Epp
High
Itanns
20
Tbac
T
Epc
Toal:
30
Clee
Rank Topics
Rank Topics
Rank Topics
Topec 3
Topie
Tope
Toolc
10
10
Fraines
20
Teck-
20
30
400
Tell:
Red
Gren
Rank Topics
Rank Topics
Figure 7: A three-layer Neural NCPD of the time-lapse video ranks 8, ro = and r: We display topics each rank for spatial (left), temporal (top right), and chromatic (bottom right) modes. Relative reconstruction loss is 0.105, 0.109, and 0. 122 respectively at each layer.
We additionally apply NMF the slices of the tensor along single mode. Slicing along the temporal spatial modes would make interpretation of the resulting topics challenging, slice along the chromatic mode, producing three matrices. In Figure we visualize a rank 3 NMF
choose to
on each of the three chromatic slices the video tensor. The chromatic factoriza- tions are nearly identical, illustrating lit- tle salient dynamic information. While similar to the rank Neural NCPD layer, the chromatic NMFs obseure much of the chromatic interaction evidenced by Neu- ral NCPD. particular, Neural NCPD il- lustrates the spatial and temporal dynam- ies of multi-colored features and their co- occurrence hierarchy, while NMF provides only single-colored features and requires far more work to glean multi-cole nulti-colored fea- ture co-occurrence information.
Tooic
Topic
Topie
Red
10
:0
20
Frames
Green
20
20
25
25
Blue
30
35
35
123
123
Figure 8: A decomposition of the time-lapse video by rank NMF on slices of the tensor along the chromatic mode. For each color, display the three topics in spatial (left) and temporal (right) modes. Relative re- construction loss 0.101.
4 CONCLUSIONS
In this paper, we introduced the hierarchi- cal NCPD model and presented novel algorithm, Neural NCPD, train this decomposition. We empirically demonstrate the promise of this method on both real and synthetic datasets; in particular, this model reveals hierarchy topics learned different NCPD ranks, which is not available to standard NCPD or based approaches.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ali Anaissi, Basem Suleiman, and Seid Miad Zandavi. NeCPD: An online tensor decomposition with optimal stochastic gradient descent. arXiv preprint WwXiv:2003.08844 2020.
J. Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an N-way generalization "Eckart- "Eckart-Young" decomposition. Psychometrika, 35(3):283-319 1970.
Andrzej Cichocki, Rafal Zdunek, and Shun-ichi Amari. Hierarchical als algorithms for nonnegative matrix and 3d tensor factorization. In International Conference on Independent Component Analysis and Signal Separation, pp. 169-176. Springer, 2007a.
Andrzej Cichocki, Rafal Zdunek, and Shun-ichi Amari. Nonnegative matrix and tensor factorization [lecture notes]. IEEE Signal Proc. Mag., 25(1):142-145) 20070.
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative matrix and fensor factorizations: applications to exploratory mulfi-way data analysis and blind source separation. John Wiley & Sons, 2009.
Jennifer Flenner and Blake Hunter. A deep non-negative matrix factorization neural network, 2018. Unpublished.
Mengdi Gao, Jamie Haddock, Denali Molitor, Deanna Needell, Eli Sadovnik, Tyler Will, and Runyu Zhang. Neural nonnegative matrix factorization for hierarchical multilayer topic modeling. In Proc. Interational Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2019.
Eric Gaussier and Cyril Goutte. Relation between PLSA and NMF and implications. In Proc. 28th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 601-602, 2005.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Lars Grasedyck. Hierarchical singular value decomposition tensors. SIAM J. Matrix Anal. A., 31 (4):2029-2054, 2010.
David Guillamet and Jordi Vitria. Non-negative matrix factorization for face recognition. Proc. Catalonian Conference on Artificial Intelligence, pp. 336-344. Springer, 2002.
Richard A. Harshman et al. Foundations the PARAFAC procedure: Models and conditions for an "explanatory" multimodal factor analysis. 1970.
Patrik O. Hoyer. Non-negative sparse coding. In Proc. 12th IEEE Workshop on Neural Networks for Signal Processing, pp. 557-565. IEEE, 2002.
Dongmin Kim, Suvrit Sra, and Inderjit S. Dhillon. Fast projection- methods for the least squares nonnegative matrix approximation problem. Stat. Anal. Data Min., 1(1):38 1-,3, 2008.
Hyunsoo Kim and Haesun Park. Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM Matrix Anal. A., 30(2):713-730, 2008.
Jingu Kim, Yunlong He, and Haesun Park. Algorithms for nonnegative matrix and tensor factor- izations: A unified view based on block coordinate descent framework. Global Optim., 58(2): 285-319, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Tamara Kolda and Brett Bader. Tensor decompositions and applications. SIAM Rev., 51(3): 455-500, 2009.
10

---

Under review as a conference paper at ICLR 2021
Tamara G. Kolda and David Hong. Stochastic gradients for large-scale tensor decomposition. arXiv preprint arXiv:1906.01657 2019.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. TensorLy: Tensor learning in Python. CoRR, abs/1610.00555, 2018.
Charles L. Lawson and Richard J. Hanson. Solving least squares problems. SIAM, 1995.
Jonathan Le Roux, John Hershey, and Felix Weninger. Deep NMF for speech separation. In Proc. Int. Conf. Acoust. Spee, pp. 66-76 IEEE, 2015.
Daniel D. Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788 791, 1999.
Daniel Lee and H. Sebastian Seung. Algorithms for non-negative matrix factorization. In Proc. Adv. Neur. In. pp. 556-562, 2001.
Hyekyoung Lee, Jiho Yoo, and Seungjin Choi. Semi-supervised nonnegative matrix factorization. IEEE Signal Proc. Let., 17(1):4-7, 2009.
Chih-Jen Lin. Projected gradient methods for nonnegative matrix factorization. Neural Comput., 19 (00:2756-22799 2007.
Justin Littman, Laura Wrubel, and Daniel Kerchner. 2016 United States Presidential Election Tweet Ids, 2016. URL httphhh////oo... wttp:///...///////////..
Herbert Robbins and Sutton Monro. A stochastic approximation method. Ann. Math. Stat., pp- 400-407, 1951.
Ankan Saha and Vikas Sindhwani. Learning evolving and emerging topies in social media: a dynamic NMF approach with temporal regularization. In Proc. ACM International Conference Web Search and Data Mining, pp. 693-702, 2012.
Farial Shahnaz, Michael W Berry, V Paul Pauca, and Robert J Plemmons. Document clustering using nonnegative matrix factorization. Information Processing & Management, 42(2):373-33 2006.
Eirik Solheim. One year in 40 seconds. URL httt:::/www.youuueecoo.... www.yy/www..../wwctcc...c.e.
1mIFXIX0O haaabacchmme--iikiklllllee
Le Song, Mariya Ishteva, Ankur Parikh, Eric Xing, and Haesun Park. Hierarchical tensor decomposi- tion of latent tree graphical models. In Proc. Int. Conf. Mach. Learn., pp. 334 342, 2013.
Xiaoxia Sun, Nasser M. Nasrabadi, and Trac D. Tran. Supervised multilayer sparse coding networks for image classification. CoRR,
2017. URL http://arxivvooo///bt
1701.0833 08349
Abraham Traoré, Maxime Berar, and Alain Rakotomamonjy. Non-negative tensor dictionary learning. 2018.
George Trigeorgis, Konstantinos Bousmalis, Stefanos Zafeiriou, and Björ W. Schuller. A deep matrix factorization method for learning attribute representations. IEEE T. Pattern Anal., 39(3): 417-429, 2016.
M Alex O Vasilescu and Eric Kim. Compositional hierarchical tensor factorization: Representing hierarchical intrinsic and extrinsic causal factors. arXiv preprint arXiv: 1.04//0/ 2019.
Max Welling and Markus Weber. Positive tensor factorization. Pattern Recogn. Lett., 22(12): 1255-1266 1261, 2001.
Wei Xu, Xin Liu, and Yihong Gong. Document clustering based on non-negative matrix factorization. In Proc, ACM SIGIR Conference on Research and Development Informaion Retrieval, pp- 267-273, 2003.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
In this supplementary material, we provide the details of the example derivative computations from Section 2.2 give a justification the NCPD expansion formula exploited Section 2.1 and provide further experimental results that not able include in Section 3
EXAMPLE DERIVATIONS
Here, we justify the derivations provided in the example in Section 2.2 We note that Anaissi et al. (2020); Kolda & Hong (2019) provide similar derivations for the CP tensor decomposition, but their decompositions do not attempt to further decompose the CP factor matrices, and thus, their results are not sufficient for providing derivatives with respect to the and matrices. Consider the full reconstruction loss function for the order-k tensor X,
C = X [X1, X X
aC
where for some fixed 1 S S k; X; ABC and consider the gradient
Let
aB
H; X
@ Xit1 @ X;-1 .... o X1
Now, we let
[ x,...., x&). Then, if X(1) is the mode-í matricization of X (see e.g., Kolda
Bader 2009
we have
( X. ..... X O X ....
Thus, we let X(a) (5) be the mode-i matricization X, have that
C = X
X (i) (ABC) C X
o x
Now, we compute the desired gradient through series of applications of the chain rule. We then see that
ac a0 T. ÒABC
0B H. 9ABC ÓB A aC OXHH, )
OX 0X;
2A T (X(0) - X.H H,C
ac
Now, using the calculations above we can proceed in calculating
Gao (019 show that
if
ac (A we have
denotes derivative of C with respect (()),
holding the S matrices constant, then
S
OC
Oc
(A (G)
SSBSC-2 ISJ5r
relates C ((1)) pends upon
where (lfi
(6)) through S (f2) and S is defined column-wise (j), and de-
ac (2)
the derivative of C' with respect to s(((a) holding s ((fa+1)
((C-2)
S
constant. Thus,
and
are sufficient to calculate
AA T We calculate the
80
A 7
s
gradient
ac ?
where
04
C = X - X X
X
12

---

Under review as a conference paper at ICLR 2021
and X; A A,A (A (c-)) s((-))
Since assume that A A is independent of all other
A's and S's, we have that
0C = ! A 11)
......((--1) 1 (x ).H. H. (A (m)))) ....A((2) !A.......! s(c-2))
0A(41)
(18)
Now, we calculate
8c 8s!'')
Since can assume that 5/60 is independent of all other A 's and
S's, we have that
ac
2 (A (...4.
!!1i) (X()
x.H, H,
(19)
as!
Thus, have the required derivatives to evaluate
ac dA, (())"
HNCPD EXPANSION
We now provide brief justification of the expansion of the NCPD in terms of later factorizations used in Section 2.1 that is,
X X2, Xa1=
Σ
Jai (AA )iss (A!")..) )ijo @ (A ):.
Jih Sr:00
where O'ji.a.....h Ep=1 (S 00)
(S 12. (S
(0)
00) 'je.
We have that by definition,
XX X2, .2. X = Σ (Xi).p [((i)) @ ( .... (x.))))....(
(Xn)..).
p=1
We also have that X; A ( s(0) for sis ki we have that for each column p, 1. p of
X,
(0)
X
Σ
S
00
)in
(A,
)..--
j=1
Thus, by the linearity of the outer product have that
(X1)s. @ (22).pp @ ... (Xa).p)
Σ
ap.iwaa.. (1A @ ( 2 ... & (
where Op (5 ) (S ) M1p.... (S (0) Din.p- Now, by noting that
a da
apai
p=1
we arrive at the original statement.
SYNTHETIC EXPERIMENT
In this section, we provide the additional views the synthetic tensor and computed approximations from Section 3.1 In Figure the main text, for visualization we displayed the projection of each tensor onto the third mode. In Figure 9 we display the projections of these tensors onto all three modes. We see that due to the simple block structure used to produce the synthetic data tensor, the three modes all tell a similar story; that is, Neural NCPD able recover meaningful structure along all three modes.
13

---

Under review as a conference paper at ICLR 2021
Original Mode
Rank 7
Mode
Mode
Mode
Mode 2
Mode
High
0.05
=
0.5
Low
Neural NCPD
Rank 5
Rank Mode
Mode
Mode 2
Mode
Mode
Mode 3
0.05
0.5
Standard HNCPD
Rank 5
Rank Mode 2
Mode
Mode 2
Mode
Mode
Mode
à = 0.05
0.5
HNTF
Rank 5
Rank Mode 2
Mode
Mode 2
Mode
Mode
Mode
0.05
0.5
Figure 9: Here we display the projections onto all three modes for the original data tensor X and approximations of ranks r = 7, p0)) = 5, and p(() produced by Neural NCPD, Standard
HNCPD, and HNTF two levels of noise.
14

---

Under review as a conference paper at ICLR 2021
TEMPORAL DOCUMENT ANALYSIS EXPERIMENT
HNTF-1
Topic
Topic 2
Topic I
billary
tederuz
Clinton
tramp hillary
Clinton
Feb
trump
Kaine
Mar
Kaine
Mar
realdomaldirump
led
Aar
tederuz
Apr
crooked
internet
O'Malley
May
people
O'Malley
May
hillaryclinton
chouseruz
Jun
bemiesanndrrs
Sanders
Jun
Topic
Topic 4
Cruz
Jul
Topic 2
Cruz
Jul
marcorubio
Icammarco
berniesanders
people
Kasich
Aug
marcorabio
teammarco vote flsen click
Aug
Sep
vote
bemie
Rubio
Rubio
Oct
flsen
Trump
Nov
Trump
Nov
click
change
3
HNTF-2
Topic
Topic
Topic marcorubio
johnkasich
marcorubio
Clinton
Feb
Clirton
Feb
kasich
leammarco
Kaine
Kaine
Mar
obio
vote
O'Malley
Apr
vote
O'Mailey
john
florida
florida
May
gov
Ilsen
Sanders
fisen
Sanders
Jun
Topic
Topic 4
Jul
Topic
Cruz
tederuz
herniesanders
Kasich
Aug
berniesanders
Kasich
Aug
cruz
people
bernie
Rubin
Sep
Oct
people bemie must
Rubia
Sep
Oct
internet
Trump
Nov
Trump
Nov
choosecruz
change
change
HNTF-3
Topic
Topic marcorubio teammarco
Topic tederuz cruz
trump
Clinton
Feb
Clinton
Feb
president
dunald
hillary
flsen
vote
O'Malley
Kaine
ted hillary crooked
O'Malley
Kaine
May
Mar
Apr
timkaine
click
Sanders
Jun
Sanders
Jun
Topic 3
Tople 4
Topic
Cruz
berniesanders
tederuz
Kasich
Aug
berniesundeess
Kasich
Aug
people
eruz
Sep
people
Sep
bernie
ted
Rublo
Oct
bemnie
Rubii
Oct
crooked
Trump
Now
must
Trump
Nov
country
thank
Figure 10: Here we display a three-layer HNTF on the Twitter dataset from Section 3.2 at ranks
(0) 4, and y((1) 2, run separately for each of the possible ordering on the data tensor. We display the top keywords and heatmaps of topics in the candidate and temporal modes at ranks 4 (left) and 2 (right). We note that the rank 8 factorization identical to that of Neural NCPD, so we do not re-display it here (see Section? 3.2
display the results from running HNTF the Twitter dataset in Section 3.2 excluding the topics at rank because they are identical to those learned by Neural NCPD (see Section 3.2 We see that while the factorization for the first possible ordering similar to that of Neural NCPD and contains significant meaningful topic modeling information, the other two orderings lose significant information by the last layer and, and have topic presence and from only 2 or of the eight candidates.
In Figure 10
15

---

Under review as a conference paper at ICLR 2021
VIDEO DATA EXPERIMENT
Frame
Frame
Frame
Frame
Frame 5
Frame
Frame 13
Frame
Frame
Frame
Frame 17
Frame 18
Frame 19
Frame 20
Frame 21
Frame 22
Frame 23
Frame 24
Frame 25
Frame 26
Frame 27
Frame 28
Frame 29
Frame 30
Frame 31
Frame
Frame
Frame
Frame
Figure 11: Here display the first 36 of 37 frames of the time lapse video dataset from Section 3.3
(The 37th frames is included in Figure 6
display the first 36 of 37 frames of the time lapse video dataset from Section 3.3 (the 37th frame is included in Figure (6) in order to make it clear how seasons progress throughout the frames. We see that the video begins the white winter months, transitions to spring at around frame 16, and stays green until it transitions to fall around frame 28.
In Figure 11
In Figure 12 we display the ( matrix (top) and S/ matrix (bottom) produced by Neural NCPD on the time-lapse video tensor described in Section 3.3 By examining the S matrices from our also able to see the hierarchical relationship between the topics Neural NCPD algorithm, s(0) matrix, we see the hierarchical relationship between the rank and from different ranks. In the
rank 8 topics. In the 5 matrix, see the hierarchical relationship between the rank 3 and rank 8 topics. We note the ((() matrix (top) illustrates that topic one of rank 6 NCPD is closely related to topic eight of rank NCPD, and S( (bottom) similarly illustrates that topic two of rank 3 NCPD is closely related to topic eight in rank NCPD; these relationships are unsurprising because, as seen Figure in the main text, these topics are present temporally during winter and fall spatially in the sky behind the trees.
16

---

Under review as a conference paper at ICLR 2021
1
Rank 6 Topics
3
4
5
6
1
Rank 3 Topies
1
2
3
1
2
3
4
Rank Topics
Figure 12: The ( matrix (top) and S matrix (bottom) produced by Neural NCPD on the time-lapse video tensor described Section 3.3
17


Reviews:
[{"review_id": "v4AVLiYSJXi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Short summary:\n\nThe paper introduces a promising new method, hierarchical nonnegative CP decomposition (HNCPD), as well as a training method for the HNCPD, neural NCPD, for topic modeling problems. After presenting some fundamentals on the topic, as well as some related work, a detailed and formal mathematical description of the suggested method is given. Finally, results of experiments on three data sets (synthetic data, image data and text data) are utilized to evaluate the suggested methods.\n\nPotential Improvements:\n- Equation (4) introduces the forward propagation for a NNMF, which is crucial for the HNCPD and Neural NCPD. However, the method of determining \n\n$\\argmin\\ limits_{\\boldsymbol{S}}$\n\n is not introduced. Further, it is not shown how the derivative of the argmin function with respect to \n\n$\\boldsymbol{A}^{(l)}$\n\n is formed, which is later on used for the backpropagtion.\n\n- The introduction mentions the importance of topic modeling given large amounts of data, as well as multi-modal tensor data. However, the present experiments only utilise rather small uni-modal data sets (for instance a selection of 37 forest images). It would therefore be of value to answer 1) how the experiments would scale to higher dimensions (for instance for images of the forest time laps) and 2) how the suggested approach would work on a multi-modal database.\n\n- The analysis of experiments in section 3.2 and 3.3 is rather qualitatively than quantitatively. A quantitative measure like the reconstruction loss would most likely be of value.- Sections 3.2 and 3.3 also lack a comparison to other methods.\n\n- The conclusions section could include a short summary of future work on this novel approach.\n\n\nMinor Comments:\n- The for-loop over the two variables i and l in Algorithm 1 could be expressed as two for-loops to be more precise.\n- In section 3: presidential election --> United States presidential election\n- In section 3: combines the the rank 8 --> combines the rank 8 \n- In section 3: we display the the results --> we display the results\n\nOverall Evaluation:\nThe paper is well and clearly written, the significance of this work, however, needs to be further proven by experiments. The originality seems moderate, as already existing concepts of neural NMF and Hierarchical NMF are applied to NCPD.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vn_oLejGtUy", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\n\nThis paper presents a hierarchical nonnegative CP tensor decomposition method. It also proposes a training method that leverages forward and backward propagation. The method is tested on both synthetic and real datasets. These experiments illustrate how the method can be used to discover topics and how they vary over time. The hierarchical nature of the method makes it possible to group the topics into supertopics in multiple steps.\n\nI thinks the paper is well-written for the most part. The topic is interesting and should be of interest to the ICLR community. It seems like the authors combine existing ideas (hierarchical and neural NMF, NCPD) into a new method. While this may be somewhat incremental, it seems like the authors are doing something that hasn't been done before. My main issue with the paper is that there are a few portions that are currently difficult to follow. In particular, in the discussion on approximation in Section 2.1, it is not clear if this idea is used in the implementation, which might make it difficult to replicate the results. Fixing this would help improve the quality of the paper. I provide more details on which parts are unclear below.\n\n\nADVANTAGES:\n\n- Introduction provides a clear overview of what has been done before. It is easy to see where the present paper fits in with the existing literature. It also does a good job of connecting to applications.\n\n- The experiments in Sections 3.2 and 3.3 are convincing and nicely explained, especially the illustrations of how topics vary over time. The last sentence in Section 3.3 help explain the benefit of a hierarchical method.\n\n- Well-written for the most part. \n\n- Should be of interest to ICLR community.\n\n\nCONCERNS/QUESTIONS:\n\n- In the last sentence on page 2, it is not clear which loss function is used for Neural NMF when doing the backpropagation. Is it the expression in the Frobenius norm in the paragraph titled \"Hierarchical NMF (HNMF)\" above?\n\n- In Section 2.1, I follow the discussion up to Equation (6). But the rest of the first paragraph is difficult to follow. For example, it's not clear to me how the columns of the hierarchical NMF factors are used to form NCPDs of ranks $r^{(0)}$, $r^{(1)}$, ..., $r^{(L-2)}$. The subsequent discussion about the dependencies and indices is also unclear. Perhaps this discussion could be made more clear with the help of an example?\n\n- In Section 2.1, I think the second paragraph is clear until Equation (7), but the rest of the paragraph is difficult to follow. The discussion about approximating the relationship between the columns of $A_i^{(0)}$ and $A_k^{(0)}$ is unclear. What do you mean by \"relationship\"? In what sense is $(W_i)_{p_1, p_2}$ approximating it? It is not clear why the definition in Equation (9) is chosen. Is the idea that $[[ \\tilde{A}_1^{(0)}, \\ldots, \\tilde{A}_k^{(0)} ]] \\approx [[ \\tilde{X}_1, \\ldots, \\tilde{X}_k$ ]]? In the experiments, do you use this approximation, or do you use the NCPD combined with HNMF for each factor matrix as discussed in the beginning of Section 2.1? \n\n- In the last sentence of Section 2.1, you mention that \"Neural NCPD allows factor matrices for all other modes to influence the factorization of a given mode.\" It makes sense that this is the case when all parameters, including the initial NCPD, are trained together via backpropagation. But does this still remain true when the initial NCPD is computed independently at first and then kept fixed throughout the fitting of the HNMFs, like in Algorithm 2? As far as I can tell, the approach in Algorithm 2 of keeping the initial NCPD fixed through the training is also what you do in the experiments.\n\n- In the experiments, do you use a combination of the loss functions in Equations (11) and (13) (e.g., C+E), or just one of them? This is not clear from the discussion. Also, since the NCPD is computed by itself at the start of the algorithm and then kept fixed, isn't the term $|| X - [[ X_1, \\ldots, X_k ]] ||_F$ in (13) a constant that could be ignored?\n\n- For the discussion in Section 2.2 about the derivatives, it would be a good idea to let the reader know that there is a more in-depth explanation available in the appendix.\n\n- It is not clear how the Standard NCPD in Section 3 is computed. Could you perhaps add an explanation in the appendix or point to a relevant reference?\n\n- Below Figure 2, you say that $g \\sim N(z; 0, \\sigma^2)$. What does $z$ here mean?\n\n- For the experiment in Section 3.2, did you try keeping each frame as a matrix in the tensor? In other words, did you try reshaping the tensor into a 4-dimensional tensor of size 37 $\\times$ 3 $\\times$ (number of x pixels) $\\times$ (number of y pixels)? One motivation for using tensors is to avoid having to vectorize things like images, so it would be interesting to know if this also worked well. I don't think you need to change the current example, I'm just asking out of curiosity. \n\n\nMINOR CONCERNS/QUESTIONS:\n\n- In Figure 1, the colors are a nice addition, but they're quite muted which makes it hard to see them. Could they be made brighter? Also, should the red line along the columns of the $S_2^{(0)}$ matrix instead be along the rows of the $A_2^{(0)}$ matrix?\n\n- In Figure 1, in the caption, the tensor $X$ is not using the bold tensor notation.\n\n- Below Equation (7), in the definition of $\\alpha_{j_1, j_2, \\ldots, j_k}$, should the sum go to $r$ instead of $r^{(0)}$? It looks like it should based on the derivation in the appendix.\n\n- Equation (15) should end with a comma instead of a period since the sentence keeps going below.\n\n- In the second sentence of Section 3.1, the word \"size\" appears twice in a row.\n\n- At the bottom of page 7, should the tensor size be 8 x 100 x 12721 instead of 8 x 10 x 12721 since the number of tweets are capped at 100?\n\n- In the text to the left of Figure 7, the words \"the\" appears twice in a row in two places.\n\n- In the appendix, in the section \"HNCPD expansion\", in the 2nd sentence (starting with \"We have that by definition...\"), I think the square bracket \"]\" should be removed on the right hand side of the equation?\n\n#######################\n\nUpdate:\n\nI thank the reviewers for their responses. I appreciate the effort they put into clarifying the paper. However, I still think Section 2.1 in particular is difficult to follow. I will therefore keep my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QNyHwqLn7t", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nIn this paper, an extension of nonnegative CP decomposition called hierarchical nonnegative CP decomposition (HNCPD) is proposed. This method is designed to capture the hierarchical structure in e.g., topic modeling. Also, an optimization method called neural NCPD is proposed. Several decomposition results of synthetic data, video data, and Twitter data are presented.\n\nStrong points:\n- S1. A new algorithm is proposed.\n- S2. Experimental results with real data are reported.\n\nWeak points:\n- W1. The technical contributions are small.\n- W2. The motivation of HNCPD is unclear.\n- W3. The paper is not clearly written.\n\nMy recommendation: \nAlthough this paper contains some interesting ideas, I feel the overall quality is not high enough to be accepted in the ICLR community. Here I would like to elaborate on the reasons. \n\nW1. The formulation of HNCPD is not new. Cichocki et al. (2007) proposed a more general form of HNCPD; see Eq. (13) of the following paper. So the technical contribution of this paper is summarized to the development of neural NCPD, but it is a direct extension of Gao et al. (2019). \n````\nCichocki A., Zdunek R., Amari S. (2007) Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization. In: Davies M.E., James C.J., Abdallah S.A., Plumbley M.D. (eds) Independent Component Analysis and Signal Separation. ICA 2007. Lecture Notes in Computer Science\n```````\nW2. It is not clearly explained how HNCPD is beneficial to real-world applications. For example, the pictures in e.g. Fig. 4 seem to tell us some interesting information, but how can we use that? In topic modeling, what kind of knowledge/insights can we get from the results? And also, how can we quantify the benefits?\n\nW3. Some part is not easy to read. \n- In Section 2, $r$ is defined as the number of super topics. However, in Table 1, $r$ seems to be used as the number of subtopics, which is previously defined as $r_0, r_1, r_2$. A similar inconsistency is observed in Fig4, Fig6, etc.\n- The interpretation of experimental results is unclear. In Section 3.2, it is claimed that HNCPD is better than NMF because \"the chromatic NMFs obscure much of the chromatic interaction\". Here, the true structure of chromatic interaction is hidden and we cannot observe it. So in what sense can we show the superiority of HNCPD?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a hierarchical version of NMF for the CP decomposition of tensors.

The idea is similar to Chinocki etal 2007 and extends Gao etal 2019, and in Chinocki was presented for the standard linear formulation with regularisation terms.  The extension here doesn't use the standard ALS algorithm but rather presents a neural network analogue, though the functions are still linear, its just that back-prop etc. are used for the computation.  The authors point out their formulation is a more flexible representation and optimisation (in response to AnonReviewer5), and thus represents an improvement.  While this is an interesting implementation, in NNs, the model is still fairly simple.  

Moreover the experimental results are restricted to a few data sets.  There are literally hundreds of NMF variants in publication and many different evaluations are done.  The experimental work here, while showcasing the work, is not extensive.  For instance, more empirical comparisons should have been made against prior hierarchical NMF on a battery of data.

So this is good, publishable work, and the authors have repaired many of the issues raised by the reviewers.  The work, however, is borderline in empirical work and the contribution is not strong.
Decision: Reject