Title: Dynamic Tensor Decomposition via Neural Diffusion-Reaction Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tensor Decomposition

Representation Learning
Abstract: Tensor decomposition is an important tool for multiway data analysis. In practice, the data is often sparse yet associated with rich temporal information. Existing methods, however, often under-use the time information and ignore the structural knowledge within the sparsely observed tensor entries. To overcome these limitations and to better capture the underlying temporal structure, we propose Dynamic EMbedIngs fOr dynamic Tensor dEcomposition (DEMOTE). We develop a neural diffusion-reaction process to estimate dynamic embeddings for the entities in each tensor mode. Specifically, based on the observed tensor entries, we build a multi-partite graph to encode the correlation between the entities. We construct a graph diffusion process to co-evolve the embedding trajectories of the correlated entities and use a neural network to construct a reaction process for each individual entity. In this way, our model can capture both the commonalities and personalities during the evolution of the embeddings for different entities. We then use a neural network to model the entry value as a nonlinear function of the embedding trajectories. For model estimation, we combine ODE solvers to develop a stochastic mini-batch learning algorithm. We propose a stratified sampling method to balance the cost of processing each mini-batch so as to improve the overall efficiency. We show the advantage of our approach in both simulation studies and real-world applications. The code is available at https://github.com/wzhut/Dynamic-Tensor-Decomposition-via-Neural-Diffusion-Reaction-Processes.

Dynamic Tensor Decomposition via Neural Diffusion-Reaction Processes

Abstract

Tensor decomposition is an important tool for multiway data analysis. In practice, the data is often sparse yet associated with rich temporal information. Existing methods, however, often under-use the time information and ignore the structural knowledge within the sparsely observed tensor entries. To overcome these limitations and to better capture the underlying temporal structure, we propose Dynamic EMbedIngs fOr dynamic Tensor dEcomposition (DEMOTE). We develop a neural diffusion-reaction process to estimate dynamic embeddings for the entities in each tensor mode. Specifically, based on the observed tensor entries, we build a multi-partite graph to encode the correlation between the entities. We construct a graph diffusion process to co-evolve the embedding trajectories of the correlated entities and use a neural network to construct a reaction process for each individual entity. In this way, our model can capture both the commonalities and personalities during the evolution of the embeddings for different entities. We then use a neural network to model the entry value as a nonlinear function of the embedding trajectories. For model estimation, we combine ODE solvers to develop a stochastic mini-batch learning algorithm. We propose a stratified sampling method to balance the cost of processing each mini-batch so as to improve the overall efficiency. We show the advantage of our approach in both simulation study and real-world applications. The code is available at https://github.com/wzhut/Dynamic-Tensor-Decompositionvia-Neural-Diffusion-Reaction-Processes .

1 Introduction

Multiway data is common in real-world applications and naturally represented by tensors. For example, online shopping and promotion activities can be expressed as a three-mode tensor (customer, commodity, online merchant) . Tensor decomposition is an important tool for multiway data analysis. It estimates embeddings for the entities in each tensor mode, with which to recover the observed

entry values. The embeddings can reflect the underlying structures within the entities and can be used as predictive features, such as for recommendation and ads auction.

In practice, tensor data is often very sparse. That is, the observed entries only take a tiny portion of all possible entries, say, 0 . 01% . In addition, the data often includes timestamps for the observed entry values, which imply rich, complex temporal variation patterns. Current tensor decomposition approaches often ignore the structure knowledge within the sparsely observed entries and under-use the temporal information, e.g. , simply binning the timestamps into crude time steps (Xiong et al., 2010; Rogers et al., 2013; Zhe et al., 2016a, 2015; Du et al., 2018). More important, standard tensor decomposition estimates a static embedding for each entity. However, as the representation of entities, these embeddings summarize the underlying properties of the entities, which can naturally evolve with time, such as customer interests, user income, product popularity, and fashion. Learning static embeddings can miss capturing these interesting, important temporal knowledge. While the most recent work (Wang et al., 2022) has proposed the first decomposition method to estimate embedding trajectories, it never considers the structural knowledge within the data.

To overcome these limitations, we propose DEMOTE, a dynamic embedding approach for dynamic tensor decomposition. We construct a nonlinear diffusion-reaction process in an Ordinary Differential Equation (ODE) framework to estimate embedding trajectories for tensor entities. The ODE framework is known to be flexible and convenient to handle irregularly sampled timestamps and sparsely observed data (Rubanova et al., 2019). In addition, since ODE models focus on learning the dynamics ( i.e. , time derivatives) of the target function, they have promising potential for providing robust, accurate long-term predictions (via integration with the dynamics). Specifically, to leverage the structural knowledge within the data, we first build a multi-partite graph based on the observed entries. The graph encodes the correlations between entities at different modes in terms of their interactions. We then construct a graph diffusion process in the ODE to co-evolve the embedding trajectories of correlated entities. Next, we use a neural network to construct a reaction process to model the individual-specific evolution for each entity. In this way, our neural diffusion-reaction process captures both the commonalities and personalities of the entities in learning their dynamic embeddings. Given the embedding trajectories, we model the entry value as a latent function of the associated entities' trajectories. We use another neural network to flexibly estimate the function and to capture the complex relationships of the entities. For efficient training, we base on ODE solvers to develop a stochastic mini-batch learning algorithm. We develop a stratified sampling scheme, which can balance the cost of executing the ODE solvers in each mini-batch so as to improve the efficiency.

We evaluated our method in both simulation and real-world applications. The simulation experiments show that DEMOTE can successfully capture the underlying dynamics of the entities from their temporal interactions and recover the hidden clustering structures within the trajectories. Then in three real-world applications, we tested the accuracy in predicting the tensor entry values at different time points. DEMOTE consistently outperforms the state-of-the-art decomposition methods that incorporate temporal information, often by a large margin. We also demonstrated that both the diffusion and reaction processes contribute to the learning performance. Finally, we investigated the learned embedding trajectories and found interesting evolution paths and hidden structures.

2 Notations and Background

Suppose we have collected data for a K -mode tensor. Each mode k includes d k entities, which we index by 1 , . . . , d k . We then index each tensor entry by a tuple glyph[lscript] = ( l 1 , . . . , l K ) where for each k , we have 1 ≤ l k ≤ d k . Suppose we observed N tensor entry values and timestamps. The dataset is denoted by D = { ( glyph[lscript] 1 , t 1 , y 1 ) , . . . , ( glyph[lscript] N , t N , y N ) } where { t n } and { y n } are the timestamps and entry values, respectively. Our goal is for each entity j of mode k , to estimate a dynamic embedding u k j ( t ) : R + → R R . That is, the embedding is a time function (trajectory) with an R -dimensional output. Standard tensor decomposition introduces a static embedding representation for each entity, namely, u k j is considered as time invariant. Tensor decomposition aims to estimate the embeddings (or factors) to reconstruct the tensor. For example, the classical Tucker decomposition (Tucker, 1966) employs a multilinear factorization model, M = W× 1 U 1 × 2 . . . × K U K , where M∈ R d 1 × ... × d k is the entire tensor, W ∈ R R 1 ×···× R K is the tensor-core parameter, U k comprises all the embeddings of the entities in mode k , and × k is the tensor-matrix product at mode k (Kolda, 2006). The popular CANDECOMP/PARAFAC (CP) decomposition (Harshman, 1970) can be viewed as a simplified version of Tucker decomposition, where we set R 1 = . . . = R K = R and the tensor-core W to be

…

…

…

…

…

…

𝒖𝒊

𝟏

(t)

𝒖𝟏

𝟏

(t)

𝟏

(t)

𝒖𝒋

𝟐

(t)

𝒖𝟏

𝟐

(t)

𝒖𝒅𝟐

𝟐

(t)

𝒖𝒌

𝟑

(t)

𝒖𝟏

𝟑

(t)

𝒖𝒅𝟑

𝟑

(t)

: Embeddings of entities of each mode

: Edges defined by observed entries (

𝒊, 𝒋, 𝒌

)

: Diffusion process along edges

: Reaction process on individual entities

Time trajectory

Mode 1

Mode 2

Mode 3

Time-

stamp

Value

𝒊

𝒋

𝒌

𝒕

𝒚𝒊𝒋𝒌 (𝒕)

…

…

…

…

…

Table of observed data

Mode 1

Mode 2

Mode 3

diagonal. Hence, each entry value is factorized as m glyph[lscript] = ( u 1 l 1 ◦ . . . ◦ u K l K ) glyph[latticetop] λ , where ◦ is the Hadamard (element-wise) product, and λ corresponds to diag( W ) . While CP and Tucker decomposition are popular, their multilinear modeling can be oversimplistic for complex applications. To estimate nonlinear relationships of the entities, Xu et al. (2012); Zhe et al. (2015, 2016a) used a Gaussian process (GP) (Rasmussen and Williams, 2006) to model the entry value as a random function of the embeddings, m glyph[lscript] = g ( u 1 l 1 , . . . , u K l K ) , where g ∼ GP (0 , κ ( x glyph[lscript] , x glyph[lscript] ′ )) , x glyph[lscript] = [ u 1 l 1 ; . . . ; u K l K ] and x glyph[lscript] ′ = [ u 1 l ′ 1 ; . . . ; u K l ′ K ] are the embeddings of the entities in entry glyph[lscript] and glyph[lscript] ′ , respectively, and κ ( · , · ) is the covariance (kernel) function. Given the GP prior, any finite set of N entry values follow a multi-variate Gaussian distribution, m ∼ N ( 0 , K ) , where m = [ m glyph[lscript] 1 , . . . , m glyph[lscript] N ] , K is the N × N kernel matrix, and each [ K ] i,j = κ ( x glyph[lscript] , x glyph[lscript] ′ ) . Suppose we have collected continuous observations for the N entries y = [ y 1 , . . . , y N ] . We can use a Gaussian noise model: y n = m glyph[lscript] n + glyph[epsilon1] n where glyph[epsilon1] n ∼ N (0 , σ 2 ) . The marginal likelihood is p ( y ) = N ( y | 0 , K + σ 2 I ) , which we can maximize to estimate the embeddings and the model parameters.

Practical data often includes temporal information, i.e. , the timestamp when each observed entry value is generated. To leverage this information, existing methods often bin the timestamps into a series of steps, say, by weeks or months (Xiong et al., 2010; Rogers et al., 2013; Zhe et al., 2016a; Song et al., 2017). The tensor is then expanded with an additional time-step mode, and one can apply any decomposition algorithm to estimate embeddings for both the entities and time steps. To capture the temporal dependency, a conditional model is often used (Xiong et al., 2010), say, p ( t j +1 | t j ) = N ( t j +1 | t j , τ I ) where t j is the embedding of j -th step. To leverage the continuous time information, Zhang et al. (2021) recently developed continuous CP decomposition, where the coefficients λ are modeled as a time function with polynomial splines.

3 Model

Standard tensor decomposition assumes the embeddings are static and time-invariant. However, the embeddings summarize and extract the properties of entities, which can evolve with time, such as customer interests, health status, and product popularity. Therefore, only estimating static embeddings can miss important temporal variations of the entities' properties, resulting in poor representations and predictive performance. In addition, practical tensor data are typical sparse, and only a small portion of entries actually have data. Within these entries can be valuable structural knowledge. Current methods, however, are rarely aware of such knowledge. To overcome these limitations, we propose DEMOTE, a novel dynamic embedding approach.

Specifically, we propose an ODE model to learn the embedding trajectories { u k j ( t ) | 1 ≤ k ≤ K, 1 ≤ j ≤ d k } . The ODE framework is known to be amenable for irregularly sampled, sparsely observed data. More important, ODE models concentrate on learning the time derivative d u k j / d t ( i.e. , dynamics), rather than the trajectory function itself. Therefore, they have a promising potential to give reliable, long-term trajectory prediction (via numerical integration) even at time points far away from the training timestamps, provided the time derivative is well captured. We construct a joint ODE model for all the embedding trajectories. The ODE consists of a diffusion process and a

reaction process. The diffusion process leverages the structural knowledge in data to co-evolve the embeddings of correlated entities, so as to better overcome the data sparsity. The reaction process models the entity-specific evolution so that it can capture the individual differences in the embedding evolution. The ODE model synergizes the two processes to capture both the commonalities and personalities of these embedding trajectories.

Diffusion Process on Multi-Partite Graphs. First, we construct a graph-based diffusion process to exploit the entity correlations reflected in data. Intuitively, if an observed entry involves entity A ( e.g. , customer A) and B ( e.g. , commodity B), the two entities are likely correlated. Thus, we can draw an edge between A and B to express the correlation. We then generalize this intuition to create a K -partite undirected graph G ( E,V ) , to encode such correlations across all the entities in the K tensor modes. Each vertex represents a particular entity, and the entire collection of the entities is partitioned into K groups, V = V 1 ∪ . . . ∪ V K , where group V k = { v k 1 , . . . , v k d k } represents the entities of mode k . Two entities (at different modes) are connected if they were observed to interact, namely, ( v k j , v k ′ j ′ ) ∈ E if ∃ glyph[lscript] n ∈ D such that glyph[lscript] n = ( . . . , j, . . . , j ′ , . . . ) where j and j ′ are indices at mode k and k ′ , respectively. See Fig. 1 for an illustration. This graph naturally implies underlying information diffusion across the entities within their interactions. For example, if customer A connects to products B and C, it might mean that A distributes their interests/willingness/budgets to purchase B and C. The edges between one merchant A and a list of products {B, C, . . . } might indicate the diffusion of willingness to increase the inventory of these products.

glyph[negationslash]

To flexibly estimate the diffusion rate, we introduce a weight w k,k ′ j,j ′ for each edge ( v k j , v k ′ j ′ ) ∈ E . We then arrange these weights into K ( K -1) adjacent matrices, W = { W k,k ′ | 1 ≤ k, k ′ ≤ K,k = k ′ } where W k,k ′ = ( W k ′ ,k ) glyph[latticetop] . Each W k,k ′ is a sparse d k × d k ′ matrix that represents the edges and edge weights between V k and V k ′ , i.e. , [ W k,k ′ ] j,j ′ = w k,k ′ j,j ′ if ( v k j , v k ′ j ′ ) ∈ E and 0 otherwise. We now construct a diffusion process based on the K -partite graph. We view the embedding trajectory as a kind of concentration. For each entity j at mode k , the change rate of its concentration (embedding) u k j ( t ) is determined by the difference from the concentrations of its neighbors. Since the neighbors come from entities of all the other K -1 modes, we have

where w k,s j is the j -th row of W k,s , U s ( t ) = [ u s 1 ( t ) , . . . , u s d s ( t )] glyph[latticetop] is the embeddings of all the entities at mode s , of size d s × R , and a k,s j = ∑ d s j ′ =1 [ W k,s ] j,j ′ is the degree of vertex j in W k,s . We can see that the evolution of the embeddings for different modes are coupled. Hence, it is natural to formulate the diffusion process jointly for all the embeddings, d U ( t ) ∂t = d ( U 1 ( t ) , . . . , U K ( t ) ) / d t = WU ( t ) -AU ( t ) = ( W-A ) U ( t ) where

A = diag ( ∑ s ∈{ 1 ...K }\ 1 A 1 ,s , . . . , ∑ s ∈{ 1 ...K }\ K A K,s ) , and each A k,s = diag( a k,s 1 , . . . , a k,s d k ) is the degree matrix of W k,s .

Reaction Process of Individual Entities. Next, to capture the individual difference of each entity in evolving their embeddings, we model a local reaction process for each entity, f θ k ( u k j ( t ) , t ) , where f ( · ) is a neural network (NN), and θ k are the NN (reaction) parameters for modek entities. The metaphor from the chemical physics is as follows. While substances are being diffused across different sites, at each site a chemical reaction process happens concurrently, which varies the concentration locally. We extend the model as a joint diffusion-reaction process,

Entry Value Generation. Given the embedding trajectories, to obtain the tensor entry value m glyph[lscript] at arbitrary time t , we model m glyph[lscript] ( t ) as a function of the relevant embeddings at time t ,

While one can follow (Xu et al., 2012; Zhe et al., 2016b) to assign a GP prior over g ( · ) , the GP model needs to compute a giant kernel matrix over all the observed entry values (see Sec. 2). It is computationally too expensive or infeasible when the number of observations is large. Hence one has to seek for complex low-rank approximations. To avoid this problem, we model g with another neural network, which is not only as flexible as GP, but is more scalable and convenient for computation. Since now, the input to g ( · ) consists of the trajectory values, which vary with time, our NN model for g can flexibly capture the complex temporal relationship between the entities. We finally sample the observed entry values with a Gaussian noise model, p ( y | m ) = N ( y | m , σ 2 I ) where y = [ y 1 , . . . , y N ] glyph[latticetop] and m = [ m glyph[lscript] 1 ( t 1 ) , . . . , m glyph[lscript] N ( t N )] glyph[latticetop] . We focus on real-valued data in this paper. However, it is straightforward to extend our approach to other types of data.

4 Model Estimation

Given data D = { ( glyph[lscript] 1 , t 1 , y 1 ) , . . . , ( glyph[lscript] N , t N , y N ) } , the joint probability of our model is

where β is the NN parameters for g , each θ k is the NN reaction parameters for modek entities, p ( β ) and p ( θ k ) are element-wise standard Gaussian, and y = ( y 1 , . . . , y N ) glyph[latticetop] . To obtain the trajectory values in the Gaussian likelihood of each y n , we need to solve the ODE in (1) to time t n ,

where Θ = {W , θ 1 , . . . , θ K } consists of the ODE parameters. Our goal is to estimate Θ , the initial state U 0 , the NN parameters β , and the noise variance σ 2 .

Stratified Mini-Batch Sampling. We use stochastic mini-batch optimization to maximize the log joint probability so as to estimate all the required parameters,

where log( Prior ) = log p ( β ) + ∑ K k =1 log p ( θ k ) , and x n = ( u 1 l n 1 ( t n ) , . . . , u K l nK ( t n ) ) . Each time, we sample a mini-batch of observations B , and obtain an unbiased stochastic estimate of the log probability, ̂ L = log( Prior ) -N B ∑ n ∈B [ log N ( y n | g ( x n ) , σ 2 ) ] . We compute ∇ ̂ L as the stochastic gradient to update all the parameters.

For each data point n in the mini-batch, we need to run ODE solving (4) to obtain x n = ( u 1 l n 1 ( t n ) , . . . , u K l nK ( t n ) ) . To back-propagate the gradient so as to compute the gradient w.r.t the ODE parameters Θ and initial state U 0 , we can either construct a computational graph during the running of the solver ( e.g. , the Runge-Kutta method (Dormand and Prince, 1980)), or use the adjoint state method (Pontryagin, 1987; Chen et al., 2018) that solves an adjoint backward ODE to compute the gradient. In whichever case, we need to sort the time points in the mini-batch and solve the ODE sequentially for these time points. As a result, the number of unique time points in the mini-batch greatly influences the speed of processing the mini-batch. The standard mini-batch sampling (based on the training example indices) can result in an uneven allocation of the computational cost across the mini-batches - some mini-batch is fast and some including more unique time points is much slower. To address this issue, we use a simple stratified sampling approach.

We collect the unique time points in the whole dataset, T = { τ 1 , τ 2 , . . . } at the beginning.

To conduct each stochastic update, we first sample B unique time points C from T , then for each time point τ j ∈ C , we look at all the observed entry values produced at τ j , namely D τ j = { ( glyph[lscript] n , t n , y n ) ∈ D| t n = τ j } .

We randomly sample one example from each D τ j to collect the mini-batch B .

In this way, we ensure the cost of running ODE solvers and related gradient computation in each mini-batch is identical. There are no fluctuations in cost/running time when processing different min-batches. Empirically, we found that the overall speed of our method is much faster than vanilla stochastic mini-batch optimization (see Table 2 in Appendix).

5 Related Work

There have been many works on tensor decomposition, such as (Yang and Dunson, 2013; Rai et al., 2014; Zhe et al., 2015, 2016b,a; Tillinghast et al., 2020; Pan et al., 2020b; Fang et al., 2021a,b; Tillinghast and Zhe, 2021; Tillinghast et al., 2022; Fang et al., 2022). To integrate temporal information, most approaches augment the tensor with a time mode (Xiong et al., 2010; Rogers et al., 2013; Zhe et al., 2016b; Ahn et al., 2021; Zhe et al., 2015; Du et al., 2018), which includes a list of time steps. To estimate the temporal dependencies, existing methods often employ a dynamic model over the time steps, such as a conditional Gaussian prior (Xiong et al., 2010), recurrent neural networks(Wu et al., 2019), and kernel smoothing/regularization (Ahn et al., 2021). To leverage continuous timestamps, Zhang et al. (2021) modeled the CP coefficients as a time function, and Fang et al. (2022) modeled the tensor-core as a time function in the Tucker decomposition. The most recent work (Wang et al., 2022) places a GP prior in the frequency domain, and construct a bi-level GP model to learn factor or embedding trajectories as a combination of Fourier bases.

Another set of works factorize the interaction events (Schein et al., 2015, 2016; Zhe and Du, 2018; Pan et al., 2020a; Wang et al., 2020; Pan et al., 2021; Wang et al., 2022), and they cannot predict the interaction results ( e.g. , tensor entry values like purchase amount and product ratings). These works mainly leverage Poisson processes, Hawkes processes, or more general point processes to estimate the event rate. Like the standard tensor factorization, these methods also estimate static embeddings for the event participants.

Our model can be viewed as an extension of the neural ODE model (Chen et al., 2018). If we only employ the reaction process for each entity, our model is a latent neural ODE (we have an additional NN that combines the latent trajectories to predict the tensor entry values). However, we further leverage the structural knowledge in data to construct a multi-partite graph so as to encode the correlations of the entities. Based on the multi-partite graph, we construct a diffusion process to co-evolve the embeddings of the entities. In doing so, we can better overcome the data sparsity issue. The most recent work (Li et al., 2022) uses a neural ODE to model the tensor entry value as a function of the involved embeddings and time. It differs from our method in that (1) it still learns a static (time-invariant) embedding for each entity, and (2) its modeling does not use the structure knowledge within the tensor data like our method . Many other works have developed graph diffusion processes based on graph data. For example, Chamberlain et al. (2021) proposed a graph neural network (GNN) by using multi-head attention to construct the adjacent matrix for a graph diffusion equation over the graph nodes. Atwood and Towsley (2016) introduced a diffusion operator to develop diffusion-convolutional neural networks. Huang et al. (2021) developed a GNN-based ODE to model both the nodes and edges in dynamic graphs.

6 Experiment

6.1 Simulation Study

We first examined DEMOTE on a synthetic task. Specifically, we considered a two-mode tensor, where each mode includes 20 entities. Each entity has one underlying embedding trajectory. In the first mode, the trajectory of each entity is an exponential function, u 1 j ( t ) = c 1 j exp(0 . 5 c 1 j t ) ( 1 ≤ j ≤ 20 ), while in the second mode is a linear function, u 2 j ( t ) = c 2 j +2 πc 2 j t . We generated two clusters of trajectories for each mode, where those of the first ten entities form the first cluster and the remaining the second cluster. To this end, for mode 1, we sampled the coefficients of the first ten entities' trajectories, namely, [ c 1 1 , . . . , c 1 10 ] glyph[latticetop] , from N ( [ -5 , . . . , -5] glyph[latticetop] , 0 . 1 I ) , and the remaining ten coefficients [ c 1 11 , . . . , c 1 20 ] glyph[latticetop] from N ( [0 . 5 , . . . , 0 . 5] glyph[latticetop] , 0 . 1 I ) . Then for mode 2, we sampled each coefficient c 2 j conditioned on its counter-part for mode 1, c 2 j | c 1 j ∼ N ( c 2 j | c 1 j , 0 . 1) . That means, the coefficients of cluster-1 trajectories across the two modes are close, and so are those of cluster-2 trajectories. The value for a particular entry glyph[lscript] = ( l 1 , l 2 ) is generated by

(a) Ground-truth: Mode 1

(b) NONFAT: Mode 1

(c) DEMOTE: Mode 1

(d) Ground-truth: Mode 2

(e) NONFAT: Mode 2

(f) DEMOTE: Mode 2

where 1 ( · ) is the indicator function. When l 1 + l 2 is even, the entry value is the trajectory value of the first entity; otherwise, it is the trajectory value of the second entity. To generate the training data, we randomly sampled entries from { ( l 1 , l 2 ) | 1 ≤ l 1 , l 2 ≤ 10 } ∪ { ( l 1 , l 2 ) | 11 ≤ l 1 , l 2 ≤ 20 } (namely, interactions between cluster-1 entities of the two modes, and between cluster-2 entities). We then sampled t ∼ Unifrom [0 , 5] , to obtain the corresponding entry values. We randomly generated 6,400 entry values and the timestamps for training, and another 1,600 data points for testing.

We implemented our method with Pytorch (Paszke et al., 2019). We used torchdiffeq library ( https://github.com/rtqichen/torchdiffeq ) to solve ODEs and to compute the gradient w.r.t ODE parameters and initial states via automatic differentiation. For the NN of the reaction process, we used one hidden layer, with 10 neurons and tanh activation, and for the NN to predict the interaction result, we used two hidden layers, 50 neurons per layer and tanh activation.

We compared with NONFAT (NONparametric Factor Trajectory learning) (Wang et al., 2022), a bi-level latent GP model that uses Fourier bases to estimate factor trajectories for dynamic tensor decomposition. To our knowledge, this work is the only method (and also the most recent) that estimates trajectories. We used the original implementation ( https://github.com/wzhut /NONFAT ) and the default settings. We set the mini-batch size to 50, and used ADAM (Kingma and Ba, 2014) algorithm for stochastic optimization. The learning rate was automatically adjusted in [10 -4 , 10 -1 ] by the ReduceLROnPlateau scheduler (Al-Kababji et al., 2022). The maximum number of epochs is 2K, which is enough for convergence. The estimated trajectories are shown in Fig. 2a-f. As we can see, our estimation (Fig. 2c and 2f) well matches the ground-truth and accurately recovers the cluster structure of the trajectories. The root-mean-square error (RMSE) on the test set is 0.032. By contrast, although the test error of NONFAT is close to DEMOTE (0.034), its learned trajectories (Fig. 2b and 2e) are far from the ground-truth, and fail to reflect the cluster structure. These have shown the advantage of DEMOTE in capturing complex relationships within data to recover the underlying trajectories and their structure.

6.2 Prediction Accuracy

Datasets. We next evaluated the predictive performance of DEMOTE in three real-world applications. (1) CA Weather (Moosavi et al., 2019) ( https://smoosavi.org/datasets/lstw ), weather conditions in California from August 2016 to December 2020. We extracted a four-mode tensor for 7 different weather types , 6 severity levels , 30 latitudes and 30 longitudes in GPS coordinates. The entry value is the count of the particular weather condition. We collected 15 K observed tensor entry values and the timestamps. (2) CA Traffic (Moosavi et al., 2019) ( https://smoosavi.org/dataset s/lstw ), traffic accidents in California from January 2018 to December 2020. We extracted a four

2

3

5

7

R

0.6

0.65

0.7

nRMSE

Reaction

Diffusion

DEMOTE

(a)

2

3

5

7

R

0.3

0.4

0.5

nRMSE

(b)

CA Traffic

2

3

5

7

R

0.04

0.06

0.08

nRMSE

(c)

Server Room

mode tensor ( traffic type , severity level , latitudes , longitude ). There are 7 traffic types, 6 severity levels, 20 latitudes and 20 longitudes. We collected 30 K entry values (accident counts) at different time points. (3) Server Room ( https://zenodo.org/record/3610078#.XlNpAigzaM8 ), temporal temperature records of Poznan Supercomputing and Networking Center. The temperatures were measured at 34 locations, under different air-condition modes ( 24 ◦ , 27 ◦ , and 30 ◦ ) and power usage settings (50%, 75% and 100%). Hence, we extracted a three-mode tensor ( location , aircondition mode , power level ). In total, 10 K observed entry values and their timestamps were collected.

Competing Methods. The following popular and/or state-of-the-art temporal decomposition approaches were compared. (1) CP-DTLD, discrete-time CP decomposition with linear dynamics, where a conditional prior is placed over successive time steps, p ( t j +1 | t j ) = N ( t j +1 | At j + b , v I ) ; A , b and v were jointly estimated during the CP decomposition. Note that (Xiong et al., 2010) is an instance of this model where A = I and b = 0 . (2) GP-DTLD and (3) NN-DTLD, similar to CP-DTLD, except using GP (Zhe et al., 2016b) and NN decomposition models (similar to (Liu et al., 2019)), respectively. (4) CP-DTND, (5) GP-DTND and (6) NN-DTND - CP, GP and NN decomposition with nonlinear dynamics, where the conditional prior is p ( t j +1 | t j ) = N ( t j +1 | σ ( At j )+ b , v I ) where σ ( · ) is a nonlinear activation. The dynamics can therefore be viewed as an RNN transition. (7) CP-CT (Zhang et al., 2021), continuous-time CP factorization, which models the CP coefficients as a time-varying function, with polynomial splines. (8) GP-CT, continuous-time GP decomposition that extends (Xu et al., 2012; Zhe et al., 2016b) by plugging the time in the GP kernel so as to estimate the entry value as a function of the embeddings and time, m glyph[lscript] = g ( u 1 glyph[lscript] 1 , . . . , u K glyph[lscript] K , t ) . (9) NN-CT, continuous-time NN decomposition, where the input consists of both the embeddings and time t . (10) THIS-ODE (Li et al., 2022), a continuous-time decomposition, where a neural ODE is used to estimate the tensor entry values given the static embeddings and time. (11) NONFAT (Wang et al., 2022), a bi-level latent GP model that uses Fourier bases to estimate factor trajectories for dynamic tensor decomposition.

Settings and Results. All the approaches were implemented with PyTorch. The Square Exponential kernel was used for all the GP-related methods, including GP-{DTLD, DTND, CT}. We used the same variational sparse approximation (Hensman et al., 2013) to fulfill scalable posterior inference. Following (Zhe et al., 2016b), the number of inducing point was set to 100 . For the NN decomposition methods, we employed a three-layer network with tanh activation, and for THIS-ODE, we used a one-layer network. The layer width was chosen from {10, 25, 50, 75, 100}. We used tanh as the activation function in the nonlinear dynamic baselines, including {CP, GP, NN}-DTND. For our method, we used the same NN architecture for both the reaction process and entry value prediction, which includes two hidden layers with 50 neurons per layer. For CP-CT, we employed 100 knots to fulfill the polynomial splines. For each discrete-time method, the number of time steps was chosen from {25, 50, 75, 100} via the cross-validation on the training set. We trained all the models with stochastic mini-batch optimization. We used the ADAM algorithm, and the mini-batch size was set to 100 . We ran every method with 10K epochs to ensure convergence. The learning rate was automatically adjusted in [10 -4 , 10 -1 ] by the ReduceLROnPlateau scheduler. We varied the dimension of the embeddings R from {2, 3, 5, 7}. For DEMOTE, R is the number of embedding trajectories; we used computational graphs to obtain the gradient. We followed (Kang et al., 2012; Zhe et al., 2016b) to randomly draw 80% observed entries and their time stamps for training, with the remaining for test. We computed the normalized root-mean-square error (nRMSE). We repeated the evaluation five times and computed the average nRMSE and standard deviation.

As shown in Table 1, DEMOTE consistently achieves the best prediction accuracy and in many cases outperforms the competing methods by a large margin. Although learning an embedding trajectory is much more challenging than learning a fixed-value embedding, the experimental results have demonstrated the advantage of our method in predictive performance. To investigate the effect of the two processes in our model, we also examined our method with the diffusion process only and with the reaction process only on all the datasets. Their performance, as compared with DEMOTE, is shown in Fig. 3. We can see that each individual component can lead to good prediction accuracy. However, each component is worse than DEMOTE that synergizes the two components together. Therefore, the results show that each process is effective, and more important, the two processes can bolster each other to further improve the performance when they are combined.

6.3 Learning Result Investigation

Next, we looked into the learned embedding trajectories and checked if they exhibit patterns. To do so, we set R = 3 and ran DEMOTE on Server Room dataset. In Fig. 4, we show the learned embedding trajectories for the first location (a-c), the first air condition mode (d-f) and the first power usage level (g-i). As we can see, even for the same object, e.g. , a particular location, the corresponding embedding trajectories vary quite differently, implying the evolution of different underlying properties, such as the workload, memory usage, and network latency. We further found there are underlying structures within the embeddings during their evolution. We listed the results in Appendix (Sec. A).

Time

-2

-1

0

1

2

3

(a)

u

1

1

,

1

(

t

)

Time

-2

-1.5

-1

-0.5

0

0.5

(b)

u

1

1

,

2

(

t

)

Time

-6

-4

-2

0

2

(c)

u

1

1

,

3

(

t

)

Time

-4

-2

0

2

4

6

(d)

u

2

1

,

1

(

t

)

Time

1

2

3

4

5

6

(e)

u

2

1

,

2

(

t

)

Time

-10

-8

-6

-4

-2

0

(f)

u

2

1

,

3

(

t

)

Time

0

0.5

1

1.5

2

2.5

(g)

u

3

1

,

1

(

t

)

Time

-10

-8

-6

-4

-2

0

(h)

u

3

1

,

2

(

t

)

Time

1

1.2

1.4

1.6

1.8

(i)

u

3

1

,

3

(

t

)

We showcase the temporal predictions for two tensor entries. As we can see from Fig. 5, given only a few training points (blue), our method can predict the test points (green) much more accurately, as compared with GPCT, and the predictive uncertainty (reflected by the noise variance σ 2 ) is much smaller. This might be due to that via the diffusion-reaction process, and our method can more effectively extract the temporal knowledge from sparse data. For example, DEMOTE successfully captured the periodic nature in the first entry (Fig. 5b) while GPCT treated the fluctuation as noises and ended up with much inaccurate predictions and larger predictive variances.

Computational Efficiency. We compared the perepoch/iteration running time DEMOTE with the other methods. We tested all the methods in a workstation with one NVIDIA GeForce RTX 3090 Graphics Card, 10th Generation Intel Core i910850K Processor, 32 GB RAM, and 1 TB SSD. The results are shown in Table 2 in Appendix. We can see that the running speed of DEMOTE is comparable to NONFAT and other NN decomposition methods. We also compared with running DEMOTE with naive sampling (DEMOTE-NS). The stratified sampling led to 4x to 22x speed-up.

7 Conclusion

We have presented DEMOTE, a neural diffusion-reaction process model to learn dynamic embeddings for dynamic tensor decomposition. The predictive performance is encouraging, and the learned embedding trajectories exhibit interesting patterns. Currently, our method is limited to a small number of entities since it has to integrate the entire multi-partite graph to construct the diffusion process. In the future work, we plan to develop graph cut algorithms to partition the graph into a set of small sub-graphs so that we can construct multiple diffusion processes in parallel so as to scale up our model to big graphs and to large tensors.

Acknowledgments

This work has been supported by NSF CAREER Award IIS-2046295.

Time

-4

-3

-2

-1

0

DEMOTE

GP-CT

Training Data

Testing Data

(a) (2, 1, 1)

Time

-2

-1

0

1

(b) (3, 1, 1)

References

Ahn, D., Jang, J.-G., and Kang, U. (2021). Time-aware tensor decomposition for sparse tensors. Machine Learning, pages 1-22.

Al-Kababji, A., Bensaali, F., and Dakua, S. P. (2022). Scheduling techniques for liver segmentation: Reducelronplateau vs onecyclelr. arXiv preprint arXiv:2202.06373.

Atwood, J. and Towsley, D. (2016). Diffusion-convolutional neural networks. Advances in neural information processing systems, 29.

Chamberlain, B., Rowbottom, J., Gorinova, M. I., Bronstein, M., Webb, S., and Rossi, E. (2021). Grand: Graph neural diffusion. In International Conference on Machine Learning, pages 14071418. PMLR.

Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.

Dormand, J. R. and Prince, P. J. (1980). A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26.

Du, Y., Zheng, Y., Lee, K.-c., and Zhe, S. (2018). Probabilistic streaming tensor decomposition. In 2018 IEEE International Conference on Data Mining (ICDM), pages 99-108. IEEE.

Fang, S., Kirby, R. M., and Zhe, S. (2021a). Bayesian streaming sparse Tucker decomposition. In Uncertainty in Artificial Intelligence, pages 558-567. PMLR.

Fang, S., Narayan, A., Kirby, R., and Zhe, S. (2022). Bayesian continuous-time Tucker decomposition. In International Conference on Machine Learning, pages 6235-6245. PMLR.

Fang, S., Wang, Z., Pan, Z., Liu, J., and Zhe, S. (2021b). Streaming Bayesian deep tensor factorization. In International Conference on Machine Learning, pages 3133-3142. PMLR.

Harshman, R. A. (1970). Foundations of the PARAFAC procedure: Model and conditions for an'explanatory'multi-mode factor analysis. UCLA Working Papers in Phonetics, 16:1-84.

Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 282-290. AUAI Press.

Huang, Z., Sun, Y., and Wang, W. (2021). Coupled graph ode for learning interacting system dynamics. In 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021, pages 705-715.

Kang, U., Papalexakis, E., Harpale, A., and Faloutsos, C. (2012). Gigatensor: scaling tensor analysis up by 100 times-algorithms and discoveries. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 316-324. ACM.

Ketchen, D. J. and Shook, C. L. (1996). The application of cluster analysis in strategic management research: an analysis and critique. Strategic management journal, 17(6):441-458.

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.

Kolda, T. G. (2006). Multilinear operators for higher-order decompositions, volume 2. United States. Department of Energy.

Li, S., Kirby, R., and Zhe, S. (2022). Decomposing temporal high-order interactions via latent odes. In International Conference on Machine Learning, pages 12797-12812. PMLR.

Liu, H., Li, Y., Tsang, M., and Liu, Y. (2019). CoSTCo: A Neural Tensor Completion Model for Sparse Tensors, page 324-334. Association for Computing Machinery, New York, NY, USA.

Moosavi, S., Samavatian, M. H., Nandi, A., Parthasarathy, S., and Ramnath, R. (2019). Short and long-term pattern discovery over large-scale geo-spatiotemporal data. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2905-2913.

Pan, Z., Wang, Z., Phillips, J. M., and Zhe, S. (2021). Self-adaptable point processes with nonparametric time decays. Advances in Neural Information Processing Systems, 34:4594-4606.

Pan, Z., Wang, Z., and Zhe, S. (2020a). Scalable nonparametric factorization for high-order interaction events. In International Conference on Artificial Intelligence and Statistics, pages 4325-4335. PMLR.

Pan, Z., Wang, Z., and Zhe, S. (2020b). Streaming nonlinear Bayesian tensor decomposition. In Conference on Uncertainty in Artificial Intelligence, pages 490-499. PMLR.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037.

Pontryagin, L. S. (1987). Mathematical theory of optimal processes. CRC press.

Rai, P., Wang, Y., Guo, S., Chen, G., Dunson, D., and Carin, L. (2014). Scalable Bayesian lowrank decomposition of incomplete multiway tensors. In Proceedings of the 31th International Conference on Machine Learning (ICML).

Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.

Rogers, M., Li, L., and Russell, S. J. (2013). Multilinear dynamical systems for tensor time series. Advances in Neural Information Processing Systems, 26:2634-2642.

Rubanova, Y., Chen, R. T. Q., and Duvenaud, D. K. (2019). Latent ordinary differential equations for irregularly-sampled time series. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Schein, A., Paisley, J., Blei, D. M., and Wallach, H. (2015). Bayesian poisson tensor factorization for inferring multilateral relations from sparse dyadic event counts. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1045-1054. ACM.

Schein, A., Zhou, M., Blei, D. M., and Wallach, H. (2016). Bayesian poisson tucker decomposition for learning the structure of international relations. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pages 2810-2819. JMLR.org.

Song, Q., Huang, X., Ge, H., Caverlee, J., and Hu, X. (2017). Multi-aspect streaming tensor completion. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 435-443.

Tillinghast, C., Fang, S., Zhang, K., and Zhe, S. (2020). Probabilistic neural-kernel tensor decomposition. In 2020 IEEE International Conference on Data Mining (ICDM), pages 531-540. IEEE.

Tillinghast, C., Wang, Z., and Zhe, S. (2022). Nonparametric sparse tensor factorization with hierarchical Gamma processes. In International Conference on Machine Learning, pages 2143221448. PMLR.

Tillinghast, C. and Zhe, S. (2021). Nonparametric decomposition of sparse tensors. In International Conference on Machine Learning, pages 10301-10311. PMLR.

Tucker, L. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31:279311.

Wang, Z., Chu, X., and Zhe, S. (2020). Self-modulating nonparametric event-tensor factorization. In International Conference on Machine Learning, pages 9857-9867. PMLR.

Wang, Z., Xu, Y., Tillinghast, C., Li, S., Narayan, A., and Zhe, S. (2022). Nonparametric embeddings of sparse high-order interaction events. In International Conference on Machine Learning, pages 23237-23253. PMLR.

Wu, X., Shi, B., Dong, Y., Huang, C., and Chawla, N. V. (2019). Neural tensor factorization for temporal interaction learning. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 537-545.

Xiong, L., Chen, X., Huang, T.-K., Schneider, J., and Carbonell, J. G. (2010). Temporal collaborative filtering with bayesian probabilistic tensor factorization. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 211-222. SIAM.

Xu, Z., Yan, F., and Qi, Y . A. (2012). Infinite tucker decomposition: Nonparametric bayesian models for multiway data analysis. In ICML.

Yang, Y. and Dunson, D. (2013). Bayesian conditional tensor factorizations for high-dimensional classification. Journal of the Royal Statistical Society B, revision submitted.

Zhang, Y., Bi, X., Tang, N., and Qu, A. (2021). Dynamic tensor recommender systems. Journal of Machine Learning Research, 22(65):1-35.

Zhe, S. and Du, Y. (2018). Stochastic nonparametric event-tensor decomposition. In Advances in Neural Information Processing Systems, pages 6856-6866.

Zhe, S., Qi, Y., Park, Y ., Xu, Z., Molloy, I., and Chari, S. (2016a). Dintucker: Scaling up Gaussian process models on large multidimensional arrays. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.

Zhe, S., Xu, Z., Chu, X., Qi, Y., and Park, Y . (2015). Scalable nonparametric multiway data analysis. In Artificial Intelligence and Statistics, pages 1125-1134. PMLR.

Zhe, S., Zhang, K., Wang, P., Lee, K.-c., Xu, Z., Qi, Y., and Ghahramani, Z. (2016b). Distributed flexible nonlinear tensor factorization. Advances in neural information processing systems, 29.

Appendix

A Investigation of Embedding Dynamics

We investigated if there are underlying structures within the embeddings during their evolution. To this end, we looked into the embeddings of the 34 locations on Server Room dataset at five time points ( t = 1, 20, 50, 80, 100). At each time point, we ran the k-means algorithm over the embeddings to extract the clustering structures. We used the elbow method (Ketchen and Shook, 1996) to select the number of clusters. We can see that at earlier time ( t ≤ 50 ), the clusters are more compact, while at the later stages, the clusters become more scattered. This reflects how the structure of those entities ( i.e. , locations) evolves along with time. It is interesting to see that some locations are in the same cluster all the time, like location {5,7} and location {16, 32}. It implies that their underlying properties might have quite similar (or correlated) evolution patterns. Some locations are grouped in the cluster at the beginning, e.g. , location {32, 34} (at t = 1 ), but later moves to different clusters ( t > 1 ). It implies their evolution patterns can vary significantly, leading to the change of the cluster memberships.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

Cluster 1

Cluster 2

Cluster 3

(a)

t

= 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

(b)

t

= 20

2 1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

(c)

t

= 50

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

(d)

t

= 80

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

(e)

t

= 100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the framework to generate dynamic embeddings for multi-entity (e.g., traffic type, severity, latitude, and longitude) interactions with timestamps. The temporal high-order interactions are modeled via neural diffusion-reaction processes, and to tack the temporal drift of underlying node, authors propose the dynamic embeddings that follows a diffusion-based ODE model. The diffusion process uses a multi-partite graph to coevolve the trajectories, and the reaction process models the individual evolution of each object. The combination of these two processes can capture both the commonalities and personalities in learning the embedding trajectories for different objects. Authors demonstrate that the proposed model can capture the temporal dynamic well through both synthetic and real-world datasets with comparison to a large number of existing methods.\n\nSTRENGTHS:\n1. The proposed ODE-based model looks like a reasonable way of representing the dynamics of embeddings.\n\n2. The dynamic embeddings for entities are generated in continuous, rather than discrete time, unlike most existing approaches.\n\n3. Numerical experiments demonstrate that the proposed method outperforms baselines on 3 real-world datasets.\n\nWEAKNESSES:\n1. It seems that the text can be further improved to present better. Some phrases are repeated several times unnecessarily, and some points, on the contrary, are covered too briefly. I would recommend the authors to carry out additional proofreading of the text.\n\n2. The source code is not provided, and this complicates the reproducibility of the results.\n\nQUESTIONS:\n1. [Line 6] It seems that \"I\" in \"EMbedIngs\" should be lowercase.\n\n2. [Lines 92-100] It seems that there are not enough conclusions to this paragraph. What are the advantages and disadvantages of the indicated \"continuous CP decomposition\" approach?\n\n3. [Line 106] \"typical\" -> \"typically\".\n\n4. [Line 270] \"We compared <our approach>...\"\n\nLIMITATIONS:\nThe proposed method is limited to a small number of entities, however, further development of the approach using the graph cut algorithms could potentially improve its scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors introduce DEMOTE which can both consider both structure information and temporal information via dynamic tensor decomposition. Structure information is captured by multi-partite graph diffusion process which in substance is a special ODE, whereas temporal information is analyzed using another ODE. Experimental results demonstrate DEMOTE in general outperforms one baseline for synthetic data and many baselines for real-world data.\n\nSTRENGTHS:\n1. It is interesting to model the spatial interactions for values at the same time point using multi-partite graphs through graph diffusion, for the small-scale datasets.\n\n2. Addressing the sparsity in tensorial data is another strength of this work. It has been an important problem in tensorial data and appeal to the tensor learning community.\n\nWEAKNESSES:\n1. There have been some existing work on dynamic tensor decomposition using neural ODEs and relational information in the recent years, whereas not discussed and compared in this work, for example, TANGO [1].\n\n2. The presentation should be improved. The notations should be defined in a clear manner, for example, the adjacency tensor W needs more clarification on its order and how it is computed, from its indices. \n\nReference\n\n[1] Han, Z., Ding, Z., Ma, Y., Gu, Y., & Tresp, V. (2021, November). Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In *Proceedings of the 2021 conference on empirical methods in natural language processing* (pp. 8352-8364).\n\nQUESTIONS:\n1. For the statement in Page 3 Section 3 Lines 113-115, why can learning dynamics produce more reliable and long-term trajectory predictions? The time derivative can provide the general dynamics of the whole time series and mainly focus on the global sequential information. Please check this paper \"Neural Rough Differential Equations for Long Time Series\" [1].\n\n2. Could the authors provide any explanation and examples on why K should be involved and represent structure information using multi-partite graphs is necessary? It seems that including the multi-partite graphs is very complicated, and in what circumstances the multi-partite graphs should be applied to describe what types of intercorrelation among multi-dimensional data, without incurring overfitting?\n\n3. In Figure 2(c), why do the red lines decline? In the long-term, the performance of DEMOTE is worse than the baseline model NONFAT. Could the authors articulate it?\n\n4. For the results of DEMOTE in Figure 5(a) and (b), why are the entry value predictions with high and sharp spikes across time? It seems that DEMOTE can capture the periodic fluctuations, but DEMOTE by nature is based on two ODEs. In specific, ODE and neural ODE assume that the modeled trajectory should be smooth, or at least Lipschitz continuous [2,3]. Could the authors also provide their code?\n\nReferences\n\n[1] Morrill, J., Salvi, C., Kidger, P., & Foster, J. (2021, July). Neural rough differential equations for long time series. In International Conference on Machine Learning (pp. 7829-7838). PMLR.\n\n[2] Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.\n\n[3] Süli, E., & Mayers, D. F. (2003). An introduction to numerical analysis. Cambridge university press.\n\nLIMITATIONS:\nThe limitations are sufficiently addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, to capture higher-order interaction among different modes of tensor data and their evolution process over time, the authors proposed an ODE-based method called DEMOTE to capture the dynamics of latent embeddings.   In detail, the authors propose to use a diffusion-reaction process to model the evolution of these embeddings.  What's more, structure knowledge within tensor data is considered in this diffusion-reaction process, which enhances the model's representation ability.  Experiments show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The experimental results show an advantage compared with related work.\n\n2. The way of using the structure knowledge is novel and effective in modeling tensor data. It is particularly interesting to embed this structured knowledge with the framework of ODE.\n\nWEAKNESSES:\n1. The main concern lies in its novelty since there are some efforts studying the interaction results of higher order continuous time data, as clarified in the paper (that is [1]). Actually, in [1], they use ODE to model the entry values' trajectories and depend on $m_{i}$. In this work, the authors use ODE to model $m_{i}$ directly. Although these two papers correspond to different ways of handling problems and the authors have stated the advantage of the proposed method, they do share many similar intuitions towards the same task, which hinders their novelty.\n\nTypo: Page 3, last paragraph: $(v_{j}^{k}, v_{j'}^{k'})\\in \\mathbb{E}$ $\\rightarrow$  $(v_{j}^{k}, v_{j'}^{k'})\\in E$.\n\n[1] Li S, Kirby R, Zhe S. Decomposing Temporal High-Order Interactions via Latent ODEs[C]//International Conference on Machine Learning. PMLR, 2022: 12797-12812.\n\nQUESTIONS:\n1. I think the contribution should be further clarified.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a dynamic tensor decomposition (TD) model, in which the latent factors are functions of time stamps described by neural ODEs. Moreover, they encode an adjacent matrix into the diffusion process to better model the correlations among different tensor modes. Finally, they use an additional neural network to map latent factors to tensor entries. Using differentiable ODE solvers, all the parameters can be learned by optimizing the joint likelihood with SGD. For empirical evaluations, the authors first conduct a simulation study to show that the proposed model can learn latent factor trajectories. Then, they demonstrate the completion performance on several dynamic tensor datasets.

STRENGTHS:
1. The authors propose to use a diffusion process to learn tensor factor trajectories, which is new in the field.

2. The experiments show that the proposed model can learn factor and entry trajectories. Besides, it achieves a significant improvement on the completion performance.

WEAKNESSES:
1. In the paper, the authors construct a complex TD with ODEs and NN to parameterize the latent relation. Besides, the model parameters are learnt from MAP estimates (Eq. 3). I am wondering if such a model suffers from overfitting for sparse tensor data?

2. In Equation 1, the authors define the ODE of the factor trajectories. Since $\mathcal{F}(\mathcal{U}, t)$ is already parameterized by flexible neural networks, I am wondering if it is necessary to add the multi-partite graph term, i.e., $(\mathcal{W} - \mathcal{A})\mathcal{U(t)}$? Can we construct a specific network structure that is equivalent to Eq. 1?

QUESTIONS:
1. In the equation between Line 178 and 179, does $\mathcal{L}$ equal to $\log (Prior)$ **plus** the likelihood?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06