Dynamic Embeddings of Temporal High-Order Interactions via Neural Diffusion-Reaction Processes
Authors: Zheng Wang
shikai fang
Shibo Li
Shandian Zhe
Venue: Submitted to ICLR 2023 (2023)
Keywords: Embedding Trajectory
Tensor Decomposition

Abstract: High-order interactions of multiple entities are ubiquitous in practical applications. The associated data often includes the participants, interaction results, and the timestamps when each interaction occurred. While tensor factorization is a popular tool to analyze such data, it often ignores or underuses valuable timestamp information. More important, standard tensor factorization only estimates a static representation for each entity and ignores the temporal variation of the representations. However, such variations might reflect important evolution patterns of the underlying properties of the entities. To address these limitations, we propose Dynamical eMbedIngs of TempoRal hIgh-order interactions (DMITRI). We develop a neural diffusion-reaction process model to estimate the dynamic embeddings for the participant entities. Specifically, based on the observed interactions, we build a multi-partite graph to encode the correlation between the entities. We construct a graph diffusion process to co-evolve the embedding trajectories of the correlated entities and use a neural network to construct a reaction process for each individual entity. In this way, our model is able to capture both the commonalities and personalities during the evolution of the embeddings for different entities. We then use a neural network to model the interaction result as a nonlinear function of the embedding trajectories. For model estimation, we combine ODE solvers to develop a stochastic mini-batch learning algorithm. We propose a simple stratified sampling method to balance the cost of processing each mini-batch so as to improve the overall efficiency. We show the advantage of our approach in both the ablation study and real-world applications. 

Under review as a conference paper at ICLR 2023
DYNAMIC EMBEDDINGS OF TEMPORAL HIGH-ORDER INTERACTIONS VIA NEURAL DIFFUSION-REACTION PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
High-order interactions of multiple entities are ubiquitous in practical applications. The associated data often includes the participants, interaction results, and the timestamps when each interaction occurred. While tensor factorization is popular tool to analyze such data, it often ignores or underuses the valuable timestamp information. More important, standard tensor factorization only estimates static representation for each entity, and ignores the temporal variation the representa- tions. However, such variations might reflect important evolution patterns of the underlying properties of the entities. To address these limitations, we propose Dy- namical eMbedIngs of TempoRal high-order interactions (DMITRI). We develop a neural diffusion-reaatioon process model to estimate the dynamic embeddings for the participant entities. Specifically, based on the observed interactions, we build multi-partite graph to encode the correlation between the entities. We construct graph diffusion process to co-evolve the embedding trajectories of the correlated entities, and use a neural network to construct a reaction process for each individual entity. In this way, our model is able to capture both the commonalities and personalities during the evolution of the embeddings for different entities. We then use neural network to model the interaction result as a nonlinear function of the embedding trajectories. For model estimation, combine ODE solvers develop stochastic mini-batch learning algorithm. We propose a simple stratified sampling method to balance the cost of processing each mini-batch the overall efficiency. We show the advantage our approach in both the ablation study and real-world applications.
improve
Introduction
Many real-world applications are about interactions of multiple entities. For example, online shopping and promotion activities are interactions among customers, commodities and online merchants. A commonly used tool to analyze these high-order interactions is tensor factorization, which places the participant entities/objects in different tensor modes (or dimensions), and considers the interaction results as values of the observed tensor entries. Tensor factorization estimates an embedding repre- sentation for each entity, with which reconstruct the observed entries. The learned embeddings can reflect the underlying structures within the entities, such as communities and outliers, and can be used effective features for predictive tasks, such recommendation and ads auction.
Practical data often includes the timestamps when each multiway interaction occurred. These timestamps imply rich, complex temporal variation patterns. Despite the popularity of tensor factorization, current methods often ignore the timestamps, simply bin them into crude time steps (e.g., by weeks or months) and jointly estimate embeddings for the time steps (Xiong et al.) 2010 Rogers et al. 2013, Zhe et al. 2016a 2015) Du et al., 2018). Therefore, the current methods might severely under-use the valuable temporal information in data. More important, standard tensor factorization always estimates static embedding for each entity. However, the representation of entities, these embeddings summarize the underlying properties of the entities, and can naturally evolve along with time, such customer interests and preferences, user income and health, product popularity, and fashion. Learning static embeddings can miss capturing these interesting, important temporal knowledge.

---

Under review as a conference paper at ICLR 2023
To address these issues, we propose DMITRI, a dynamic embedding approach for temporal high-order interactions. We construct a nonlinear ceftusion-reaction process in an Ordinary Differential Equation (ODE) framework to jointly estimate embedding trajectories for the participant entities. The ODE framework is known to be flexible and convenient to handle irregularly sampled timestamps and sparsely observed data (Rubanova et al., 2019), which is often the case in practice. In addition, since ODE models focus on learning the dynamics (i.e., time derivatives) of the target function, they have promising potential for providing robust, accurate long-term predictions (via integration with the dynamics). Specifically, to leverage the structural knowledge within the data, we first build a multi partit- graph based on the observed interactions. The graph encodes the correlations between different types of entities in terms of their interaction history. We then construct graph diffusion process in the ODE to co-evolve the embedding trajectories of correlated entities. Next, we use neural network to construct reaction process to model the individual-specifft evolution for each entity. In this way, our neural diffusion-reaction process captures both the commonalities and personalities of the entities in learning their dynamic embeddings. Given the embedding trajectories, we model the interaction result as a latent function of the participants' trajectories. We use another neural network to flexibly estimate the function and to capture the complex relationships of the participant entities. For efficient training, we base on ODE solvers to develop a stochastic mini-batch learning algorithm. We develop a simple stratified sampling scheme, which can balance the cost of executing the ODE solvers in each mini-batch so as to improve the efficiency.
We evaluated our method in both simulation and real-world applications. The simulation experiments show that DMITRI can successfully capture the underlying dynamics of the entities from their temporal interactions, and recover the hidden clustering structures within the trajectories. Then in three real-world applications, we tested the accuracy in predicting the interaction results at different time points. DMITRI consistently outperforms the of-te-of-the-ar tensor factorization methods that incorporate temporal information, often by large margin. We also demonstrated that both the diffusion and reaction processes contribute to the learning and predictive performance. Finally, we investigated the learned embedding trajectories and found interesting evolution paths.
2 Notations and Background
Suppose we have collected data of interactions results between K types of entities (e.g., customers, commodities and merchants). Each type k includes da entities, and we index these entities by 1,...,dy. dg. We then index each interaction by tuple l = (42,...,kk) where for each k, we have 1 1 1x < dz. Suppose we observed N interactions. their results and timestamps. The dataset denoted by
{(e,, t1, y.)...., (Cy. tN. }N where {tn) and {yn} are the timestamps and interaction results. Our goal is for each entity j of each type k: to estimate a dynamic embedding (²(() RR + R. That is, the embedding a time function (trajectory) of R-dimensional outputs.
High-order interaction data can be organized as multidimensional arrays or tensors. For example, we can create a K K-mode tensor, and place the entities of type & in mode k. Each interaction is considered as an entry of the tensor, and the interaction result as the entry value. Hence, tensor factorization is a popular approach to process and analyze high-order interaction data. Standard tensor factorization introduces a static embedding representation for each entity, namely, už is considered as time invariant. Tensor factorization aims to estimate the embeddings (or factors) to reconstruct the tensor. For example, the classical Tucker decomposition (Tucker 1966) employs a multilinear factorization model, M = W x1 U X2... X UK where M E IR x is the entire tensor, W E R Re xxxxxxxxxxx is the tensor-core parameter, U comprises all the embeddings of the entities in mode kt and X is the tensor-matrix multiplication at mode & (Kolda 2006). The popular CANDECOMPIPARAEAS (CP) decomposition (Harshman) 1970) can be viewed as a simplified version of Tucker decomposition, where we set R1 = Rx R and the tensor-core W to be diagonal. Hence, each entry value is factorized as mę (u), ................... .u lx ) A. where is the Hadamard (element-wise) product, and À corresponds to diag (V). While CP and Tucker decomposition are popular and elegant, their multilinear modeling can be oversimplistic for complex applications. To estimate nonlinea relationships of the entities, Xu et al. (2012): Zhe et al.) (2015) (2] used a Gaussian process (GP) (Rasmussen and Williams, 2006) to model the entry value of the embeddings, me
random function u where 9 ~ GP (0, xex,, X2 = {u),- ...., / J and X = [u] ..... u } are the embeddings of the entities in entry l and e', respectively, and <(--)) is the covariance (kernel) function. Given the GP prior, any finite set of N entry values follow vartiav Gaussian distribution, m ~ (0, K), where m = [m,...... ,,,y I. K is the N x N kernel matrix, and each [K]. n(xe, ). Suppose we have collected continuous observations
x
2

---

Under review as a conference paper at ICLR 2023
Embeddinng entiiies of each type
Type1
Edges diffined by observed interaction
-7,07
Diftusion process along eóges
Reaction individual entibies
Time triectory
Table of observed data
uãto
t/ *0,
u²t0 uztt
Type
Type
Figure 1: The illustration of the embedding model in DMITRI.
for the N entries y= [1 Wi...... MN We can use Gaussian noise model: 1/n = me, + En where en ~ (O )²). The marginal likelihood of the observations is p(y) = (N + o IT)). We can maximize the likelihood to estimate the model parameters.
Practical interaction data often includes temporal information, i.e., the timestamp when each observed interaction occurred. To integrate this information, current methods often bin the timestamps into series of steps, say, by weeks or months (Xiong et al.] 2010 Rogers et al.) 2013 Zhe et al.] 20166 Song et al., 2017). The tensor then expanded with an additional time-step mode. We can apply an arbitrary tensor Factorization algorithm to estimate embeddings for both the entities and time steps. To learn the temporal dependency between the steps, conditional model is often used (Xiong et al.) 2010), say. p(t)+1/t,) = N(t;+1lty, rI) where ty is the embedding of j-th step. To leverage the continuous time information, Zhang et al. (202T) recently developed continuous CP decomposition, where the coefficients À are modeled as a time function with polynomial splines.
3 Model
While successful, current tensor factorization methods assume the embeddings are static and time- invariant. However, the embeddings essentially summarize/extract the properties of entities to give representation, and these properties can often evolve with time, such as customer interests, health status, and product popularity. Therefore, only estimating static embeddings can miss capturing im- portant temporal variations of the entities' properties, resulting in poor representations and predictive performance. To address this issue, we propose DMITRI, a novel dynamic embedding approach.
Specifically, we propose an ODE model to learn the embedding trajectories {uf (f)]1 S k S K, 1 j S J. The ODE framework is known to be amenable for irregularly sampled, sparsely observed data, which is often the case in practice. More important, ODE models concentrate on learning the time derivative du /dt (i.e., dynamics), rather than the trajectory function itself. Therefore, they have a promising potential to give reliable, long-term trajectory prediction (via numerical integration) even at time points far away from the training timestamps, provided the time derivative is well captured. We construct joint ODE model for all the embedding trajectories. The ODE consists of diffusion process and reaction process. The diffusion process leverages the structural knowledge in data to co-evolve the embeddings of correlated entities, so as to better overcome the data sparsity. The reaction process models the entity-speciff evolution so that it capture the individual differences in the embedding evolution. The ODE model synergizes the two processes to capture both the commonalities and personalities of these embedding trajectories.
Diffusion Process on Multi-Partite Graphs. First, we construct graph-based diffusion process to exploit the entity correlations reflected in data D. Intuitively, if a K-way interaction involves entity A (e.g., customer A) and B (e.g., commodity B), the two entities are likely correlated. Thus, we can draw an edge between A and B express the correlation. We can then generalize this intuition to create K --partite (undirected) graph G V), to encode all such correlations across all the entities in the data. Each vertex represents a particular entity, and the entire collection of the entities is partitioned into K groups, V = V' U... ......P Vk where group V = {uf ...., o represents the entities of type k. Two entities (of different types) are connected if they were observed to interact, namely, (vf ) E IE there is some interaction e, in D such that l, = (............ J......) where j and j' are the indices of k-th and k'thh participants, respectively. That is, In j and Ln = j'- See Fig. for an example. This graph can naturally imply some underlying information diffusion across the entities within their interactions. For example, if customer A connects to products B and C, might mean that A distributes their interest/willlinness//eee to purchase B and C. The edges
3

---

Under review as a conference paper at ICLR 2023
between one merchant A and list of products (B, C.... might indicate the diffusion of willingness to increase the inventory of these products.
k.k' To flexibly estimate the diffusion rate, we introduce a weight "j for each edge (vf) of.) E E. We can then arrange these weights into K(K + 1)/2 adjacent matrices, W = (W 1 S k I K, k * k.. Each Wk. is a sparse dg x dæ matrix that represents the edges and edge weights between V and VKK, i.e., [W 13,y =
k.k'' ", if (uf, ot)) E and otherwise. We now construct j.
diffusion process based on the K --partite graph. We view the embedding trajectory as kind of concentration. For each entity of type k the change rate of its concentration (embedding) uf is determined by the difference from the concentrations of its neighbors. Since the neighbors can come from entities of all the other K types, we have
du dt
Σ (E(L....KKKAA
Σ Wh bs (uf-(1--ujae uu(())
E (w) ((()) SE(L....KK.
aa-" uuf
where w is the j-th row of Whe (t) = (t). uz, (t)) is the embeddings of all the entities of type of size d, x R. and aj E [W is the degrec of vertex in Wk..s Consider (L) the embeddings of all the entities of type k. therefore have
dt) = E EOE(LL.KRA Wk()U((t A U*)
(1)
dt
k..) where A R.s diag(a ..... dd is the degree matrix of W We can see that the evolution of the embeddings for different types of entities are coupled. Hence, natural formulate the diffusion process jointly for all the embeddings.
(t) :
du(t)) =d Øt
/d = wu(() At((t) (W- - A))(t)
(2)
(K(
where
EveLL. A
0
W1,2
WIK
W
W2,1
0
A =ddiag
Exe(I..KrAA A
WK-1K
W WKK-1
EEELL A K.s
Reaction Process of Individual Entities. Next, to capture the individual difference of each entity in evolving their embeddings, we model local reaction process for each entity, fe, (u, (t), t), where f(-) a neural network (NN), and 0₂ are the NN (reaction) parameters for type-k entities") The metaphor from the chemical physics is as follows. While substances are being diffused across different sites, at each site a chemical reaction process happens concurrently, which varies the concentration locally. We extend the model
du Σ &e(L...,
Σ de E/=1 y=
W "is"" (1² (()))()((())) (t) fo, (u²,
(3)
dt
KYW
Our joint diffusion-reaction ODE model is therefore specified as follows,
au(() (W - A)U(()) + F(U, u(0) = Uo.
(4)
ØL
where F(U,t) [fe, (u) t).................... ... fo, (u), t), ..... fox((u (u t).... ..... fex (u t)]
Note that while the reaction model is the same and each type of entities share the same set of reaction parameters, those entities will have different reaction results due to the difference the input f.
4

---

Under review as a conference paper at ICLR 2023
Interaction Result Generation. Given the embedding trajectories, to obtain the interaction result me at arbitrary time t, we model me (t) as a function of the relevant embeddings at time t.
me(t)
9 (u), (t).....
K uix (t))
(5)
While one can follow (Xu al.
2012 Zhe et al. 2016b) to assign GP prior over g()), the GP model needs to compute a giant kernel matrix over all the observed interaction results (see Sec.
2}. It is computationally too expensive or infeasible when the number of observations is large. Hence one has to seek for complex sparse approximations. To avoid this problem, we model g with another neural network, which is not only as flexible as GP, but is more scalable and convenient for computation. Since now, the input to g(.) consists of the trajectory values, which vary with time, our NN model for can flexibly capture the complex temporal relationship of the entities. We finally sample the observed interaction results with Gaussian noise model. p(y m) = (y | o I²I) where y [vi,-.., y and m [me, (f1),... me, (tw)] We focus on real-valued data in this paper. However, is straightforward to extend approach to other types of data.
4 Model Estimation
We now present the model estimation algorithm. Given data D ((1,, BI)...., (EN. UN) the joint probability of our model is
[0x)
p(3) II p(Oz)
N Wynlo (ynlg (u..,((m)),... (bx),.... II
u (In)) o TT))
(6)
where 3 is the NN parameters for g. each 02 is the NN reaction parameters for type-k entities, their prior p((3) and p(O)) is element-win wise standard Gaussian, and y = [.::...: trajectory values in the Gaussian likelihood of each Ya, we need to solve the ODE in 4 to time t.
: yv To obtain the
U(t,) ODESolve(24) 0, θ)
where θ (W. 0r..,...,0xx
0xx consists of the ODE parameters. Our goal is to estimate the ODE parameters e, the initial state Uo, the NN parameters B, and the noise variance o².
Stratified Mini-Bu Batch Sampling, We use stochastic mini-batch optimization maximize the log joint probability so as to estimate all the required parameters,
f log p(3, (0)).) = log((rior)) -E .N
N (xn) (xx), o²
where log (ogiorir = log p()) + E log)(O>). and x, [u], (tx)...., u (tn)]. Each time, we sample mini-batch of interaction results B, and obtain unbiased stochastic estimate of the log probability, - (Prior) N/ EnEB [log (Vvnlg(xx)) We compute VL the stochastic gradient to update all the parameters.
K
For each data point n in the mini-batch, we need to run ODE solving [u/ (lx),..., u (In)]- To back-propagate the gradient so as to compute the gradient w.r.t the ODE parameters and initial state 24, we can either construct a computational graph during the running of the solver (e.g., the Runge- Kutta method (Dormand and Prince 1980)). use the adjoint state method (Pontryagin) 1987) Chen et al.] 2018) that solves an adjoint backward ODE to compute the gradient. In whichever case, we need sort the time points in the mini-batch and solve the ODE sequentially for these time points. As a result, the number of unique time points in the mini-batch greatly influences the speed of processing the mini-batch. The standard mini-batch sampling (based on the training example indices) can result in an uneven allocation of the computational cost across the mini- maiibbsthes some address this issue, we use a simple stratified sampling approach.
obtain X,,
is fast and some including more time points much slower. To
We collect the unique time points in the whole dataset, T= (TI, ......) .) the beginning. To conduct each stochastic update, we first sample B unique time points C from T. then for each time point т; E C, we look at all the interactions occurred at Tj, namely Dr, = [Le., Inx 3n) D)t, = T.
We randomly sample one interaction example from each Dr, to collect the mini-batch B.
In this way, ensure the cost of running ODE solvers and related gradient computation in each mini-batch is identical. There are no fluctuations in cost/running time when processing different min-batches. Empirically, we found that the overall speed of our method is much faster than vanilla stochastic bini- optimization.
5

---

Under review as a conference paper at ICLR 2023
5 Related Work
To integrate the temporal information, most tensor factorization approaches augment the tensor with
time mode (Xiong al. 2010 Rogers et al.) 2013 Zhe al., 2016bb Ahn et al. 2021; Zhe et al.) 2015; Du et al. 2018) 8). which includes a list of time steps. They jointly estimate the embeddings for entities and time steps. To estimate the temporal dependencies, existing methods often employ
dynamic model over the time steps, such as a conditional Gaussian prior (Xiong et al. (2110) recurrent neural networks (uu et al. al. 2019), and kernel smoothinggregulatzzionn (Ahn et al. (021). To leverage continuous timestamps, /Zhang et al. (2021) modeled the CP coefficients X as a time function, and estimated it with polynomial splines. The most recent work (Wang and Zhe| 2022) places GP prior in the frequency domain, and construct bi-level GP model to learn factor trajectories combination of Fourier bases. Another set of works factorize the interaction events (Schein et al.
2015 2016: Zhe and Du 2018 Pan et al.) 2020: Wang et al.) 2020), rather than the interaction results (e.g., purchase amount and product ratings). These works mainly leverage Poisson processes, Hawkes processes, more general point processes to estimate the event rate. Like the standard tensor factorization, these methods also estimate static embeddings for the event participants.
Our model can be viewed only employ the reaction process for each entity, our model is a latent neural ODE (we have an additional NN that combines the latent trajectories to predict the interaction results). However, we further leverage the structural knowledge in data to construct multi-partite graph so as to encode the correlations of the participants in their high-order interactions. Based on the multi multi-partite graph, we construct a diffusion process to co-evolve the embeddings of the entities. In doing so, we can better overcome the data sparsity issue. Our work is also related to (Rubanova et al. 2019). which uses neural ODEs to model the state transition of recurrent neural networks (RNNs) or to build auto-regressive models for time series analysis. The major difference that our work deals with time series of high-order interactions, and our ODE is used to model the evolution of the embeddings of the interaction participants (rather than the RNN states). Many other works have leveraged/devcloped graph diffusion processes given the graph data. For example. Chamberlain et al. (2021) proposed a graph neural network (GNN) by using multi-head attention to construct the adjacent matrix for a graph diffusion equation over the graph nodes. Atwood and Towsley (2016) introduced diffusion operator to develop diffusion-connolutionn
extension of the neural ODE model (Chen al. 2018). If we
neural networks. |Huang et al. (202)) developed a GNN based ODE to model both the nodes and edges in dynamic graphs.
6
Experiment
6.1 Ablation Study
We first examined DMITRI on a synthetic task. Specifically, we considered interactions among two types of entities, where each type includes 20 entities. Each entity has one underlying embedding trajectory. For type-1, the trajectory of each entity is an exponential function, nt((t) = cj txp(0.5c).) (1 و < 20), while for type-2 is linear function, u² (t) = of + 2T.²t. We generated two clusters of trajectories for each entity type, where those of the first ten entities form the first cluster and the remaining the second cluster. To this end, for type 1, we sampled the coefficients of the first ten entities' trajectories, namely,
ciu]" from A/ ([-5...... -5] 0.1I). and the remaining ten coefficients [ct....., cboo from N (10.5.... 0.5) 0.IIT. Then for type 2, we sampled each coefficient conditioned on its counter-part for type 1, cgle] ~ NNec" le), 0.1). That means, the coefficients of cluster-I I trajectories across the two event types are close, and so are those of cluster-2 trajectories. The result for particular interaction l = (42,42 is generated by
(1()) = (ui, (0) 1(1+412 mod 2=0) (uZ, (1)) 111+12 mod 2=1)
where (-)) is the indicator function. When 1 + 12 is even, the interaction result is the trajectory value of the first entity; otherwise, it is the trajectory value of the second entity. To generate the training data, we randomly sampled interactions from {(4 (2)] < 12 I 10) U ((4, 42]11 < 41 20} (namely, interactions between cluster-I entities of the two types, and between cluster-2 entities). We then sampled t ~ Unifrom[0,, 5), to obtain the interaction results. We randomly generated 6,400 interaction results and the timestamps for training, and another 1,600 data points for testing.
We implemented method with Pytorch (Paskke et al. 2019). We used torchdi ffeq library tht : httpps://gthhli //ghthbb.tt.t///// com/ wrgtichene tt/ corchdifeege to solve ODEs and to compute the gra- dient w.r.t ODE parameters and initial states via automatic differentiation. For the NN of the

---

Under review as a conference paper at ICLR 2023
reaction process, we used one hidden layer, with 10 neurons and tanh activation, and for the NN to predict the interaction result, we used two hidden layers, 50 neurons per layer and tanh activation. We compared with NONFAT (NONparametric Factor Tra- jectory learning) (Wang and Zhe, 2022), a bi-level la- tent GP model that uses Fourier bases estimate factor trajectories for dynamic tensor decomposition. To our knowledge, this work the only method (and also the
most recent) that also estimates trajectories. We used com/
the original implementation wzhut / and the default settings. We set the mini-batch size 50, and used ADAM (Kingma and Ba 2014) algorithm for stochastic optimization. The learn- ing rate was automatically adjusted in [10-4 1001 by the ReduceLROnP Setonnlattes lat scheduler (al-Kababji et al. 2022). The maximum number of epochs is 2K, which is enough for convergence. The estimated trajectories are
(a) Ground- trunt: type(b) Ground-trute type
shown in Fig. 2 As we can see, our estimation (Fig. and 2f well matches the ground-truth and accurately recovers the cluster structure of the trajectories. The root-
(c) NONFAT: Type
(d) NONFAT: Type 2
mean-square error (RMSE) on the test is 0.032. By
contrast, although the test eπrоr of NONFAT is close to DMITRI (0.034), its learned trajectories (Fig. and 2d] are far from the ground-truth, and fail to reflect the cluster structure. This might be due to that GP models are difficult to capture the interaction function (8). which is switch function. These have shown the advantage of DMITRI in capturing complex relationships within data to recover the underlying trajectories and their structure.
(e) DMITRI: Type (f) DMITRI: Type
The estimated embedding trajecto- ries for each entity type. The color indicates the ground-truth cluster memhership
Figure 2:
6.2 Prediction Accuracy
Datasets. We next evaluated the predictive performance of DMITRI in three real-world applications.
(1)
CA Weather
(Moosavi
et
al.
2019)
(https:
0.08
//smoosavi.o org/
0.5
FRASE
datasets/ista. weather
0.4
0.06
conditions in California from
0.65
August 2016 December 2020. We extracted four-way interac- tions among different weather types, 6 severity levels, 30 lati- tudes and 30 longitudes in GPS coordinate. The interaction re- sult the count of the particular weather condition. We collected 15K interactions and the timestamps. (2) CA Traffic (Moosavi et al. 2019) /tttps::/smoossass accidents in California from January 2018 to December 2020. We extracted four-way interactions between traffic type, severity level, latitudes, longitude. There are 7 traffic types, 6 severity levels, 20 latitudes and 20 longitudes. We collected 30K interaction results (accident counts) at different time XINNAi gzaM8.
0.3
0.04
5
(a) CA Weather
(b) CA Traffic
(c) Server Room
Figure 3: Predictive performance of the diffusion and reaction processes
org/ / wtt.aatttteetttee / stw),, traffic
points. (3) Server Room (https: /twpoooo.ooooo//oooooooooooooo. org/ http://eooodo1coee////
temporal temperature records of Poznan Supercomputing and Networking Center. The temperatures were measured at 34 locations, under different air-condition modes (24 27", and 30") and power usage settings (50%, 75% and 100%). Hence, we extracted three-way interactions (location, air-condition mode, power level). In total, 10K interactions and their timestamps were collected.
Competing Methods. The following popular and/or state-of-the-art temporal factorization ap- proaches were compared. (1) CP-DTLD, discrete-time CP decomposition with linear dynamics, where a conditional prior is placed over successive time steps, p(t;+ilt,)) N NN(+++) At, + b, el); A, b and v were jointly estimated during the CP decomposition. Note that (Xiong al. 2010) is an instance of
this model where A = I and b = O. (2) GP-DTLD and (3) NN-DTLD, similar to CP-DTLD, except using GP (Zhe et 2016b) and NN decomposition models (similar to (Liu et al. 2019) ) respectively.
7

---

Under review as a conference paper at ICLR 2023
CA Weather
R 2
R 3
R=3
R
CP-DTLD
0.7440 0.0035
0.7372 1 0.0040
0.7290 1 0.0042
0.7270 1 0.0044
GP-DTLD
0.7417 1: 0.0031
0.7414 + 0.0036
0.7444 1: 0.0036
0.7449 +: 0.0039
NN-DTLD
0.7228 + 0.0054
0.7116 0.0033
0.7070 = 0.0041
0.7065 0.0038
CP-DTND
0.7448 0.0031
0.7360 = 0.0035
0.7273 = 0.0037
0.7280 0.0044
GP-DTND
0.7399 0.0034
0.7346 = 0.0082
0.7448 = 0.0037
0.7467 0.0031
NN-DTND
0.7113 + 0.0045
0.6979 + 0.0126
0.6659 + 0.0122
0.65-13 + 0.0155
CP-CT
1.0000 + 0.0096
0.9959 = 0.0067
1.0010 + 0.0017
1.0060 +: 0.0034
GP-CT
0.7433 0.0038
0.7354 0.0027
0.7359 + 0.0034
0.7377 + 0.0033
NN-CT
0,8697 0.0014
0.8679 = 0.0022
0.8676 = 0.0018
0.8695 0.0016
NONFAT
0.7444 + 0 0.0042
0.7460 = 0.0032
0.7645 = 0.0061
0.7553 - 0.0029
DMITRI
0.6327 + 0.0119
0.6109 = 0.0056
0.6172 = 0.0075
0. 0.6354 0.0085
CA Traffic
CP-ITLD
0.6498 0.0257
0.6424 = 0.0266
0.6436 + 0.0268
0.6405 + 0.0262
GP-DTLD
0.6309 0.0167
0.6290 0.0185
0.6383 + 0.0204
0.6496 + 0.0193
NN-DTLD
0.6528 0.0230
0.6545 0.0244
0.6401 + 0.0282
0.6136 0.0338
CP-DTND
0.6497 0.0245
0.6456 + 0.0265
0.6431 1 0.0263
0.6419 1 0.0259
GP-DTND
0.6544 1: 0.0213
0.6559 + 0.0224
0.6604 1 0.0243
0.6674 1: 0.0214
NN-DTND
0.6578 + 0.0248
0.6528 = 0.0256
0.6519 0.0249
0.6482 = 0.0261
CP-CT
0.9858 0.0120
0.9972 = 0.0056
0.9816 0.0136
0.9991 0.0120
GP-CT
0.6610 0.0207
0.6668 = 0.0191
0.6756 = 0.0190
0.6768 + 0.0196
NN-CT
0.9804 + 0.0017
0.9815 = 0.0015
0.9791 + 0.0012
0.9802 + 0.0017
NONFAT
0.4461 + 0.02/17
0.4610 = 0.0231
0.5031 + 0.0155
0.6307 + 0.0817
DMITRI
0.3601 0.0334
0.2972 0.0099
0.3174 + 0.0118
0.3269 0.0162
Server Room
CP-DTLD
0.4211 0.0029
0.4209 0.0031
0.4208 = 0.0028
0,4208 :: 0.0028
GP-DTLD
0.0914 0.0020
0.0791 1 0.0010
0.0739 = 0.0014
0.0753 - 0.0013
NN-DTLD
0.4213 +: 0.0032
0.4213 = 0.0032
0,4212 : 0.0034
0.4205 +: 0.0030
CP-DTND
0.2835 + 0.0160
0.1751 = 0.0020
0.1174 = 0.0011
0.0829 + 0.0044
GP-DTND
0.0925 + 0.0013
0.0784 0.0011
0.0739 0.0009
0.0774 + 0.0009
NN-DTND
0.4213 0.0033 0.0032
0.4212 0.0030
0.4211 = 0.0032
0.4205 + 0.0030
CP-CT
0.9919 1 0.0099 0.0096
0.9951 + 0.0050
0.9862 1 0.0109
1.0121 1: 0.0070
GP-CT
0.1385 1 0.0020
0.1223 + 0.0016
0.1275 1 0.0014
0.1365 1: 0.0014
NN-CT
0.1193 1 0.0030
0.1140 0.0015
0.1113 1 0.0027
0.1149 1 0.0028
NONFAT
0.1468 0.0026
0.1407 = 0.0023
0.1396 = 0.0022
0.1409 0.0030
DMITRI
0.0536 0.0031
0.0403 = 0.0014
0.0393 = 0.0018
0.0403 + 0.0027
Table 1: Normalized Root Mean-Square Error (nRMSE), The results averaged from five runs.
(4) CP-DTND, (5) GP-DTND and (6) NN-DTND CP, GP and NN de- composition with nonlinear dynam- ies, where the conditional prior is ((t)+ilt,) ((tt++io((tt)
vI) where a()) is an nonlinear activa- tion. The dynamics can therefore be viewed as an RNN transition. (7) CP- CT (Zhang et al. 2021), continuous- time CP factorization, which models the CP coefficients as a time-varying function, with polynomial splines. (8) GP-CT, continuous-time GP decompo- sition that extends (Xu al.) 2012] Zhe et al.] 2016b) by plugging the time in the GP kernel so as to esti- mate the interaction result as func- tion of the embeddings and time, me =g(u), ..... u t). (9) NN- CT, continuous-tim
(a) (t)
(b) 2
(c) (1,)
(d) (t)
(c) (t)
( a
Time
NN decomposi- tion, where the input consists of both the embeddings and time t. (10) NON-
(g) x
(h) (t)
(t)
Figure 4: The learned embedding trajectories for location 1 (a-c), air conditional mode (d-f), and power usage level I (g-i) in Server Room dataset.
FAT (Wang and Zhe 2022
8

---

Under review as a conference paper at ICLR 2023
Settings and Results. All the approaches were implemented with PyTorch. The Square Exponential kernel was used for all the GP related methods, including GP-(DDLD, DTND, CT). We used the same variational sparse approximation (Hensman et al. 2013) to fulfill scalable posterior inference. Following (Zhe et a 2016b), the number of inducing point was set to 100. For the NN decomposition methods, we employed three-layer network with tanh activation. The layer width was chosen from 110, 25, 50, 75, 100). We used tanh as the activation function in the nonlinear dynamic baselines, including (CP, GP, -N For our method, we used the same NN architecture for both the reaction process and interaction result prediction, which includes two hidden layers with 50 neurons per layer. For CP-CT, we employed 100 knots to fulfill the polynomial splines. For each discrete-time method, the number of time steps was chosen from (25, 50, 75, 100) via the cross-validation on the training set. We trained all the models with stochastic mini-batch optimization. We used the ADAM algorithm, and the mini-batch size was set to 100. We ran every method with 10K epochs to ensure convergence. The learning rate was automatically adjusted in [10- 10 by the ReduceLRon ea scheduler. We varied the dimension of the embeddings R from (2. 3. 7). For DMITRI, R is the number of embedding trajectories; we used computational graphs to obtain the gradient. We followed (Xu et al. 2012 KKng al. [012] Zhe et al., 2016b) to randomly draw 80% observed interactions and their time stamps for training, with the remaining for test. We computed the normalized root-mean-square error (nRMSE). We repeated the evaluation for five times and computed the average nRMSE and standard deviation.
As shown in Table | DMITRI consistently achieves the best prediction accuracy. and in many cases outperforms the competing methods by a large margin. Although learning an embedding trajectory much more challenging than learning fixed-value embedding, the experimental results have demonstrated the advantage of our method in predictive performance. To investigate the effect of the two processes in our model, we also examined our method with the diffusion process only and reaction process only on all the datasets. Their predictive performance, as compared with DMITRI, shown in Fig. 3 We can see that each individual component can lead to good prediction accuracy. However, each component is worse than DMITRI that synergizes the two components together. Therefore, the results show that each process is effective, and more important, the two processes can bolster each other further improve the performance when they combined.
6.3 Learning Result Investigation
Next, we looked into the learned embedding trajec- tories and checked if they exhibit patterns. To do so, we set R and ran DMITRI on Server Room dataset. In Fig. 4 we show the learned embedding trajectories for the first location (a-c), the first air condition mode (d-f) and the first power usage level (g-i). As we can see, even for the same object, e.g., a particular location, the corresponding embedding trajectories vary quite differently, implying the evo- lution of different underlying properties, such as the Figure 5: Interaction result prediction on Server workload, memory usage, and network latency.
DMTTR Training Testing Data
(a) (2, 1, 1)
(b) (3, 1, 1)
Room.
Finally, we showease the temporal predictions for two interactions. As we can see from Fig. [ given only few training points (blue), our method can predict the test points (green) much more accurately, as compared with GPCT, and the predictive uncertainty (reflected by the noise variance a²) is much smaller. This might be due to that via diffusion-reaction process, our method can more effectively extract the temporal knowledge from sparse data. For example, DMITRI successfully captured the periodic nature in the first interaction (Fig. 5a) while GPCT treated the fluctuation as noises and ended up with much inaccurate predictions and much larger predictive variances.
7 Conclusion
We have presented DMITRI, neural diffusion-reaction process model to learn dynamic embeddings from high-order interaction time series. The predictive performance is encouraging and the learned embedding trajectories exhibit interesting patterns. Currently, our method is limited to a small number of entities since it has to integrate the entire multi martiipartiiee graph to construct the diffusion process. In the future work, plan to develop graph cut algorithms to partition the graph into set of small sub-gu aab-graphs, so that we can construct multiple diffusion processes in parallel so as to scale up to big graphs and a large number of entities.

---

Under review as a conference paper at ICLR 2023
References
Ahn, D., Jang, J.-G., and Kang, U. (2021). Time-aware tensor decomposition for sparse tensors. Machine Learning, pages 1-22.
Al-Kababji, A., Bensaali, F., and Dakua, (2022). Scheduling techniques for liver segmentation: Reducelronplateau vs onecycleir. arXiv preprint arXiv:2202.06373.
Atwood, J. and Towsley, D. (2016). Diffusionn-oonvlutionaa neural networks. Advances in neural information processing systems, 29.
Chamberlain, B., Rowbottom, Gorinova, M. I., Bronstein, M., Webb, S., and Rossi, E. (2021). Grand: Graph neural diffusion. In International Conference on Machine Learning, pages 1407- 1418. PMLR.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems. 31.
Dormand, R. and Prince, (1980), family embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26.
Du, Y., Zheng, Y., Lee, K.-.,, and Zhe, S. (2018). Probabilistic streaming tensor decomposition. In 2018 IEEE International Conference on Data Mining (ICDM), pages 99-108. IEEE.
Harshman, R. A. (1970). Foundations of the PARAFAC procedure: Model and conditions for an" "neeellanaarr"" ixxplantory"multi-modo muuli-mooe factor analysis. UCLA Working Papers in Phonetics, 16:1-84.
Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 282-2901 AUAI Press.
Huang, Z., Sun, Y., and Wang, W. (2021). Coupled graph ode for learning interacting system dynamies. In 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021, pages 705-715.
Kang, U., Papalexakis, E., Harpale, A., and Faloutsos, C. (2012). Gigatensor: scaling tensor analysis up by 100 times- algorithss and discoveries. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 316-324. ACM.
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. cite 6980Comment: Published as a conference paper at the 3rd International Conference for arxi:11412.6800Cmmeen
Learning Representations, San Diego, 2015.
Kolda, G. (2006). Multilinear operators for higher-order decompositions, volume 2. United States. Department of Energy.
Liao, R., Zhao, Z., Urtasun, R., and Zemel, R. S. (2019). Lanczosnet: Multi-scale deep graph convo-lutional networks. In 7th International Conference on Learning Representations, ICLR 2019.
Liu, H., Li, Y., Tsang, M., and Liu, Y. (2019). CoSTCo: A Neural Tensor Completion Model for Sparse Tensors, page 324-334. Association for Computing Machinery, New York, NY, USA.
Moosavi, S., Samavatian, M. H., Nandi, A., Parthasarathy, S., and Ramnath, R. (2019). Short and long-term pattern discovery large-scale geo-spatiotemporal data. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2905-2913.
Pan, Z., Wang, Z., and Zhe, S. (2020). Scalable nonparametric factorization for high high-order interaction events. In International Conference on Artificial Intelligence and Statistics, pages 2325-4335. PMLR.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.. Chanan, G., Killeen, T., Lin. Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8033
10

---

Under review as a conference paper at ICLR 2023
Pontryagin, L. S. (1987). Mathematical theory of optimal processes. CRC press.
Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.
Rogers, M., Li, L., and Russell, S. J. (2013). Multilinear dynamical systems for tensor time series. Advances in Neural Information Processing Systems, 26:2634-2642.
Rubanova, Y., Chen, T. Q.. and Duvenaud, D. K. (2019). Latent ordinary differential equations for irreglarly-samplee time series. In Wallach, H., Larochelle, H., Beygelzimer, A., dAlché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Schein, A., Paisley, J., Blei, D. M., and Wallach, H. (2015). Bayesian poisson tensor factorization for inferring multilateral relations from sparse dyadic event counts. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1045-1054. ACM.
Schein, A.. Zhou, M., Blei, D. M., and Wallach, H. (2016). Bayesian poisson tucker decomposition for learning the structure of international relations. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, pages 2810-2819. JMI..orrg
Song, Q. Huang, X., Ge, H., Caverlee, J., and Hu, (2017). Multi-aspect streaming tensor completion. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 435 443.
Tucker, L. (1966). Some mathematical notes on mhree-mode factor analysis. Psychometrika, 31:279- 311.
Wang, Z., Chu, X., and Zhe, S. (2020). Self-modulating nonparametric event-tensor factorization. In International Conference on Machine Learning, pages 9867.9967.. PMLR.
Wang, Z. and Zhe, S. (2022). Nonparametric factor trajectory learning for dynamic tensor decompo- sition. In International Conference on Machine Learning, pages 23459-23469, PMLR.
Wu, X., Shi, B., Dong, Y., Huang. C., and Chawla. N. V. (2019). Neural tensor factorization for temporal interaction learning. In Proceedings of the Twelfth ACM International Conference Web Search and Data Mining, pages 537-545.
Xiong, L., Chen, X., Huang, T.-K., Schneider, J., and Carbonell, J. G. (2010). Temporal collabora- tive filtering with bayesian probabilistic tensor factorization. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 211-222. SIAM.
Xu, Z., Yan, F., and Qi, Y. A. (2012). Infinite tucker decomposition: Nonparametric bayesian models for multiway data analysis. In ICML.
Zhang, Y., Bi, X., Tang, N., and Qu, A. (2021). Dynamic tensor recommender systems. Journal of Machine Learning Research, 22(65):1-35.
Zhe, S. and Du, Y. (2018). Stochastic nonparametric event-tensor decomposition. In Advances in Neural Information Processing Systems, pages 6856-6866.
Zhe, S. Qi Y., Park, Y., Xu, Z., Molloy, I., and Chari, S. (2016a). Dintucker: Scaling up gaussian process models on large multidimensional arrays. In Thirtieth AAAI conference on artificial intelligence.
Zhe, S., Xu, Z., Chu, X., Qi, Y., and Park, Y. (2015). Scalable nonparametric multiway data analysis. In Proceedings of the Fighteenth International Conference on Artificial Intelligence and Statisties, pages 1125-1134
Zhe, S., Zhang, K., Wang, P., Lee, K.-c., Xu, Z., Qi, Y., and Ghahramani, Z. (2016b). Distributed flexible nonlinear tensor factorization. In Advances in Neural Information Processing Systems, pages 928-936.
11


Reviews:
[{"review_id": "v1a0A8wasi", "reviewer": "Reviewer_cuTX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the DMITRI framework to generate dynamic embeddings for multi-entity interactions with timestamps. A multi-entity interaction involves potentially many types of entities, e.g. traffic type, severity, latitude, and longitude, and consists of a timestamped observation for those entities, such as the number of traffic accidents under those conditions at different times. The DMITRI framework uses an ordinary differential equation (ODE) model to learn embedding trajectories for each entity of each type. The authors consider both a diffusion process across multi-partite graphs, representing interactions between different types, and a reaction process for the individual entities. They demonstrate improvements in predictive accuracy on 3 real datasets compared to a large number of existing methods.", "summary_of_the_review": "I find this to be a novel and interesting contribution with strong empirical results for prediction accuracy, but also a few weaknesses in the experimental evaluation that prevent me from strongly recommending acceptance.\n", "strength_and_weaknesses": "Strengths:\n- Proposed DMITRI framework generates dynamic embeddings for entities in continuous, rather than discrete time, unlike most existing dynamic tensor factorization approaches.\n- Use of neural ODEs to generate embeddings for multi-entity interactions appears to be novel.\n- Comparisons to a wide range of competing methods yields impressive predictive accuracy on 3 real datasets.\n\nWeaknesses:\n- Minimal interpretations of the learned embedding trajectories. Beyond predicted observations, an additional output of DMITRI is the continuous-time embedding trajectories. The authors don't provide many interpretations of the trajectories, with only an example in Section 6.3.\n- No empirical results on computation time. The authors state that their approach is limited to a small number of entities due to the multi-partite graph approach for the diffusion process. I would have liked to see some experiments (perhaps in the supplementary material) comparing wall clock time to competing methods.\n\nQuestion:\n1. On page 4, you state that there are $K(K+1)/2$ adjacency matrices $\\mathbf{W}^{k,k'}$. Why $K(K+1)/2$ rather than $K(K-1)$? It appears that we can't have edges within the same type $k$, so no diagonal, and that there is no symmetry between $\\mathbf{W}^{k,k'}$ and $\\mathbf{W}^{k',k}$.\n\nTypo:\n- Page 6, second last paragraph: Unifrom\\[0,5\\] -> Uniform\\[0,5\\]", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "up7llHlqs09", "reviewer": "Reviewer_eUMJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a neural diffusion-reaction process model, which can take into account the timestamps and model the temporal embedding of participants behind high-order interactions, compared with existing related tensor factorization methods.\n\n- To that end, they build the model upon the ODE framework that can handle continuously observed timestamps.\n- They build a multi-partite graph on observed interactions, and learn a graph diffusion process to allow the dynamic embeddings of correlated participants to co-evolve over time.\n- A separate neural net is used to construct a reaction process to capture individual difference.\n- The final interaction result is captured using a Gaussian observation model, of which the mean is a function of the dynamic embedding.", "summary_of_the_review": "This paper develops a new model to capture timestamps and to model dynamic embedding of entities behind high-order interaction time series, which cannot be well captured using existing tensor factorization solutions.\n\nTo me, the method looks interesting and solid, but somewhat novel in the motivated context.\n\nIn addition, the final experiment is not entirely clear to me.", "strength_and_weaknesses": "Strength :\n\nThe paper targets an interesting question, and the method looks solid.\n\nWeaknesses :\n\nTo me, the technical part of Sec. 3 is not that easy to digest. For instance, it is necessary to thoroughly explain why to construct the diffusion and reaction processes in this way, for someone else to take it up.\n\nFor the results in Table 1, it is entirely unclear that what does the RMSE error reflect? On pp.9, you only mention use remaining for test. But, I cannot find what are you predicting in the test set!", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TVL0LMj-Cz", "reviewer": "Reviewer_Desm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets modeling the temporal high-order interactions via neural diffusion-reaction processes. \nIt proposes a Dynamical eMbedIngs of TempoRal hIgh-order interactions (DMITRI) framework. \nA multi-partite graph is built to encode the correlation between the entities. \nA graph diffusion process is constructed to co-evolve the embedding trajectories of the correlated entities and a neural network is used to capture the reaction process for each individual entity. ", "summary_of_the_review": "Considering [1], I’m not sure about the theoretical contributions and the results are SOTA. ", "strength_and_weaknesses": "Strengths\n\n(1) This paper provides sufficient details in both theoretical and experimental sections. Although the authors didn’t release their code, I think it’s not hard to implement their work.\n\n(2) The experimental results seem quite encouraging compared with the current baselines.\n\nWeaknesses\n\n(1) I suggest the authors reorganize this paper carefully. The author should emphasize their contributions concisely and accurately. In the current version, both the abstract and the introduction are filled with too many details and make me hard to capture what are the key contributions that the authors would like to present. It would be even better if the author could add a figure to demonstrate the architecture of the proposed framework. \n\n(2) Could the authors discuss the correlations and differences with the work of [1]? It seems to have quite similar ideas with this paper, both the motivations and the methodologies. The authors should at least take it as a baseline. \n\n[1] Li, Shibo, Robert Kirby, and Shandian Zhe. \"Decomposing Temporal High-Order Interactions via Latent ODEs.\" In International Conference on Machine Learning, pp. 12797-12812. PMLR, 2022.\n\n(3) The authors mention in the conclusion the proposed method is limited to a small number of entities. I suggest the authors provide some discussions about the algorithm complexity. Although the authors plan to develop graph cut algorithms to improve the scalability, I wonder how effective this algorithm can be.\n\n(4) Typos: The last line of page 1, “these” should be “this”.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FV4B5iC6mvR", "reviewer": "Reviewer_U7s8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors address the problem of temporal high-order interactions where more than two entities -- such as customers, commodities, and merchants -- correspond to one event. To tack the temporal drift of underlying node, authors propose the dynamic embeddings that follows a diffusion-based ODE model. Authors show that the proposed DMITRI model can capture the temporal dynamic well through both synthetic and real-world datasets. ", "summary_of_the_review": "While the proposed method is incremental to existing work, authors translate the states over multi-dimensional space into a graph among multi-entities, apply the extended ODE, and show the good performance. The proposed method is worth being discussed in the community.", "strength_and_weaknesses": "* Strengths\n- ODE is a reasonable way of representing the dynamics of embeddings.\n- Authors use both synthetic and real-world datasets and show the proposed method can track the temporal patterns well.\n- Empirical evaluations show that the proposed method outperforms the other baseline models.\n\n* Weaknesses\n- The manuscript can be further improved to present better. Crucial information -- such as the experimental task setup -- or explanation is often not very clear.\n- The authors verify the end-to-end predictive power, but it would be great if the authors could analyze the insights from the dynamics of embeddings.\n- For the dynamic embedding setting, it would be great to have some forecasting tasks for evaluations.\n- It would be great if the authors evaluate on the other kinds of data, where the actual events are sparse like social interaction datasets. The provided datasets have the characteristics that graphs are generated from different forms of dataset, not naturally given such as social interactions or citations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject