Title: Streaming Factor Trajectory Learning for Temporal Tensor Decomposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tensor Decomposition

streaming method

Bayesian model
Abstract: Practical tensor data is often along with time information. Most existing temporal decomposition approaches estimate a set of fixed factors for the objects in each tensor mode, and hence cannot capture the temporal evolution of the objects' representation. More important, we lack an effective approach to capture such evolution from streaming data, which is common in real-world applications.  To address these issues, we propose Streaming Factor Trajectory Learning (SFTL) for temporal tensor decomposition. We use Gaussian processes (GPs) to model the trajectory of  factors so as to flexibly estimate their temporal evolution. To address the computational challenges in handling streaming data, we convert the GPs into a state-space prior by constructing an equivalent stochastic differential equation (SDE).  We develop an efficient online filtering algorithm to estimate a decoupled running posterior of the involved factor states upon receiving new data. The decoupled estimation enables us to conduct standard Rauch-Tung-Striebel smoothing to compute the full posterior of all the  trajectories in parallel, without the need for revisiting any previous data. We have shown the advantage of SFTL in both synthetic tasks and real-world applications.

Streaming Factor Trajectory Learning for Temporal Tensor Decomposition

Abstract

Practical tensor data is often along with time information. Most existing temporal decomposition approaches estimate a set of fixed factors for the objects in each tensor mode, and hence cannot capture the temporal evolution of the objects' representation. More important, we lack an effective approach to capture such evolution from streaming data, which is common in real-world applications. To address these issues, we propose Streaming Factor Trajectory Learning (SFTL) for temporal tensor decomposition. We use Gaussian processes (GPs) to model the trajectory of factors so as to flexibly estimate their temporal evolution. To address the computational challenges in handling streaming data, we convert the GPs into a state-space prior by constructing an equivalent stochastic differential equation (SDE). We develop an efficient online filtering algorithm to estimate a decoupled running posterior of the involved factor states upon receiving new data. The decoupled estimation enables us to conduct standard Rauch-Tung-Striebel smoothing to compute the full posterior of all the trajectories in parallel, without the need for revisiting any previous data. We have shown the advantage of SFTL in both synthetic tasks and real-world applications. The code is available at https://github. com/xuangu-fang/Streaming-Factor-Trajectory-Learning .

1 Introduction

Tensor data is common in real-world applications. For example, one can extract a three-mode tensor (patient, drug, clinic) from medical service records and a four-mode tensor (customer, commodity, seller, web-page) from the database of an online shopping platform. Tensor decomposition is a fundamental tool for tensor data analysis. It introduces a set of factors to represent the objects in each mode, and estimate these factors by reconstructing the observed entry values. These factors can be viewed as the underlying properties of the objects. We can use them to search for interesting structures within the objects ( e.g., communities and outliers) or as discriminate features for predictive tasks, such as personalized treatment or recommendation.

Real-world tensor data is often accompanied with time information, namely the timestamps at which the objects of different modes interact to produce the entry values. Underlying the timestamps can be rich, valuable temporal patterns. While many temporal decomposition methods are available,

most of them estimate a set of static factors for each object - they either introduce a discrete time mode [Xiong et al., 2010, Zhe et al., 2016a] or inject the timestamp into the decomposition model [Zhang et al., 2021, Fang et al., 2022, Li et al., 2022]. Hence, these methods cannot learn the temporal variation of the factors. Accordingly, they can miss important evolution of the objects' inner properties, such as health and income. In addition, practical applications produce data streams at a rapid pace [Du et al., 2018]. Due to the resource limit, it is often prohibitively expensive to decompose the entire tensor from scratch whenever we receive new data. Many privacy-protecting applications ( e.g., SnapChat) even forbid us to preserve or re-access the previous data. Therefore, not only do we need a more powerful decomposition model that can estimate the factor evolution, we also need an effective method to capture such evolution from fast data streams.

To address these issues, we propose SFTL, a Bayesian streaming factor trajectory learning approach for temporal tensor decomposition. Our method can efficiently handle data streams, with which to estimate a posterior distribution of the factor trajectory to uncover the temporal evolution of the objects' representation. Our method never needs to keep or re-visit previous data. The contribution of our work is summarized as follows.

First, we use a Gaussian process (GP) prior to sample the factor trajectory of each object as a function of time. As a nonparametric function prior, GPs are flexible enough to capture a variety of complex temporal dynamics. The trajectories are combined through a CANDECOMP/PARAFAC (CP) [Harshman, 1970] or Tucker decomposition [Tucker, 1966] form to sample the tensor entry values at any time point.

Second, to sidestep the expensive covariance matrix computation in the GP, which further causes challenges in streaming inference, we use spectral analysis to convert the GP into a linear time invariant (LTI) stochastic differential equation (SDE). Then we convert the SDE into an equivalent state-space prior over the factor (trajectory) states at the observed timestamps. As a result, the posterior inference becomes easier and computationally efficient.

Third, we take advantage of the chain structure of the state-space prior and use the recent conditional expectation propagation framework [Wang and Zhe, 2019] to develop an efficient online filtering algorithm. Whenever a collection of entries at a new timestamp arrives, our method can efficiently estimate a decoupled running posterior of the involved factor states, with a Gaussian product form. The decoupled Gaussian estimate enables us to run standard RTS smoothing [Särkkä, 2013] to compute the full posterior of each factor trajectory independently and in parallel, without revisiting any previous data. Our method at worst has a linear scalability with respect to the number of observed timestamps.

We first evaluated our method in a simulation study. On synthetic datasets, SFTL successfully recovered several nonlinear factor trajectories, and provided reasonable uncertainty estimation. We then tested our method on four real-world temporal tensor datasets for missing value prediction. In both online and final predictive performance, SFTL consistently outperforms the state-of-the-art streaming CP and Tucker decomposition algorithms by a large margin. In most cases, the prediction accuracy of SFTL is even higher than the recent static decomposition methods, which have to pass through the dataset many times. Finally, we investigated the learned factor trajectories from a real-world dataset. The trajectories exhibit interesting temporal evolution.

2 Preliminaries

Tensor Decomposition. We denote an M -mode tensor by Y ∈ R d 1 ×···× d M , where each mode m has d m dimensions, corresponding to d m objects. Each tensor entry is indexed by a tuple glyph[lscript] = ( glyph[lscript] 1 , . . . , glyph[lscript] M ) , and the value is denoted by y glyph[lscript] . For decomposition, we introduce a set of latent factors u j m ∈ R R m to represent each object j in mode m ( 1 ≤ m ≤ M ). One most popular tensor decomposition model is the CANDECOMP/PARAFAC (CP) decomposition [Harshman, 1970], which sets R 1 = . . . = R M = R , and uses the following element-wise form, y glyph[lscript] ≈ 1 glyph[latticetop] ( u 1 glyph[lscript] 1 ◦ . . . ◦ u glyph[lscript] M M ) = ∑ R r =1 ∏ M m =1 u m glyph[lscript] m ,r , where ◦ is the element-wise product. Another commonly used model is Tucker decomposition [Tucker, 1966], y glyph[lscript] ≈ vec( W ) glyph[latticetop] ( u 1 glyph[lscript] 1 ⊗ . . . ⊗ u glyph[lscript] M M ) = ∑ R 1 r 1 =1 . . . ∑ R M r M =1 [ w r · ∏ M m =1 u m glyph[lscript] m ,r m ] , where W ∈ R R 1 ×···× R M is the tensor-core parameter, vec( · ) is the vectorization, ⊗ is the Kronecker product, and r = ( r 1 , . . . , r M ) .

Gaussian Process (GP)s are nonparametric function priors. For a function f ( x ) , if we place a GP prior, f ∼ GP (0 , κ ( x , x ′ )) , it means f ( · ) is sampled as a realization of the GP with covariance function κ , which is often chosen as a kernel function. The GP prior only models the correlation between the function values, namely, cov ( f ( x ) , f ( x ′ )) = κ ( x , x ′ ) , and does not assume any parametric form of the function. Hence, GPs are flexible enough to estimate various complex functions from data, e.g., from multilinear to highly nonlinear. The finite projection of the GP is a Gaussian distribution. That is, given an arbitrary collection of inputs { x 1 , . . . , x N } , the corresponding function values f = [ f ( x 1 ) , . . . , f ( x N )] glyph[latticetop] follow a multi-variate Gaussian prior distribution, p ( f ) = N ( f | 0 , K ) where K is the covariance matrix and each [ K ] mn = κ ( x m , x n ) .

3 Bayesian Temporal Tensor Decomposition with Factor Trajectories

In real-world applications, tensor data is often associated with time information, namely, the timestamps at which the objects of different modes interact to generate the entry values. To capture the potential evolution of the objects' inner properties, we propose a Bayesian temporal tensor decomposition model that can estimate a trajectory of the factor representation. Specifically, for each object j in mode m , we model the factors as a function of time, u j m : [0 , ∞ ] → R R . To flexibly capture a variety of temporal evolution, we assign a GP prior over each element of u j m ( t ) = [ u m j, 1 ( t ) , . . . , u m j,R ( t )] glyph[latticetop] , i.e., u m j,r ( t ) ∼ GP (0 , κ ( t, t ′ )) ( 1 ≤ r ≤ R ). Given the factor trajectories, we then use the CP or Tucker form to sample the entry values at different time points. For the CP form, we have

where U ( t ) = { u j m ( t ) } includes all the factor trajectories, and τ is the inverse noise variance, for which we assign a Gamma prior, p ( τ ) = Gam ( τ | α 0 , α 1 ) . For the Tucker form, we have p ( y glyph[lscript] ( t ) |U ( t ) , W ) = N ( y glyph[lscript] ( t ) | vec( W ) glyph[latticetop] ( u 1 glyph[lscript] 1 ( t ) ⊗ . . . ⊗ u glyph[lscript] M M ( t )) , τ -1 ) where we place a standard normal prior over the tensor-core, p (vec( W )) = N (vec( W ) | 0 , I ) . In this work, we focus on continuous observations. It is straightforward to extend our method for other types of observations.

Suppose we have a collection of observed entry values and timestamps, D = { ( glyph[lscript] 1 , y 1 , t 1 ) , . . . , ( glyph[lscript] N , y N , t N ) } where t 1 ≤ ·· · ≤ t N . We denote the sequence of timestamps when a particular object j of mode m participated in the observed entries by s m j, 1 < . . . < s m j,c j m , where c j m is the participation count of the object. Note that it is a sub-sequence of { t n } . From the GP prior, the values of each u m j,r ( t ) at these timestamps follow a multi-variate Gaussian distribution, p ( u m j,r ) = N ( u m j,r | 0 , K j m ) where u m j,r = [ u m j,r ( s m j, 1 ) , . . . , u m j,r ( s m j,c j m )] glyph[latticetop] 2 and K j m is the covariance/kernel matrix computed at these timestamps. The joint probability with the CP form is

The joint probability with the Tucker form is the same except that we use the Tucker likelihood instead and multiply with the prior of tensor-core p ( W ) .

While this formulation is straightforward, it can introduce computational challenges. There are many multi-variate Gaussian distributions in the joint distribution (2), i.e., {N ( u m j,r | 0 , K j m ) } . The time and space complexity to compute each N ( u m j,r | 0 , K j m ) is O ( ( c j m ) 3 ) and O ( ( c j m ) 2 ) , respectively. With the increase of N , the appearance count c j m for many objects can grow as well, making the computation cost very expensive or even infeasible. The issue is particularly severe when we handle streaming data - the number of timestamps grows rapidly when new data keeps coming in, so does the size of each covariance matrix.

3.1 Equivalent Modeling with State-Space Priors

To sidestep expensive covariance matrix computation and ease the inference with streaming data, we follow [Hartikainen and Särkkä, 2010] to convert the GP prior into an SDE via spectral analysis.

We use a Matérn kernel κ ν ( t, t ′ ) = a ( √ 2 ν ρ ∆ ) ν Γ( ν )2 ν -1 K ν ( √ 2 ν ρ ∆ ) where Γ( · ) is the Gamma function, ∆ = | t -t ′ | , a > 0 , ρ > 0 , K ν is the modified Bessel function of the second kind, and ν = p + 1 2 ( p ∈ { 0 , 1 , 2 , . . . } ) as the GP covariance. Via the analysis of the power spectrum of κ ν , we can show that if f ( t ) ∼ GP (0 , κ ν ( t, t ′ )) , it can be characterized by a linear time-invariant (LTI) SDE, with state z = ( f, f (1) , . . . , f ( p ) ) glyph[latticetop] where f ( k ) ∆ = d k f/ d t k ,

where β ( t ) is a white noise process with diffusion σ 2 ,

Both σ 2 and A are obtained from the parameters in κ ν . Due to the space limit, we leave the detailed derivation in Appendix (Section A). The LTI-SDE is particularly useful in that its finite set of states follow a Gauss-Markov chain, i.e., the state-space prior. Given arbitrary t 1 < . . . < t L , we have

where p ( z ( t 1 )) = N ( z ( t 1 ) | 0 , P ∞ ) , p ( z ( t k +1 ) | z ( t k )) = N ( z ( t k +1 ) | F k z ( t k ) , Q k ) , P ∞ is the stationary covariance matrix computed by solving the matrix Riccati equation [Lancaster and Rodman, 1995], F n = exp(∆ k · A ) where ∆ k = t k +1 -t k , and Q k = P ∞ -A k P ∞ A glyph[latticetop] k . Therefore, we do not need the full covariance matrix as in the standard GP prior, and the computation is much more efficient. The chain structure is also convenient to handle streaming data as we will explain later.

We therefore convert the GP prior over each factor trajectory u m j,r ( t ) into an LTI-SDE. We denote the corresponding state by z m j,r ( t ) . For example, if we choose p = 1 , then z m j,r ( t ) = [ u m j,r ( t ); d u m j,r ( t ) / d t ] . For each object j in mode m , we concatenate all its trajectory states into one, z j m ( t ) = [ z m j, 1 ( t ); . . . ; z m j,R ( t )] . Then on all of its timestamps s m j, 1 < . . . < s m j,c j m , we obtain a state-space prior

The joint probability of our model with the CP form now becomes

Note that in the likelihood, each u m glyph[lscript] nm ( t n )(1 ≤ j ≤ M ) is contained in a corresponding state vector h m glyph[lscript] nm ,k such that s m glyph[lscript] nm ,k = t n (by definition, we then have h m glyph[lscript] nm ,k = z m glyph[lscript] nm ( t n ) ). The joint probability with the Tucker form is similar, which we omit to save the space.

4 Trajectory Inference from Streaming Data

In this section, we develop an efficient, scalable algorithm for factor trajectory estimation from streaming data. In general, we assume that we receive a sequence of (small) batches of observed tensor entries, {B 1 , B 2 , . . . } , generated at different timestamps, { t 1 , t 2 , . . . } . Each batch B n is generated at timestamp t n and t n < t n +1 . Denote by D t n all the data up to timestamp t n , i.e., D t n = B 1 ∪ . . . ∪ B n . Upon receiving B n +1 , we intend to update our model without revisiting D t n

<latexi sh

1_b

64="

+VU

Qu rLv0B7

w

qHSWc

>A

CKX

D

gM

F

3

2

k

jP

yO

J

f

T

d

GZ

o

/

Y

R

n

p

8

9

z

I

5

m

E

N

<latexi sh

1_b

64="U

c9dV

p

+Q

X

E

>A

C

H

FNT

27

O

Y

SB

K

W

k

g

3

v

jR

5/

f

P

n

u

m

o

J

r

<latexi sh

1_b

64="c fu2

z

IYQU3jW

8

F

9

>A

C

H

V

N

E

p

ZKqT

y

/

n

Bdg

P

O

o

+

7

S

w

M

v

k

5

X

m

L

0

J

r

R

G

D

<latexi sh

1_b

64="5n0y

Qr u8

j

3Odo mS

>A

B7

c

VDL

gN

E

z

G

M

H

Y

R

W

ZT

q

vUJ

PC

/

f

I

2

X

F

w

K

p

+

k

9

<latexi sh

1_b

64="NFnEPMf9

+Uj

CTO

0rJwXG

>A

H

c

V

27

Lz

Y

SBQZK

W

k

g

3

I

v

R

5/

d

q

p

8

u

m

y

D

o

<latexi sh

1_b

64="dD7HqAmv93c zgJG

5

XuM8

o

>

Cf

VFNT

E

2

Q

I

S

UyK

pB

W

j

r

/

0

Z

P

k

L

Y

O

R

+

n

w

<latexi sh

1_b

64="

L

/

Xf7S

A2TrU

+E

8d

W

9

>

B

n

c

VD

N

F

p

q

I

gC

R

y

0

mk3

oZ

J

Q

5

G

wz

P

K

v

u

jH

O

Y

M

<latexi sh

1_b

64="YUGELC

/M2jF7

g8

J

R

I

B

o

>A

N3

c

ZD

S

nXmu9V

y

v

0

q

5

W

p

Q

O

H

w

P

z

d

k

T

K

+

f

r

…

Data Stream

…

…

…

…

…

…

…

…

Time

: Trajectory States

: State Transition

: Old data

: New data

Factor

Trajectory

Θ

n

B

n

+1

D

t

n

=

B

1

∪

. . .

∪ B

n

Θ

n

+1

t

1

t

2

t

n

t

n

+1

to provide the trajectory posterior estimate, { p ( u j m ( t ) |D t n +1 ) |∀ t ≥ 0 , 1 ≤ m ≤ M, 1 ≤ j ≤ d m } , where D t n +1 = D t n ∪ B n +1 .

To this end, we first observe that the standard state-space model with a Gaussian likelihood has already provided a highly efficient streaming inference framework. Denote by x n and y n the state and observation at each step n , respectively. To handle streaming observations { y 1 , y 2 , . . . } , we only need to compute and track the running posterior p ( x n | y 1: n ) upon receiving each y n , where y 1: n denotes the total data up to step n . This is called Kalman filtering [Särkkä, 2013], which only depends on the running posterior at step n -1 , i.e., p ( x n -1 | y 1: n -1 ) , and is highly efficient. After all the data is processed (suppose it stops at step N ), we can use Rauch-Tung-Striebel (RTS) smoother [Särkkä, 2013] to efficiently compute the full posterior of each state, p ( x n | y 1: N ) , from backward, which does not need to re-access any previous observations (see Section B in Appendix).

However, one cannot apply the above framework outright to our model, since the tensor decomposition likelihood of each observed entry couples the states of multiple factor trajectories, see (1) - which correspond to the participated objects at different modes. That means, the factor-state chains of different objects are dynamically intertwined through the received data. The multiplicative form of these states in the likelihood render the running posterior of each trajectory intractable to compute, not to mention running RTS smoother. To address this challenge, we take advantage of the chain structure and use the recent conditional Expectation propagation (CEP) framework [Wang and Zhe, 2019] to develop an efficient online filtering algorithm, which approximates the running posterior of the involved factor states as a product of Gaussian. Thereby, we can decouple the involved factor state chains, and conduct standard RTS smoothing for each chain independently.

Specifically, denote the sequence of timestamps when each object j of mode m has showed up in the data stream up to t n , by s m j, 1 < s m j, 2 < . . . < s m j,c m j,n where c m j,n is the object's appearance count up to t n . Denote by I m n the indexes of all the objects of mode m appearing in B n . Hence, for every object j ∈ I m n , we have s m j,c m j,n = t n , and h m j,c m j,n ∆ = z j m ( t n ) is the factor state of the object at t n . Upon receiving each B n , we intend to approximate the running posterior of all the involved factor states and noise inverse variance τ with the following decoupled form,

where q ( τ |D t n ) = Gam ( τ | a n , b n ) , and q ( h m j,c m j,n |D t n ) = N ( h m j,c m j,n | ̂ µ m j,c m j,n , ̂ V m j,c m j,n ) . To this end, let us consider given the approximation at t n , how to obtain the new approximation at t n +1 ( i.e., upon

receiving B n +1 ) in the same form of (6). To simplify the notation, let us define the preceding states of the involved factors by Θ n = { h m j,c m j,n | j ∈ I m n +1 } m , and the current states by Θ n +1 = { h m j,c m j,n +1 | j ∈ I m n +1 } m . First, due to the chain structure of the prior over each { h m j,c m j,k | k = 0 , 1 , 2 , . . . } , we can see that conditioned on { Θ n , τ } , the current states Θ n +1 and the new observations B n +1 are independent of D t n . This is because in the graphical model representation, { Θ n , τ } have blocked all the paths from the old observations D t n to the new state and observations [Bishop, 2007]; see Fig. 1 for an illustration. Then, we can derive that p (Θ n +1 , Θ n , τ |D t n +1 ) ∝ p (Θ n +1 , Θ n , τ, B n +1 |D t n ) = p (Θ n , τ |D t n ) p (Θ n +1 , B n +1 | Θ n , τ, D t n ) = p (Θ n , τ |D t n ) p (Θ n +1 | Θ n ) p ( B n +1 | Θ n +1 , τ ) , (7) where p (Θ n , τ |D t n ) is the running posterior at t n ,

each p ( h m j,c m j,n +1 | h m j,c m j,n ) is a conditional Gaussian distribution defined in (4), and p ( B n +1 | Θ n +1 , τ ) = ∏ ( glyph[lscript] ,y ) ∈B n +1 N ( y | 1 glyph[latticetop] ( u 1 glyph[lscript] 1 ( t n +1 ) ◦ · · · ◦ u glyph[lscript] M M ( t n +1 ) ) , τ -1 ) . Since p (Θ n , τ |D t n ) takes the form of (6), we can analytically marginalize out each h m j,c m j,n ∈ Θ n , and obtain

If we view the R.H.S of (8) as a joint distribution with B n +1 , then our task amounts to estimating the posterior distribution, i.e., the L.H.S of (8). The product in the CP likelihood (and also Tucker likelihood) renders exact posterior computation infeasible, and we henceforth approximate

where ∝ ∼ means approximately proportional to. To optimize these approximation terms, we use the recent conditional Expectation propagation (CEP) framework [Wang and Zhe, 2019] to develop an efficient inference algorithm. It uses conditional moment matching to update each approximation in parallel and conducts fixed point iterations, and hence can converge fast. We leave the details in the Appendix (Section C). Once it is done, we substitute the approximation (9) into (8). Then the R.H.S of (8) becomes a product of Gaussian and Gamma terms over each state and τ . We can then immediately obtain a closed-form estimation in the form as (6). At the beginning, when estimating p (Θ 1 , τ |D t 1 ) , since the preceding states Θ 0 = ∅ , we have a n = α 0 , b n = α 1 , ̂ µ m j,c m j,n +1 = 0 , and

In this way, we can continuously filter the incoming batches {B 1 , B 2 , . . . } . As a result, for the factor state chain of every object j in every mode m , along with each timestamp s m j,k , we can online estimate and track a running posterior approximation { q ( h m j,k |D t s m j,k ) | k = 1 , 2 , . . . } , which is a Gaussian distribution. Hence, we can run the standard RTS smoother, to compute the full posterior of every factor state, with which we can compute the posterior of the trajectory at any time point t [Bishop, 2006]. Our method is summarized in Algorithm 1.

Algorithm Complexity. The time complexity of our algorithm processing a batch B n is O ( |B n | R 3 ) where | · | is the size. The time complexity of RTS smoother for a particular object j in mode m is O ( R 3 c m j,N ) , where N is the total number of timestamps. The space complexity of our algorithm is O ( ∑ M m =1 ∑ d m j =1 c m j,N R 2 ) , which is to track the running posterior of the factor state at each appearing timestamp for every object. Since c m j,N ≤ N , the complexity of our algorithm is at worst linear in N .

5 Related Work

Many tensor decomposition methods have been developed, such as [Yang and Dunson, 2013, Rai et al., 2014, Zhe et al., 2015, 2016b,a, Tillinghast et al., 2020, Pan et al., 2020b, Fang et al., 2021a,b,

Tillinghast and Zhe, 2021, Tillinghast et al., 2022, Fang et al., 2022, Zhe and Du, 2018, Pan et al., 2020a, Wang et al., 2020, Pan et al., 2021, Wang et al., 2022]. For temporal decomposition, most existing methods augment the tensor with a discrete time mode to estimate additional factors for time steps, e.g., [Xiong et al., 2010, Rogers et al., 2013, Song et al., 2017, Du et al., 2018, Ahn et al., 2021]. The most recent works have conducted continuous-time decomposition. Zhang et al. [2021] used polynomial splines to model a time function as the CP coefficients. Li et al. [2022] used neuralODE [Chen et al., 2018] to model the entry value as a function of latent factors and time point. Fang et al. [2022] performed continuous-time Tucker decomposition, and modeled the tensor-core as a time function. To our knowledge, [Wang et al., 2022] is the first work to estimate factor trajectories. It places a GP prior in the frequency domain, and samples the factor trajectories via inverse Fourier transform. It then uses another GP to sample the entry values. While successful, this method cannot handle streaming data, and the black-box GP decomposition lacks interpretability.

Current Bayesian streaming tensor decomposition methods include [Du et al., 2018, Fang et al., 2021a, Pan et al., 2020b, Fang et al., 2021b], which are based on streaming variational Bayes [Broderick et al., 2013] or assumed density filtering (ADF) [Boyen and Koller, 1998]. ADF can be viewed as an instance of Expectation Propagation (EP) [Minka, 2001] for streaming data. EP approximates complex terms in the probability distribution with exponential-family members, and uses moment matching to iteratively update the approximations, which essentially is a fixed point iteration. To address the challenge of intractable moment matching, Wang and Zhe [2019] proposed conditional EP (CEP), which uses conditional moment matching and Taylor expansion to compute the moments for factorized approximations. The theoretical guarantees and error bound analysis for EP and ADF have been studied for a long time, such as [Boyen and Koller, 1998, Dehaene and Barthelmé, 2015, 2018]. The most recent work [Fang et al., 2022] also uses SDEs to represent GPs and CEP framework for inference, but their GP prior is placed on the tensor-core, not for learning factor trajectories, and their method is only for static decomposition, and cannot handle streaming data.

6 Experiment

6.1 Simulation Study

We first conducted a simulation study, for which we simulated a two-mode tensor, with two nodes per mode. Each node is represented by a time-varying factor: u 1 1 ( t ) = -sin 3 (2 πt ) , u 1 2 ( t ) = ( 1 -sin 3 ( 1 2 πt ) ) sin 3 (3 πt ) , u 2 1 ( t ) = sin(2 πt ) , and u 2 2 ( t ) = -cos 3 (3 πt ) sin(3 πt ) sin(2 πt ) . Given

/

/

/

/

⋂]⌉˜

⊕˜∐√{˜̂(√√∐⌊˜̂√}√glyph[arrowbt]

(a)

u

1

1

(

t

)

/

/

/

⋂]⌉˜

(b)

u

1

2

(

t

)

/

/

/

⋂]⌉˜

(c)

u

2

1

(

t

)

/

/

⋂]⌉˜

(d)

u

2

2

(

t

)

these factors, an entry value at time t is generated via y ( i,j ) ( t ) ∼ N ( u 1 i ( t ) u 2 j ( t ) , 0 . 05 ) . We randomly sampled 500 (irregular) timestamps from [0 , 1] . For each timestamp, we randomly picked two entries, and sampled their values accordingly. Overall, we sampled 1,000 observed tensor entry values.

We implemented SFTL with PyTorch [Paszke et al., 2019]. We used ν = 3 2 and a = ρ = 0 . 3 for the Matérn kernel. We streamed the sampled entries according to their timestamps, and ran our streaming factor trajectory inference based on the CP form. The estimated trajectories are shown in Fig. 2. As we can see, SFTL recovers the ground-truth pretty accurately, showing that SFTL has successfully captured the temporal evolution of the factor representation for every node. It is interesting to observe that when t is around 0 , 0 . 5 and 1 , the posterior standard deviation (the shaded region) increases significantly. This is reasonable: the ground-truth trajectories overlap at these time points, making it more difficult to differentiate/estimate their values at these time points. Accordingly, the uncertainty of the estimation increases. In Section D of Appendix, we further provide the root-mean-square error (RMSE) in recovering the four trajectories, and sensitivity analysis of the kernel parameters.

0

20000

40000

Number of Entries

0.45

1.00

1.50

RMSE

SFTL-CP

SFTL-Tucker

ADF-CP

POST

(a)

FitRecord

0

4000

8000

Number of Entries

0.25

0.60

1.00

RMSE

(b)

ServerRoom

0

8000

16000

Number of Entries

0.2

0.7

1.2

RMSE

(c)

BeijingAir-2

0

6000

12000

Number of Entries

0.45

0.70

1.00

RMSE

(d)

6.2 Real-World Applications

Next, we examined SFTL in four real-world datasets: FitRecord , ServerRoom , BeijingAir-2 , and BeijingAir-3 . We tested 11 competing approaches. We compared with state-of-the-art streaming tensor decomposition methods based on the CP or Tucker model, including (1) POST [Du et al., 2018], (2) BASS-Tucker [Fang et al., 2021a] and (3) ADF-CP [Wang and Zhe, 2019], the state-ofthe-art static decomposition algorithms, including (4) P-Tucker [Oh et al., 2018], (5) CP-ALS and (6) Tucker-ALS [Bader and Kolda, 2008]. For those methods, we augment the tensor with a time mode, and convert the ordered, unique timestamps into increasing time steps. We also compared with the most recent continuous-time decomposition methods. (7) CT-CP [Zhang et al., 2021], (8) CT-GP, (9) BCTT [Fang et al., 2022], (10) THIS-ODE [Li et al., 2022], and (11) NONFAT [Wang et al., 2022], nonparametric factor trajectory learning, the only existing work that also estimates factor trajectories for temporal tensor decomposition. Note that the methods 4-11 cannot handle data streams. They have to iteratively access the data to update the model parameters and factor estimates. The details about the competing methods and datasets are provided in Appendix (Section E).

For all the competing methods, we used the publicly released implementations of the original authors. The hyper-parameter setting and turning follows the original papers. For SFTL, we chose ν from { 1 2 , 3 2 } , a from [0.5, 1] and ρ from [0.1, 0.5]. For our online filtering, the maximum number of CEP iterations was set to 50 and the tolerance level to 10 -4 . For numerical stability, we re-scaled the timestamps to [0 , 1] . We examined the number of factors (or factor trajectories) R ∈ { 2 , 3 , 5 , 7 } .

Final Prediction Accuracy. We first examined the final prediction accuracy with our learned factor trajectories. To this end, we followed [Xu et al., 2012, Kang et al., 2012], and randomly sampled 80% observed entry values and their timestamps for streaming inference and then tested the prediction error on the remaining entries. We also compared with the static decomposition methods, which need to repeatedly access the training entries. We repeated the experiment five times, and computed the average root mean-square-error (RMSE), average mean-absolute-error (MAE), and their standard deviations. We ran our method based on both the CP and Tucker forms, denoted by SFTL-CP and SFTL-Tucker, respectively. We report the results for R = 5 in Table 1. Due to the space limit, we leave the other results in the Appendix (Table 4,5, and 6). As we can see, SFTL outperforms all the streaming approaches by a large margin. SFTL even obtains significantly better prediction accuracy than all the static decomposition approaches, except that on Server Room , SFTL is second to THIS-ODE and NONFAT. Note that SFTL only went through the training entries for once. Although NONFAT can also estimate the factor trajectories, it uses GPs to perform black-box nonlinear decomposition and hence loses the interpretability. Note that NONFAT in most case also outperforms the other static decomposition methods that only estimate time-invariant factors. The superior performance of SFTL and NONFAT shows the importance of capturing factor evolution.

Online Predictive Performance. Next, we evaluated the online predictive performance of SFTL. Whenever a batch of entries at a new timestamp has been processed, we examined the prediction accuracy on the test set, with our current estimate of the factor trajectories. We repeated the evaluation for five times, and examine how the average prediction error varies along with the number of processed entries. We show the results for R = 5 in Fig. 3, and the others in Appendix (Fig. 5, 6, and 7). It is clear that SFTL in most cases outperforms the competing streaming decomposition algorithms by a large margin throughout the course of running. Note that the online behavior of BASS-Tucker was quite unstable and so we excluded it in the figures. It confirms the advantage of our streaming trajectory learning approach - even in the streaming scenario, incrementally capturing the timevariation of the factors can perform better than updating fixed, static factors. To confirm the advantage of SFTL in computational efficiency, we report the running time in Section G of Appendix.

Time

0.5

0.0

0.5

(a)

u

1

1

,

1

(

t

)

Time

1.0

0.5

0.0

(b)

u

1

1

,

2

(

t

)

Time

0.5

0.0

0.5

(c)

u

1

1

,

3

(

t

)

Time

1.4

0.5

0.4

(d)

u

2

2

,

1

(

t

)

Time

0.5

0.0

0.8

(e)

u

2

2

,

2

(

t

)

Time

0.7

0.0

0.7

(f)

u

2

2

,

3

(

t

)

Time

0

1

2

(g)

u

3

3

,

1

(

t

)

Time

1

0

1

(h)

u

3

3

,

2

(

t

)

Time

1

0

1

(i)

u

3

3

,

3

(

t

)

Investigation of Learning Results. Finally, we investigated our learned factor trajectories. We set R = 3 and ran SFTL-CP on ServerRoom . In Fig. 4, we visualize the trajectory for the 1st air conditioning mode, the 2rd power usage level, and the 3rd location. The shaded region indicates the standard deviation, and the red dashed line the biggest timestamp in the data. First, we can see that the posterior variance of all the trajectories grow quickly when moving to the right of the red line. This is reasonable because it is getting far away from the training region. Second, for each object, the first and second trajectories (1st and 2nd column in Fig. 4) exhibit quite different time-varying patterns, e.g., the local periodicity in u 1 1 , 2 ( t ) and u 2 2 , 2 ( t ) , the gradual decreasing and increasing trends in u 3 3 , 1 ( t ) and u 3 3 , 2 ( t ) , respectively, which imply different inner properties of the object. Third, it is particularly interesting to see that the third trajectory for all the objects appear to be close to zero, with relatively large posterior variance all the time. This might imply that two factor trajectories have been sufficient to represent each object. Requesting for a third trajectory is redundant. More important, our model is empirically able to detect such redundancy and returns a zero-valued trajectory.

7 Conclusion

We have presented SFTL, a probabilistic temporal tensor decomposition approach. SFTL can efficiently handle streaming data, and estimate time-varying factor representations. On four realworld applications, SFTL achieves superior online and final prediction accuracy.

Acknowledgments

This work has been supported by MURI AFOSR grant FA9550-20-1-0358 and NSF CAREER Award IIS-2046295.

References

Dawon Ahn, Jun-Gi Jang, and U Kang. Time-aware tensor decomposition for sparse tensors. Machine Learning, pages 1-22, 2021.

Brett W Bader and Tamara G Kolda. Efficient matlab computations with sparse and factored tensors. SIAM Journal on Scientific Computing, 30(1):205-231, 2008.

Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.

Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2007.

Xavier Boyen and Daphne Koller. Tractable inference for complex stochastic processes. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, pages 33-42, 1998.

Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. Streaming variational bayes. Advances in neural information processing systems, 26, 2013.

Chris Chatfield. The analysis of time series: an introduction. Chapman and hall/CRC, 2003.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.

Guillaume Dehaene and Simon Barthelmé. Expectation propagation in the large data limit. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 80(1):199-217, 2018.

Guillaume P Dehaene and Simon Barthelmé. Bounding errors of expectation-propagation. Advances in Neural Information Processing Systems, 28, 2015.

Yishuai Du, Yimin Zheng, Kuang-chih Lee, and Shandian Zhe. Probabilistic streaming tensor decomposition. In 2018 IEEE International Conference on Data Mining (ICDM), pages 99-108. IEEE, 2018.

Shikai Fang, Robert M Kirby, and Shandian Zhe. Bayesian streaming sparse Tucker decomposition. In Uncertainty in Artificial Intelligence, pages 558-567. PMLR, 2021a.

Shikai Fang, Zheng Wang, Zhimeng Pan, Ji Liu, and Shandian Zhe. Streaming Bayesian deep tensor factorization. In International Conference on Machine Learning, pages 3133-3142. PMLR, 2021b.

Shikai Fang, Akil Narayan, Robert Kirby, and Shandian Zhe. Bayesian continuous-time Tucker decomposition. In International Conference on Machine Learning, pages 6235-6245. PMLR, 2022.

A. Harshman. Foundations of the PARAFAC procedure: Model and conditions for an'explanatory'multi-mode factor analysis. UCLA Working Papers in Phonetics, 16:1-84, 1970.

Jouni Hartikainen and Simo Särkkä. Kalman filtering and smoothing solutions to temporal gaussian process regression models. In 2010 IEEE international workshop on machine learning for signal processing, pages 379-384. IEEE, 2010.

U Kang, Evangelos Papalexakis, Abhay Harpale, and Christos Faloutsos. Gigatensor: scaling tensor analysis up by 100 times-algorithms and discoveries. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 316-324. ACM, 2012.

Peter Lancaster and Leiba Rodman. Algebraic riccati equations. Clarendon press, 1995.

Shibo Li, Robert Kirby, and Shandian Zhe. Decomposing temporal high-order interactions via latent odes. In International Conference on Machine Learning, pages 12797-12812. PMLR, 2022.

Thomas P Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pages 362-369, 2001.

Gary W Oehlert. A note on the delta method. The American Statistician, 46(1):27-29, 1992.

Sejoon Oh, Namyong Park, Sael Lee, and Uksong Kang. Scalable tucker factorization for sparse tensors-algorithms and discoveries. In 2018 IEEE 34th International Conference on Data Engineering (ICDE), pages 1120-1131. IEEE, 2018.

Zhimeng Pan, Zheng Wang, and Shandian Zhe. Scalable nonparametric factorization for high-order interaction events. In International Conference on Artificial Intelligence and Statistics, pages 4325-4335. PMLR, 2020a.

Zhimeng Pan, Zheng Wang, and Shandian Zhe. Streaming nonlinear Bayesian tensor decomposition. In Conference on Uncertainty in Artificial Intelligence, pages 490-499. PMLR, 2020b.

Zhimeng Pan, Zheng Wang, Jeff M Phillips, and Shandian Zhe. Self-adaptable point processes with nonparametric time decays. Advances in Neural Information Processing Systems, 34:4594-4606, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.

Piyush Rai, Yingjian Wang, Shengbo Guo, Gary Chen, David Dunson, and Lawrence Carin. Scalable Bayesian low-rank decomposition of incomplete multiway tensors. In Proceedings of the 31th International Conference on Machine Learning (ICML), 2014.

Mark Rogers, Lei Li, and Stuart J Russell. Multilinear dynamical systems for tensor time series. Advances in Neural Information Processing Systems, 26:2634-2642, 2013.

Simo Särkkä. Bayesian filtering and smoothing. Number 3. Cambridge University Press, 2013.

Qingquan Song, Xiao Huang, Hancheng Ge, James Caverlee, and Xia Hu. Multi-aspect streaming tensor completion. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 435-443, 2017.

Conor Tillinghast and Shandian Zhe. Nonparametric decomposition of sparse tensors. In International Conference on Machine Learning, pages 10301-10311. PMLR, 2021.

Conor Tillinghast, Shikai Fang, Kai Zhang, and Shandian Zhe. Probabilistic neural-kernel tensor decomposition. In 2020 IEEE International Conference on Data Mining (ICDM), pages 531-540. IEEE, 2020.

Conor Tillinghast, Zheng Wang, and Shandian Zhe. Nonparametric sparse tensor factorization with hierarchical Gamma processes. In International Conference on Machine Learning, pages 21432-21448. PMLR, 2022.

Ledyard Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31: 279-311, 1966.

Zheng Wang and Shandian Zhe. Conditional expectation propagation. In UAI, page 6, 2019.

Zheng Wang, Xinqi Chu, and Shandian Zhe. Self-modulating nonparametric event-tensor factorization. In International Conference on Machine Learning, pages 9857-9867. PMLR, 2020.

Zheng Wang, Yiming Xu, Conor Tillinghast, Shibo Li, Akil Narayan, and Shandian Zhe. Nonparametric embeddings of sparse high-order interaction events. In International Conference on Machine Learning, pages 23237-23253. PMLR, 2022.

Liang Xiong, Xi Chen, Tzu-Kuo Huang, Jeff Schneider, and Jaime G Carbonell. Temporal collaborative filtering with bayesian probabilistic tensor factorization. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 211-222. SIAM, 2010.

Zenglin Xu, Feng Yan, and Yuan Alan Qi. Infinite tucker decomposition: Nonparametric bayesian models for multiway data analysis. In ICML, 2012.

Yang and D.B. Dunson. Bayesian conditional tensor factorizations for high-dimensional classification. Journal of the Royal Statistical Society B, revision submitted, 2013.

Yanqing Zhang, Xuan Bi, Niansheng Tang, and Annie Qu. Dynamic tensor recommender systems. Journal of Machine Learning Research, 22(65):1-35, 2021.

Shandian Zhe and Yishuai Du. Stochastic nonparametric event-tensor decomposition. In Advances in Neural Information Processing Systems, pages 6856-6866, 2018.

Shandian Zhe, Zenglin Xu, Xinqi Chu, Yuan Qi, and Youngja Park. Scalable nonparametric multiway data analysis. In Artificial Intelligence and Statistics, pages 1125-1134. PMLR, 2015.

Shandian Zhe, Yuan Qi, Youngja Park, Zenglin Xu, Ian Molloy, and Suresh Chari. Dintucker: Scaling up Gaussian process models on large multidimensional arrays. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016a.

Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-chih Lee, Zenglin Xu, Yuan Qi, and Zoubin Ghahramani. Distributed flexible nonlinear tensor factorization. Advances in neural information processing systems, 29, 2016b.

Appendix

A Spectral Analysis and LTI-SDE

We consider the Matérn kernel family,

where ∆ = | t -t ′ | , Γ( · ) is the Gamma function, a > 0 and ρ > 0 are the amplitude and length-scale parameters, respectively, K ν is the modified Bessel function of the second kind, ν > 0 controls the smoothness. Since κ ν is a stationary kernel, i.e., κ ν ( t, t ′ ) = κ ν ( t -t ′ ) , according to the Wiener-Khinchin theorem [Chatfield, 2003], if

the energy spectrum density of f ( t ) can be obtained by the Fourier transform of κ ν (∆) ,

where ω is the frequency, and α = √ 2 ν ρ . We consider the commonly used choice ν = p + 1 2 where p ∈ { 0 , 1 , 2 , . . . } . Then we can observe that

where σ 2 = a 2 √ π Γ( p +1) Γ( p + 1 2 ) α 2 p +1 , and i indicates an imaginary number. We expand the polynomial

where { c k | 0 ≤ k ≤ p } are the coefficients. From (12) and (13), we can construct an equivalent system to generate the signal f ( t ) . That is, in the frequency domain, the system output's Fourier transform ̂ f ( ω ) is given by

where ̂ β is the Fourier transform of a white noise process β ( t ) with spectral density (or diffusion) σ 2 . The reason is that by construction, ̂ f ( ω ) = ̂ β ( ω ) ( α + iω ) p +1 , which gives exactly the same spectral density as in (12), S ( ω ) = | ̂ f ( ω ) | 2 . We then conduct inverse Fourier transform on both sides of (14) to obtain the representation in the time domain,

which is an SDE. Note that β ( t ) has the density σ 2 . We can further construct a new state z = ( f, f (1) , . . . , f ( p ) ) glyph[latticetop] (where each f ( k ) ∆ = d k f/ d t k ) and convert (15) into a linear time-invariant (LTI) SDE,

where

For a concrete example, if we take p = 1 (and so ν = 3 2 ), then A = [0 , 1; -α 2 , -2 α ] , η = [0; 1] , and σ 2 = 4 aα 3 .

The LTI-SDE is particularly useful in that its finite set of states follow a Gauss-Markov chain, namely the state-space prior. Specifically, given arbitrary t 1 < . . . < t L , we have

where p ( z ( t 1 )) = N ( z ( t 1 ) | 0 , P ∞ ) , p ( z ( t k +1 ) | z ( t k )) = N ( z ( t k +1 ) | F k z ( t k ) , Q k ) , P ∞ is the stationary covariance matrix computed by solving the matrix Riccati equation [Lancaster and Rodman, 1995], F n = exp(∆ k · A ) where ∆ k = t k +1 -t k , and Q k = P ∞ -A k P ∞ A glyph[latticetop] k . Therefore, we do not need the full covariance matrix as in the standard GP prior, and the computation is much more efficient. The chain structure is also convenient to handle streaming data as explained in the main paper.

Note that for other type of kernel functions, such as the square exponential (SE) kernel, we can approximate the inverse spectral density 1 /S ( ω ) with a polynomial of ω 2 with negative roots, and follow the same way to construct an LTI-SDE (approximation) and state-space prior.

B RTS Smoother

Consider a standard state-space model with state x n and observation y n at each time step n . The prior distribution is a Gauss-Markov chain,

Suppose we have a Gaussian observation likelihood,

Then upon receiving each y n , we can use Kalman filtering to obtain the exact running posterior,

which is a Gaussian. After all the data has been processed - suppose it ends after step N -wecan use Rauch-Tung-Striebel (RTS) smoother [Särkkä, 2013] to efficiently compute the full posterior of each state from backward, which does not need to re-access any data: p ( x n | y 1: N ) = N ( x n | m s n , P s n ) , where

As we can see, the computation only needs the running posterior p ( x n | y 1: n ) = N ( ·| m n , P n ) and the full posterior of the next state p ( x n +1 | y 1: N ) = N ( ·| m n +1 , P n +1 ) . It does not need to revisit previous observations y 1: N

C Details about Online Trajectory Inference

In this section, we provide the details about how to update the running posterior according to (8) and (9) (in the main paper) with the conditional EP (CEP) framework [Wang and Zhe, 2019].

C.1 EP and CEP framework

We first give a brief introduction to the EP and CEP framework. Consider a general probabilistic model with latent parameters θ . Given the observed data D = { y 1 , . . . , y N } , the joint probability distribution is

Our goal is to compute the posterior p ( θ |D ) . However, it is usually infeasible to compute the exact marginal distribution p ( D ) , because of the complexity of the likelihood and/or prior. EP therefore seeks to approximate each term in the joint probability by an exponential-family term,

where c n and c 0 are constants to ensure the normalization consistency (they will get canceled in the inference, so we do not need to calculate them), and

where λ n is the natural parameter and φ ( θ ) is sufficient statistics. For example, if we choose a Gaussian term, f n = N ( θ | µ n , Σ n ) , then the sufficient statistics is φ ( θ ) = { θ , θθ glyph[latticetop] } . The moment is the expectation of the sufficient statistics.

We therefore approximate the joint probability with

Because the exponential family is closed under product operations, we can immediately obtain a closed-form approximate posterior q ( θ ) ≈ p ( θ |D ) by merging the approximation terms in the R.H.S of (20), which is still a distribution in the exponential family.

Then the task amounts to optimizing those approximation terms { f n ( θ ) | 0 ≤ n ≤ N } . EP repeatedly conducts four steps to optimize each f n .

Step 1. We obtain the calibrated distribution that integrates the context information of f n ,

where q ( θ ) is the current posterior approximation.

Step 2. We construct a tilted distribution to combine the true likelihood,

Step 3. We project the tilted distribution back to the exponential family,

Note that if n = 0 , we have ˜ p ( θ ) ∝ q \ n ( θ ) · p ( θ ) .

where q belongs to the exponential family. This can be done by moment matching,

That is, we compute the expected moment under ˜ p , with which to obtain the parameters of q ∗ . For example, if q ∗ ( θ ) is a Gaussian distribution, then we need to compute E ˜ p [ θ ] and E ˜ p [ θθ glyph[latticetop] ] , with which to obtain the mean and covariance for q ∗ ( θ ) . Hence we obtain q ∗ ( θ ) = N ( θ | E ˜ p [ θ ] , E ˜ p [ θθ glyph[latticetop] ] -E ˜ p [ θ ] E ˜ p [ θ ] glyph[latticetop] )

Step 4. We update the approximation term by

In practice, EP often updates all the f n 's in parallel, and uses damping to avoid divergence. It iteratively runs the four steps until convergence. In essence, this is a fixed point iteration to optimize a free energy function (a mini-max problem) [Minka, 2001].

The critical step in EP is the moment matching (21). However, in many cases, it is analytically intractable to compute the moment under the tilted distribution ˜ p , due to the complexity of the likelihood. To address this problem, CEP considers the commonly used case that each f n has a factorized structure,

where each f nm is also in the exponential family, and { θ m } are mutually disjoint. Then at the moment matching step, we need to compute the moment of each θ m under ˜ p , i.e., E ˜ p [ φ ( θ m )] . The first key idea of CEP is to use the nested structure,

where θ \ m = θ \ θ m . Therefore, we can first compute the inner expectation, i.e., conditional moment,

and then seek for computing the outer expectation, E ˜ p ( θ \ m ) [ g ( θ \ m )] . The inner expectation is often easy to compute ( e.g., with our CP/Tucker likelihood). When f n is factorized individually over each element of θ , this can always be efficiently and accurately calculated by quadrature. However, the outer expectation is still difficult to obtain because ˜ p ( θ \ m ) is intractable. The second key idea of CEP is that since the moment matching is also between q ( θ \ m ) and ˜ p ( θ \ m ) , we can use the current marginal posterior to approximate the marginal titled distribution and then compute the outer expectation,

If it is still analytically intractable, we can use the delta method [Oehlert, 1992] to approximate the expectation. That is, we use a Taylor expansion of g ( · ) at the mean of θ \ m . Take the first-order expansion as an example,

where J is the Jacobian of g at E q ( θ \ m ) [ θ \ m ] . Then we take the expectation on the Taylor approximation instead,

The above computation is very convenient to implement. Once we obtain the conditional moment g ( θ \ m ) , we simply replace the θ \ m by its expectation under current posterior approximation q , i.e., E q ( θ \ m ) [ θ \ m ] , to obtain the matched moment g ( E q ( θ \ m ) [ θ \ m ]) , with which to construct q ∗ in Step 3 of EP (see (21)). The remaining steps are the same.

C.2 Running Posterior Update

Now we use the CEP framework to update the running posterior p (Θ n +1 , τ |D t n +1 ) in (8) via the approximation (9). To simplify the notation, let us define v l m m ∆ = u m glyph[lscript] m ( t n +1 ) , and hence for each ( glyph[lscript] , y ) ∈ B n +1 , we approximate

If we substitute (9) into (8), we can immediately obtain a Gaussian posterior approximation of each v m glyph[lscript] m and a Gamma posterior approximation of the noise inverse variance τ . Then dividing the current posterior approximation with the R.H.S of (28), we can obtain the calibrated distribution,

where 1 ≤ m ≤ M . Next, we construct a tilted distribution,

To update each N ( v m glyph[lscript] m | γ glyph[lscript] m m , Σ m glyph[lscript] m ) in (28), we first look into the conditional tilted distribution,

where V \ m glyph[lscript] is { v j glyph[lscript] j | 1 ≤ j ≤ M,j = m } , and glyph[negationslash]

The conditional tilted distribution is obviously Gaussian, and the conditional moment is straightforward to obtain,

where S denotes the conditional covariance. Next, according to (27), we simply replace τ , v \ m glyph[lscript] , and v \ m glyph[lscript] ( v \ m glyph[lscript] ) glyph[latticetop] by their expectation under the current posterior q in (32) and (33), to obtain the moments, i.e., the mean and covariance matrix, with which we can construct q ∗ in Step 3 of the EP framework. The computation of E q [ τ ] is straightforward, and

Similarly, to update Gam ( α glyph[lscript] , ω glyph[lscript] ) in (28), we first observe that the conditional titled distribution is also a Gamma distribution,

where v glyph[lscript] = v 1 glyph[lscript] 1 ◦ . . . ◦ v glyph[lscript] M M , and

Since the conditional moments (the expectation of τ and log τ ) are functions of α and ω , when using the delta method to approximate the expected conditional moment, it is equivalent to approximating the expectation of ˜ α and ˜ ω first, and then use the expected ˜ α and ˜ ω to recover the moments. As a result, we can simply replace v glyph[lscript] and v glyph[lscript] v glyph[latticetop] glyph[lscript] in (35) by their expectation under the current posterior, and we obtain the approximation of E q [ ˜ α ] and E q [ ˜ ω ] . With these approximated expectation, we then construct q ∗ ( τ ) = Gam ( τ | E q [ α ] , E q [ ω ]) at Step 3 in EP. The remaining steps are straightforward. The running posterior update with the Tucker form likelihood follows a similar way.

D More Results on Simulation Study

D.1 Accuracy of Trajectory Recovery

We provide the quantitative result in recovering the factor trajectories. Note that there is only one competing method, NONFAT, which can also estimate factor trajectories. We therefore ran our method and NONFAT on the synthetic dataset. We then randomly sampled 500 time points in the domain and evaluate the RMSE of the learned factor trajectories for each method. As shown in Table 2, the RMSE of NONFAT on recovering u 1 1 ( t ) and u 2 1 ( t ) is close to SFTL, showing NONFAT achieved the same (or very close) quality in recovering these two trajectories. However, on u 1 2 ( t ) and u 2 2 ( t ) , the RMSE of NONFAT is much larger, showing that NONFAT have failed to capture the other two trajectories. By contrast, SFTL consistently well recovered them.

D.2 Sensitive Analysis on Kernel Parameters

To examine the sensitivity to the kernel parameters, we used the synthetic dataset, and randomly sampled 100 entries and new timestamps for evaluation. We then examined the length-scale ρ and

(b) Prediction RMSE with ρ = 0 . 3 and varying a .

amplitude a , for two commonly-used Matérn kernels: Matérn-1/2 and Matérn-3/2. The study was performed on SFTL based on both the CP and Tucker forms. The results are reported in Table 3. Overall, the predictive performance of SFTL is less sensitive to the amplitude parameter a than to the length-scale parameter ρ . But when we use Matérn-1/2, the performance of both SFTL-CP and SFTL-Tucker is quite stable to the length-scale parameter ρ . When we use Matérn-3/2, the choice of the length-scale is critical.

E Real-World Dataset Information and Competing Methods

We tested all the methods in the following four real-world datasets.

FitRecord 3 , workout logs of EndoMondo users' health status in outdoor exercises. We extracted a three-mode tensor among 500 users, 20 sports types, and 50 altitudes. The entry values are heart rates. There are 50 K observed entry values along with the timestamps.

ServerRoom 4 , temperature logs of Poznan Supercomputing and Networking Center. We extracted a three-mode tensor between 3 air conditioning modes (24 ◦ , 27 ◦ and 30 ◦ ), 3 power usage levels (50%, 75%, 100%) and 34 locations. We collected 10K entry values and their timestamps.

BeijingAir-2 5 , air pollution measurement in Beijing from year 2014 to 2017. We extracted a two-mode tensor (monitoring site, pollutant), of size 12 × 6 , and collected 20 K observed entry values (concentration) and their timestamps.

BeijingAir-3 , extracted from the same data source as BeijingAir-2 , a three-mode tensor among 12 monitoring sites, 12 wind speeds and 6 wind directions. The entry value is the PM2.5 concentration. There are 15 K observed entry values at different timestamps.

We first compared with the following state-of-the-art streaming tensor decomposition methods based on the CP or Tucker model. (1) POST [Du et al., 2018], probabilistic streaming CP decomposition via mean-field streaming variational Bayes [Broderick et al., 2013] (2) BASS-Tucker [Fang et al., 2021a] Bayesian streaming Tucker decomposition, which online estimates a sparse tensor-core via a spike-and-slab prior to enhance the interpretability. We also implemented (3) ADF-CP, streaming CP

decomposition by combining the assumed density filtering and conditional moment matching [Wang and Zhe, 2019].

Next, we tested the state-of-the-art static decomposition algorithms, which have to go through the data many times. (4) P-Tucker [Oh et al., 2018], an efficient Tucker decomposition algorithm that performs parallel row-wise updates. (5) CP-ALS and (6) Tucker-ALS [Bader and Kolda, 2008], CP/Tucker decomposition via alternating least square (ALS) updates. The methods (1-6) are not specifically designed for temporal decomposition and cannot utilize the timestamps of the observed entries. In order to incorporate the time information for a fair comparison, we augment the tensor with a time mode, and convert the ordered, unique timestamps into increasing time steps.

We then compared with the most recent continuous-time temporal decomposition methods. Note that none of these methods can handle data streams. They have to iteratively access the data to update the model parameters and factor estimates. (7) CT-CP [Zhang et al., 2021], continuoustime CP decomposition, which uses polynomial splines to model a time-varying coefficient λ for each latent factor, (8) CT-GP, continuous-time GP decomposition, which extends [Zhe et al., 2016a] to use GPs to learn the tensor entry value as a function of the latent factors and time y glyph[lscript] ( t ) = g ( u 1 glyph[lscript] 1 , . . . , u K glyph[lscript] K , t ) ∼ GP (0 , κ ( · , · )) , (9) BCTT [Fang et al., 2022], Bayesian continuoustime Tucker decomposition, which estimates the tensor-core as a time-varying function, (10) THISODE [Li et al., 2022], which uses a neural ODE [Chen et al., 2018] to model the entry value as a function of the latent factors and time, d y glyph[lscript] ( t ) d t = NN ( u 1 glyph[lscript] 1 , . . . , u K glyph[lscript] K , t ) where NN is short for neural networks. (11) NONFAT [Wang et al., 2022], nonparametric factor trajectory learning, the only existing work that also estimates factor trajectories for temporal tensor decomposition. It uses a bi-level GP to estimate the trajectories in the frequency domain and applies inverse Fourier transform to return to the time domain.

F More Results about Prediction Accuracy

We report for R = 2 , R = 3 and R = 7 , the final prediction error (after the data has been processed) of all the methods in Table 4, Table 5, and Table 6, respectively. We report for R = 2 , R = 3 and R = 7 , the online predictive performance of the streaming decomposition approaches in Fig. 5, Fig. 6, and Fig. 7, respectively.

0

20000

40000

Number of Entries

0.45

1.00

1.50

RMSE

SFTL-CP

SFTL-Tucker

ADF-CP

POST

(a)

FitRecord

0

4000

8000

Number of Entries

0.25

0.60

1.00

RMSE

(b)

ServerRoom

0

8000

16000

Number of Entries

0.2

0.7

1.2

RMSE

(c)

BeijingAir-2

0

6000

12000

Number of Entries

0.45

0.70

1.00

RMSE

(d)

0

20000

40000

Number of Entries

0.45

1.00

1.50

RMSE

(a)

FitRecord

0

4000

8000

Number of Entries

0.25

0.60

1.00

RMSE

(b)

ServerRoom

0

8000

16000

Number of Entries

0.2

0.7

1.2

RMSE

(c)

BeijingAir-2

0

6000

12000

Number of Entries

0.45

0.70

1.00

RMSE

(d)

G Running Time

As compared with static (non-streaming) methods, such as BCTT, our method is faster and more efficient. That is because whenever new data comes in, the static methods have to retrain the model from scratch and iteratively access the whole data accumulated so far, while our method only performs

0

20000

40000

Number of Entries

0.45

1.00

1.50

RMSE

(a)

FitRecord

0

4000

8000

Number of Entries

0.25

0.60

1.00

RMSE

(b)

ServerRoom

0

8000

16000

Number of Entries

0.2

0.7

1.2

RMSE

(c)

BeijingAir-2

0

6000

12000

Number of Entries

0.45

0.70

1.00

RMSE

(d)

incremental updates and never needs to revisit the past data. To demonstrate this point, we compared the training time of our method with BCTT on BeijingAir2 dataset. All the methods were run on a Linux workstation. From Table 7, we can see a large speed-up of our method with both the CP and Tucker form. The higher the rank ( R ), the more significant the speed-up.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn traditional tensor decomposition, the factor of decomposition is static. Therefore, it was not possible to fit time-evolving data well. This paper develops SFTL using a Gaussian process to capture the time-developing trajectory of the factors. Their naive implementation would have increased computational complexity. Therefore, they reduce the computational complexity by replacing the Matern kernel in GP with an SDE.\n\nSTRENGTHS:\n- Standard tensor decomposition can only extract linear patterns. SFTL, however, uses a Gaussian process to extract nonlinear features in the time direction.\n- Elegant techniques are used to reduce the computational complexity. It is unfortunate that these techniques are not applicable to non-matern kernels, i.e., RBF kernels.\n- They provide a theoretical discussion of the computational complexity of the proposed method and include additional experimental results in the appendix.\n- Notation of the symbols is clear and not misleading. The paper is well written.\n- It is good to have a discussion about the sensitivity of hyperparameters.\n\nWEAKNESSES:\nThere are some known tricks to reduce the computational complexity of the Gaussian process from O(n^3), such as Inducing Variable Method, Deterministic Training Conditional (DTC) Approximation, and Fully Independent Training Conditional (FITC) Approximation. All of these tricks can be adapted to non-Matern kernels. With these tricks, I think it is possible to achieve fast computations without Hartikainen's elegant tricks. Although the elegant method of this paper is to be evaluated, it is a drawback that there is no comment on this point.\n\nQUESTIONS:\n- I struggled to understand Fig 1. Since there are 3 trajectories in Fig 1, does this correspond to the SFTL for a tensor that has three 3 modes?\n- In my understanding, the size of vec(z) depends on the rank R, where R is a hyperparameter. In Algorithm 1, R is not included in Input. That point needs to be clarified.\n- In section 6.1, the word `node` suddenly appears, but I am unclear what this means. My understanding is that the input to the experiment in section 6.1 is a full-rank 2x2 time-evolving matrix with noise, is this correct?\n\nLIMITATIONS:\nThe authors appropriately mention in the Appendix that the speedup of the proposed method is only applicable to Matern kernels.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors highlight that real-world tensor data often includes time information, representing the timestamps of interactions between objects in different modes. However, most existing decomposition methods for tensors with temporal information only estimate static factors for each object and cannot capture the temporal variation of the factors. To address this limitation, the authors introduce Streaming Factor Trajectory Learning (SFTL), which can efficiently handle data streams and estimate the temporal evolution of objects' representation without storing or re-accessing previous data. \n\nThey use a Gaussian process (GP) prior to model the factor trajectory of each object as a function of time. GPs are flexible nonparametric function priors that can capture complex temporal dynamics. The factor trajectories are combined using either the CANDECOMP/PARAFAC (CP) or Tucker decomposition to sample the tensor entry values at any given time.\n\nThe authors evaluate SFTL using both simulation studies and real-world temporal tensor datasets. In the simulation studies, SFTL recovers nonlinear factor trajectories and provides reasonable uncertainty estimation. When applied to real-world datasets for missing value prediction, SFTL consistently outperforms state-of-the-art streaming CP and Tucker decomposition algorithms in both online and final predictive performance. In some cases, SFTL even surpasses the prediction accuracy of recent static decomposition methods that require multiple passes through the dataset. The learned factor trajectories from a real-world dataset also exhibit interesting temporal evolution.\n\nSTRENGTHS:\nThe main advantage of this method is its computational efficiency. To overcome the computational challenges of the GP's covariance matrix computation in streaming inference, they employ spectral analysis to convert the GP into a linear time-invariant (LTI) stochastic differential equation (SDE). This conversion enables them to transform the SDE into an equivalent state-space prior over the factor states at observed timestamps, simplifying posterior inference and making it computationally efficient.\n\nIn addition, they increase the scalability of tensor decomposition as in the following. They leverage the chain structure of the state-space prior and utilize the conditional expectation propagation framework to develop an efficient online filtering algorithm. This algorithm allows for the estimation of a decoupled running posterior of the involved factor states when new entries at a new timestamp arrive. The decoupled Gaussian estimate enables independent and parallel computation of the full posterior of each factor trajectory without revisiting previous data. The method scales linearly with the number of observed timestamps.\n\nWEAKNESSES:\nI think the paper is not so clear and it is hard to follow the contribution especially in Section 3 and 4. The simulation study looks fine, but not a big difference is observed between SOTA and the STL on the real-world applications.  Besides the real-world applications are not explained, so it is also not clear to understand what are the application areas of this method.\n\nQUESTIONS:\nCan you please explain the real-world applications (we only see the dataset names, but it is not sufficient to understand the application)?\nCan you also add how much gain you provide with STL compared to baselines, both in efficiency and accuracy, e.g. in percent?\nCan you please add more explanations to Figure 1? It is not so easy to understand how the overall method works from that figure.\n\nLIMITATIONS:\nThe authors can discuss more the scalability and the applicability of STL. What are the applications that STL can be used in real-world?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the dynamic tensor factorization under the streaming data setting. In particular, the authors propose to use a state-space Gaussian process (GP) to parameterize the CP or Tucker factors as functions of time. To efficiently learn the GP and handle streaming data, an equivalent SDE and a conditional expectation propagation algorithm are then derived. For experiments, the authors show that the model can learn factor and tensor entry trajectories. Besides, it achieves good competition performance, even compared with model non-streaming settings.

STRENGTHS:
1. This work presents the first model that attempts to learn factor trajectories under streaming data setting.

2. The performance of completion is good. Even though the proposed model runs under the online setting, the error is smaller than many models that use the whole dataset.

WEAKNESSES:
1. Although the problem setting is new in the tensor decomposition field, most techniques used in this work seem standard, such as state-space Gaussian process and expectation propagation.

2. The notations are too complex and hard to follow. There are too many super/sub-scripts and sometimes the meanings are unclear. For examples, for latent variables $z$, sometimes, the second subscript is the rank (e.g., in Line 128, $z^m_{j, r}(t)$, where $r$ is the rank). Then, in Line 141, it becomes the time stamp, e.g., $\bar{z}^m_{j, k}$ with $k$ being the time index.

QUESTIONS:
1. In Equation 9, for the first Gaussian distribution, is the variance missing?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a new approach for the decomposition of a tensor with a temporal dimension. The developed method can apply to streaming data where decomposing the entire tensor from scratch would be prohibitive. \n\nThe key idea is to model each entry (or parameter) in the decomposition factor as a function of time, thus representing a factor trajectory in time, and to assume that this function is sampled from a Gaussian process. Then, given the factor trajectories, initial tensor values are recovered by sampling from a specific Normal distribution.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The authors clearly state the necessary background, problem, and notations.\nThe considered problem of handling streaming data is relevant. However, many technical choices were left unjustified.\n\nWEAKNESSES:\nThe primary concerns are about (implicit) assumptions made in the paper:\n- Parameters of the decomposition factor are sampled from a GP with the same kernel. That says that the smoothness of the factor trajectories is assumed to be similar. That might be a restrictive assumption on the data the method is applied for, but it is not discussed.\n- Relates to the previous concern. Continuity in the latent space of the factors translates to the continuity in the space of the initial tensor values that follow from (1). In practice, a slight change in the factors might hugely affect the initial tensor values and vice versa. It would be essential to see the results of the method on more diverse problems, e.g., streaming point cloud data. \n- Finally, the trade-off between the decomposition rank and sustaining several Gaussian processes (i.e., better modeling of the factors). The rank R used in the experiments is relatively small.\n\n\nSome of the technical choices for the method are not well justified:\n- why is Gamma distribution used?\n- how are the hyperparameters of the models chosen? Do the data dictate those, or is there a reason for them?\n\nA simple baseline would tackle the problem as a regression task, thus possibly learned by a neural network.  It would be great to add it as a baseline.\n\nQUESTIONS:\n- The authors use CP and Tucker decomposition. Why Tensor Train or Tensor Ring decompositions, are not compared to or not used?\nFor example, [1] propose a simple method incorporating temporal component into the decomposition. It is interesting to see the number of parameters comparison, compression time, and quality metrics.\n\n[1] T4DT: Tensorizing Time for Learning Temporal 3D Visual Data\nMikhail Usvyatsov, Rafael Ballester-Rippoll, Lina Bashaeva, Konrad Schindler, Gonzalo Ferrer, Ivan Oseledets\n\nLIMITATIONS:\nThe limitations arise from the set of assumptions made by the methods. Is this method applicable to the streaming 3D visual data, where the change in scenes might be significant from scene to scene? How the chosen rank R affects the quality, and what is the trade-off for it? These questions are not discussed in the paper though critical for the method to be applicable in practice.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}