Title: Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamic networks

representation learning

spectral methods
Abstract: We present a new representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. Given triples $(i,j,t)$, each representing a time-stamped ($t$) interaction between two entities ($i,j$), our procedure returns a continuous-time trajectory for each node, representing its behaviour over time. The framework consists of three stages: estimating pairwise intensity functions, e.g. via kernel smoothing; learning a projection which minimises a notion of intensity reconstruction error; and constructing evolving node representations via the learned projection. The trajectories satisfy two properties, known as structural and temporal coherence, which we see as fundamental for reliable inference. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses. The theory also elucidates the role of smoothing as a bias-variance trade-off, and shows how we can reduce the level of smoothing as the signal-to-noise ratio increases on account of the algorithm `borrowing strength' across the network.

Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks

Abstract

We present a new representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. Given triples ( i, j, t ) , each representing a time-stamped ( t ) interaction between two entities ( i, j ), our procedure returns a continuous-time trajectory for each node, representing its behaviour over time. The framework consists of three stages: estimating pairwise intensity functions, e.g. via kernel smoothing; learning a projection which minimises a notion of intensity reconstruction error; and constructing evolving node representations via the learned projection. The trajectories satisfy two properties, known as structural and temporal coherence, which we see as fundamental for reliable inference. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noisesensitive follow-on analyses. The theory also elucidates the role of smoothing as a bias-variance trade-off, and shows how we can reduce the level of smoothing as the signal-to-noise ratio increases on account of the algorithm 'borrowing strength' across the network.

1 Introduction

Making sense of patterns of connections occurring over time is a common theme of modern data analysis and is often approached in one of two ways. On the one hand, we may see dynamic network data as a graph , in which connections between the same entities over time are somehow treated as one, e.g. through weighting. This view evokes methodological ideas such as community detection [1, 2], topological data analysis [3], or manifold learning [4, 5]. On the other, we may see the data as a set of point processes [6], each modelling the event times of connections between two entities. This view evokes temporal notions such as trend, changepoints and periodicity.

In this paper, we develop a representation learning framework for continuous-time dynamic network data in which ideas from both the graph and temporal domains can be combined. The framework obtains a continuously evolving trajectory for each node which represents its continuously evolving behaviour in the network. These trajectories can be used for data exploration, inference and prediction tasks such as spatio-temporal clustering, behavioural analytics, detecting network trends and periodicities, and forecasting. Our framework provides two basic guarantees which could reasonably be viewed as minimum requirements for these tasks:

structural coherence : when two nodes exhibit similar behaviour at a point in time, their representations at that time are close;

temporal coherence : when a node exhibits similar behaviours at two points in time, its representations at those times are close.

Despite this, almost all existing embedding algorithms fail to satisfy both properties, and there is a perception that some trade-off between the two is necessary [7, 8]. To our knowledge, unfolded spectral embedding [9, 10] is the only exception, but the method is limited to discrete time.

At its core, our procedure is a spectral method and can be implemented at scale using a sparse singular value decomposition. We give a preliminary motivation for the algorithm as minimising a notion of reconstruction error, before developing more rigorous estimation theory: a non-asymptotic, uniform error bound under minimal assumptions on the data generating process, which draws on recent developments in entrywise eigenvector estimation for random matrices [10-14]. This tight control of the behaviour of estimated trajectories means that even quite noise-sensitive follow-on analyses, such as topological data analysis, could plausibly be shown to be consistent.

With appropriate development, the ideas of this paper could be brought to bear on several problems. The first is building a coherent narrative about the nodes based on an embedding. In contact-tracing data for interactions between children at school (Section 5), we see trajectories mixing during lunch hours and then returning to their earlier positions, which represent physical classrooms. Both types of coherence are necessary for this description to be possible. We cannot see this, for example, just by looking at the pattern of clustering over time. Given richer and larger datasets it could be possible to automate this 'story-telling' process. Among many possible applications, such a technology could transform how we use contact-tracing data in the future. Second, the jump from discrete to continuous time has substantial implications for mathematical modelling, since it makes notions such as continuity and smoothness possible. This work could pave the way to describing structural dynamics in formal mathematical and quantitative terms, for example, what we mean by 'communities splitting' [15], including the exact time at which this occurs, or by 'network polarisation' [16], and how it could be measured, e.g. as a type of derivative. Finally, in criminal investigations, dynamic networks are often analysed to locate an entity, such as a victim of humantrafficking, whose pseudonym or other identifier is changing [17]. Similarly, in dynamic networks of corporate contracting, it is useful to detect when one company is acting like another in the past, e.g. when a shell company takes over the illegal operations of a sanctioned company [18]. Since two nodes acting the same way, at different epochs, are still embedded to the same position, our framework could lead to novel matching and tracking technologies.

Related work. While modelling and performing inference on continuous-time dynamic networks has a well-established literature [6, 19-21], the majority of existing methods for representation learning obtain a single, static representation of each node [22-30], and we only aware of two existing methods which learn continuously evolving node representations from the data we consider [31, 32]. Representation learning for discrete-time dynamic networks has a more established literature, and algorithms broadly fall under community detection methods [33-36], fitting latent position models [37-39], spectral methods [9, 10, 40-42] and word-embedding-based methods [43, 44]. For a specific choice of intensity estimator (the histogram), our method can be viewed as a weighted graph analogue of unfolded spectral embedding [9, 10], a spectral method for discrete-time and multilayer networks, but those papers consider different data and models. Given how limited the options are for handling continuous time, in our method comparison we also include some discrete-time methods which could reasonably be used as alternatives.

2 Intensity Profile Projection

Data. We consider dynamic network data, denoted G , representing instantaneous undirected interactions between nodes over time, which we define formally as G = ( V , E ) on a time domain T = (0 , T ] , containing a vertex set V = [ n ] and a set of triples E = { ( i e , j e , t e ) } e ≥ 1 , each corresponding to an undirected interaction event, where i e < j e ∈ V , t e ∈ T . We let E ij := { t : ( i, j, t ) ∈ E} denote the interaction events between nodes i and j .

Model. We assume the interaction events E ij are driven by an independent inhomogeneous Poisson process with intensity λ ij ( t ) . Informally:

Algorithm 1 Approximate Intensity Profile Projection

Input: Continuous time dynamic graph G , dimension d .

1: Construct intensity estimates ̂ λ ij ( · ) of λ ij ( · ) from E ij for all i < j .

2: Compute the top d left singular vectors ̂ u 1 , . . . , ̂ u d of

where t 1 < · · · < t B are equally spaced points on (0 , T ] .

3: Define the trajectory of node i as

Output: Node trajectories ̂ X 1 ( t ) , . . . , ̂ X n ( t ) .

We represent these intensities in a symmetric time-varying matrix Λ ( · ) : T → R n × n + .

Procedure. Intensity Profile Projection can be summarised as follows.

Intensity estimation. Construct intensity estimates ̂ λ ij ( · ) of λ ij ( · ) from E ij for all i < j .

Subspace learning. Compute the top d eigenvectors ̂ U d = ( ̂ u 1 , . . . , ̂ u d ) of

where ̂ Λ ( t ) has symmetric entries ̂ λ ij ( t ) , and rows denoted ̂ Λ i ( t ) called intensity profiles .

Projection. For a query node i at time t , project the intensity profile ̂ Λ i ( t ) onto the subspace spanned by ̂ u 1 , . . . , ̂ u d , to obtain ̂ X i ( t ) = ̂ U ⊤ d ̂ Λ i ( t ) .

While we develop more principled statistical justifications for the procedure in future sections, it is inspired by a simple reconstruction argument. For an arbitrary d -dimensional subspace spanned by the orthonormal columns of a matrix V d ∈ R n × d , let

denote the reconstruction error of node i at time t , and define the integrated residual sum of squares as

Lemma 1. Among all d -dimensional subspaces of R n , the column span of ̂ U d minimises the integrated residual sum of squares criterion ̂ R 2 .

Lemma 1 may be viewed as a dynamic analogue to the classical Eckart-Young theorem on low-rank matrix approximation [45]. A proof is given in Section E of the appendix.

2.1 Intensity estimation

The choice of intensity estimator is left fully open, but our theory makes two important recommendations. First, there are computational gains to be made using sparse estimators for subspace learning. Second, the procedure borrows strength across the network, and can give precise representations even when the individual intensity estimates are noisy (e.g. inconsistent). In our experiments, we focus on standard non-parametric estimators such as the histogram or kernel smoothers, and choose kernels with finite support to induce sparse estimates.

2.2 Subspace learning

The subspace learning step of our procedure involves the computation of an integral, and computing the eigendecomposition of the resulting dense matrix ̂ Σ , both of which may be infeasible for large

networks. If a sparse intensity estimator is employed in step 1 of the procedure and we approximate the integral (1) using a numerical quadrature scheme, then step 2 can be rephrased as a single sparse, truncated singular value decomposition, which can be computed quickly for very large networks using a efficient solver [46, 47].

Consider the numerical approximation

where t 1 < · · · < t B are equally spaced points on (0 , T ] . The top d eigenvectors of the right-handside of (2) are then equal 1 to the top d left singular vectors of the matrix

the row concatenation of ̂ Λ ( t 1 ) , . . . , ̂ Λ ( t B ) . This procedure is presented in Algorithm 1, and we discuss its computational complexity in Section D of the appendix.

2.3 Projection

The inductive nature of the Intensity Profile Projection allows us to obtain representations ̂ X i ( t ) on demand, for example, the full trajectory for a particular node, or the representations of the entire graph at a point in time. It is possible to obtain representations for intensity profiles outside the training sample, corresponding to new nodes or times outside the training domain, allowing online inference. In practice, one will need to retrain occasionally, i.e. return to step 2, although we leave the discussion of this computational and statistical trade-off for future work (see, for example, [48] in the context of static networks).

3 Estimation theory

In this section, we develop estimation theory showing the sense in which ̂ X i ( t ) is a 'good' estimator of X i ( t ) := U ⊤ d Λ i ( t ) where U d = ( u 1 , . . . , u d ) ∈ R n × d is the matrix containing the topd orthonormal eigenvectors of

In this section, we assume, without loss of generality, that T = (0 , 1] . We now introduce some quantities which appear in our main theorem. Firstly, we assume that each λ ij ( · ) is Lipschitz with constant L , and is upper bounded by λ max . Secondly, we define the (reduced) condition number and the eigengap,

respectively, where σ 2 1 ≥ ·· · ≥ σ 2 n are eigenvalues of Σ . Finally, we introduce the subspace coherence parameter

which is small when, informally, information about a single entry of Σ is 'spread out' across the matrix [49].

Rather than attempt to develop a theoretical framework encompassing all intensity estimators, we choose arguably the most rudimentary, the histogram, and we expect more powerful estimators will only improve matters. This choice of estimator is also attractive because it allows us pinpoint the crucial practical considerations at play.

Notation. We say an event E occurs with overwhelming probability if P ( E ) ≥ 1 -n -c for any constant c > 0 . We use a ≲ b to denote that a ≤ Cb where C > 0 is a universal constant which, when qualified with the prior probabilistic statement, may depend on the constant c . Additionally, we write a ≍ b if a ≲ b and a ≳ b .

We now state the assumptions we require for our theorem. Our first assumption is that the intensities are bounded.

Assumption 1 (Bounded intensities) . The intensities are upper bounded by a constant which doesn't depend on the other quantities in the problem; i.e. λ max ≲ 1 .

Our second assumption is on the population integrated residuals. It ensures that the intensity profiles Λ 1 ( t ) , . . . , Λ n ( t ) do not deviate 'too much' from a common low-dimensional subspace.

Assumption 2 (Small population residuals) . The population residuals satisfy

for all t ∈ [0 , 1] .

Our third assumption is a technical condition on the eigengap which, broadly speaking, ensures that there is 'enough signal'. √

Assumption 3 (Enough signal) . The eigengap satisfies δ log( δ/ nλ max ) ≳ κnλ max .

Our final assumption is on the bin size, and ensures that the bins are not chosen 'too small'.

Assumption 4 (Large enough bins) . The number of bins satisfies M ≲ nλ max / log 3 n .

These assumptions are weaker than those typically required in the literature (e.g. on stochastic block models and random dot product graphs [2, 50]). To emphasise this point, consider the following stronger alternative assumptions which imply Assumptions 1, 2 and 3:

Assumption 1a. The intensities λ ij ( t ) are of comparable order, i.e. λ ij ( t ) ≍ ρ for some ρ ≲ 1 and all i, j ∈ [ n ] , t ∈ (0 , 1] .

Assumption 2a. The matrix Σ has rank d ≍ 1 ; is incoherent, i.e. µ ≍ 1 ; and its non-zero eigenvectors are of comparable order, i.e. σ 1 ≍ σ d > σ d +1 = 0 .

It is immediate that Assumption 1a implies Assumption 1 and under Assumption 2a, the population residuals are all exactly zero, κ ≍ 1 and δ ≍ nρ , which implies Assumptions 2 and 3.

Assumption 4 requires that the expected number of events involving each node in each bin is at least of the order log 3 n . This is analogous to the log n degree growth required for perfect clustering under the binary stochastic block model. Since the latter is an information-theoretic bound [51] and the additional logarithmic powers in our work stem from the sub-exponential tails of the Poisson distribution, we do not think this assumption can be weakened.

We now state our main theorem, which under Assumptions 1-4, provides a non-asymptotic bound on the error between the learned representations and their population counterparts, which holds uniformly over the whole node-set and the time domain.

Theorem 1. Suppose that ̂ λ ij ( t ) are histogram estimates with M equally-spaced bins and that Assumptions 1-4 hold. Then with overwhelming probability, there exists an orthogonal matrix W such that

The proof of Theorem 1 is given in Section F of the appendix. As a corollary to Theorem 1, we state a simplified version of this result in which we replace Assumptions 1, 2 and 3 with the stronger Assumptions 1a and 2a. Since the Lipschitz constant L scales with the order of the intensities, and we define the quantity L 0 satisfying L = ρL 0 which is invariant to the rescaling of intensities.

Corollary 1. Suppose that ̂ λ ij ( t ) are histogram estimates with M equally-spaced bins and that Assumptions 1a, 2a and 4 hold. Then with overwhelming probability, there exists an orthogonal matrix W such that

0

2

4

6

8

10

12

bias

5 bins

0

2

4

6

8

10

12

20 bins

0

2

4

6

8

10

12

200 bins

0

2

4

6

8

10

12

time

variance

0

2

4

6

8

10

12

time

0

2

4

6

8

10

12

time

(

t

)

(

t

)

(

t

)

3.1 A bias-variance trade-off

The first term in the bound corresponds to the bias between ¯ X i ( t ) and X i ( t ) , where ¯ X i ( t ) is a histogram approximation to X i ( t ) (modulo orthogonal transformation, see Section F of the appendix). The second term corresponds to the variance of the estimate.

Theorem 1 gives some theoretical guidance on how to select the number of bins in the histogram estimator. For simplicity, we consider the setting of Corollary 1. Ignoring logarithmic terms in n , the bound in (5) is optimised by choosing

Figure 1 illustrates this bias-variance trade-off with an example. We simulate a dynamic network with 100 nodes with common intensities λ ij ( t ) = 0 . 7 ×{ 2 + cos( t ) } , for all i, j , on the time domain (0 , 4 π ] .

The top row shows the population representation X i ( t ) of a single node (gray) and its histogram approximation ¯ X i ( t ) (blue) for a variety of bin sizes. The more bins that are chosen, the smaller the bias and the more ¯ X i ( t ) resembles X i ( t ) . The bottom rows shows the histogram approximation ¯ X i ( t ) , and the estimate ̂ X i ( t ) (orange) obtained using Intensity Profile Projection. The fewer bins that are chosen, the smaller the variance and the more that ̂ X i ( t ) resembles ¯ X i ( t ) .

4 Structural and temporal coherence

For many practical inference tasks, it is desirable for a representation learning procedure to possess the following two properties:

Structural coherence. If two nodes exhibit statistically indistinguishable behaviour at a given time, then their representations at that time are close. That is, if Λ i ( t ) = Λ j ( t ) , then ̂ X i ( t ) ≈ ̂ X j ( t ) ;

Temporal coherence. If a node exhibits statistically indistinguishable behaviour at two distinct points in time, then its representations at both these times are close. That is, if Λ i ( s ) = Λ i ( t ) , then ̂ X i ( s ) ≈ ̂ X i ( t ) .

0.2

0.4

0.6

0.8

1

time

(a) IPP (kernel smoothing)

0.2

0.4

0.6

0.8

1

time

(b) IPP (histogram) / USE

0.2

0.4

0.6

0.8

1

time

(c) Aligned

0.2

0.4

0.6

0.8

1

time

(d) Average

0.2

0.4

0.6

0.8

1

time

(e) CLPM

0.2

0.4

0.6

0.8

1

time

(f) Omnibus

0.2

0.4

0.6

0.8

1

time

(g) PisCES

0.2

0.4

0.6

0.8

1

time

(h) MultiNeSS

Community 1

Community 2

It has been observed in a recent survey of [7] that almost all existing dynamic network embedding procedures possess only one of these properties, but not both.

In the following lemma, we formally define ̂ X i ( s ) ≈ ̂ X j ( t ) to mean that X i ( s ) = X j ( t ) , referring to equality 'up to statistical noise' in the sense of Theorem 1.

Lemma 2. Intensity Profile Projection is both structurally and temporally coherent.

This follows from the simple observation that X i ( t ) is a fixed function of Λ i ( t ) for all i and t . To the best of our knowledge, Intensity Profile Projection is the only existing continuous-time procedure which satisfies these desiderata.

4.1 Simulated example: a bifurcating block model

To illustrate these properties, we simulate a two-community dynamic stochastic block model (i.e. where Λ ( t ) is block structured) in which the intra-community intensities and inter-community intensities are initially distinct, they then gradually merge, remain indistinguishable for some time, and finally diverge. We refer to this model as a bifurcating block model and provide full details of the simulation in Section A of the appendix.

We apply Intensity Profile Projection to the simulated network, using both a histogram intensity estimator, and a kernel smoother, to produce two-dimensional representations. For visualisation, we reduce the dimension from two to one using a dynamic adaptation of principal component analysis (see Section C of the appendix), and the resulting representations are shown in Figures 2(a) and (b).

In both cases, the estimated trajectories mirror the underlying dynamics of the network: the two communities are in well separated to begin with, gradually merge, remain relatively constant before returning to the positions in which they started.

We now illustrate the potential pitfalls of some more naive approaches for embedding dynamic networks. We find that most existing methodology can be viewed as some combination of the two techniques:

Alignment [42]. Obtain a sequence of static snapshots of the network, embed each of the networks snapshots separately and subsequently align the embedding from window t +1 with the embedding from window t .

Averaging [25]. Obtain a static summary of the network by averaging it over time, and to embed this to obtain constant node representations.

Alignment is structurally coherent, however can fail to be temporally coherent. Averaging is temporally coherent, but can fail to be structurally coherent. To illustrate this point, we apply both approaches, using adjacency spectral embedding into two dimensions, orthogonal Procrustes alignment and linear interpolation, to a network simulated from the bifurcating block model. Figures 2(c) and (d) show visualisations of the trajectories obtained from each approach.

4.2 Method comparison

In this section, we demonstrate how our procedure compares to some existing methods on the simulated data described above. Due to the limited number of continuous-time methods, we include a number of discrete-time methods (Omnibus, PisCSE and MultiNeSS) which we give as an input a discrete sequence of snapshots A (1) , . . . , A ( M ) of our simulated continuous-time networks. We compare the following methods:

IPP (kernel smoothing) . Algorithm 1 applied with intensities estimated using kernel smoothing.

IPP (histogram) / USE [10]. Algorithm 1 applied with intensities estimated using a histogram estimator. Equivalent to a weighted extension of the Unfolded Spectral Embedding algorithm of [10].

CLPM [31]. Fits a continuous latent position model log λ ij ( t ) = β -∥ Z i ( t ) -Z j ( t ) ∥ 2 with a penalty on large velocities in the latent space.

Omnibus [40]. Approximately factorises the matrix A with blocks A [ k, l ] = 1 2 ( A ( k ) + A ( l )) , using a spectral decomposition.

PisCES [41]. Minimises the objective function

for L ⋆ (1) , ..., L ⋆ ( M ) , where α ∈ [0 , 1] and L ( k ) are the Laplacian normalisations of A ( k ) . Then, approximately factorises each L ⋆ (1) , ..., L ⋆ ( M ) using spectral decompositions.

MultiNeSS [52]. Fits a latent position model A ij ( k ) ∼ Q {· ; f ( Z i ( k ) , Z j ( k )) , ϕ } , where Q ( · ; θ, ϕ ) is a parametric distribution.

We use an embedding dimension of d = 2 for all methods, and for visualisation we reduce this to one using PCA. Additional details such as hyperparameter selection, where applicable, are given in the Section A of the appendix.

The CLPM and Omnibus methods produce representations which are temporally coherent, however both fail to capture the complete merging of the communities, shown by Figures 2(e) and (f), and are therefore not structurally coherent. The PisCES and MultiNeSS methods produce representations which are structurally coherent, however both are unstable when the communities are indistinguishable, shown by Figures 2(g) and (h), and are therefore not temporally coherent.

5 Real data

We demonstrate Intensity Profile Projection on a dataset containing the face-to-face interactions of the pupils of a primary school in Lyon over two days in October 2009 [53]. During the study, discreet radio-frequency identification devices were worn by 232 pupils and 10 teachers which recorded their face-to-face interactions. When two participants were in close proximity over an interval of 20 seconds, the timestamped interaction event was recorded. The school contains five year groups, each divided into two classes, and each class has an assigned room and an assigned teacher. The school day runs from 8:30am to 4:30pm, with a lunch break from 12:00pm to 2:00pm, and no data was gathered on contacts taking place outside the school or during sports activities. For more details about the study and dataset, we refer the reader to [53].

We apply Intensity Profile Projection to the data corresponding to each day of the study using a kernel smoother with an Epanechnikov kernel, choosing a bandwidth of 5 minutes and computing 30 dimensional trajectories.

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

1A & 1B

Day 1

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Day 2

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

2A & 2B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

3A & 3B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

4A & 4B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

Classes

5A & 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

20

0

20

40

20

0

20

40

(a) 9:30am, Day 1

20

0

20

(b) 9:30am, Day 2

20

0

20

(c) 12:30pm, Day 2

20

0

20

(d) 3:30pm, Day 2

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

To visualise the node trajectories, we first rescale them to have unit norm, which has the effect of removing information about the 'activeness' of a node from its representation (see, for example, [54]), and apply two dimension reduction techniques. The first is principal component analysis (PCA), which we adapt to our dynamic setting by projecting the (centered) representations onto the the direction of maximum average variance over the time domain. This visualisation gives us a temporally coherent view on the trajectories (more details are given in Section C of the appendix). In Figure 3, we visualise the trajectories of each pair of classes in each year group using PCA, and for clarity, we just plot the average trajectory for each class, along with one standard deviation above and below.

The second is t-Distributed Stochastic Neighbor Embedding (t-SNE), a popular non-linear dimensionreduction tool which provides enough flexibility to visualise the whole set of representations at each point in time. Figure 4 shows t-SNE visualisations of the node representations at a collection of times throughout the study. In Section B of the appendix, we include analogous figures for aligned spectral embedding and Omnibus embedding for comparison.

Figure 3 clearly shows the mixing of classes during the lunch hours, and from Figures 4, we see that the the representations are much more fragmented during the lunch hour (12:30pm, Day 2) than they are during lessons at the other times, where they form tighter clusters corresponding to classes.

While it is reassuring that the geometry of the trajectories reflects the known class and timetable structures of the school, it also allows us to uncover structure in the data that was not known from the report on the study. For example, classes 5A and 5B (olive and cyan, respectively) merge into a single cluster at approximately 9:30am on Day 1, and classes 3A and 3B (brown and pink, respectively) do the same at approximately 9:30am on Day 2. One might conjecture that this corresponds to a joint lesson, which is taken by the students of both classes in a year group.

6 Discussion

We have presented an algorithmic framework to learn continuous-time, low-dimensional trajectories representing the evolving behaviours of nodes in a dynamic network. We view our framework as providing a platform on which novel inference procedures can be developed, particularly combining graph and temporal concepts. For example, in dynamic networks with continuously evolving community structure, it might be interesting to develop procedures for detecting branching points (see bifurcating block model example, Section 4), or measures of polarisation and cohesion in the network via the velocities of the trajectories. More generally, we believe there is much left to understand and exploit in the time-evolving topology and geometry of these representations.

A limitation of our framework is the need for bandwidth and dimension selection. These decisions are difficult because they are trade-offs, bias versus variance in the case of bandwidth selection (as seen here), and statistical versus computational in the case of dimension selection (see e.g. [55]). In the presence of a specific supervised downstream task, both decisions could be assisted by crossvalidation. In unsupervised settings with reasonably-sized networks, our method is very fast, allowing expedient exploration of different choices.

Our theory suggests selecting a dimension which corresponds to an 'eigengap' in the spectrum. In practice, it is possible than the spectrum does not decay quickly and no eigengap is present. This likely corresponds to the violation of Assumption 3. Some possible solutions are to take an entrywise transformation of the intensity profiles, such as the square root, to temper heavy tails [56], or to employ a robust subspace estimator, such as robust PCA [57].

Our method might be viewed as a dynamic analogue of adjacency spectral embedding for static graphs [58] and, as a result, in future research it could be profitable to find dynamic analogues of other variants of spectral embedding, e.g. applying Laplacian normalisation [59-61] or regularisation [54, 62].

We believe improved dynamic network analysis can be used for societal good, in applications such as cyber-security, or combating human-trafficking, fraud, and corruption. However, one should also be aware of the risks, particularly to individual privacy and targeted influence.

References

Mark EJ Newman. Modularity and community structure in networks. Proceedings of the National Academy of Sciences , 103(23):8577-8582, 2006.

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Larry Wasserman. Topological data analysis. Annual Review of Statistics and Its Application , 5: 501-532, 2018.

Patrick Rubin-Delanchy. Manifold structure in graph embeddings. Advances in Neural Information Processing Systems , 33:11687-11699, 2020.

Avanti Athreya, Minh Tang, Youngser Park, and Carey E. Priebe. On estimation and inference in latent structure random graphs. Statistical Science , 36(1):68 - 88, 2021. doi: 10.1214/ 20-STS787. URL https://doi.org/10.1214/20-STS787 .

Patrick O Perry and Patrick J Wolfe. Point process modelling for directed interaction networks. Journal of the Royal Statistical Society: Series B: Statistical Methodology , pages 821-849, 2013.

Guotong Xue, Ming Zhong, Jianxin Li, Jia Chen, Chengshuai Zhai, and Ruochen Kong. Dynamic network embedding survey. Neurocomputing , 472:212-223, 2022.

Paulo E. Rauber, Alexandre X. Falcão, and Alexandru C. Telea. Visualizing time-dependent data using dynamic t-sne. EuroVis '16, page 73-77, Goslar, DEU, 2016. Eurographics Association.

Ian Gallagher, Andrew Jones, and Patrick Rubin-Delanchy. Spectral embedding for dynamic networks with stability guarantees. Advances in Neural Information Processing Systems , 34: 10158-10170, 2021.

Andrew Jones and Patrick Rubin-Delanchy. The multilayer random dot product graph. arXiv preprint arXiv:2007.10455 , 2020.

Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Annals of Statistics , 48(3):1452, 2020.

Lihua Lei. Unified ℓ 2 →∞ eigenspace perturbation theory for symmetric random matrices. arXiv preprint arXiv:1909.04798 , 2019.

Jianqing Fan, Weichen Wang, and Yiqiao Zhong. An ℓ ∞ eigenvector perturbation bound and its application to robust covariance estimation. Journal of Machine Learning Research , 18(207): 1-42, 2018.

Fangzheng Xie. Entrywise limit theorems of eigenvectors for signal-plus-noise matrix models with weak signals. arXiv preprint arXiv:2106.09840 , 2021.

Jörg Gross and Carsten KW De Dreu. The rise and fall of cooperation through reputation and group polarization. Nature Communications , 10(1):776, 2019.

Joan-Maria Esteban and Debraj Ray. On the measurement of polarization. Econometrica: Journal of the Econometric Society , pages 819-851.

Pedro Szekely, Craig A Knoblock, Jason Slepicka, Andrew Philpot, Amandeep Singh, Chengye Yin, Dipsy Kapoor, Prem Natarajan, Daniel Marcu, Kevin Knight, et al. Building and using a knowledge graph to combat human trafficking. In The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part II 14 , pages 205-221. Springer, 2015.

Darren Edge. Revealing the hidden structure of corruption. https://www. microsoft.com/en-us/research/group/societal-resilience/articles/ revealing-the-hidden-structure-of-corruption/ , 2021. Accessed: 12 May 2023.

Carter T Butts. 4. a relational event framework for social action. Sociological Methodology , 38 (1):155-200, 2008.

Duy Vu, David Hunter, Padhraic Smyth, and Arthur Asuncion. Continuous-time regression models for longitudinal networks. Advances in Neural Information Processing Systems , 24, 2011.

Francesco Sanna Passino and Nicholas A Heard. Mutually exciting point process graphs for modeling dynamic networks. Journal of Computational and Graphical Statistics , pages 1-15, 2022.

Christopher DuBois, Carter Butts, and Padhraic Smyth. Stochastic blockmodeling of relational event dynamics. In Artificial Intelligence and Statistics , pages 238-246. PMLR, 2013.

Marco Corneli, Pierre Latouche, and Fabrice Rossi. Block modelling in dynamic networks with non-homogeneous poisson processes and exact icl. Social Network Analysis and Mining , 6: 1-14, 2016.

Catherine Matias, Tabea Rebafka, and Fanny Villers. A semiparametric extension of the stochastic block model for longitudinal networks. Biometrika , 105(3):665-680, 2018.

Sharmodeep Bhattacharyya and Shirshendu Chatterjee. Spectral clustering for multiple sparse networks: I. arXiv preprint arXiv:1805.10594 , 2018.

Ruthwik Junuthula, Maysam Haghdan, Kevin S Xu, and Vijay Devabhaktuni. The block point process model for continuous-time event-based dynamic networks. In The World Wide Web Conference , pages 829-839, 2019.

Makan Arastuie, Subhadeep Paul, and Kevin Xu. CHIP: a Hawkes process model for continuoustime networks with scalable and consistent estimation. Advances in Neural Information Processing Systems , 33:16983-16996, 2020.

Jiasen Yang, Vinayak A Rao, and Jennifer Neville. Decoupling homophily and reciprocity with latent space network models. In UAI , 2017.

Zhipeng Huang, Hadeel Soliman, Subhadeep Paul, and Kevin S Xu. A mutually exciting latent space hawkes process model for continuous-time networks. In Uncertainty in Artificial Intelligence , pages 863-873. PMLR, 2022.

Giang Hoang Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. Continuous-time dynamic network embeddings. In Companion proceedings of the Web Conference 2018 , pages 969-976, 2018.

Riccardo Rastelli and Marco Corneli. Continuous latent position models for instantaneous interactions. arXiv preprint arXiv:2103.17146 , 2021.

Igor Artico and Ernst Wit. Fast inference of latent space dynamics in huge relational event networks. arXiv preprint arXiv:2303.17460 , 2023.

Wenjie Fu, Le Song, and Eric P Xing. Dynamic mixed membership blockmodel for evolving networks. In Proceedings of the 26th annual International Conference on Machine Learning , pages 329-336, 2009.

Kevin S Xu and Alfred O Hero. Dynamic stochastic blockmodels for time-evolving social networks. IEEE Journal of Selected Topics in Signal Processing , 8(4):552-562, 2014.

Catherine Matias and Vincent Miele. Statistical clustering of temporal networks through a dynamic stochastic block model. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 79(4):1119-1141, 2017.

James D Wilson, Nathaniel T Stevens, and William H Woodall. Modeling and detecting change in temporal networks via the degree corrected stochastic block model. Quality and Reliability Engineering International , 35(5):1363-1378, 2019.

Purnamrita Sarkar and Andrew W Moore. Dynamic social network analysis using latent space models. Advances in Neural Information Processing Systems , 18:1145, 2006.

Daniel K Sewell and Yuguo Chen. Latent space models for dynamic networks. Journal of the American Statistical Association , 110(512):1646-1657, 2015.

Nial Friel, Riccardo Rastelli, Jason Wyse, and Adrian E Raftery. Interlocking directorates in Irish companies using a latent space model for bipartite networks. Proceedings of the National Academy of Sciences , 113(24):6629-6634, 2016.

Keith Levin, Avanti Athreya, Minh Tang, Vince Lyzinski, Youngser Park, and Carey E Priebe. A central limit theorem for an omnibus embedding of multiple random graphs and implications for multiscale network inference. arXiv preprint arXiv:1705.09355 , 2017.

Fuchen Liu, David Choi, Lu Xie, and Kathryn Roeder. Global spectral clustering in dynamic networks. Proceedings of the National Academy of Sciences , 115(5):927-932, 2018.

Joshua Cape. Spectral analysis of networks with latent space dynamics and signs. Stat , 10(1): e381, 2021.

Lun Du, Yun Wang, Guojie Song, Zhicong Lu, and Junshan Wang. Dynamic network embedding: An extended approach for skip-gram based network embedding. In IJCAI , volume 2018, pages 2086-2092, 2018.

Sedigheh Mahdavi, Shima Khoshraftar, and Aijun An. dynnode2vec: Scalable dynamic network embedding. In 2018 IEEE International Conference on Big Data (Big Data) , pages 3762-3765. IEEE, 2018.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, 1936.

Richard Bruno Lehoucq. Analysis and implementation of an implicitly restarted Arnoldi iteration . Rice University, 1995.

James Baglama and Lothar Reichel. Augmented implicitly restarted Lanczos bidiagonalization methods. SIAM Journal on Scientific Computing , 27(1):19-42, 2005.

Keith Levin, Fred Roosta, Michael Mahoney, and Carey Priebe. Out-of-sample extension of graph adjacency spectral embedding. In International Conference on Machine Learning , pages 2975-2984. PMLR, 2018.

Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimization. Communications of the ACM , 55(6):111-119, 2012.

Avanti Athreya, Donniell E Fishkind, Minh Tang, Carey E Priebe, Youngser Park, Joshua T Vogelstein, Keith Levin, Vince Lyzinski, and Yichen Qin. Statistical inference on random dot product graphs: a survey. The Journal of Machine Learning Research , 18(1):8393-8484, 2017.

Emmanuel Abbe and Colin Sandon. Community detection in general stochastic block models: Fundamental limits and efficient algorithms for recovery. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 670-688. IEEE, 2015.

Peter W MacDonald, Elizaveta Levina, and Ji Zhu. Latent space models for multiplex networks with shared structure. Biometrika , 109(3):683-706, 2022.

Juliette Stehlé, Nicolas V oirin, Alain Barrat, Ciro Cattuto, Lorenzo Isella, Jean-François Pinton, Marco Quaggiotto, Wouter Van den Broeck, Corinne Régis, Bruno Lina, et al. High-resolution measurements of face-to-face contact patterns in a primary school. PloS one , 6(8):e23176, 2011.

Tai Qin and Karl Rohe. Regularized spectral clustering under the degree-corrected stochastic blockmodel. Advances in Neural Information Processing Systems , 26, 2013.

Matthias Löffler, Anderson Y Zhang, and Harrison H Zhou. Optimality of spectral clustering in the gaussian mixture model. The Annals of Statistics , 49(5):2506-2530, 2021.

Ian Gallagher, Andrew Jones, Anna Bertiger, Carey Priebe, and Patrick Rubin-Delanchy. Spectral embedding of weighted graphs. Journal of the American Statistical Association , (just-accepted):1-19, 2023.

Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM) , 58(3):1-37, 2011.

Daniel L Sussman, Minh Tang, Donniell E Fishkind, and Carey E Priebe. A consistent adjacency spectral embedding for stochastic blockmodel graphs. Journal of the American Statistical Association , 107(499):1119-1128, 2012.

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17:395-416, 2007.

Minh Tang and Carey E Priebe. Limit theorems for eigenvectors of the normalized laplacian for random graphs. The Annals of Statistics , 46(5):2360-2415, 2018.

Yilin Zhang and Karl Rohe. Understanding regularized spectral clustering via graph conductance. Advances in Neural Information Processing Systems , 31, 2018.

Francesco Sanna Passino, Nicholas A Heard, and Patrick Rubin-Delanchy. Spectral clustering on spherical coordinates under the degree-corrected stochastic blockmodel. Technometrics , 64 (3):346-357, 2022.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation-based t-sne for improved visualization of single-cell rna-seq data. Nature Methods , 16(3):243-245, 2019.

Fangzheng Xie. Entrywise limit theorems of eigenvectors and their one-step refinement for sparse random graphs. arXiv preprint arXiv:2106.09840 , 2021.

Stanislav Minsker. On some extensions of bernstein's inequality for self-adjoint operators. Statistics & Probability Letters , 127:111-119, 2017.

Afonso S Bandeira and Ramon Van Handel. Sharp nonasymptotic bounds on the norm of random matrices with independent entries. Annals of Probability , 44(4):2479-2506, 2016.

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning , 14(5):566-806, 2021.

David Pollard. Empirical processes: Theory and applications. In NSF-CBMS Regional Conference Series in Probability and Statistics , pages i-86. JSTOR, 1990.

Appendix to 'Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks'

A Details of the simulated example and method comparison of Sections 4.1 and 4.2

We simulate data according to the following generative process, which might be viewed as describing as a dynamic, two-community stochastic block model. Assign to each node i a variable z i ∈ { 1 , 2 } denoting its community (which does not change with time). If nodes i and j are in the same community, i.e., z i = z j , the point process E ij follows a homogeneous Poisson process with (fixed) intensity η 0 . Otherwise, E ij follows an inhomogeneous Poisson process with intensity

where 0 < s 1 < s 2 < T . This model describes two communities gradually coming together until fully merging by time s 1 , before splitting at time s 2 and then gradually drifting apart. We simulate from this model using the parameters T = 1 , n = 100 , z 1 , . . . , z 50 = 1 , z 51 , . . . , z 100 = 2 , η 0 = 100 , η 1 = 10 , s 1 = 0 . 3 and s 2 = 0 . 7 .

In our method comparison, we used an embedding dimension of d = 2 for all methods, unless otherwise stated. For the discrete-time methods, we construct a series of 20 snapshots of the continuous-time network, each a weighted static network whose edge weights are the number of events which occur on the edge in the corresponding time window. The selection of hyperparameters for each method is outlined below:

Intensity Profile Projection (histogram) : We used a bin size of 1 M = 1 20 .

Intensity Profile Projection (kernel smoothing) : We used a Epanechnikov kernel with bandwidth 0 . 1 and applied the approximate Intensity Profile Projection algorithm with B = 20 . Different values of bandwidth gave similar results in terms of embedding structure; we chose this bandwidth to achieve the desired smoothness.

CLPM [31]: The dimension is automatically computed by the algorithm as d = 2 . The hyperparameters are chosen equal to the ones used in 'Simulation C' in [31] which is a similar simulated example with two communities. We used 19 changes point which correspond to 20 windows. The implementation was obtained from the Github repository https://github.com/marcogenni/CLPM .

PisCES [41]: The dimension is automatically selected by the algorithm as d = 2 . The smoothing parameter is chosen with cross-validation which results in equivalent loglikelihood values for α from 0.00001 to 0.001. We choose α = 0 . 001 which is the larger value for which the algorithm converges. The implementation was obtained from the Github repository https://github.com/xuranw/PisCES .

MultiNeSS [52]: The dimension is automatically selected by the algorithm as d = 2 for all windows except windows 5 to 16 for which d = 1 is selected. For these windows, we set missing the second dimension to zeros. The implementation is obtained from the multiness R package (available on CRAN) and hyperparameters are set to their default values.

1.0

1.5

2.0

2.5

(a) 9:30am, Day 1

(b) 12:30pm, Day 1

(c) 3:30pm, Day 1

0.5

1.0

1.5

2.0

1.0

1.5

2.0

2.5

(d) 9:30am, Day 2

0.5

1.0

1.5

2.0

(e) 12:30pm, Day 2

0.5

1.0

1.5

2.0

(f) 3:30pm, Day 2

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

0.5

1.0

1.5

2.0

(a) 9:30am, Day 1

(b) 12:30pm, Day 1

(c) 3:30pm, Day 1

1.5

2.0

2.5

3.0

0.5

1.0

1.5

2.0

(d) 9:30am, Day 2

1.5

2.0

2.5

3.0

(e) 12:30pm, Day 2

1.5

2.0

2.5

3.0

(f) 3:30pm, Day 2

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

1A & 1B

Day 1

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Day 2

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

2A & 2B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

3A & 3B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

4A & 4B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

Classes

5A & 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

20

0

20

40

20

0

20

40

(a) 9:30am, Day 1

20

0

20

(b) 9:30am, Day 2

20

0

20

(c) 12:30pm, Day 2

20

0

20

(d) 3:30pm, Day 2

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

1A & 1B

Day 1

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Day 2

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

2A & 2B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

3A & 3B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

4A & 4B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

Classes

5A & 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

1A & 1B

Day 1

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Day 2

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

2A & 2B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

3A & 3B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

Classes

4A & 4B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

Classes

5A & 5B

9:00am

11:00am

1:00pm

3:00pm

5:00pm

time of day

class 1A

class 1B

class 2A

class 2B

class 3A

class 3B

class 4A

class 4B

class 5A

class 5B

B Additional real data analysis

We provide further experiments and details of the Intensity Profile Projection analysis of the Lyon primary school dataset described in Section 5. As a comparison to the analysis in the paper, we apply Intensity Profile Projection to the data corresponding to each day of the study with a histogram intensity estimator, choosing a bin size of 10 minutes and computing 30-dimensional trajectories.

Figures 5 and 6 show the first two spherical coordinates [63] of the trajectories obtained using the histogram intensity estimator and the Epanechnikov kernel smoother, respectively. The six plots correspond to the morning, lunchtime and afternoon across both days.

Figure 7 (equivalent to Figure 3) visualises the trajectories of each pair of classes in each year group using PCA where we plot the average trajectory for each class, along with one standard deviation above and below. Since every trajectory using the histogram intensity estimator is piece-wise constant, so are the resulting PCA averages. The pairs of trajectories merge and split in a similar way to those obtained using the kernel smoother.

Figure 8 shows t-SNE visualisations of the node representations at a collection of times throughout the study. The plots are very similar to the equivalent Figure 4 for the kernel smoother with almost identical clusters of students before and after lunch, albeit placed differently by the t-SNE algorithm.

We apply the aligned spectral embedding method and the Omnibus spectral embedding method, described in Section 4.2 on the main text, to snapshots of the data. Figures 9 and 10 visualises the respective trajectories of each pair of classes in each year group using PCA where we plot the average trajectory for each class, along with one standard deviation above and below. These representations fail to capture the behaviour of the network and fail in the same ways as they did for the simulated data in Section 4.

C Visualisation

In this section, we give a short overview of the two dimension reduction techniques employed for visualisation in this paper.

For the trajectory visualisation in Figures 2 and 3, we use a principal component analysis which we extend to the dynamic setting by computing a projection using the leading eigenvectors of the average covariance matrix, which we apply to the (globally centered) trajectories. This has a similar flavour to our Intensity Profile Projection algorithm, and since we reduce dimension using a common projection, it gives a temporally coherent view of the trajectories.

The second visualisation technique we apply is t-SNE [64], using the Flt-SNE implementation [65], which we used to obtain Figure 4. This visualisation method is not naturally extended to dynamic data, so we initialise the algorithm using the aforementioned dynamic extension PCA, which results in the visualisations at different times being approximately aligned.

D Computational complexity

In this section, we given a brief discussion of the computational complexity of Algorithm 1. Suppose we use a kernel with finite support of width 2 h , then the expected time complexity of the kernel intensity estimate for a single edge at a single point in time is O ( hλ max ) . Evaluation of ̂ Λ ( t ) is O ( n 2 hλ max ) and of the matrix (3) is O ( Bn 2 hλ max ) . Consider the setting of Corollary 1, where λ ij ( t ) ≍ ρ for all i, j, t and L ≍ ρL 0 where µ, d, L 0 ≍ 1 are fixed, our theory suggests choosing h ≍ ( nρ ) -1 / 3 , and if we suppose nρ ≍ log 3 n , the sparsest regime our theory allows, then evaluation of (3) is O ( Bn log 3 n ) , i.e. log-linear in n .

In practice the top singular vectors of (3) can be computed using the Augmented Implicitly Restarted Lanczos Bidiagonalization algorithm [47] implemented in the irlba package in R, or the irlbpy in Python. The time complexity of this algorithm has not been studied theoretically although in practice it can be incredibly fast. For example, the package author performs a simulated experiment in which they compute the first 2 singular vectors of a sparse 10M x 10M matrix with 1M non-zero entries which takes approximately 6 seconds on a computer with two Intel Xeon CPU E5-2650

processors (16 physical CPU cores) running at 2 GHz equipped with 128 GB of ECC DDR3 RAM (see https://bwlewis.github.io/irlba/comparison.html ).

E Proof of Lemma 1

We begin by writing

and since ( I -VV ⊤ ) is the projection onto the orthogonal complement of the columns space of V , we have that

Therefore, minimising ̂ R 2 ( V ) is equivalent to maximising

where the equality holds due to the invariance of the Euclidean norm under orthogonal transformations. As a result, we have

where the final equality follows from the Courant-Fisher min-max theorem. This concludes the proof.

F Proof of Theorem 1

F.1 Prerequisites

F.1.1 Additional notation

In this proof, we use the notation a n P ≲ b n to mean a n ≲ b n with overwhelming probability.

F.1.2 Symmetric dilation with change of basis 'trick'

Symmetric dilation is a proof technique which allows statements about the eigenvectors of a symmetric matrix to be easily extended to hold for the singular vectors of a (potentially rectangular) asymmetric

matrix. Let M be an n 1 × n 2 matrix with non-zero singular values { σ i } r i =1 and corresponding orthonormal left singular vectors { u i } r i =1 and right singular vectors { v i } r i =1 . Its symmetric dilation is the n × n matrix (with n = n 1 + n 2 ) constructed as

One can easily verify that D ( M ) has eigenvalues {± σ i } r i =1 and eigenvectors { ( u ⊤ i , ± v ⊤ i ) ⊤ } r i =1 . We stack the first d left and right singular vectors into matrices U ∈ R n 1 × d and V ∈ R n 2 × d , and stack the first 2 d eigenvectors of D ( M ) into a matrix

We then have

While this standard construction is very useful when n 1 ≍ n 2 , it can lead to suboptimal bounds when n 2 ≫ n 1 , or n 1 ≫ n 2 , due to an issue about incoherence, which was first raised in [13]. The incoherence of a subspace U 0 spanned by the orthonormal columns of a matrix U 0 ∈ R n 0 × d is

To obtain a good entrywise eigenvector bound under a signal-plus-noise matrix model it is typically necessary that µ ( ¯ U ) ≍ 1 . Observe that

If µ ( U ) , µ ( V ) ≍ 1 and n 1 ≍ n 2 , then µ ( ¯ U ) ≍ 1 and it is typically possible to obtain good bounds. However, when n 2 ≫ n 1 , we have µ ( ¯ U ) ≫ 1 , and a good bound can typically not be obtained. The imbalance of n 1 and n 2 can cause similar issues when obtaining spectral norm bounds.

This issue can be overcome by changing to a basis which balances the contribution from its first n 1 and second n 2 elements of each column. Specifically, let π 1 = √ 2 n 1 / ( n 1 + n 2 ) and π 2 = √ 2 n 2 / ( n 1 + n 2 ) , and consider the basis ˜ e 1 , . . . , ˜ e n 1 + n 2 , such that

where { e i } n 1 + n 2 i =1 are the standard basis vectors in R n 0 . Let | | |·| | | η denote a norm with respect to the column basis { ˜ e i } n 1 + n 2 i =1 , then one can verify that

As a result, if ˜ µ ( U 0 ) = √ n 0 /d | | | U 0 | | | 2 , ∞ , then

regardless of the relative sizes of n 1 and n 2 . We use this symmetric dilation with change-of-basis 'trick' to apply some existing theorems for symmetric matrices to our setting.

F.1.3 Concentration inequalities

In this section, we state a collection of lemmas which we will make use of throughout the proof. We begin with a tail bound for a Poisson random variable.

Lemma 3. Let X ∼ Poisson( λ ) . Then

For t ≥ λ ,

The bound can be established by approximating the Poisson distribution with mean λ as the sum of k Bernoulli random variables with mean λ/k , applying Bernstein's inequality, and taking k → 0 .

Our next result is a concentration bound which adapts Lemma A.1 of [66] and can be proved using a vector version of the Bernstein inequality (Corollary 4.1 in [67]).

Lemma 4. Let X i ∼ Poisson( λ i ) independently for all i = 1 , . . . , n , and suppose Q ∈ R n × d is a deterministic matrix whose rows we denote Q i . Let λ max := max i ∈ [ n ] λ i , then with probability 1 -28 n -3

Next, we state a concentration bound for the spectral norm of random matrices with independent entries which appears as Corollary 3.12 in [68]. The original statement of this lemma is for symmetric random matrices, although we general it to arbitrary random matrices using the symmetric dilation with change-of-basis trick described in Section F.1.2.

Lemma 5 (Corollary 3.12 of [68]) . Let X be an n 1 × n 2 matrix whose entries x ij are independent random variables which obey

Then there exists a universal constant c > 0 such that for any t ≥ 0

where and π k = 2 n k / ( n 1 + n 2 ) .

F.1.4 Weyl's inequality and Wedin's sin Θ theorem

The next two lemmas are classical results matrix perturbation theory. Weyl's inequality shows that the singular values of a matrix are stable with respect to small perturbations.

Lemma 6 (Weyl's inequality) . Let M , E be n 1 × n 2 real-valued matrices. Then for every 1 ≤ i ≤ ( n 1 ∧ n 2 ) , the i th largest singular value of M and M + E obey

One way to measure the distance between two subspaces U and ̂ U is via principal angles. Let U , ̂ U be matrices whose orthonormal columns span U and ̂ U respectively, and let { ξ i } d i =1 denote the singular values of U ⊤ ̂ U . Then the principal angles { θ i } d i =1 between U and ̂ U are defined by ξ i = cos( θ i ) . Let sin Θ( U , ̂ U ) := diag(sin θ 1 , . . . , sin θ d ) . Another way to measure the distance between U and ̂ U is via the difference between the projection operators UU ⊤ and ̂ U ̂ U ⊤ , and in fact, these two characterisations are equivalent. Specifically,

We will use this equivalence without mention throughout the proof. Wedin's sin Θ theorem shows that the singular vectors of a matrix are stable with respect to small perturbations.

Lemma 7. Let M and ̂ M = M + E be two n 1 × n 2 real-valued matrices, and denote by U , ̂ U (respectively V , ̂ V ) the matrices whose columns contain d orthonormal left (respectively, right) singular vectors, corresponding to the d largest singular values of M and ̂ M . Let δ = σ d ( M ) -σ d +1 ( M ) and suppose that ∥ E ∥ < (1 -1 / √ 2) δ , then

See [69] for a proof.

F.2 Implications of Assumptions 1-4

We state here some inequalities involving the parameters of our problem which follow from Assumptions 1-4, and elementary linear algebra. We will use these facts throughout the proof without mention.

The inequality (6) holds since δ ≤ σ 1 / 2 1 ( Σ ) ≤ √ n ∥ Σ ∥ 1 / 2 max ≤ nλ max , and

where we invoked Assumption 4. (7) holds by noting that the previous bound implies log( δ/ √ nλ max ) ≲ log n and invoking Assumption 3. (8) follows from (6) since κ ≲ δ log n/nλ max ≲ log n.

F.3 Setup

We begin by defining M equally spaced bins in (0 , 1] ,

and define the piecewise approximation of λ i ( t ) ,

We then define t 1 , . . . , t m ∈ (0 , 1] such that ¯ λ ij ( t ) = λ ij ( t m ) for all t ∈ B m , which exist by the continuity of λ ij ( t ) , and define the piecewise constant approximation of Y i ( t ) as ¯ Y i ( t ) = U ⊤ ¯ Λ i ( t ) . Our strategy to obtain the bound in Theorem 1 is to decompose it into bias and variance terms:

Section F.6 is dedicated to bounding the bias term, and the rest of this section is dedicated to bounding the variance term. Define the unfolding matrices ̂ Λ and Λ (without arguments) and their (thin) singular value decompositions as

Then one has that for t ∈ B m , m ∈ [ M ] ,

where ̂ V m , ¯ V m denote the m th blocks of ̂ V and ¯ V respectively. Therefore it follows that,

.

For ease of exposition, we drop the subscript 1 on W 1 in this section. Our bound is based on the following decomposition of ̂ V ̂ S -¯ V ¯ SW .

Proposition 1. We have the decomposition

Proof of Proposition 1. We begin by adding and subtracting terms to obtain

Then, noting that ̂ V ̂ S = ̂ Λ ⊤ ̂ U and ( I -¯ V ¯ V ⊤ ) ¯ Λ ⊤ ¯ U = 0 , we have

Next, we decompose ( I -¯ V ¯ V ⊤ )( ̂ Λ -¯ Λ ) ⊤ ̂ U by adding and subtracting terms to obtain

F.4 Technical propositions

We now outline a series of technical propositions which we require to bound terms (9)-(12) which we prove in Section G.

Our first proposition is a 1-norm and spectral norm bound for ̂ Λ .

Proposition 2. The bounds

hold with overwhelming probability.

The spectral norm bound is obtained using Lemma 5, and the 1-norm bound is obtained via an application of the classical Bernstein inequality. The next proposition provides control on the singular values of ̂ Λ .

Proposition 3. Let σ i ( · ) denote the i th ordered singular value of a matrix. The singular values of ̂ Λ satisfy

The result is obtained using Weyl's inequality. The next proposition provides control of the spectral norm of Q ⊤ ( ̂ Λ -¯ Λ ) R , where Q , R are conformable, deterministic unit-norm matrices.

Proposition 4. For conformable, deterministic unit-norm matrices Q , R , the bound

holds with overwhelming probability.

The proof of Proposition 4 employs a classical ε -net argument to the spectral norm of an appropriately constructed symmetric dilation matrix.

The next proposition states that both the matrices ¯ U ⊤ ̂ U and ¯ V ⊤ ̂ V are well approximated by a common orthogonal matrix.

Proposition 5. There exists an orthogonal matrix W such that

hold with overwhelming probability.

To prove Proposition 5, we empoy the Wedin sin Θ theorem to obtain a bound on ∥ ¯ U ⊤ ̂ U -W ∥ 2 . We then obtain a bound on ∥ ¯ U ⊤ ̂ U -¯ V ⊤ ̂ V ∥ 2 , and combine these bounds to establish the proposition.

The next technical tool we require is the ability to 'swap' W , ¯ S and ̂ S .

Proposition 6. The bound

holds with overwhelming probability.

This result follows by applying the previous propositions to an appropriately constructed decomposition.

Part of the challenge of obtaining a good bound on the term (12) is that ( ˜ A -˜ Λ ) and ( ̂ U -¯ UW ) are dependent, and this dependence must be decoupled in order to apply the standard suite of matrix perturbation tools. For m = 1 , . . . , n , let

and construct the auxiliary matrices ̂ Λ (1) , . . . , ̂ Λ ( n ) defined by

In words, ̂ Λ ( m ) ij is the matrix obtained by replacing the m th row and columns of each of its blocks with its expectation. In this way, the m th row of ( ̂ Λ -¯ Λ ) and ̂ Λ ( m ) are independent. Let ̂ U ( m ) denote the matrix of leading left singular values of ̂ Λ ( m ) .

We apply a result due to [11], which provides ℓ 2 , ∞ control of ∥ ̂ U ∥ 2 , ∞ , ∥ ̂ U ( m ) ∥ 2 , ∞ , and ∥ ̂ U ( m ) W ( m ) -U ∥ 2 , ∞ .

Proposition 7. The bounds

hold with overwhelming probability.

In addition, we require control on the spectral norm difference between the projection matrices ̂ U ( m ) ( ̂ U ( m ) ) ⊤ and the projection matrices ¯ U ¯ U ⊤ and ̂ U ̂ U ⊤ , which is provided in the following proposition.

Proposition 8. The bounds

hold with overwhelming probability.

The proof of Proposition 8 requires a delicate 'leave-one-out'-style argument.

F.5 Bounding terms (9) -(12)

Firstly observe that

and therefore term (9) can be bounded as

where the third inequality follows from Assumption 4 that √ M log 3 / 2 n ≲ nλ max , and the definition κ := √ σ 1 ( Σ ) /σ d ( Σ ) , and the fourth inequality follows from Assumption 3 that δ log n ≥ δ log( δ/ √ nλ max ) ≳ κnλ max .

To bound (10), we first apply Wedin's sin Θ theorem to obtain

Then, we use Assumption 2 to obtain the bound

Putting these two bounds together, we bound (10) as

To bound term (11), we set E = ̂ Λ -¯ Λ and note that each column of M -1 E contains independent Poisson random variables with means no greater that M -1 λ max . We will use Lemma 4 to bound the rows E ¯ U as

where the third inequality uses Assumption 4 and a union bound over i = 1 , . . . , n . Therefore, we have

Noting that ∥ ∥ I -¯ V ¯ V ⊤ ∥ ∥ ∞ ≤ ∥ ∥ I -¯ V ¯ V ⊤ ∥ ∥ 2 ≲ 1 , we bound (11) as

Finally, we bound term (12). Let ̂ Λ (1) , . . . , ̂ Λ ( n ) denote the auxiliary matrices described in (14), and let ̂ U ( m ) denote the matrix of leading left singular values of ̂ Λ ( m ) . We can then decompose the Euclidean norm of ( ̂ Λ -¯ Λ ) ⊤ · ,m ( ̂ U -UW ) as

The first term (18) is bounded as

To bound the second term (19), we employ Proposition 8 to obtain

We now set about bounding the third term (20). Let Ω 1 ΞΩ ⊤ 2 denote a singular value decomposition of ( ̂ U ( m ) ) ⊤ ¯ U , and set W ( m ) := Ω 1 Ω ⊤ 2 . Let θ ( m ) i denote the principal angles between the column spaces of ̂ U ( m ) and ¯ U defined by ξ ( m ) i = cos( θ ( m ) i ) , where ξ ( m ) i are the singular values of ( ̂ U ( m ) ) ⊤ ¯ U . We invoke Wedin's theorem to show that

We define H ( m ) := ̂ U ( m ) ( ̂ U ( m ) ) ⊤ -UU ⊤ and note that H ( m ) is independent of ( ̂ Λ -¯ Λ ) m, · and that

Then, using Lemma 4 we have that

Combining these bounds and taking a union bound over m ∈ [ n ] , we have

and the term (12) is bounded as

Combining the bounds on (9)-(12), we have

which completes the proof.

F.6 Controlling the bias term

F.6.1 Edge level bias

We begin by studying the edge-level bias of the histogram intensity estimator. Let ρ ij ( t ) = ∫ t 0 λ ij ( s ) d s denote the cumulative intensity of edge i, j . Now we have, for t ∈ B ℓ ,

for some t ⋆ ∈ B ℓ , which follows by an application of the mean value theorem, where ρ ′ ( t ) = λ ij ( t ) . We then apply the L -Lipschitz continuity of λ ij ( t ) to obtain

F.6.2 A subspace perturbation bound

Define the operator A : ( T → R n ) → R n by

and define the operator A ⋆ : R n → ( T → R n ) by

Then Σ ≡ AA ⋆ since

Denote its eigenvalues σ 2 1 , . . . , σ 2 n , and its corresponding orthonormal eigenvectors u 1 , . . . , u n , and define v i ( · ) = Λ ( · ) u i /ξ i for all i = 1 , . . . , n . Then, Λ ( · ) admits the (functional) singular value decomposition

Define ¯ A and its corresponding parameters analogously. By definition,

Therefore, by (a functional version of) Wedin's sin Θ theorem

F.6.3 Controlling the bias term

Combining the above bounds, we have that uniformly for all i, j, t ,

where the final inequality follows from the fact that δ ≤ nλ max .

G Proofs of the technical propositions

G.1 Proof of Proposition 2

We have that M -1 times the lower-triangular elements of each block of ̂ Λ are independent Poisson random variables with mean given by M -1 times the lower-triangular elements of each block of ¯ Λ . Define the matrices ̂ Λ L and ̂ Λ U with the upper and lower triangles, respectively, of each block set to zero, and the diagonals of each block halved, and define ¯ Λ L and ¯ Λ U similarly, so that M -1 ̂ Λ L (respectively M -1 ̂ Λ U ) has independent Poisson entries with means M -1 ¯ Λ L (respectively M -1 ¯ Λ U ), and ̂ Λ -¯ Λ = ( ̂ Λ L -¯ Λ L ) + ( ̂ Λ U -¯ Λ U ) .

We condition on the event that ( ̂ Λ L -¯ Λ L ) ij ≲ M log n for all i, j , which occurs with overwhelming probability by Lemma 3 and a union bound. Now, we employ Lemma 5 with B := M log n and ν := Mnλ max to obtain

Setting t = M log 3 / 2 n , we have that

where the final inequality follows from Assumption 4. We obtain an analogous bound for ∥ ∥ ∥ ̂ Λ U -¯ Λ U ∥ ∥ ∥ 2 and combine the with the triangle inequality:

We now establish a bound on ∥ ∥ ∥ ̂ Λ -¯ Λ ∥ ∥ ∥ 1 . We condition on the event | ̂ Λ ij -¯ Λ ij | ≲ M log n for all i, j , which occurs with overwhelming probability due to Lemma 3 and a union bound, and note that we have ∑ n j =1 E ( ̂ Λ ji -¯ Λ ji ) 2 ≤ Mnλ max . Then, by the classical Bernstein inequality, we have for any t > 0 ,

and setting t = √ nMλ max log n , we obtain

A union bound establishes that

which establishes Proposition 2.

G.2 Proof of Proposition 3

Proposition 3 follows from an application of Weyl's inequality. We have

since σ 1 ( ¯ Λ ) = √ Mσ 1 ( Σ ) ≳ √ Mδ ≳ √ Mnλ max . Similarly, we have

which establishes the proposition.

G.3 Proof of Proposition 4

We begin by constructing matrices ¯ Q and ¯ E , via a symmetric dilation trick, such that the spectral norms of Q ⊤ ( ̂ Λ -¯ Λ ) R and ¯ Q ⊤ ¯ E ¯ Q coincide, and then apply a classical ε -net argument to the spectral norm of ¯ Q ⊤ ¯ E ¯ Q , following the proof of Lemma D.1 in [66].

First, we set ¯ E := D ( ̂ Λ -¯ Λ ) , where D is the dilation operator (see Section F.1.2) and ¯ Q = ( QR ) , and observe that

where the second equality follows from the Courant-Fischer min-max theorem. Now, let S d -1 ε be an ε -net of the d -1 -dimensional unit sphere S d -1 := { v : ∥ v ∥ 2 = 1 } . By definition, for any v ∈ S d -1 , there exists some w ( v ) ∈ S d -1 ε such that ∥ v -w ( v ) ∥ 2 < ε and

With ε = 1 / 3 , we have

Now, S d -1 1 / 3 can be selected so that its cardinality can be upper bounded by |S d -1 1 / 3 | ≤ 18 d (see, for example, Pollard [70]). For a fixed w ∈ S d -1 1 / 3 , we let z = ¯ Q w and note that since S d -1 1 / 3 ⊂ S d -1 , that ∥ z ∥ 2 ≤ 1 , and

Now, over the event that entries e ij ≲ M log n , for all i, j , which occurs which overwhelming probability by Lemma 3, Hoeffding's inequality and a union bound over w ∈ S d -1 1 / 3 gives

Setting t = M log 3 / 2 n gives

completing the proof.

G.4 Proof of Proposition 5

Denote the singular value decomposition of ¯ U ⊤ ̂ U by Ω 1 ΞΩ ⊤ 2 , where Ξ = diag( ξ 1 , . . . , ξ d ) , and let W := Ω 1 Ω ⊤ 2 . The principal angles { θ i } d i =1 between the column spaces of ¯ U and ̂ U are defined by ξ i = cos( θ i ) , and by the Wedin sin Θ theorem, we have

We apply the Wedin sin Θ theorem again to obtain a bound which we will require later:

We now establish a bound on ∥ ∥ ∥ ¯ U ⊤ ̂ U -¯ V ⊤ ̂ V ∥ ∥ ∥ 2 . We start by showing that

and then we employ the decomposition

Therefore we have

Combining this with (21), we have

G.5 Proof of Proposition 6

We begin by decomposing W ̂ S -¯ SW as

and therefore we have that

which completes the proof.

G.6 Proof of Proposition 7

Akey tool in proving Proposition 7 is a theorem due [11], providing entrywise eigenvector bounds for random matrices. The original statement is given for the eigenvectors of symmetric random matrices

with row and column-wise independence. We state a generalisation for the singular vectors of rectangular matrices with block-wise independence structure. The extension to block-wise independence structure has been handled in [12] (see Proposition 2.1(b) of that paper), although the exposition of the results in this paper is more complicated. For this reason, we choose to state the result due to [11] with this generalisation, which can be seen by following through the relevant parts of their proof.

Lemma 8 (A slight generalisation of Theorem 2.1 of [11]) . Let M 0 be an n 1 × n 2 real-valued random matrix. Define n 0 = n 1 + n 2 and let π 1 = √ 2 n 1 /n 0 and π 2 = √ 2 n 2 /n 0 . Define κ 0 := σ 1 ( E M 0 ) /σ d ( E M 0 ) , δ 0 = σ d ( E M 0 ) -σ d +1 ( E M 0 ) . Suppose there exists some γ > 0 and a function φ : R + → R + which is continuous and non-decreasing on R + , with φ (0) = 0 and φ ( x ) /x non-increasing on R + , such that the following conditions hold:

B1 (Incoherence) . ∥ E M 0 ∥ 2 , ∞ ∨ ∥ ∥ E M ⊤ 0 ∥ ∥ 2 , ∞ ≤ γδ 0 .

B2 (Block-wise independence) . Assume that for any k ∈ [ n 1 ] , ℓ ∈ [ n 2 ] , there exists N 1 k ⊂ [ n 1 ] and N 2 ℓ ⊂ [ n 2 ] , such that the k th row of M 0 is independent of the columns { j : j / ∈ N 1 k } , and the ℓ th column of M 0 is independent of the rows { i : i / ∈ N 2 ℓ } . Let m 0 = max k,ℓ { |N 1 k | ∨ |N 2 ℓ | } and assume m 0 ≲ δ 0 .

B3 (Spectral norm concentration) . κ 0 max { γ, φ ( γ ) } ≲ 1 and P ( ∥ M 0 -E M 0 ∥ 2 > γ ∆) ≤ η 0 for some η 0 ∈ (0 , 1) .

B4. [Row and column concentration] There exists some η 1 ∈ (0 , 1) such that for any matrices Q ∈ R n 1 × d , R ∈ R n 2 × d and i ∈ [ n 1 ] , j ∈ [ n 2 ] ,

and

where b ∞ := π 1 ∥ Q ∥ 2 , ∞ ∨ π 2 ∥ R ∥ 2 , ∞ , and b F := ( π 1 ∥ Q ∥ 2 F + π 2 ∥ R ∥ 2 F ) 1 / 2 .

Let ̂ U 0 , U 0 (respectively ̂ V 0 , V 0 ) be the matrices containing the left (respectively, right) singular vectors corresponding to the d leading singular values of M 0 and E M 0 . Then, with probability at least 1 -η 0 -2 η 1 , we have

The following is an adaptation of Lemma D.2 of [66] (see also Lemma 7 of [11]) who showed an analogous result for Bernoulli random variables.

Lemma 9. Let Y i ∼ Poisson( λ i ) independently for all i = 1 , . . . , n, and suppose Q is a deterministic matrix. The Q i denote the i th row of Q , and set λ max := max i ∈ [ n ] λ i . Then for any α > 0 ,

We omit the proof of Lemma 9, which is identical to the proof of Lemma D.2 of [66] with the Bernoulli moment generating function with the Poisson moment generating function.

With these tools to hand, we begin by obtaining a bound on ∥ ̂ U ∥ 2 , ∞ using Lemma 8, with M 0 := ̂ Λ . We set γ := √ nλ max /δ and

First observe that n 0 = n + nM ≍ nM and π 1 ≍ M -1 / 2 and π 2 ≍ 1 , and that κ 0 = κ and δ 0 = √ Mδ . B1 holds since ∥ ¯ Λ ∥ 2 , ∞ ≤ √ nλ max ≲ √ nλ max since λ max ≲ 1 by Assumption 1. Using Assumptions 3 and 4, we have

and therefore B2 holds. B3 holds from Proposition 2, and observing that by Assumption 3, κ 0 max { γ, ϕ ( γ ) } ≲ 1 .

To see that B4 holds, note that each row and column of M -1 ( ̂ Λ -¯ Λ ) contains independent Poisson random variables with means not exceeding nλ max /M . Then for Q ∈ R n × d , R ∈ R nM × d , setting α = log n/nλ max in Lemma 9 implies that

Similarly, setting α = M log n/nλ max we have

which establishes B4 . Having established B1 -B4 , we are ready to apply Lemma 8:

∥ ∥ ∥ ̂ U ∥ ∥ ∥ 2 , ∞ P ≲ √ M { κ 0 + φ (1) } ( π 1 ∥ U ∥ 2 , ∞ ∨ π 2 ∥ V ∥ 2 , ∞ )+ √ Mγ ( π 1 ∥ ∥ ¯ Λ ∥ ∥ 2 , ∞ ∨ π 2 ∥ ∥ ¯ Λ ⊤ ∥ ∥ 2 , ∞ ) /δ 0 We have

where we used Assumption 3 in the final inequality. Therefore and we have

and so the first term satisfies

To control the second term, we first observe that

where the final inequality follows from κ ≲ log n ≤ √ n and λ max ≲ 1 . Similarly we obtain π 2 ∥ ¯ Λ ⊤ ∥ 2 , ∞ ≲ µδ √ dλ max log n using (22), and therefore

We then have

Combining these bounds, we obtain

We now apply Lemma 8 with M 0 = ̂ Λ ( m ) . We set γ and φ ( x ) as before and verify Assumptions B1 -B4 in the same way. By analogous calculations to the above, we obtain the bound

The final bound is shown in the same way, requiring the additional observation that κ { γ ∨ φ ( γ ) } ≲ 1 which follows from Assumption 3, and we obtain

G.7 Proof of Proposition 8

We show (15) using a simple application of Wedin's inequality:

The proof of (16) requires a more delicate argument. We apply Wedin's theorem to obtain

By Weyl's inequality and

and therefore

We now focus our attention on obtaining a bound for ∥ ( ̂ Λ ( m ) -̂ Λ ) ̂ U ( m ) ∥ F . Let

The ij th entry of ̂ Λ -̂ Λ ( m ) is

and so ̂ Λ ( m ) -̂ Λ is independent of ̂ Λ ( m ) and hence ̂ Λ ( m ) -̂ Λ is independent of ̂ U ( m ) . We can then write

The (square root of the) first term is easily bounded as

and to bound the second term, we employ Lemma 4 to obtain

where we used Assumption 4 in the third inequality. Therefore

Similar analysis yields an analogous bound for ∥ ( ̂ Λ ( m ) -̂ Λ ) ̂ V ( m ) ∥ 2 , and combining this, with (23) and (24) we have

which establishes the proposition.

Peer reviews:
comments: SUMMARY:
The paper proposes a method to learn low-dimensional continuous-time representations of network nodes, based on the collection of interaction events among them. More precisely, the events are in the form of $(i,j,t)$, where $(i,j)$ is the pair of nodes involved in the interaction event, and $t$ is the occurrence time. The proposed method first estimate the intensity function $\lambda_{i,j}(t)$ of events between each pair of nodes $(i,j)$ at every time instant $t$, then project the intensities of each node at time $t$ onto a learned lower dimensional subspace to obtain a representation. Theoretical results on the recovery error of the representation is provided. Numerical experiments using real data shows the effectiveness of the proposed method.

STRENGTHS:
The paper proposes to estimate the representation of nodes using continuous-time events, which seems to be a novel type of data.

WEAKNESSES:
I find the presentation of the paper generally vague and hand-wavy. See the following.

1. The introduction is way too high-level. The authors should be more specific about the problem setting in this paper, for example, why we care about dynamic models, continuous-time event data, low-dimensional representation of nodes etc.

2. The related work is not specific. The authors should use a sentence to summarize the contribution of the mentioned papers and explain the difference from your work.

3. Lemma 1 is not correct. $\widehat U_d$ minimizes the residual sum of squares at $B$ chosen time instants, but not the integrated one. 

4. In Section 3, notation part, what is the difference between $\gg$ and $\gtrsim$? Also is the universal constant multiplicative or additive?

5. It's not clear what `$\approx$' means in Section 4.

QUESTIONS:
.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework called Intensity Profile Projection (IPP) for continuous-time representation learning in dynamic networks. The authors aim to address the challenge of capturing temporal dynamics and evolving relationships in dynamic networks with both high statistical precision and interpretability. The model leverages the concept of intensity profiles, which encode the temporal changes and interactions between nodes in a network. The model provides a uniform error bound for learned node representations and preserves a novel \"temporal coherence\" property compared to existing baselines. Empirical results on real-world dynamic network datasets demonstrate that IPP outperforms existing methods in various tasks such, highlighting its ability to capture continuous-time representations and uncover temporal patterns in dynamic networks.\n\nSTRENGTHS:\n1. The paper introduces the Intensity Profile Projection (IPP) framework, which offers a unique and innovative approach to continuous-time representation learning for dynamic networks. It introduces the concept of intensity profiles and effectively utilizes them to capture temporal dynamics. \n\n2. Theoretical analysis towards the model shows that the model can achieve high statistical precision and preserve interpretability in terms of \"\"temporal coherence\".\n\n3. The paper is in general easy to follow.\n\nWEAKNESSES:\n1. Lack of comparison with state-of-the-art methods: Although the paper claims improved performance over existing methods, it does not provide a comprehensive comparison with some existing continuous models such as GraphODEs[1,2,3,4] which combines neuralODE with GNNs to model network evolution over time.\n\n2. Scalability: The scalability of the IPP framework is not extensively discussed. It would be valuable to address the computational requirements and scalability limitations of the proposed approach, especially when dealing with large-scale dynamic networks.\n\n3. The related work section is too short to provide a comprehensive background of the research topic.\n\n\n[1] Huang, Zijie, Yizhou Sun, and Wei Wang. \"Learning continuous system dynamics from irregularly-sampled partial observations.\" Advances in Neural Information Processing Systems 33 (2020): 16177-16187.\n\n[2] Song Wen, Hao Wang, and Dimitris Metaxas. 2022. Social ODE: Multi-agent Trajectory Forecasting with Neural Ordinary Differential Equations. In Computer Vision–ECCV 2022: 17th European Conference.\n\n[3]Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In\n401 ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 705–715, 2021.\n\n[4] Zang, Chengxi, and Fei Wang. \"Neural dynamics on complex networks.\" In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 892-902. 2020.\n\nQUESTIONS:\n1. What would be the time complexity of the proposed method?\n2. How would the model performance be affected by different network topology?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo represent the continuous dynamic network, authors provide the framework based on the intensity profile. First, the intensity between nodes is estimated, which produces the intensity profile. Low dimension reduction via SVD is applied on the intensity, and then each node embedding is obtained by the low dimensional subspace.\nAuthor also provide various theoretical analysis about the error bound and the bias-various trade-off. Theoretical analysis as well as empirical analysis on the simulated data demonstrates that the proposed method capture structural preserving and temporally coherent properties. Case study on the real data is conducted to explain the outcome of the proposed framework qualitatively\n\nSTRENGTHS:\n- Simple but powerful method is proposed\n- Based on the mathematical model, theoretical bound is analyzed and explained.\n- IPP can capture the behavior of a bifurcating block model.\n\nWEAKNESSES:\n- The proposed method is not novel enough. SVD decomposition is a very common technique for the reduction of dimensions, and it often suffers from the long-tailed singular values. \n- Comparison is too limited. The analysis has been made only for the simulated data with figures. More experiments as well as some qualitative results would be great to have.\n- SVD decomposition does not prevent producing negative values at the reconstruction.\n- The proposed projection space is very dependent on the fixed dataset. At least, how to leverage the given embeddings for predictions is not straightforward. Given this, the potential application value is not very clear.\n\nQUESTIONS:\n- Figure numbers are all wrong. \n- Section 4 is true for any global subspace projection. Also, both properties could be debatable, not necessarily ideal. For instance, when \\Labmda_{i}(s) = \\Lambda_{i}(t), X_{i}(s) = \\alpha * X_{i}(t) could be more ideal, depending on the interactions among the other nodes. \n- It would be great if authors compare the embedding trajectory for more real data, beyond the specific simulated ones.\n\nLIMITATIONS:\nOften, the meaning of each dimension from the SVD decomposition is not clear. This interpretability is not necessarily required for the representation, but this should be addressed when presenting the case study.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach for learning time-varying node embeddings from continuous-time dynamic network data, which consist of a set of instantaneous timestamped relational events between nodes (e.g., messages from one social media user to another). Their proposed approach learns a projection that minimizes reconstruction error of the pairwise intensities between nodes and comes with theoretical guarantees on estimation error. They also show that their approach generates embeddings that both preserve network structure at a given time and is temporally coherent. They demonstrate strong empirical performance on simulated data compared to other dynamic network embeddings. Furthermore, they use their approach to analyze a real network data set on face-to-face interactions of primary school students, which is quite enlightening due to the interpretability of their model.\n\n*After rebuttal:* The authors have clarified the one question I had about the meaning of \"inductive\" in their setting. I continue to strongly support the paper.\n\nSTRENGTHS:\n- Proposed approach learns time-varying node embeddings from continuous-time networks with theoretical guarantees, which is among the first, if not the first, in the literature.\n- Proposed embeddings can satisfy two good properties of structure preservation and temporal coherence.\n- Very well written and organized paper that provides highlights of theoretical analysis in the main paper followed by details, including proofs, in the supplementary.\n\nWEAKNESSES:\n- There's a large body of related literature on probabilistic generative models for continuous-time networks using point process models such as Hawkes processes that should be discussed. Many of these models are based on stochastic block models or latent space models and are thus also learning node embeddings. See suggested references below.\n- No quantitative evaluation. This is only a minor weakness in my opinion because I view the main contribution to be theoretical.\n\nTypos and minor issues:\n- Supplementary Section C heading: Visualsation -> Visualisation\n\nReferences:\n- Arastuie, M., Paul, S., & Xu, K. S. (2020). CHIP: A Hawkes process model for continuous-time networks with scalable and consistent estimation. In Advances in Neural Information Processing Systems 33 (pp. 16983-16996).\n- Corneli, M., Latouche, P., & Rossi, F. (2018). Multiple change points detection and clustering in dynamic networks. Statistics and Computing, 28(5), 989-1007. doi:10.1007/s11222-017-9775-1\n- Huang, Z., Soliman, H., Paul, S., & Xu, K. S. (2022). A mutually exciting latent space Hawkes process model for continuous-time networks. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence (Vol. 180, pp. 863-873).\n- Junuthula, R. R., Haghdan, M., Xu, K. S., & Devabhaktuni, V. K. (2019). The Block Point Process Model for continuous-time event-based dynamic networks. In Proceedings of the World Wide Web Conference (pp. 829-839).\n- Matias, C., Rebafka, T., & Villers, F. (2018). A semiparametric extension of the stochastic block model for longitudinal networks. Biometrika, 105(3), 665-680. doi:10.1093/biomet/asy016\n- Yang, J., Rao, V., & Neville, J. (2017). Decoupling homophily and reciprocity with latent space network models. In Proceedings of the Conference on Uncertainty in Artificial Intelligence.\n\nQUESTIONS:\n1. The authors mention several times that their approach is inductive, allowing one to obtain a node representation profile outside of the training sample. If the task is to obtain the node representation for the future, how would the Intensity Profile Projection approach handle it? Would it require some data from other nodes at that future time?\n\nLIMITATIONS:\nLimitations are thoroughly discussed in Section 6. I commend the authors for being very forthcoming with these limitations. I don't view the limitations as weaknesses, because they are mostly limitations that apply to all unsupervised problems.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}