SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexander Modell<sup>1</sup> Ian Gallagher<sup>2</sup> Emma Ceccherini<sup>2</sup> Nick Whiteley<sup>2</sup> Patrick Rubin-Delanchy<sup>2</sup>
<sup>1</sup>Imperial College London, U.K. <sup>2</sup>University of Bristol, U.K.  
a.modelell@imperial.ac.uk, {ian.gallagher,emma.ceccherini, nick.whiteley,patrick.rubin-delanchy}@bristol.ac.uk
We present a new representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. Given triples  $(i,j,t)$ , each representing a time-stamped  $(t)$  interaction between two entities  $(i,j)$ , our procedure returns a continuous-time trajectory for each node, representing its behaviour over time. The framework consists of three stages: estimating pairwise intensity functions, e.g. via kernel smoothing; learning a projection which minimises a notion of intensity reconstruction error; and constructing evolving node representations via the learned projection. The trajectories satisfy two properties, known as structural and temporal coherence, which we see as fundamental for reliable inference. Moreover, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses. The theory also elucidates the role of smoothing as a bias-variance trade-off, and shows how we can reduce the level of smoothing as the signal-to-noise ratio increases on account of the algorithm 'borrowing strength' across the network.
Making sense of patterns of connections occurring over time is a common theme of modern data analysis and is often approached in one of two ways. On the one hand, we may see dynamic network data as a graph, in which connections between the same entities over time are somehow treated as one, e.g. through weighting. This view evokes methodological ideas such as community detection [1, 2], topological data analysis [3], or manifold learning [4, 5]. On the other, we may see the data as a set of point processes [6], each modelling the event times of connections between two entities. This view evokes temporal notions such as trend, changepoints and periodicity.
In this paper, we develop a representation learning framework for continuous-time dynamic network data in which ideas from both the graph and temporal domains can be combined. The framework obtains a continuously evolving trajectory for each node which represents its continuously evolving behaviour in the network. These trajectories can be used for data exploration, inference and prediction tasks such as spatio-temporal clustering, behavioural analytics, detecting network trends and periodicities, and forecasting. Our framework provides two basic guarantees which could reasonably be viewed as minimum requirements for these tasks:
1. structural coherence: when two nodes exhibit similar behaviour at a point in time, their representations at that time are close;
2. temporal coherence: when a node exhibits similar behaviours at two points in time, its representations at those times are close.
Despite this, almost all existing embedding algorithms fail to satisfy both properties, and there is a perception that some trade-off between the two is necessary [7, 8]. To our knowledge, unfolded spectral embedding [9, 10] is the only exception, but the method is limited to discrete time.
At its core, our procedure is a spectral method and can be implemented at scale using a sparse singular value decomposition. We give a preliminary motivation for the algorithm as minimising a notion of reconstruction error, before developing more rigorous estimation theory: a non-asymptotic, uniform error bound under minimal assumptions on the data generating process, which draws on recent developments in entrywise eigenvector estimation for random matrices [10-14]. This tight control of the behaviour of estimated trajectories means that even quite noise-sensitive follow-on analyses, such as topological data analysis, could plausibly be shown to be consistent.
With appropriate development, the ideas of this paper could be brought to bear on several problems. The first is building a coherent narrative about the nodes based on an embedding. In contact-tracing data for interactions between children at school (Section 5), we see trajectories mixing during lunch hours and then returning to their earlier positions, which represent physical classrooms. Both types of coherence are necessary for this description to be possible. We cannot see this, for example, just by looking at the pattern of clustering over time. Given richer and larger datasets it could be possible to automate this "story-telling" process. Among many possible applications, such a technology could transform how we use contact-tracing data in the future. Second, the jump from discrete to continuous time has substantial implications for mathematical modelling, since it makes notions such as continuity and smoothness possible. This work could pave the way to describing structural dynamics in formal mathematical and quantitative terms, for example, what we mean by "communities splitting" [15], including the exact time at which this occurs, or by "network polarisation" [16], and how it could be measured, e.g. as a type of derivative. Finally, in criminal investigations, dynamic networks are often analysed to locate an entity, such as a victim of humantrafficking, whose pseudonym or other identifier is changing [17]. Similarly, in dynamic networks of corporate contracting, it is useful to detect when one company is acting like another in the past, e.g. when a shell company takes over the illegal operations of a sanctioned company [18]. Since two nodes acting the same way, at different epochs, are still embedded to the same position, our framework could lead to novel matching and tracking technologies.
Related work. While modelling and performing inference on continuous-time dynamic networks has a well-established literature [6, 19-21], the majority of existing methods for representation learning obtain a single, static representation of each node [22-30], and we only aware of two existing methods which learn continuously evolving node representations from the data we consider [31, 32]. Representation learning for discrete-time dynamic networks has a more established literature, and algorithms broadly fall under community detection methods [33-36], fitting latent position models [37-39], spectral methods [9, 10, 40-42] and word-embedding-based methods [43, 44]. For a specific choice of intensity estimator (the histogram), our method can be viewed as a weighted graph analogue of unfolded spectral embedding [9, 10], a spectral method for discrete-time and multilayer networks, but those papers consider different data and models. Given how limited the options are for handling continuous time, in our method comparison we also include some discrete-time methods which could reasonably be used as alternatives.
2 Intensity Profile Projection
Data. We consider dynamic network data, denoted  $\mathcal{G}$ , representing instantaneous undirected interactions between nodes over time, which we define formally as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  on a time domain  $\mathcal{T} = (0,T]$ , containing a vertex set  $\mathcal{V} = [n]$  and a set of triples  $\mathcal{E} = \{(i_e,j_e,t_e)\}_{e\geq 1}$ , each corresponding to an undirected interaction event, where  $i_e < j_e\in \mathcal{V},t_e\in \mathcal{T}$ . We let  $\mathcal{E}_{ij}\coloneqq \{t:(i,j,t)\in \mathcal{E}\}$  denote the interaction events between nodes  $i$  and  $j$ .
Model. We assume the interaction events  $\mathcal{E}_{ij}$  are driven by an independent inhomogeneous Poisson process with intensity  $\lambda_{ij}(t)$ . Informally:
$$
\lambda_ {i j} (t) \mathrm {d} t = \mathbb {P} \left\{\text {i n t e r a c t i o n b e t w e e n n o d e s} i \text {a n d} j \text {i n} (t, t + \mathrm {d} t ] \right\}.
$$
Algorithm 1 Approximate Intensity Profile Projection
Input: Continuous time dynamic graph  $\mathcal{G}$ , dimension  $d$ .
1: Construct intensity estimates  $\widehat{\lambda}_{ij}(\cdot)$  of  $\lambda_{ij}(\cdot)$  from  $\mathcal{E}_{ij}$  for all  $i < j$ .
2: Compute the top  $d$  left singular vectors  $\widehat{u}_1,\ldots ,\widehat{u}_d$  of
$$
\left[ \begin{array}{c c c c} \widehat {\boldsymbol {\Lambda}} (t _ {1}) & \widehat {\boldsymbol {\Lambda}} (t _ {2}) & \dots & \widehat {\boldsymbol {\Lambda}} (t _ {B}) \end{array} \right]
$$
where  $t_1 < \dots < t_B$  are equally spaced points on  $(0, T]$ .
3: Define the trajectory of node  $i$  as
$$
\widehat {X} _ {i} (t) := \widehat {\mathbf {U}} _ {d} ^ {\top} \widehat {\Lambda} _ {i} (t)
$$
where  $\widehat{\mathbf{U}}_d\coloneqq (\widehat{u}_1,\ldots ,\widehat{u}_d)$
Output: Node trajectories  $\widehat{X}_1(t),\ldots ,\widehat{X}_n(t)$
We represent these intensities in a symmetric time-varying matrix  $\Lambda (\cdot):\mathcal{T}\rightarrow \mathbb{R}_{+}^{n\times n}$
Procedure. Intensity Profile Projection can be summarised as follows.
1. Intensity estimation. Construct intensity estimates  $\widehat{\lambda}_{ij}(\cdot)$  of  $\lambda_{ij}(\cdot)$  from  $\mathcal{E}_{ij}$  for all  $i < j$ .
2. Subspace learning. Compute the top  $d$  eigenvectors  $\widehat{\mathbf{U}}_d = (\widehat{u}_1,\dots ,\widehat{u}_d)$  of
$$
\hat {\boldsymbol {\Sigma}} := \frac {1}{T} \int_ {0} ^ {T} \hat {\boldsymbol {\Lambda}} ^ {2} (t) \mathrm {d} t, \tag {1}
$$
where  $\widehat{\Lambda} (t)$  has symmetric entries  $\widehat{\lambda}_{ij}(t)$ , and rows denoted  $\widehat{\Lambda}_i(t)$  called intensity profiles.
3. Projection. For a query node  $i$  at time  $t$ , project the intensity profile  $\widehat{\Lambda}_i(t)$  onto the subspace spanned by  $\widehat{u}_1, \ldots, \widehat{u}_d$ , to obtain  $\widehat{X}_i(t) = \widehat{\mathbf{U}}_d^\top \widehat{\Lambda}_i(t)$ .
While we develop more principled statistical justifications for the procedure in future sections, it is inspired by a simple reconstruction argument. For an arbitrary  $d$ -dimensional subspace spanned by the orthonormal columns of a matrix  $\mathbf{V}_d \in \mathbb{R}^{n \times d}$ , let
$$
\widehat {r} _ {i} (t; \mathbf {V} _ {d}) := \left\| \mathbf {V} _ {d} \mathbf {V} _ {d} ^ {\top} \widehat {\Lambda} _ {i} (t) - \widehat {\Lambda} _ {i} (t) \right\| _ {2}
$$
denote the reconstruction error of node  $i$  at time  $t$ , and define the integrated residual sum of squares as
$$
\widehat {R} ^ {2} (\mathbf {V} _ {d}) := \int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \widehat {r} _ {i} ^ {2} (t; \mathbf {V} _ {d})   \mathrm {d} t.
$$
Lemma 1. Among all  $d$ -dimensional subspaces of  $\mathbb{R}^n$ , the column span of  $\widehat{\mathbf{U}}_d$  minimises the integrated residual sum of squares criterion  $\widehat{R}^2$ .
Lemma 1 may be viewed as a dynamic analogue to the classical Eckart-Young theorem on low-rank matrix approximation [45]. A proof is given in Section E of the appendix.
2.1 Intensity estimation
The choice of intensity estimator is left fully open, but our theory makes two important recommendations. First, there are computational gains to be made using sparse estimators for subspace learning. Second, the procedure borrows strength across the network, and can give precise representations even when the individual intensity estimates are noisy (e.g. inconsistent). In our experiments, we focus on standard non-parametric estimators such as the histogram or kernel smoothers, and choose kernels with finite support to induce sparse estimates.
2.2 Subspace learning
The subspace learning step of our procedure involves the computation of an integral, and computing the eigendecomposition of the resulting dense matrix  $\widehat{\Sigma}$ , both of which may be infeasible for large
networks. If a sparse intensity estimator is employed in step 1 of the procedure and we approximate the integral (1) using a numerical quadrature scheme, then step 2 can be rephrased as a single sparse, truncated singular value decomposition, which can be computed quickly for very large networks using a efficient solver [46, 47].
Consider the numerical approximation
$$
\widehat {\boldsymbol {\Sigma}} \approx \frac {1}{B} \sum_ {b = 1} ^ {B} \widehat {\boldsymbol {\Lambda}} ^ {2} \left(t _ {b}\right) \tag {2}
$$
where  $t_1 < \dots < t_B$  are equally spaced points on  $(0, T]$ . The top  $d$  eigenvectors of the right-hand-side of (2) are then equal to the top  $d$  left singular vectors of the matrix
$$
\left[ \begin{array}{l l l l} \widehat {\boldsymbol {\Lambda}} (t _ {1}) & \widehat {\boldsymbol {\Lambda}} (t _ {2}) & \dots & \widehat {\boldsymbol {\Lambda}} (t _ {B}) \end{array} \right], \tag {3}
$$
the row concatenation of  $\widehat{\Lambda}(t_1), \ldots, \widehat{\Lambda}(t_B)$ . This procedure is presented in Algorithm 1, and we discuss its computational complexity in Section D of the appendix.
2.3 Projection
The inductive nature of the Intensity Profile Projection allows us to obtain representations  $\widehat{X}_i(t)$  on demand, for example, the full trajectory for a particular node, or the representations of the entire graph at a point in time. It is possible to obtain representations for intensity profiles outside the training sample, corresponding to new nodes or times outside the training domain, allowing online inference. In practice, one will need to retrain occasionally, i.e. return to step 2, although we leave the discussion of this computational and statistical trade-off for future work (see, for example, [48] in the context of static networks).
3 Estimation theory
In this section, we develop estimation theory showing the sense in which  $\widehat{X}_i(t)$  is a "good" estimator of  $X_{i}(t)\coloneqq \mathbf{U}_{d}^{\top}\Lambda_{i}(t)$  where  $\mathbf{U}_d = (u_1,\ldots ,u_d)\in \mathbb{R}^{n\times d}$  is the matrix containing the top-  $d$  orthonormal eigenvectors of
$$
\boldsymbol {\Sigma} := \frac {1}{T} \int_ {0} ^ {T} \boldsymbol {\Lambda} ^ {2} (t) \mathrm {d} t.
$$
In this section, we assume, without loss of generality, that  $\mathcal{T} = (0,1]$ . We now introduce some quantities which appear in our main theorem. Firstly, we assume that each  $\lambda_{ij}(\cdot)$  is Lipschitz with constant  $L$ , and is upper bounded by  $\lambda_{\mathrm{max}}$ . Secondly, we define the (reduced) condition number and the eigengap,
$$
\kappa := \frac {\sigma_ {1}}{\sigma_ {d}}, \qquad \text {a n d} \qquad \delta := \sigma_ {d} - \sigma_ {d + 1},
$$
respectively, where  $\sigma_1^2 \geq \dots \geq \sigma_n^2$  are eigenvalues of  $\pmb{\Sigma}$ . Finally, we introduce the subspace coherence parameter
$$
\mu := \sqrt {\frac {n}{d}} \| \mathbf {U} _ {d} \| _ {2, \infty},
$$
which is small when, informally, information about a single entry of  $\pmb{\Sigma}$  is "spread out" across the matrix [49].
Rather than attempt to develop a theoretical framework encompassing all intensity estimators, we choose arguably the most rudimentary, the histogram, and we expect more powerful estimators will only improve matters. This choice of estimator is also attractive because it allows us pinpoint the crucial practical considerations at play.
<sup>1</sup>Up to signs, rotations in the eigenspaces in the case of repeated eigenvalues, and assuming a gap between the  $d$ th and  $(d + 1)$ th eigenvalues.
Notation. We say an event  $E$  occurs with overwhelming probability if  $\mathbb{P}(E) \geq 1 - n^{-c}$  for any constant  $c > 0$ . We use  $a \lesssim b$  to denote that  $a \leq C b$  where  $C > 0$  is a universal constant which, when qualified with the prior probabilistic statement, may depend on the constant  $c$ . Additionally, we write  $a \asymp b$  if  $a \lesssim b$  and  $a \gtrsim b$ .
We now state the assumptions we require for our theorem. Our first assumption is that the intensities are bounded.
Assumption 1 (Bounded intensities). The intensities are upper bounded by a constant which doesn't depend on the other quantities in the problem; i.e.  $\lambda_{\mathrm{max}} \lesssim 1$ .
Our second assumption is on the population integrated residuals. It ensures that the intensity profiles  $\Lambda_1(t),\ldots ,\Lambda_n(t)$  do not deviate "too much" from a common low-dimensional subspace.
Assumption 2 (Small population residuals). The population residuals satisfy
$$
r _ {1} (t), \dots , r _ {n} (t) \lesssim \sqrt {\frac {d}{n}} \mu \delta \log^ {5 / 2} n
$$
for all  $t\in [0,1]$
Our third assumption is a technical condition on the eigengap which, broadly speaking, ensures that there is "enough signal".
Assumption 3 (Enough signal). The eigengap satisfies  $\delta \log (\delta /\sqrt{n\lambda_{\mathrm{max}}})\gtrsim \kappa n\lambda_{\mathrm{max}}$
Our final assumption is on the bin size, and ensures that the bins are not chosen "too small".
Assumption 4 (Large enough bins). The number of bins satisfies  $M \lesssim n\lambda_{\max} / \log^3 n$ .
These assumptions are weaker than those typically required in the literature (e.g. on stochastic block models and random dot product graphs [2, 50]). To emphasise this point, consider the following stronger alternative assumptions which imply Assumptions 1, 2 and 3:
Assumption 1a. The intensities  $\lambda_{ij}(t)$  are of comparable order, i.e.  $\lambda_{ij}(t) \asymp \rho$  for some  $\rho \lesssim 1$  and all  $i,j \in [n], t \in (0,1]$ .
Assumption 2a. The matrix  $\Sigma$  has rank  $d\asymp 1$ ; is incoherent, i.e.  $\mu \asymp 1$ ; and its non-zero eigenvectors are of comparable order, i.e.  $\sigma_{1}\asymp \sigma_{d} > \sigma_{d + 1} = 0$ .
It is immediate that Assumption 1a implies Assumption 1 and under Assumption 2a, the population residuals are all exactly zero,  $\kappa \asymp 1$  and  $\delta \asymp n\rho$ , which implies Assumptions 2 and 3.
Assumption 4 requires that the expected number of events involving each node in each bin is at least of the order  $\log^3 n$ . This is analogous to the  $\log n$  degree growth required for perfect clustering under the binary stochastic block model. Since the latter is an information-theoretic bound [51] and the additional logarithmic powers in our work stem from the sub-exponential tails of the Poisson distribution, we do not think this assumption can be weakened.
We now state our main theorem, which under Assumptions 1-4, provides a non-asymptotic bound on the error between the learned representations and their population counterparts, which holds uniformly over the whole node-set and the time domain.
Theorem 1. Suppose that  $\widehat{\lambda}_{ij}(t)$  are histogram estimates with  $M$  equally-spaced bins and that Assumptions 1-4 hold. Then with overwhelming probability, there exists an orthogonal matrix  $\mathbf{W}$  such that
$$
\max  _ {i \in [ n ]} \sup  _ {t \in (0, 1 ]} \left\| \mathbf {W} \widehat {X} _ {i} (t) - X _ {i} (t) \right\| _ {2} \lesssim \frac {n ^ {3 / 2} L \lambda_ {\operatorname* {m a x}}}{M \delta} + \mu \sqrt {M \lambda_ {\operatorname* {m a x}} d} \cdot \log^ {5 / 2} n. \tag {4}
$$
The proof of Theorem 1 is given in Section F of the appendix. As a corollary to Theorem 1, we state a simplified version of this result in which we replace Assumptions 1, 2 and 3 with the stronger Assumptions 1a and 2a. Since the Lipschitz constant  $L$  scales with the order of the intensities, and we define the quantity  $L_{0}$  satisfying  $L = \rho L_{0}$  which is invariant to the rescaling of intensities.
Corollary 1. Suppose that  $\widehat{\lambda}_{ij}(t)$  are histogram estimates with  $M$  equally-spaced bins and that Assumptions 1a, 2a and 4 hold. Then with overwhelming probability, there exists an orthogonal matrix  $\mathbf{W}$  such that
$$
\max  _ {i \in [ n ]} \sup  _ {t \in (0, 1 ]} \left\| \mathbf {W} \widehat {X} _ {i} (t) - X _ {i} (t) \right\| _ {2} \lesssim \frac {n ^ {1 / 2} \rho L _ {0}}{M} + \sqrt {M \rho} \cdot \log^ {5 / 2} n. \tag {5}
$$
Figure 1: A bias-variance trade-off. We simulate a network with common intensities  $\lambda_{ij}(t) = 0.7 \times \{2 + \cos(t)\}$  for all  $i, j$ , and apply Intensity Profile Projection with a histogram intensity estimator with 5, 20, and 200 bins. In the 'bias' plots, the gray lines show an estimand  $X_i(t)$ , while the blue lines show its histogram approximation. The discrepancy between the gray line and the blue line corresponds the bias of the Intensity Profile Projection estimator. In the 'variance' plots, the blues lines are as in the 'bias' plots and the orange line shows the estimate obtained using Intensity Profile Projection into one dimension. The discrepancy between the blue line and the orange line corresponds the variance of the Intensity Profile Projection estimator.
3.1 A bias-variance trade-off
The first term in the bound corresponds to the bias between  $\bar{X}_i(t)$  and  $X_{i}(t)$ , where  $\bar{X}_i(t)$  is a histogram approximation to  $X_{i}(t)$  (modulo orthogonal transformation, see Section F of the appendix). The second term corresponds to the variance of the estimate.
Theorem 1 gives some theoretical guidance on how to select the number of bins in the histogram estimator. For simplicity, we consider the setting of Corollary 1. Ignoring logarithmic terms in  $n$ , the bound in (5) is optimised by choosing
$$
M \asymp \left(n \rho L _ {0} ^ {2}\right) ^ {1 / 3}.
$$
Figure 1 illustrates this bias-variance trade-off with an example. We simulate a dynamic network with 100 nodes with common intensities  $\lambda_{ij}(t) = 0.7 \times \{2 + \cos(t)\}$ , for all  $i, j$ , on the time domain  $(0,4\pi]$ .
The top row shows the population representation  $X_{i}(t)$  of a single node (gray) and its histogram approximation  $\bar{X}_i(t)$  (blue) for a variety of bin sizes. The more bins that are chosen, the smaller the bias and the more  $\bar{X}_i(t)$  resembles  $X_{i}(t)$ . The bottom rows show the histogram approximation  $\bar{X}_i(t)$ , and the estimate  $\widehat{X}_i(t)$  (orange) obtained using Intensity Profile Projection. The fewer bins that are chosen, the smaller the variance and the more that  $\widehat{X}_i(t)$  resembles  $\bar{X}_i(t)$ .
4 Structural and temporal coherence
For many practical inference tasks, it is desirable for a representation learning procedure to possess the following two properties:
- Structural coherence. If two nodes exhibit statistically indistinguishable behaviour at a given time, then their representations at that time are close. That is, if  $\Lambda_{i}(t) = \Lambda_{j}(t)$ , then  $\widehat{X}_i(t) \approx \widehat{X}_j(t)$ ;
- Temporal coherence. If a node exhibits statistically indistinguishable behaviour at two distinct points in time, then its representations at both these times are close. That is, if  $\Lambda_{i}(s) = \Lambda_{i}(t)$ , then  $\widehat{X}_i(s) \approx \widehat{X}_i(t)$ .
(e) CLPM
(f) Omnibus
Figure 2: One-dimensional PCA visualisation of the two-dimensional node representations, obtained using a collection of methods, for a network simulated from a bifurcating block model. Colours correspond to the community membership of the node.
(g) PisCES
(h) MultiNeSS
It has been observed in a recent survey of [7] that almost all existing dynamic network embedding procedures possess only one of these properties, but not both.
In the following lemma, we formally define  $\widehat{X}_i(s) \approx \widehat{X}_j(t)$  to mean that  $X_i(s) = X_j(t)$ , referring to equality "up to statistical noise" in the sense of Theorem 1.
Lemma 2. Intensity Profile Projection is both structurally and temporally coherent.
This follows from the simple observation that  $X_{i}(t)$  is a fixed function of  $\Lambda_{i}(t)$  for all  $i$  and  $t$ . To the best of our knowledge, Intensity Profile Projection is the only existing continuous-time procedure which satisfies these desiderata.
4.1 Simulated example: a bifurcating block model
To illustrate these properties, we simulate a two-community dynamic stochastic block model (i.e. where  $\Lambda(t)$  is block structured) in which the intra-community intensities and inter-community intensities are initially distinct, they then gradually merge, remain indistinguishable for some time, and finally diverge. We refer to this model as a bifurcating block model and provide full details of the simulation in Section A of the appendix.