SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We apply Intensity Profile Projection to the simulated network, using both a histogram intensity estimator, and a kernel smoother, to produce two-dimensional representations. For visualisation, we reduce the dimension from two to one using a dynamic adaptation of principal component analysis (see Section C of the appendix), and the resulting representations are shown in Figures 2(a) and (b).
In both cases, the estimated trajectories mirror the underlying dynamics of the network: the two communities are in well separated to begin with, gradually merge, remain relatively constant before returning to the positions in which they started.
We now illustrate the potential pitfalls of some more naive approaches for embedding dynamic networks. We find that most existing methodology can be viewed as some combination of the two techniques:
- Alignment [42]. Obtain a sequence of static snapshots of the network, embed each of the networks snapshots separately and subsequently align the embedding from window  $t + 1$  with the embedding from window  $t$ .
- Averaging [25]. Obtain a static summary of the network by averaging it over time, and to embed this to obtain constant node representations.
Alignment is structurally coherent, however can fail to be temporally coherent. Averaging is temporally coherent, but can fail to be structurally coherent. To illustrate this point, we apply both approaches, using adjacency spectral embedding into two dimensions, orthogonal Procrustes alignment and linear interpolation, to a network simulated from the bifurcating block model. Figures 2(c) and (d) show visualisations of the trajectories obtained from each approach.
4.2 Method comparison
In this section, we demonstrate how our procedure compares to some existing methods on the simulated data described above. Due to the limited number of continuous-time methods, we include a number of discrete-time methods (Omnibus, PisCSE and MultiNeSS) which we give as an input a discrete sequence of snapshots  $\mathbf{A}(1),\ldots ,\mathbf{A}(M)$  of our simulated continuous-time networks. We compare the following methods:
- IPP (kernel smoothing). Algorithm 1 applied with intensities estimated using kernel smoothing.
- IPP (histogram) / USE [10]. Algorithm 1 applied with intensities estimated using a histogram estimator. Equivalent to a weighted extension of the Unfolded Spectral Embedding algorithm of [10].
- CLPM [31]. Fits a continuous latent position model  $\log \lambda_{ij}(t) = \beta -\| Z_i(t) - Z_j(t)\|^2$  with a penalty on large velocities in the latent space.
- Omnibus [40]. Approximately factorises the matrix  $\mathbf{A}$  with blocks  $\mathbf{A}[k,l] = \frac{1}{2} (\mathbf{A}(k) + \mathbf{A}(l))$ , using a spectral decomposition.
- PisCES [41]. Minimises the objective function
$$
\sum_ {k = 1} ^ {M} \left\| \mathbf {L} (k) - \mathbf {L} ^ {\star} (k) \right\| _ {F} ^ {2} + \alpha \sum_ {k = 1} ^ {M - 1} \left\| \mathbf {L} ^ {\star} (k) - \mathbf {L} ^ {\star} (k + 1) \right\| _ {F} ^ {2},
$$
for  $\mathbf{L}^{\star}(1),\dots,\mathbf{L}^{\star}(M)$ , where  $\alpha \in [0,1]$  and  $\mathbf{L}(k)$  are the Laplacian normalisations of  $\mathbf{A}(k)$ . Then, approximately factorises each  $\mathbf{L}^{\star}(1),\dots,\mathbf{L}^{\star}(M)$  using spectral decompositions.
- MultiNeSS [52]. Fits a latent position model  $\mathbf{A}_{ij}(k) \sim Q\{\cdot; f(Z_i(k), Z_j(k)), \phi\}$ , where  $Q(\cdot; \theta, \phi)$  is a parametric distribution.
We use an embedding dimension of  $d = 2$  for all methods, and for visualisation we reduce this to one using PCA. Additional details such as hyperparameter selection, where applicable, are given in the Section A of the appendix.
The CLPM and Omnibus methods produce representations which are temporally coherent, however both fail to capture the complete merging of the communities, shown by Figures 2(e) and (f), and are therefore not structurally coherent. The PisCES and MultiNeSS methods produce representations which are structurally coherent, however both are unstable when the communities are indistinguishable, shown by Figures 2(g) and (h), and are therefore not temporally coherent.
5 Real data
We demonstrate Intensity Profile Projection on a dataset containing the face-to-face interactions of the pupils of a primary school in Lyon over two days in October 2009 [53]. During the study, discreet radio-frequency identification devices were worn by 232 pupils and 10 teachers which recorded their face-to-face interactions. When two participants were in close proximity over an interval of 20 seconds, the timestamped interaction event was recorded. The school contains five year groups, each divided into two classes, and each class has an assigned room and an assigned teacher. The school day runs from 8:30am to 4:30pm, with a lunch break from 12:00pm to 2:00pm, and no data was gathered on contacts taking place outside the school or during sports activities. For more details about the study and dataset, we refer the reader to [53].
We apply Intensity Profile Projection to the data corresponding to each day of the study using a kernel smoother with an Epanechnikov kernel, choosing a bandwidth of 5 minutes and computing 30 dimensional trajectories.
Figure 3: One-dimensional PCA visualisation of the 30-dimensional node representations for pairs of classes in the same year group. The solid lines show the average trajectory for each class, and the dashed line shows one standard deviation above and below.
Figure 4: Two-dimensional t-SNE visualisation of the 30-dimensional node representations of all pupils and teachers evaluated at 9:30am on Day 1, and 9:30am, 12:30pm and 3:30pm on Day 2.
To visualise the node trajectories, we first rescale them to have unit norm, which has the effect of removing information about the "activeness" of a node from its representation (see, for example, [54]), and apply two dimension reduction techniques. The first is principal component analysis (PCA), which we adapt to our dynamic setting by projecting the (centered) representations onto the direction of maximum average variance over the time domain. This visualisation gives us a temporally coherent view on the trajectories (more details are given in Section C of the appendix). In Figure 3, we visualise the trajectories of each pair of classes in each year group using PCA, and for clarity, we just plot the average trajectory for each class, along with one standard deviation above and below.
The second is t-Distributed Stochastic Neighbor Embedding (t-SNE), a popular non-linear dimension-reduction tool which provides enough flexibility to visualise the whole set of representations at each point in time. Figure 4 shows t-SNE visualisations of the node representations at a collection of times throughout the study. In Section B of the appendix, we include analogous figures for aligned spectral embedding and Omnibus embedding for comparison.
Figure 3 clearly shows the mixing of classes during the lunch hours, and from Figures 4, we see that the representations are much more fragmented during the lunch hour (12:30pm, Day 2) than they are during lessons at the other times, where they form tighter clusters corresponding to classes.
While it is reassuring that the geometry of the trajectories reflects the known class and timetable structures of the school, it also allows us to uncover structure in the data that was not known from the report on the study. For example, classes 5A and 5B (olive and cyan, respectively) merge into a single cluster at approximately 9:30am on Day 1, and classes 3A and 3B (brown and pink, respectively) do the same at approximately 9:30am on Day 2. One might conjecture that this corresponds to a joint lesson, which is taken by the students of both classes in a year group.
We have presented an algorithmic framework to learn continuous-time, low-dimensional trajectories representing the evolving behaviours of nodes in a dynamic network. We view our framework as providing a platform on which novel inference procedures can be developed, particularly combining graph and temporal concepts. For example, in dynamic networks with continuously evolving community structure, it might be interesting to develop procedures for detecting branching points (see bifurcating block model example, Section 4), or measures of polarisation and cohesion in the network via the velocities of the trajectories. More generally, we believe there is much left to understand and exploit in the time-evolving topology and geometry of these representations.
A limitation of our framework is the need for bandwidth and dimension selection. These decisions are difficult because they are trade-offs, bias versus variance in the case of bandwidth selection (as seen here), and statistical versus computational in the case of dimension selection (see e.g. [55]). In the presence of a specific supervised downstream task, both decisions could be assisted by cross-validation. In unsupervised settings with reasonably-sized networks, our method is very fast, allowing expedient exploration of different choices.
Our theory suggests selecting a dimension which corresponds to an "eigengap" in the spectrum. In practice, it is possible than the spectrum does not decay quickly and no eigengap is present. This likely corresponds to the violation of Assumption 3. Some possible solutions are to take an entrywise transformation of the intensity profiles, such as the square root, to temper heavy tails [56], or to employ a robust subspace estimator, such as robust PCA [57].
Our method might be viewed as a dynamic analogue of adjacency spectral embedding for static graphs [58] and, as a result, in future research it could be profitable to find dynamic analogues of other variants of spectral embedding, e.g. applying Laplacian normalisation [59-61] or regularisation [54, 62].
We believe improved dynamic network analysis can be used for societal good, in applications such as cyber-security, or combating human-trafficking, fraud, and corruption. However, one should also be aware of the risks, particularly to individual privacy and targeted influence.
[56] Ian Gallagher, Andrew Jones, Anna Bertiger, Carey Priebe, and Patrick Rubin-Delanchy. Spectral embedding of weighted graphs. Journal of the American Statistical Association, (just-accepted):1–19, 2023.
[57] Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
[58] Daniel L Sussman, Minh Tang, Donniell E Fishkind, and Carey E Priebe. A consistent adjacency spectral embedding for stochastic blockmodel graphs. Journal of the American Statistical Association, 107(499):1119-1128, 2012.
[59] Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
[60] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17:395-416, 2007.
[61] Minh Tang and Carey E Priebe. Limit theorems for eigenvectors of the normalized laplacian for random graphs. The Annals of Statistics, 46(5):2360-2415, 2018.
[62] Yilin Zhang and Karl Rohe. Understanding regularized spectral clustering via graph conductance. Advances in Neural Information Processing Systems, 31, 2018.
[63] Francesco Sanna Passino, Nicholas A Heard, and Patrick Rubin-Delanchy. Spectral clustering on spherical coordinates under the degree-corrected stochastic blockmodel. Technometrics, 64 (3):346-357, 2022.
[64] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
[65] George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation-based t-sne for improved visualization of single-cell rna-seq data. Nature Methods, 16(3):243-245, 2019.
[66] Fangzheng Xie. Entrywise limit theorems of eigenvectors and their one-step refinement for sparse random graphs. arXiv preprint arXiv:2106.09840, 2021.
[67] Stanislav Minsker. On some extensions of bernstein's inequality for self-adjoint operators. Statistics & Probability Letters, 127:111–119, 2017.
[68] Afonso S Bandeira and Ramon Van Handel. Sharp nonasymptotic bounds on the norm of random matrices with independent entries. Annals of Probability, 44(4):2479-2506, 2016.
[69] Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning, 14(5):566-806, 2021.
[70] David Pollard. Empirical processes: Theory and applications. In NSF-CBMS Regional Conference Series in Probability and Statistics, pages i-86. JSTOR, 1990.
Appendix to "Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks"
A Details of the simulated example and method comparison of Sections 4.1 and 4.2
We simulate data according to the following generative process, which might be viewed as describing as a dynamic, two-community stochastic block model. Assign to each node  $i$  a variable  $z_{i} \in \{1,2\}$  denoting its community (which does not change with time). If nodes  $i$  and  $j$  are in the same community, i.e.,  $z_{i} = z_{j}$ , the point process  $\mathcal{E}_{ij}$  follows a homogeneous Poisson process with (fixed) intensity  $\eta_0$ . Otherwise,  $\mathcal{E}_{ij}$  follows an inhomogeneous Poisson process with intensity
$$
\lambda_ {i j} (t) = \left\{ \begin{array}{l l} \eta_ {0} \exp \{\eta_ {1} (t - s _ {1}) \} & t <   s _ {1}, \\ \eta_ {0} & s _ {1} \leq t <   s _ {2}, \\ \eta_ {0} \exp \{- \eta_ {1} (t - s _ {2}) \} & t \geq s _ {2}, \end{array} \right.
$$
where  $0 < s_1 < s_2 < T$ . This model describes two communities gradually coming together until fully merging by time  $s_1$ , before splitting at time  $s_2$  and then gradually drifting apart. We simulate from this model using the parameters  $T = 1, n = 100, z_1, \ldots, z_{50} = 1, z_{51}, \ldots, z_{100} = 2, \eta_0 = 100, \eta_1 = 10, s_1 = 0.3$  and  $s_2 = 0.7$ .
In our method comparison, we used an embedding dimension of  $d = 2$  for all methods, unless otherwise stated. For the discrete-time methods, we construct a series of 20 snapshots of the continuous-time network, each a weighted static network whose edge weights are the number of events which occur on the edge in the corresponding time window. The selection of hyperparameters for each method is outlined below:
- Intensity Profile Projection (histogram): We used a bin size of  $\frac{1}{M} = \frac{1}{20}$ .
- Intensity Profile Projection (kernel smoothing): We used a Epanechnikov kernel with bandwidth 0.1 and applied the approximate Intensity Profile Projection algorithm with  $B = 20$ . Different values of bandwidth gave similar results in terms of embedding structure; we chose this bandwidth to achieve the desired smoothness.
- CLPM [31]: The dimension is automatically computed by the algorithm as  $d = 2$ . The hyperparameters are chosen equal to the ones used in "Simulation C" in [31] which is a similar simulated example with two communities. We used 19 changes point which correspond to 20 windows. The implementation was obtained from the Github repository https://github.com/marcogenni/CLPM.
- PisCES [41]: The dimension is automatically selected by the algorithm as  $d = 2$ . The smoothing parameter is chosen with cross-validation which results in equivalent log-likelihood values for  $\alpha$  from 0.00001 to 0.001. We choose  $\alpha = 0.001$  which is the larger value for which the algorithm converges. The implementation was obtained from the Github repository https://github.com/xuranw/PisCES.
- MultiNeSS [52]: The dimension is automatically selected by the algorithm as  $d = 2$  for all windows except windows 5 to 16 for which  $d = 1$  is selected. For these windows, we set missing the second dimension to zeros. The implementation is obtained from the multiness R package (available on CRAN) and hyperparameters are set to their default values.
Figure 5: The first two dimensions of the spherical coordinates of the coordinates  $\widehat{X}_i(t)$  using the histogram intensity estimator for times corresponding to the morning, lunchtime and afternoon across both days. The colours indicate classes with black points representing teachers.
(a) 9:30am, Day 1
(b) 12:30pm, Day 1
(c) 3:30pm, Day 1
(d) 9:30am, Day 2
(e) 12:30pm, Day 2
(f) 3:30pm, Day 2
Figure 6: The first two dimensions of the spherical coordinates of the trajectories  $\widehat{X}_i(t)$  using the Epanechnikov kernel smoother for times corresponding to the morning, lunchtime and afternoon across both days. The colours indicate classes with black points representing teachers.
Figure 7: One-dimensional PCA visualisation of the 30-dimensional node representations for pairs of classes in the same year group. The solid lines show the average trajectory for each class, and the dashed line shows one standard deviation above and below.
Figure 8: Two-dimensional t-SNE visualisation of the 30-dimensional node representations of all pupils and teachers evaluated at 9:30am on Day 1, and 9:30am, 12:30pm and 3:30pm on Day 2.
Figure 9: One-dimensional PCA visualisation of the 30-dimensional node representations for pairs of classes in the same year group, obtained using aligned spectral embedding. The solid lines show the average trajectory for each class, and the dashed line shows one standard deviation above and below.
Figure 10: One-dimensional PCA visualisation of the 30-dimensional node representations for pairs of classes in the same year group, obtained using the Omnibus spectral embedding. The solid lines show the average trajectory for each class, and the dashed line shows one standard deviation above and below.
B Additional real data analysis
We provide further experiments and details of the Intensity Profile Projection analysis of the Lyon primary school dataset described in Section 5. As a comparison to the analysis in the paper, we apply Intensity Profile Projection to the data corresponding to each day of the study with a histogram intensity estimator, choosing a bin size of 10 minutes and computing 30-dimensional trajectories.
Figures 5 and 6 show the first two spherical coordinates [63] of the trajectories obtained using the histogram intensity estimator and the Epanechnikov kernel smoother, respectively. The six plots correspond to the morning, lunchtime and afternoon across both days.
Figure 7 (equivalent to Figure 3) visualises the trajectories of each pair of classes in each year group using PCA where we plot the average trajectory for each class, along with one standard deviation above and below. Since every trajectory using the histogram intensity estimator is piece-wise constant, so are the resulting PCA averages. The pairs of trajectories merge and split in a similar way to those obtained using the kernel smoother.
Figure 8 shows t-SNE visualisations of the node representations at a collection of times throughout the study. The plots are very similar to the equivalent Figure 4 for the kernel smoother with almost identical clusters of students before and after lunch, albeit placed differently by the t-SNE algorithm.
We apply the aligned spectral embedding method and the Omnibus spectral embedding method, described in Section 4.2 on the main text, to snapshots of the data. Figures 9 and 10 visualises the respective trajectories of each pair of classes in each year group using PCA where we plot the average trajectory for each class, along with one standard deviation above and below. These representations fail to capture the behaviour of the network and fail in the same ways as they did for the simulated data in Section 4.
C Visualisation
In this section, we give a short overview of the two dimension reduction techniques employed for visualisation in this paper.
For the trajectory visualisation in Figures 2 and 3, we use a principal component analysis which we extend to the dynamic setting by computing a projection using the leading eigenvectors of the average covariance matrix, which we apply to the (globally centered) trajectories. This has a similar flavour to our Intensity Profile Projection algorithm, and since we reduce dimension using a common projection, it gives a temporally coherent view of the trajectories.
The second visualisation technique we apply is t-SNE [64], using the Flt-SNE implementation [65], which we used to obtain Figure 4. This visualisation method is not naturally extended to dynamic data, so we initialise the algorithm using the aforementioned dynamic extension PCA, which results in the visualisations at different times being approximately aligned.
D Computational complexity
In this section, we given a brief discussion of the computational complexity of Algorithm 1. Suppose we use a kernel with finite support of width  $2h$ , then the expected time complexity of the kernel intensity estimate for a single edge at a single point in time is  $O(h\lambda_{\max})$ . Evaluation of  $\widehat{\Lambda}(t)$  is  $O(n^2 h\lambda_{\max})$  and of the matrix (3) is  $O(Bn^2 h\lambda_{\max})$ . Consider the setting of Corollary 1, where  $\lambda_{ij}(t) \asymp \rho$  for all  $i,j,t$  and  $L \asymp \rho L_0$  where  $\mu, d,L_0 \asymp 1$  are fixed, our theory suggests choosing  $h \asymp (n\rho)^{-1/3}$ , and if we suppose  $n\rho \asymp \log^3 n$ , the sparsest regime our theory allows, then evaluation of (3) is  $O(Bn\log^3 n)$ , i.e. log-linear in  $n$ .
In practice the top singular vectors of (3) can be computed using the Augmented Implicitly Restarted Lanczos Bidiagonalization algorithm [47] implemented in the ir1ba package in R, or the ir1by in Python. The time complexity of this algorithm has not been studied theoretically although in practice it can be incredibly fast. For example, the package author performs a simulated experiment in which they compute the first 2 singular vectors of a sparse  $10\mathrm{M}\times 10\mathrm{M}$  matrix with 1M non-zero entries which takes approximately 6 seconds on a computer with two Intel Xeon CPU E5-2650
processors (16 physical CPU cores) running at 2 GHz equipped with 128 GB of ECC DDR3 RAM (see https://bwlewis.github.io/irlba/comparison.html).
We begin by writing
$$
\begin{array}{l} \operatorname *{arg  min}_{\mathbf{V}\in \mathcal{O}(n,d)}\widehat{R}^{2}(\mathbf{V}) = \operatorname *{arg  min}_{\mathbf{V}\in \mathcal{O}(n,d)}\int_{0}^{T}\widehat{r}_{i}^{2}(t;\mathbf{V})  \mathrm{d}t \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \min } \int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \left\| \mathbf {V V} ^ {\top} \widehat {\Lambda} _ {i} (t) - \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2} \mathrm {d} t \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \min } \int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \left\| (\mathbf {I} - \mathbf {V V} ^ {\top}) \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2} \mathrm {d} t, \\ \end{array}
$$
and since  $(\mathbf{I} - \mathbf{V}\mathbf{V}^{\top})$  is the projection onto the orthogonal complement of the columns space of  $\mathbf{V}$ , we have that
$$
\left\| \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2} = \left\| \mathbf {V V} ^ {\top} \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2} + \left\| (\mathbf {I} - \mathbf {V V} ^ {\top}) \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2}.
$$
Therefore, minimising  $\widehat{R}^2 (\mathbf{V})$  is equivalent to maximising
$$
\int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \left\| \mathbf {V} \mathbf {V} ^ {\top} \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2} d t = \int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \left\| \mathbf {V} ^ {\top} \widehat {\Lambda} _ {i} (t) \right\| _ {2} ^ {2}
$$
where the equality holds due to the invariance of the Euclidean norm under orthogonal transformations.