SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiyao Huang†, Yingyue Zhang†, Zhihong Zhang†*, Edwin R. Hancock‡ † School of Informatics, Xiamen University, China
$^{\ddagger}$ Department of Computer Science, University of York, York YO10 5GH, UK. {30920211154155, 30920221154261}@stu.xmu.edu.cn zhihong@xmu.edu.cn, edwin.hancock@york.ac.uk
Temporal networks are widely used as abstract graph representations for real-world dynamic systems. Indeed, recognizing the network evolution states is crucial in understanding and analyzing temporal networks. For instance, social networks will generate the clustering and formation of tightly-knit groups or communities over time, relying on the triadic closure theory. However, the existing methods often struggle to account for the time-varying nature of these network structures, hindering their performance when applied to networks with complex evolution states. To mitigate this problem, we propose a novel framework called ESSEN, an Evolution StateS awarE Network, to measure temporal network evolution using von Neumann entropy and thermodynamic temperature. The developed framework utilizes a von Neumann entropy aware attention mechanism and network evolution state contrastive learning in the graph encoding. In addition, it employs a unique decoder the so-called Mixture of Thermodynamic Experts (MoTE) for decoding. ESSEN extracts local and global network evolution information using thermodynamic features and adaptively recognizes the network evolution states. Moreover, the proposed method is evaluated on link prediction tasks under both transductive and inductive settings, with the corresponding results demonstrating its effectiveness compared to various state-of-the-art baselines<sup>1</sup>.
Recently, graph representation learning has demonstrated excellent performance for various types of static graphs [7; 14; 28; 18]. Indeed, the success of static graph representation learning has led to a growing interest in continuous-time dynamic graph representation learning. Temporal network representation learning has emerged as an active research area focusing on learning low-dimensional representations that capture topological and temporal properties. However, learning effective representations is still a difficult task in many temporal networks, which are naturally generated in real-world systems such as social networks[12] and citation networks. The evolving nature of these networks poses a significant challenge for network analysis and modeling as the relationships between nodes and their properties evolve. The existing methods often struggle to account for the time-varying nature of these network structures, hindering their performance when applied to networks with complex time-evolving states. However, capturing the evolution states of temporal networks suffers from the following challenges: (1) Temporal networks have different types of evolution states, such as periodic, linear, or non-linear changes in their structure over time. Moreover, the evolving patterns can change at different stages in the evolution of the network. As
$^{1}$ Code is available at https://github.com/QiyaoHuang/ESSEN
(a) MathOverflow: 10th Day (b) MathOverflow: 20th Day (c) MathOverflow: 30th Day
(d) BitcoinOTC: 10th Day
(e) BitcoinOTC: 20th Day
(f) BitcoinOTC: 30th Day
Figure 1: Network snapshots of the MathOverflow website and BitcoinOTC trading platform on the 10th, 20th, and 30th day. The black nodes represent the users who have connections, and the grey nodes represent the users with no edges prior to the snapshot time.
illustrated in Fig. 1, the rate of evolution varies at different times in both datasets. The MathOverflow network evolves rapidly and has obvious central nodes, which means meaningful topics will receive long-term attention. By contrast, in the BitcoinOTC network, early active users may quickly become dormant. This phenomenon requires algorithms that can perform extensive and effective recognition of diverse evolution states. (2) As time passes, temporal networks tend to accumulate more nodes and edges, resulting in an increasing number of possible connections and a rapidly growing neighborhood size for each node. This growth in the neighborhood size can lead to significant computational challenges when analyzing and modeling evolving patterns. Besides, many connections can quickly make structure recognition computationally intractable in large and complex networks, especially those recent methods based on the anonymous walk [24; 6; 20]. The time complexity is tightly related to the length and number of paths, so it is challenging to balance time consumption and algorithm effectiveness.
To overcome these shortcomings, in this paper, we use the von Neumann network entropy to improve evolution state estimation. Network entropy is a macroscopic representation of network structures widely used to characterize the salient features of static and dynamic network systems in biology, physics, and social sciences. In particular, von Neumann entropy has been successfully used to describe the structural properties of random, small-world, and scale-free networks [1; 2], and thus plays a crucial role in understanding the structural and topological complexity of network systems. Moreover, the ability of von Neumann entropy to capture structural information content aligns well with the time-evolving nature of temporal networks. This potentially allows us to adopt an information-theoretic perspective to enhance our understanding of the laws of the network's evolution and their capacity to both transmit and store information over time.
Unfortunately, computing the required network entropies for temporal networks can be computationally burdensome due to the required spectral decomposition. Hence, we approximate the von Neumann entropy to render it tractable in the context of temporal networks. Compared with alternative graph entropies, by approximating von Neumann entropy with low time complexity we can better adapt to the constantly evolving nature of complex temporal networks. Moreover, we can compute an approximate thermodynamic temperature for the temporal network, which provides an important way to monitor changes in network structure with time. Measuring the thermodynamic temperature and the von Neumann entropy provides a better understanding of the state of network evolution with time.
On this basis, we propose an Evolution StateS awarE Network (ESSEN). As shown in Fig. 2, ESSEN encodes node embeddings with state evolution information by utilizing two proposed techniques:
a) a von Neumann entropy aware attention mechanism and b) virtual evolution node representation learning. In addition, ESSEN employs a unique mixture of thermodynamic experts (MoTE) for the purposes of decoding. Specifically, we project the global von Neumann network entropy into each edge. The proposed von Neumann entropy aware attention mechanism aggregates over neighborhoods in both a virtual evolution graph and the original graph based on the von Neumann projected edge entropy. In the virtual evolution graph, future edges are hypothesized based on a test pertaining to future instants in time. The MoTE decoder evaluates the evolution state based on both the thermodynamic temperature and von Neumann entropy over both graph representations, providing a combined result from multiple experts. The decoder adaptively recognizes the network under various evolution states. Our framework is evaluated on transductive and inductive link prediction tasks. The experimental results demonstrate our method's effectiveness compared to various state-of-the-art baselines. The overall contributions of our work are summarized as follows:
- To our best knowledge, we are the first to utilize the von Neumann entropy in temporal network representation learning. We provide a method to expand the approximate von Neumann Entropy and approximate thermodynamic temperature to temporal networks.
- We propose a novel framework, namely ESSEN. The model introduces a new perspective to encode evolution-aware node representations using the von Neumann entropy aware attention mechanism and virtual evolution node representation learning. Furthermore, the model uses a novel decoder MoTE that adaptively recognizes temporal network evolution states.
- We evaluate our framework on link prediction tasks with transductive and inductive settings. The results show the effectiveness of our proposed method compared to various state-of-the-art baselines.
Temporal Network. Formally, the temporal network can be denoted as  $G = (V, E, T)$ , where  $V$  represents the set of nodes,  $E \subseteq V \times V$  represents the set of links, and  $T$  represents the set of timestamps. Each link  $(u, v, t)$  signifies a connection between node  $u$  and node  $v$  at time  $t$ . The temporal network evolves over time, with links appearing at different timestamps. The temporal networks can also include attributes associated with nodes or links, providing further information about the entities or their interactions at specific timestamps.
Dynamic Link Prediction. In a temporal network  $G = (V, E, T)$ , the dynamic link prediction task aims to predict the presence or absence of a link at a future timestamp based on the observed network evolution history. Given a time window  $T_w \subseteq T$ , which contains the observed link data, denoted as  $E_{T_w} \subseteq E$ , the goal is to learn a function  $f: (V, E_{T_w}, T_w) \to \{0, 1\}$  that assigns a probability score to the existence of a link  $(u, v)$  at the future timestamp  $t$ . Mathematically, the function  $f$  can be defined as:
$$
f (u, v, t) = P (u, v \mid t, E _ {T _ {w}}), \tag {1}
$$
where  $P((u,v)|t, E_{T_w})$  represents the probability of the link  $(u,v)$  being present at the future timestamp  $t$  given the observed network and the link data  $E_{T_w}$  within the time window  $T_w$ .
Evolution State. Evolution state denotes the specific arrangements of nodes and edges at specific instants of time. These states can be characterized by a network topology, reflecting the evolving nature of the network over time. For example, social networks at specific evolution states will generate the clustering and formation of tightly-knit groups or communities over time via the triadic closure theory [29]. The theory is formally defined as  $\exists u,v,w,w^{\prime}\in V:(u,v),(v,w)\in E_{T_w},(u,w),(u,w'),(v,w')\notin E_{T_w}\mapsto P(u,w|E_{T_w}) > P(u,w'|E_{T_w})$ . Therefore, analyzing evolution states helps in understanding the temporal behavior of the network, identifying recurring patterns, predicting future states, and studying the impact of temporal dynamics on network properties and phenomena. In this paper, we aim to capture evolution states using thermodynamic entropy.
Von Neumann Entropy. Von Neumann Entropy is the quantum counterpart of the Shannon entropy. For a quantum system with a density matrix  $\rho$  the von Neumann entropy is  $S_{VN} = -\mathrm{Tr}(\rho \log \rho)$ . The density matrix  $\rho$  describes a system whose state is a mixture of pure quantum states  $|\psi_i\rangle$ , each with probability  $p_i$ , and is defined as  $\rho = \sum_{i=1}^{T} p_i |\psi_i\rangle \langle \psi_i|$ , where  $T$  is the number of pure states. When defined in this way, the density matrix is Hermitian, i.e.,  $\rho = \rho^\dagger$  and  $\rho \geq 0$ ,  $\mathrm{Tr}[\rho] = 1$ , where
Figure 2: The framework of ESSEN. It comprises two modules: (1) Evolution States Aware Graph Encoder. (2) Mixture of Thermodynamic Experts Decoder.
represents conjugate transpose. It plays an important role in the quantum observation process, which can be used to calculate the expectation value of measurable quantities.
Severini et al. [17] show that a density matrix for a graph or network can be obtained by scaling the combinatorial Laplacian matrix  $\tilde{L} = D - A$  (where  $A$  is the adjacency matrix and  $D$  is the diagonal degree matrix) by the reciprocal of the number of nodes in the graph  $\mathcal{V}$ , i.e.  $\rho = \frac{\tilde{L}}{|\mathcal{V}|}$ . The interpretation of the scaled normalized Laplacian as a density operator opens up the possibility of characterizing a graph using the von Neumann entropy. With the definition of the density matrix adopted, the von Neumann entropy[17] can be computed from the normalized Laplacian spectrum as follows:
$$
S _ {V N} (G) = - \operatorname {T r} (\rho \log \rho) = - \sum_ {i = 1} ^ {| \mathcal {V} |} \frac {\lambda_ {i}}{| \mathcal {V} |} \log \frac {\lambda_ {i}}{| \mathcal {V} |}, \tag {2}
$$
where  $\lambda_1, \ldots, \lambda_{|\mathcal{V}|}$  are the eigenvalues combinatorial Laplacian matrix. This form of von Neumann entropy has been shown to be effective for static network characterization.
3.1 Evolution State Estimation
Von Neumann Entropy Computation in Temporal Network. The von Neumann entropy can be computed in the static graph by Eq. 2. However, the application to temporal networks has two challenges: (1) The dynamic nature of the temporal network. Unlike static networks, where the structure remains constant, temporal networks capture time-evolving relationships and interactions. This dynamic nature introduces challenges in analyzing and modeling the network behavior, as the network topology and connectivity patterns may vary at different times. (2) The computationally expensive time complexity of obtaining the Laplacian eigenvalues. In Eq. 2, computing the eigenvalues of the Laplacian matrix is a computationally intensive task, and the time complexity is cubic in the number of nodes. In temporal networks, where the network structure changes over time, repeatedly calculating the Laplacian eigenvalues can become prohibitively expensive.
To address these challenges, we must simplify the network and efficiently approximate thermodynamic quantities. First, we select a specific time interval for the temporal network and aggregate edge weights or frequencies over this interval. The number of occurrences within the chosen time frame
determines the strength of an edge. Following this process, the temporal network can be projected to the time-independent 2-D plane, which provides a simplified representation of the underlying network structure at a specific time. Moreover, following [27], we use the approximate expression for the von Neumann entropy and reduce the computation to be quadratic in the number of nodes. The approximate von Neumann entropy is
$$
S _ {V N} \left(G _ {t}\right) = 1 - \frac {1}{| V |} - \frac {1}{| V | ^ {2}} \sum_ {(u, v) \in E} \frac {1}{d _ {u} d _ {v}}, \tag {3}
$$
where  $V$  is the node set of the temporal network, and  $d_u, d_v$  are the degree of node  $u$  and  $v$  at time  $t$ . This approximation allows the von Neumann entropy to be computed without explicitly solving the eigensystem for the normalized Laplacian. Thus, the von Neumann entropy can be computed in quadratic time using the node degrees for pairs of nodes connected by edges. We introduce the details of the approximate method in Appendix A.
Thermodynamic Temperature. In thermodynamics, the thermodynamic state of a system can be fully described by an appropriate set of thermodynamic variables. We treat the von Neumann entropy as a thermodynamic network entropy and use the thermodynamic temperature to  $\mathcal{T}$  measure fluctuations in network structure with time. Specifically, suppose that the graphs  $G_{1}$  and  $G_{2}$  represent the structure of a time-varying system at two consecutive epochs  $t_1$  and  $t_2$  respectively. For a thermodynamic system with a fixed number of particles, the change in energy is  $de = \mathcal{T}dS - P_r dV_o$ , where  $P_r$  and  $V_o$  denote the pressure and volume of the system respectively. It is important to stress that this equation holds and is valid for both reversible and irreversible processes for a closed system, since  $e, \mathcal{T}, S, P_r$  and  $V_o$  are all state functions and are independent of thermodynamic path. We assume the network evolves from time  $t_1$  to time  $t_2$  under constant-volume (isochoric process) as the time interval becomes small, i.e.  $\Delta t \to 0$ . As a result, for the path from  $G_{1}$  to  $G_{2}$  we have  $dV_o = 0$  and  $de = \mathcal{T}dS$ . The reciprocal of the temperature  $\mathcal{T}$  is the rate of change of entropy with average energy, i.e.,
$$
\mathcal {T} \left(G _ {1}, G _ {2}\right) = \frac {d e}{d S} = \frac {e \left(G _ {1}\right) - e \left(G _ {2}\right)}{S \left(G _ {1}\right) - S \left(G _ {2}\right)}, \tag {4}
$$
where  $S(G_{1})$  is the graph entropy for the graph  $G_{1}$  and  $e(G_{1})$  is the graph average energy. Moreover, the thermodynamic temperature can also be approximated when we represent the temporal network as a directed graph. The approximate computation method uses a low-order Taylor series that can be computed using the traces of powers of the normalized Laplacian matrix, avoiding explicit computation of the normalized Laplacian spectrum [26]. In summary, the temperature associated with the evolutionary transition between the two networks can be approximated as
$$
\mathcal {T} \left(G _ {1}, G _ {2}\right) = - \frac {2}{k} + \frac {2}{3 k} \cdot \frac {\mathcal {K} \left(G _ {1}\right) - \mathcal {K} \left(G _ {2}\right)}{\mathcal {J} \left(G _ {1}\right) - \mathcal {J} \left(G _ {2}\right)}, \tag {5}
$$
$$
\mathcal {J} (G) = \sum_ {u, v \in V} \frac {A _ {u v}}{d _ {u} d _ {v}}, \tag {6}
$$
$$
\mathcal {K} (G) = \sum_ {u, v, w \in V} \frac {A _ {u v} A _ {v w} A _ {w u}}{d _ {u} d _ {v} d _ {w}}, \tag {7}
$$
where  $k$  is the Boltzmann constant, and  $A$  is the adjacency matrix of the network.  $\mathcal{J}(G)$  and  $\mathcal{K}(G)$  are network structure statistics that can be interpreted as the probabilities of a random walker traversing specific edges or cycles in the graph. The computation of temperature is quadratic in the number of nodes.
In summary, by simplifying the network representation and using efficient approximations, the von Neumann entropy and the thermodynamic temperature can be computed effectively in temporal networks. These measures provide insights into the evolving nature of the network and enable the estimation of its evolution state.
3.2 Evolution States Aware Graph Encoder
Von Neumann Entropy Aware Attention Mechanism. In graph encoding, we utilize the von Neumann entropy to explore a more diverse and balanced distribution of attention weights with the
attention mechanism in the input neighborhood. This strategy helps us to learn network evolution states adaptively. According to Eq. 3, the global network entropy is a sum of contributions from individual edges. The von Neumann entropy of the edge connecting nodes  $u$  and  $v$  is
$$
S _ {V N} ^ {u v} \left(G _ {t}\right) = \frac {1}{| E |} - \frac {1}{| V | | E |} - \frac {1}{| E | | V | ^ {2}} \frac {1}{d _ {u} d _ {v}}, \tag {8}
$$
To encode entropy features into attention layers, we use the von Neumann edge entropy as a bias term in an attention module[22]. Moreover, we use a time position encoding module[25] to supplement the continuous time information for edges using the simplified von Neumann entropy. For a target node  $u$  at time  $t$ , the attention weight  $\alpha_v^{(l)}$  for the neighboring node  $v$  in the  $l^{th}$  layer is
$$
\alpha_ {v} ^ {(l)} = \frac {Q _ {u} ^ {(l)} \left(K _ {v} ^ {(l)}\right) ^ {T}}{\sqrt {d _ {n}}} + S _ {V N} ^ {u v} (G _ {t}), \tag {9}
$$
$$
Q _ {u} ^ {(l)} = \left(h _ {u} ^ {(l - 1)} \| e _ {0} \| \phi (0)\right) W _ {Q}, \tag {10}
$$
$$
K _ {v} ^ {(l)} = M _ {v, t} ^ {(l)} W _ {K}, \tag {11}
$$
$$
M _ {v, t} ^ {(l)} = \left(h _ {v} ^ {(l - 1)} \| e _ {u v, t} \| \phi \left(t _ {q} - t\right)\right) W _ {M}, \tag {12}
$$
where  $\lVert \rVert$  is the concatenation operation.  $W_{K} \in \mathbb{R}^{(d_{n} + d_{t} + d_{e}) \times d_{n}}$  and  $W_{Q} \in \mathbb{R}^{(d_{n} + d_{t} + d_{e}) \times d_{n}}$  are the projection matrices to obtain the query matrices and key matrices, where  $d_{n}, d_{t}$  and  $d_{e}$  are the dimensions of the node representation, the time code, and the entropy bias, respectively.  $e_0$  is an all-zero vector to keep the same dimension as  $K$  and  $V$ , and  $\phi (*)$  is the generic time position encoding module from [25], which encodes the difference between the edge timestamp and the query timestamp.  $M_{v,t}^{(l)}$  is the message representation at time  $t$  from node  $v$  to  $u$ , where  $h_v^{(l-1)}$  is node  $v$ 's hidden representation on the  $(l-1)^{th}$  layer,  $e_{uv,t} \in \mathbb{R}^{d_e}$  is the edge feature, and  $t_q$  is the query time. Next, the model combines values with the attention weight aware of generating hidden representation  $z_u^{(l)}(t)$  for node  $u$ . Finally, an MLP is used to combine the node representation of the previous layer with the neighborhood information:
$$
h _ {u} ^ {(l)} = M L P \left(h _ {u} ^ {(l - 1)} \| z _ {u} ^ {(l)} (t)\right), \tag {13}
$$
$$
z _ {u} ^ {(l)} (t) = \sum_ {v \in \mathcal {N} _ {u}} \operatorname {s o f t m a x} _ {v} \left(\alpha_ {v} (t)\right) V _ {v} (t), \tag {14}
$$
$$
V _ {v} ^ {(l)} = M _ {u v, t} ^ {(l)} W _ {V}, \tag {15}
$$
where  $V_{v}^{(l)}$  is the value vector of neighbor node  $v$ , and  $\mathcal{N}_u$  is a neighbor node set that connects with node  $u$  before time  $t$ .
Virtual Evolution Node Representation Learning. Temporal networks follow time-dependent evolution laws. The emergence of nodes and edges at different times can often be predicted, i.e., future network states can be predicted from their past states and together with the evolutionary laws. Virtual evolution node representation learning utilizes historical evolution information together with the future potential evolution path to generate virtual node representations. Specifically, the dynamic link prediction task aims to predict the probability of the link between two nodes appearing at a future instant in time. We assume the link has been generated at the query instant and further construct a virtual evolution graph belonging to the query node pair based on this assumption. For example, given a node pair  $(u,v)$  and the query time  $t$ , there is a virtual edge at time  $t$  that connects the nodes  $u$  and  $v$  in the virtual evolution graph  $G_{uv}^{\prime}$ . The approach makes the two node neighborhoods interconnected. We denote as  $h_u^{\prime}$  and  $h_v^{\prime}$  the virtual future node embeddings of nodes  $u$  and  $v$ . Moreover, the thermodynamic temperature of the virtual evolution path can be computed because the instantaneous virtual evolution process is also an isochoric process we introduced in Section 3.1. Based on this, the decoding process can estimate the evolution states more comprehensively.
3.3 Mixture of Thermodynamic Experts Decoder
Evolution State Feature Extractor. We use the von Neuman entropy of the original graph, the von Neuman entropy of the virtual evolution graph, and the thermodynamic temperature between the two networks to construct a unique vector for evolution state estimation. The vector represents the network evolution states in a 3-D thermodynamic space, which is spanned by the von Neumann entropy at the two instants in time together with the thermodynamic temperature. The approximate von Neumann graph entropy for the original and virtual evolution graphs are computed by Eq. 3, and the approximate thermodynamic temperature is computed by Eq. 5. Furthermore, to control evolution-aware time intervals, which is important for large temporal networks, we compute these approximate thermodynamic quantities in the node neighborhood and set the sampled neighborhood size  $N$ .
Expert Decoding Process. The mixture of thermodynamic experts decoder dynamically selects the appropriate thermodynamic expert model based on the input expert assessment feature vectors. For each expert, we use a two-layer MLP model to represent. Then the MoTE decoder combines the output embedding of each expert model using respective expert weights to produce the final target score as follows:
$$
\operatorname {s c o r e} (u, v, t) = \sum_ {i = 1} ^ {Y} \sigma \left(W _ {i} \left(h _ {u}, h _ {v}, h _ {v} ^ {\prime} - h _ {v}, h _ {u} ^ {\prime} - h _ {u}\right)\right) \pi_ {i}, \tag {16}
$$