# Exploring the Performance of Continuous-Time Dynamic Link Prediction Algorithms
## 3 Understanding the Effect of Splitting a Dynamic Graph Based on Time
### The Surprise Index  
The Birth-Death diagrams suggest that the proportion of overlap and inductive nodes and edges is highly dependent on the cutoff time \(t_{\text{split}}\). We formally assess this proportion through the _surprise index_, i.e., the proportion of inductive nodes/edges in the test set. **Definition 6** (Surprise Index). : _The node/edge **surprise index** is defined as the proportion of nodes and edges \(x\) in the test set (i.e., their Death time \(d_{\mathcal{H}}^{x}\geq t_{\text{split}}\)) that only appear in the test set (i.e., their Birth time \(b_{\mathcal{H}}^{x}>t_{\text{split}}\)). In other words, it is the ratio between the number of inductive (i.e., having a Birth time after \(t_{\text{split}}\)) and the number of inductive or overlap (i.e., having a Death time after \(t_{\text{split}}\)) nodes/edges. Mathematically, considering \(x\) to be either the nodes or edges,_  
\[surprise^{x}=\frac{\big{|}\{x|b_{\mathcal{H}}^{x}\geq t_{\text{split}}\} \big{|}}{\big{|}\{x|d_{\mathcal{H}}^{x}\geq t_{\text{split}}\}\big{|}}.\]  
_Indeed, by definition, the inductive nodes/edges are those whose Birth time is after \(t_{\text{split}}\), while objects which are either inductive or overlap are those whose Death time is after \(t_{\text{split}}\)._  
Typical ML pipelines will learn from the training data by hinting on recurring patterns (concepts present in the data) and learn to recognize these patterns. The test data may reproduce these patterns to some extent, along with some other signals previously unobserved by the model, which will be reflected in the fact that the predictions will not be imperfect. In the case of the DLP, we dispose of concrete ways of measuring the quantity of information in the test set that will be new for the model. The surprise index is a natural way of measuring that. In Figure 3, we present the node surprise index against the edge surprise index for various datasets, with train-test splitting ratios ranging from 0.1 to 0.5. By examining these indices, we can observe some interesting properties. **The Surprise Index is Not Necessarily Monotonous with The Size of The Test Set. ** It may be intuitive to assume that the surprise index increases monotonically with a larger proportion of events included in the test set (i.e., earlier cutoff times \(t_{\text{split}}\)). However, this is not always the case. For instance, in the CanParl dataset, increasing the test ratio from 0.3 to 0.4 actually decreases the edge surprise index, while substantially increasing the node surprise index. A similar non-monotonicity can be observed for the Enron dataset, where increasing the test ratio from 0.3 to 0.4 similarly decreases the edge surprise index. This paradox arises from the fact that the surprise index is a non-decreasing function of the ratio between inductive events (those that occur only in the test set) and overlapping events (those that occur in both the training and test sets). Thus, if adding more events to the test set increases the number of overlapping events faster than it increases the number of inductive events, the surprise index will actually decrease. This observation highlights the _importance of carefully selecting cutoff times_ to ensure that the evaluation setting accurately reflects the difficulty and type of task at hand. **The Edge Surprise Index is Typically Higher than the Node Surprise Index. ** All datasets evaluated here exhibit curves in the lower right of Figure 2, indicating that the edge surprise index is generally higher than the node surprise index. While this may seem obvious, we stress here that it does not have to be the case. Indeed, what is clear is that if a given node was never observed during training but starts to interact during testing, then it means that the corresponding _edge_ it forms during testing was necessarily never seen during training. As a consequence, there are always at least as many inductive edges as there are inductive nodes.