SlowGuess/NeurIPS_2023_OCR huggingface dataset:
EntEncoder. We devise the EntEncoder (illustrated by the blue block) to extract entity information. As entities often contain more local and periodic information, we learn their representations from two aspects: the frequency domain (upper line) and the time domain (lower line). In the EntEncoder's upper line, we first perform a Fast Fourier Transform (FFT) operation on the input data and then feed it into a linear layer to capture the frequency information. Subsequently, we apply the inverse Fast Fourier Transform (iFFT) to return it to the time domain. In the lower line of the EntEncoder, we utilize the self-attention mechanism to capture the intricate relationships between different time steps, thereby focusing on crucial aspects of the time domain information. This is followed by a layer normalization. By combining the frequency and time domain information, and feeding it into a linear layer, we can obtain the entity features  $H_{i}$ .
C More Details of HL Deconfounder
The HL Deconfounder is developed (see Figure 4b) to learn a surrogate that mimics the node information filtered by the dynamic causal score. To achieve an accurate representation learning for this surrogate, we use an edge-level convolution to model the ripple effect of the causal relationships, which is mathematically based on the boundary operator [24] and Hodge-Laplacian operator [16]. We provide details on these two techniques in the following part.
Boundary operator. Graphs consist of nodes and edges, which can be mathematically represented as 0-dimensional and 1-dimensional simplices, respectively [7]. The topology of a graph can be described by a boundary operator  $\partial_k$  [24], where  $\partial_1$  encodes the connections between two 0-dimensional simplices (nodes, denoted as  $\sigma_0$ ) to form a 1-dimensional simplex (edge, denoted as  $\sigma_1$ ). The second order boundary operator  $\partial_2$  represents the connections between 1-dimensional simplices (edges) to form 2-dimensional simplices (triangles, denoted as  $\sigma_2$ ). The boundary operator  $\partial_k$  can be formulated in the matrix form as follows:
$$
[ \partial_ {k} ] _ {i j} = \left\{ \begin{array}{l l} 1 & \text {i f} \sigma_ {k - 1} ^ {i} \text {i s p o s i t i v e l y o r i e n t e d w . r . t .} \sigma_ {k} ^ {j}, \\ - 1 & \text {i f} \sigma_ {k - 1} ^ {i} \text {i s n e g a t i v e l y o r i e n t e d w . r . t .} \sigma_ {k} ^ {j}, \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
where  $\sigma_k^i,\sigma_k^j\in C_k;C_k = \{\sigma_k^1,\dots ,\sigma_k^q\}$  denotes the chain complex and  $q$  is the number of simplices. For better understanding, we illustrate an example of the first-order boundary operator  $\partial_1$  in Figure 9.
Figure 9: Example of the first boundary operator  $\partial_1$ .
(1, 2)	(2, 3)	(2, 4)	(2, 5)	(5, 6)
1	-1	0	0	0	0
2	1	-1	-1	-1	0
3	0	1	0	0	0
4	0	0	1	0	0
5	0	0	0	1	-1
6	0	0	0	0	1
Hodge-Laplacian operator. The Hodge-Laplacian (HL) operator[16] is proposed to achieve node-level and edge-level representation learning based on the features of neighboring nodes and edges. The boundary operators  $\partial_k$  play a key role in achieving this goal and can be incorporated into the  $k$ -th Hodge-Laplacian operator, defined as:
$$
\mathcal {L} _ {k} = \boldsymbol {\partial} _ {k + 1} \boldsymbol {\partial} _ {k + 1} ^ {\top} + \boldsymbol {\partial} _ {k} ^ {\top} \boldsymbol {\partial} _ {k}, \tag {9}
$$
where when  $k = 0$ ,  $\mathcal{L}_0 = \partial_1\partial_1^\top$  operates over nodes and is equivalent to the Graph Laplacian. When  $k = 1$ , the HL operator  $\mathcal{L}_1 = \partial_2\partial_2^\top + \partial_1^\top \partial_1$  operates over edges. In this study, we use the  $\mathcal{L}_1$  to achieve the edge-level convolution.
D Dataset and Experiment Settings
Datasets. Our experimental design involved selecting three real-world datasets from two distinct domains. The first dataset, PEMS08 [42], contains traffic flow data that was collected by sensors deployed on the road network. Traffic flow data is often considered to be a complex and challenging type of spatio-temporal data influenced by numerous factors, such as weather, time of day, and road conditions. The PEMS08 dataset is thus an ideal choice for our study as it allows us to examine the effectiveness of our proposed method in the context of a real-world traffic flow analysis scenario. The remaining two datasets, AIR-BJ and AIR-GZ [59] contain one-year  $\mathrm{PM}_{2.5}$  readings obtained from air quality monitoring stations located in Beijing and Guangzhou, respectively. The issue of air pollution is a major concern in many cities around the world, and the AIR-BJ and AIR-GZ datasets offered us the opportunity to investigate the application of our proposed method to spatiotemporal data associated with environmental monitoring. To provide a comprehensive overview of the characteristics of the datasets used in the experiments, we present their statistics in Table 3.
Table 3: Statistics of datasets.
Dataset	#Nodes	#Edges	Data Type	Time interval	Date Range	#Samples	Train:Val:Test
AIR-BJ	34	82	PM2.5	1 hour	Jan.1, 2019 - Dec. 31, 2019	8,760	4:1:1
AIR-GZ	41	77	PM2.5	1 hour	Jan.1, 2017 - Dec. 31, 2017	8,760	4:1:1
PEMS08	170	303	Traffic flow	5 minutes	Jul. 1, 2016 - Aug.31, 2016	17,856	8:1:1
Edge Features. Edge features are built for causality extraction. For each edge, we create three types of features: (1) Euclidean-based similarity  $W_{ij}$ , (2) Pearson correlations  $\rho_{ij}$ , and (3) Time-delayed Dynamic Time Warping (DTW)  $R_{ij}$ . The first two are static values, while the third one is a dynamic variable that changes over time.
- We compute the Euclidean distances between stations and construct a weighted adjacency matrix using thresholded Gaussian kernel [41] as follows:
$$
W _ {i j} = \left\{ \begin{array}{l l} \exp \left(- \frac {\operatorname {d i s t} (v _ {i} , v _ {j}) ^ {2}}{\sigma^ {2}}\right), & \text {i f} \operatorname {d i s t} \left(v _ {i}, v _ {j}\right) \leq \kappa \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
where  $W_{ij} \in [0,1]$  denotes the edge weight between node  $v_i$  and node  $v_j$ ;  $\mathrm{dist}(v_i, v_j)$  represents the distance between nodes  $v_i$  and  $v_j$ ;  $\sigma$  is used as a standard deviation of distances and  $\kappa$  is a threshold to exclude nodes that are considered to be very distant.
- The Pearson correlation coefficient measures the linear relationship between two signals generated by  $i$ -th node and  $j$ -th node, represented mathematically as:
$$
\rho_ {i j} = \frac {\operatorname {c o v} \left(X _ {i} , X _ {j}\right)}{\sigma_ {X _ {i}} \sigma_ {X _ {j}}},
$$
which ranges from -1 to 1, where 1 indicates a perfect positive linear relationship, 0 indicates no linear relationship, and -1 indicates a perfect negative linear relationship.  $\operatorname{cov}(\cdot, \cdot)$  represents the covariance between two random variables.
- Finally, we create a time-delay DTW  $R_{ij}(\tau)$  with delay window size  $\tau$  between the source node's signal  $X_{i}^{t}$  and the target node's  $\tau$  lag series  $X_{j}^{t}$  [32]:
$$
R _ {i j} ^ {\alpha} (\tau) = D T W (X _ {i} ^ {t}, X _ {j} ^ {t - \alpha \tau}), R _ {i j} (\tau) = [ R _ {i j} ^ {1} (\tau), R _ {i j} ^ {2} (\tau), \dots , R _ {i j} ^ {N} (\tau) ]
$$
where  $N = T / \tau$ ,  $T$  refers the length of the input series and  $\alpha \in \{1, 2, \dots, N\}$ . In this work, we set  $\tau = 6$  and  $T = 24$ , resulting in 4 dimensions of  $R_{ij}$  for each edge.
Computation of HL Operator. The computation of the first-order HL operator involves the calculation of  $\partial_{2}$ , which characterizes the interaction of edges and triangles. However, for the purpose of STG forecasting in this work, we simplify the computation by disregarding triangles and thus use the down HL operator (i.e., setting  $\partial_{2}$  to zero) [39]. While in other applications where triangles or higher-order elements have explicit meanings, such as the Benzene Rings of Protein Structures in Molecular Structure Modeling,  $\partial_{2}$  should be carefully defined.
Evaluation Metrics. We leverage Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) to evaluate the performance of models. Let  $Y_{i}$  be the label of each entry of  $Y$ , and  $\hat{Y}_i$  denote the corresponding prediction result, two metrics are calculated as follows:
$$
\operatorname {M A E} (Y, \hat {Y}) = \frac {1}{| Y |} \sum_ {i = 1} ^ {| Y |} \left| Y _ {i} - \hat {Y} _ {i} \right|, \tag {10}
$$
$$
\operatorname {R M S E} (Y, \hat {Y}) = \sqrt {\frac {1}{| Y |} \sum_ {i = 1} ^ {| Y |} \left(Y _ {i} - \hat {Y} _ {i}\right) ^ {2}}, \tag {11}
$$
Final Settings of CaST. We introduce the best hyperparameter configurations for each dataset as follows.
- For the PEMS08 dataset, the number of hidden units is set at 64, the codebook size  $K$  is 20, and the number of layers in all convolutional blocks is 2. The position embedding size is set to 5, and the balancing coefficient for the mutual information loss,  $\alpha$ , is set to 0.5.
- For the AIR-BJ dataset, we configure the model with the following settings: the number of hidden units is set to 64, the codebook size  $K$  is 10, and the number of layers for all convolutional blocks is 3. The position embedding size is set to 10, and the balancing coefficient for the mutual information loss,  $\alpha$ , is set to 1.
- For the AIR-GZ dataset, we use these configurations: the number of hidden units is set at 32, the codebook size  $K$  equals 3, and the number of layers in all convolutional blocks is 3. The position embedding size is set to 5, and the balancing coefficient for the mutual information loss,  $\alpha$ , is set to 0.5.
E Details of Baselines
Description & Settings. We opted to include a selection of widely-used traditional methods and popular cutting-edge methods for comparative evaluation. We describe these baselines and outline the setting used in our experiments as follows. The number of parameters of each model is illustrated in Table 4.
- HA [63] forecasts future time series values by calculating the average of historical readings for the corresponding time periods.
- VAR [45] is a popular time series forecasting method that models the interdependence between multiple time series variables.
- DCRNN [26] is a type of neural network that uses diffusion convolution and sequence-to-sequence architecture to learn spatial dependencies and temporal relations. We utilize the source code provided on GitHub<sup>2</sup>. For each of the three datasets, we configure the model with 2 DCRNN layers and 64 hidden units. The receptive field in diffusion convolution is set to 3.
- STGCN [61] is a spatial-temporal graph convolution network that combines spectral graph convolution with 1D convolution to capture correlations between space and time. We employ the implementation shared by the authors on GitHub<sup>3</sup>. For each of the three datasets, we configure the channels of the three layers in the ST-Conv block to 64. We set the graph convolution kernel size  $K$  and temporal convolution kernel size  $K_{t}$  both to 3.
- ASTGCN [14] leverages an attention-based mechanism and a spatial-temporal convolution system to dynamically capture spatial-temporal correlations and model various temporal properties of
$^{2}$ https://github.com/liyaguang/DCRNN
<sup>3</sup>https://github.com/PKUAI26/STGCN-IJCAI-18
traffic flows, thereby providing superior traffic flow forecasting. We use the code released by the author<sup>4</sup> to implement this model. We set the number of blocks to 2 and the hidden size to 64.
- MTGNN [55] models the temporal dynamics of graph-structured data by aggregating information from spatially neighboring nodes and past time steps using a message-passing framework. We employ the official MTGNN implementation<sup>5</sup>. For all three datasets, we configure each MTGNN block with a graph convolution module and a temporal convolution module. The hidden dimension for each node in these modules is set to 40. Additionally, we set the convolution channel to 32 and the graph convolution depth to 2.
- AGCRN [1] uses Node Adaptive Parameter Learning and Data Adaptive Graph Generation modules to automatically infer inter-dependencies in traffic series and capture node-specific patterns. We use the code offered in  $\mathbf{G}\mathbf{\mathrm{h}}\mathbf{u}\mathbf{b}^{6}$  to run the experiments of this model. We set the order of Chebyshev polynomials to 2 and the hidden size to 32.
- GMSDR [30] improves upon RNNs by incorporating the hidden states of multiple historical time steps as input at each time unit. We use the available GMSDR code<sup>7</sup> for our experiments. Across all three datasets, we configure the model with 2 RNN layers and 64 hidden units.
- STGNCDE [5] is a spatio-temporal graph neural controlled differential equation model that uses two neural control differential equations to process spatial and sequential data. We employ the provided STGNCDE code<sup>8</sup> for our experiments. Across all three datasets, we configure the model with a node embedding size of 10 and a hidden vector dimensionality of 32.
Table 4: The number of parameters of models. The magnitude is Kilo.
Model	AIR-BJ	AIR-GZ	PEMS08
DCRNN	94	94	94
STGCN	163	163	163
ASTGNC	90	92	210
MTGNN	155	156	166
AGCRN	189	189	189
GMSDR	229	246	553
STGNCDE	372	372	372
CaST (ours)	350	212	391
F More Experiments Results
Experiments Results for Various Steps. We performed five runs for each model, and Table 5 presents the mean and standard deviation of the results for different future time steps on the PEMS08 dataset. We observe that CaST outperforms the majority of compared models.
Effects of Position Embedding. As delineated in Section 4, we incorporate a position embedding  $P$  alongside the edge signal to learn global location information. We execute our proposed model five times, both with and without  $P$ , for 4,000 iterations and present the loss curves for the training and evaluation sets in Figure 10. Losses are documented at 100-iteration intervals. Solid lines signify the mean loss across five runs, while the filled area indicates the standard deviation. For enhanced visualization, we employ a moving average method with a window size of 5 to smoothen the curves. Our findings suggest that models lacking position embedding may be susceptible to overfitting, as evidenced by the decreasing losses on the training set beyond 3,000 iterations, while losses on the evaluation set increase.
Effects of Loss Balance. We investigated the influence of different weightings for terms  $\alpha$  and  $\beta$  in our loss function on prediction performance, shown in Figure 11. The color intensity represents the value of the metrics, with lighter colors indicating better performances in MAE or RMSE. From the results, the combination of  $\alpha = 0.5$  and  $\beta = 1.5$  yielded the lowest MAE and RMSE, indicating this
$^{4}$ https://github.com/wanhuaiyu/ASTGCN
$^{5}$ https://github.com/nnzhan/MTGNN
$^{6}$ https://github.com/LeiBAI/AGCRN
$^{7}$ https://github.com/dcliu99/MSDR
$^{8}$ https://github.com/jeongwhanchoi/STG-NCDE
Table 5: 5-run results on PEMS08 for different time steps prediction. The bold/underlined font means the best/the second best result.
Model	1 - 8 steps	9 - 16 steps	17 - 24 steps
MAE	RMSE	MAE	RMSE	MAE	RMSE
HA	58.60	81.60	58.85	82.02	59.04	82.27
VAR	21.07	32.05	37.10	53.85	52.94	74.58
DCRNN	16.48 ± 0.1	25.86 ± 0.17	22.18 ± 0.43	34.32 ± 0.59	27.63 ± 0.82	42.00 ± 1.07
STGCN	15.13 ± 0.05	23.47 ± 0.08	18.60 ± 0.09	28.55 ± 0.16	22.07 ± 0.10	33.45 ± 0.24
ASTGCN	16.27 ± 0.16	25.24 ± 0.22	20.38 ± 0.47	31.04 ± 0.60	24.43 ± 0.87	36.53 ± 0.95
MTGNN	14.80 ± 0.05	23.55 ± 0.06	18.12 ± 0.10	28.98 ± 0.12	21.47 ± 0.17	34.18 ± 0.18
AGCRN	15.19 ± 0.10	23.68 ± 0.14	17.24 ± 0.13	27.18 ± 0.14	18.74 ± 0.20	29.6 ± 0.19
GMSDR	18.04 ± 0.58	27.67 ± 0.80	18.50 ± 0.76	28.40 ± 1.13	18.49 ± 0.71	29.00 ± 1.13
STGNCDE	15.74 ± 0.30	24.29 ± 0.31	17.66 ± 0.30	27.46 ± 0.34	19.27 ± 0.34	30.15 ± 0.48
CaST (ours)	14.39 ± 0.04	22.83 ± 0.10	16.47 ± 0.08	26.80 ± 0.22	18.47 ± 0.20	30.28 ± 0.17
Figure 10: Training losses and evaluation losses on AIR-BJ. w: with. w/o: without.
balance might be optimal for minimizing the average error magnitude and reducing the impact of larger errors. However, it's noteworthy that the trends in performance improvement are not consistent, implying that the relationship between the terms in the loss function and the prediction performance could be complex and not straightforward.
Figure 11: Heatmap of MAE and RMSE values for varying loss function weights on AIR-BJ dataset.
External Factors Distribution. The distribution of temperature and pressure in Beijing during the training period (January to August 2019) and the testing period (November to December 2019) is shown in Figure 12. The plot reveals that the testing period has a lower temperature and higher pressure than the training period, which is consistent with the observation in the
case study of temporal environments visualization in Section 5.2 that the occurrence frequency of a specific temporal environment is higher in the testing set than in the training set. Note that these external factors are just used to do the interpretation analysis. When training the model, we did not use these external factors.
Figure 12: Distribution of external features.
G More Discussions
Generalization to the Unseen Environment. We utilize the environmental codebook as a latent representation to capture information for each temporal environment type, whose goal is to represent unique and informative environments in the real world. During training, each environment is assigned an embedding from the codebook, while during testing, a soft probability is assigned to indicate the likelihood of the example belonging to each environment. This approach enables the generation of embeddings for unseen environments. For instance, if we have embeddings for high temperature and low pressure (i.e.,  $e_1$ ) and low temperature and high pressure (i.e.,  $e_2$ ), and we encounter a moderate environment with middle temperature and pressure, we can use different coefficients to generate a representative embedding  $\lambda_1e_1 + \lambda_2e_2$  for this unseen environment.
Divergence-free Property of Datasets. In our ablation study on the effects of core components (see Figure 5), the impact of removing the edge filtering module significantly affects performance on the PEMS08 dataset, less so for AIR-BJ. This can be attributed to the unique properties of the down HL operator and the datasets. In the case of incompressible traffic flow, which is naturally divergence-free [40, 4], our HL deconfounder is able to capture significant causal information via the edge signals. This indeed might explain the significant drop in performance when edge signals are removed in the ablation study on the PEMS08 dataset. Conversely, for data types where the flow is not divergence-free, like the PM2.5 of the AIR-BJ dataset, the HL deconfounder may not capture as much information, leading to less significant performance drops when edge signals are removed.
Complexity Analysis. We omit hidden dimensionality in the following analysis for simplicity. The complexity of the spatial module is  $O(M^2)$  induced by the edge convolution operation (i.e., HL Deconfounder) [21], with  $M$  denoting the number of nodes; the complexity of the temporal module (i.e., Env Disentangler) is  $O(T^2)$ , where  $T$  represents the historical signal length. Generally, we have  $T \ll M$ , leading to  $O(M^2)$  additional overheads over the original Backbone Encoder.
Limitations and Future Directions. One limitation of our proposed framework is that it requires the stratification of temporal environments, which can potentially be infinite. This could pose a challenge in real-world applications where the environmental contexts may be continuously changing and difficult to categorize into a finite set of environments. In the future, we may explore the possibility of modeling continuous environments. Besides, while we have underscored the efficacy of the edge convolution CaST in discerning the causative relations between nodes, its computational speed lags relative to alternative methods, such as the adaptive adjacency matrix [56], due to the incorporation of edge graph construction, thereby increasing computational demands. However, given it can be done in the preprocessing stage in one shot, it won't produce extra computational costs in the training phase. Additionally, although we have demonstrated the effectiveness of CaST in various real-world applications, further investigation is required to evaluate its performance in more diverse and complex scenarios.
Comparison with Existing Works. Temporal  $OoD$ : CaseQ [57] and our proposed CaST both segment the temporal environment based on the back-door adjustment to achieve generalization to a new environment. While CaseQ uses shared inference units to associate different types of contexts, CaST stores seen environments in a codebook and approximates unseen environments by the combination of vectors from the codebook. Additionally, CaseQ is not suitable for STG forecasting problems as it is designed for future event prediction and does not consider the spatial context. Spatial Causal Strength: The primary challenge in the spatial aspect of this work is how to effectively model the ripple effect of dynamic spatial causal relationships, which is essential for improving node representations for accurate prediction. Although Graph attention [47] and the self-adaptive adjacency matrix proposed by [56] are two alternative methods for learning causal strength, they cannot account for the ripple effect of causation, which involves operating convolution on edges. Specifically, we build a higher-order graph over the edge based on the boundary operator and then use the Hodge-Laplacian operator [16] to filter the edge signal. In Section 5.2, we demonstrate the advantages and effectiveness of incorporating convolution on edge signals to model this effect of causal relationships, and we also provide a case study with visualizations. While there are alternative methods for edge convolution, e.g., the switch of nodes and edges [19, 21], these approaches may face a substantial computational burden, particularly when dealing with non-sparse graphs[16].
Social Impacts. By incorporating causal inference, our proposed CaST framework possesses the substantial potential to effect meaningful change across a variety of domains, particularly within the
realm of smart cities. The interpretability offered by CaST promotes a deeper comprehension of complex spatio-temporal systems and their underlying causal dynamics. This integration of causal analysis enables stakeholders to make enlightened decisions, ultimately improving individual and communal quality of life and fostering more resilient, sustainable urban environments. For example, in the context of environmental health, CaST can provide more accurate forecasting of critical air quality indices, like  $\mathrm{PM}_{2.5}$  concentrations. As air pollution continues to pose significant economic, environmental, and health challenges, the precision of such predictions becomes increasingly paramount. This enhanced accuracy facilitates more effective human health protection measures and informs policy-making, promoting better air quality management and mitigation strategies.