SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35:27387-27401, 2022.
Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= j6zUzrapY3L.
Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266-13279, 2021.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. Advances in Neural Information Processing Systems, 32, 2019.
Xitong Zhang, Yixuan He, Nathan Brugnone, Michael Perlmutter, and Matthew Hirn. Magnet: A neural network for directed graphs. Advances in neural information processing systems, 34: 27003-27015, 2021.
Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinform., 34(13):i457-i466, 2018.
Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Gunnemann. Language-agnostic representation learning of source code from structure and context. In International Conference on Learning Representations.
Theoretical Intuition of DAG Attention
Proof. Personalized PageRank is defined as  $\pi_G(i_x) = (1 - \alpha)A_{rw}\pi_G(i_x) + \alpha i_x$ . Through rearrangement, we obtain:  $(I_n - (1 - \alpha)A_{rw})\pi_G(i_x) = \alpha i_x$ . Now we prove that  $(I_n - (1 - \alpha)A_{rw})$  is an invertible matrix. The matrix is invertible iff the determinant  $\operatorname*{det}(I_n - (1 - \alpha)A_{rw}) \neq 0$ , which is the case iff  $\operatorname*{det}(A_{rw} - \frac{1}{1 - \alpha} I_n) \neq 0$ , i.e., iff  $\frac{1}{1 - \alpha}$  is not an eigenvalue of  $A_{rw}$ . This value is always larger than 1 since the teleport probability  $\alpha \in (0,1]$ . However, the largest eigenvalue of the row-stochastic matrix  $A_{rw}$  is 1, as can be proven using the Gershgorin circle theorem. Hence,  $\frac{1}{1 - \alpha}$  cannot be an eigenvalue and  $(I_n - (1 - \alpha)A_{rw})$  is an invertible matrix. So we obtain  $\pi_G(i_x) = \alpha (I_n - (1 - \alpha)A_{rw})^{-1}i_x$ . Because the spectral radius (largest eigenvalue) of matrix  $(1 - \alpha)A_{rw}$  is less than 1, from the Neumann series theorem, we obtain
$$
\left(I _ {n} - (1 - \alpha) A _ {r w}\right) ^ {- 1} = \sum_ {m = 0} ^ {\infty} \left((1 - \alpha) A _ {r w}\right) ^ {m} = \sum_ {m = 0} ^ {\infty} \left((1 - \alpha) A D ^ {- 1}\right) ^ {m}.
$$
Because  $D$  is a diagonal matrix and  $1 - \alpha$  is a constant, they do not affect whether an element in the matrix is 0 or not. Therefore, we only need to discuss  $\sum_{m=0}^{\infty} A^{m}$ . Because  $A$  is the adjacency matrix of the DAG  $G$ , in this matrix, iff there is not a directed path from  $x$  to  $y$  ( $x \not\leq y$ ), we deduce that  $(\sum_{m=0}^{\infty} A^{m})[x, y] = 0$ . This leads us to the conclusion that  $\pi_{G}(i_{x})[y] = 0$ .
We invert the directions of the edges in  $G$  to create a reverse DAG  $\tilde{G}$ . We can also obtain that, iff  $x \not\leq y$ , then  $\pi_{\tilde{G}}(i_x)[y] = 0$ .
Therefore, for every node  $x$ , only nodes  $y$  that  $y \notin N_{\infty}(x)$  will satisfy that  $\pi_G(i_x)[y] + \pi_{\tilde{G}}(i_x)[y]$  equals 0.
Theorem 2. In a directed rooted tree  $T = (V, E)$  with the number of nodes  $|V|$ , the runtime of DAG attention is  $O(|V| \times k \times \Delta^{+} \times d)$ , where  $\Delta^{+}$  is the maximum out degree of the  $T$  and  $d$  is feature dimension. When  $k = \infty$ , the runtime of DAG attention is  $O(|V| \times \ell \times \Delta^{+} \times d)$ , where  $\ell$  is the depth of  $T$ .
Proof. In DAG attention, for each node  $v$ , we need to calculate the attention to  $N_{k}(v)$ , whose runtime is upper bounded by  $O(|N_{k}(v)| \times d)$ . So the runtime of DAG attention is linear in the sum of the sizes of receptive fields  $\sum_{v \in V} |N_{k}(v)|$ . We only need to show that  $\sum_{v \in V} |N_{k}(v)| \leqslant |V| \times k \times \Delta^{+}$ . When  $k = \infty$ ,  $\sum_{v \in V} |N_{k}(v)| \leqslant |V| \times \ell \times \Delta^{+}$ .
Next, we prove this theorem for the case of  $k = \infty$ . For bounded  $k$ , the proof is similar. We prove by induction on the depth  $\ell$  of  $T$ .
We assume that  $\Delta^{+} \geqslant 2$ , since when  $\Delta^{+} = 1$ ,  $T$  is a 1-ary tree,  $\sum_{v \in V} |N_{\infty}(v)| = |V| \times |V| \leqslant |V| \times \ell \times \Delta^{+}$ .
For the basis step, when  $\ell = 0$ ,  $T$  only has one node. So  $\sum_{v\in V}|N_{\infty}(v)| = 0\leqslant |V|\times \ell \times \Delta^{+}$ .
For the inductive step, we assume that
$\sum_{v \in V} |N_{\infty}(v)| \leqslant |V| \times \ell \times \Delta^{+}$  holds when  $\ell \geqslant 0$ . Consider a tree  $T_{new}$  of depth  $\ell + 1$  whose root is denoted by  $r$ . Then the sub-trees whose roots are the children of  $r$  have depth at most  $\ell$ . Let  $s$  be the number of these sub-trees,  $s \leqslant \Delta^{+}$ , and  $V_{i}$  be the node set of the  $i$ -th sub-tree. Since the number of reachability relations with  $r$  involved is  $2(|V| - 1)$ , we have
$$
\begin{array}{l} \sum_ {v \in V} | N _ {\infty} (v) | \\ = \sum_ {i \in [ s ]} \sum_ {v \in V _ {i}} | N _ {\infty} (v) | + 2 (| V | - 1) \\ \leqslant \left| V _ {1} \right| \times \ell \times \Delta^ {+} + \dots + \left| V _ {s} \right| \times \ell \times \Delta^ {+} + 2 (| V | - 1) \\ \leqslant \ell \times \Delta^ {+} \times (| V | - 1) + 2 (| V | - 1) \\ \leqslant \ell \times \Delta^ {+} \times (| V | - 1) + \Delta^ {+} \times (| V | - 1) \\ \leqslant \quad (\ell + 1) \times \Delta^ {+} \times | V | \\ \end{array}
$$
Table 8: The statistics of academic self-citation dataset.
Min	Avg	Max
# nodes	30	59	296
DAG depth	1	5	21
Node degree	0	1.8	83
This completes the proof of Theorem 2 for the case of  $k = \infty$ .
B.1 Computing Environment
Our implementation is based on PyG [Fey and Lenssen, 2019]. All reported results are averaged over 10 runs using different random seeds. The experiments are conducted with two RTX 3090 GPUs.
B.2 Dataset Details
We provide details of the datasets used in our experiments, including ogbg-code2 [Hu et al., 2020], Neural architectures (NA) [Zhang et al., 2019], Self-citation, Cora, Citeseer and Pubmed [Sen et al., 2008].
ogbg-code2. ogbg-code2 [Hu et al., 2020] is a dataset containing 452,741 Python method definitions extracted from thousands of popular Github repositories. It is made up of Abstract Syntax Trees as DAGs where the task is to correctly classify the sub-tokens that comprise the method name. The min/avg/max numbers of nodes in the graphs are 11/125/36123, respectively. We use the standard evaluation metric (F1 score) and splits provided by [Hu et al., 2020].
Neural architectures (NA). This dataset is created in the context of neural architecture search. It contains 19,020 neural architectures generated from the ENAS software [Pham et al., 2018]. Each neural architecture has 6 layers (i.e., nodes) sampled from 6 different types of components, plus an input and output layer. The input node vectors are one-hot encodings of the component types. The weight-sharing accuracy [Pham et al., 2018] (a proxy of the true accuracy) on CIFAR-10 [Krizhevsky et al., 2009] is taken as performance measure. Since it is a regression task, the metrics are RMSE and Pearson's r. We use the splits as is used in [Thost and Chen, 2021]. Details about the generation process can be found in [Zhang et al., 2019].
Self-citation. The academic self-citation dataset is a directed acyclic graph, representing the scholar's academic self-citation network on the natural language processing (NLP) field extracted from ACL Anthology Reference Corpus [ARC, 2021]. This dataset contains 1000 academic self-citation networks from scholars ranked top-1000 by number of papers. In a self-citation networks, each node is a paper of this scholar and each directed edge indicates that one paper is cited by another one. Each graph node includes two quantitative attributes: the publication year, the paper's total citation count. The task is to predict missing citation counts given existing citation counts. Specifically, for each scholar self-citation network, we randomly select  $10\%$  of its papers and drop their citation counts, and the node-level task is to predict whether a paper which misses its citation count is highly-cited or not. In Computer Science, Engineering, and Mathematics, for papers in ISI (Web of science database) listed journals that are 10 years old, around 20 citations gets in the top  $10\%$ . So we consider papers with a citation count greater than or equal to 20 as highly-cited papers. As the class balance is skewed (only  $27.2\%$  of data is positive), we use the Average Precision (AP) and ROC-AUC as the evaluation metrics. We randomly split the dataset into  $80\%$  training,  $10\%$  valid and  $10\%$  test sets. The statistics is shown in Table 8.
Cora, CiteSeer and Pubmed [Sen et al., 2008]. These three datasets are commonly used citation networks for evaluating models on node classification tasks. These datasets are medium-scale networks (with  $2\mathrm{K}\sim 20\mathrm{K}$  nodes) and the goal is to classify the topics of documents (instances) based on input features of each instance (bag-of-words representation of documents) and graph structure (citation links). For the baselines, the citation links are treated as undirected edges [Wu et al., 2022] and each document has a class label. Only for our method, we treated citation links as directed edges and removed a small number of cyclic citation links to make them DAGs. We use the same evaluation metric (classification accuracy) and splits provided by NodeFormer [Wu et al., 2022].
Table 9: Hyperparameter search on different datasets.
Hyperparameter	NA	Self-citation	Cora, Citeseer and Pubmed
# Layers	{2,3}	{2,3,4,5}	{2,3,4,5}
Hidden dimensions	{32,128}	{32,64,128}	{16,32,64,128}
Dropout	0.2	{0.1,0.2,0.5}	0.0
Learning rate	1e-3	{1e-4,5-e4,1e-3,5e-3}	{1e-3,1e-2}
# Epochs	100	{50,100}	1000
Weight decay	None	1e-6	5e-3
# Attention heads	{2,4}	{2,4,8}	{2,4}
Table 10: Number of parameters for DAG+ models.
Model	ogbg-code2	NA	Self-citation	Cora	Citeseer	Pubmed
DAG+Transformer	127,06k	399k	478k	291k	404k	14k
DAG+SAT	149,53k	631k	710k	276k	165k	14k
B.3 Hyperparameters and Reproducibility
ogbg-code2. We implemented DAG attention on vanilla Transformer, GraphTrans, GraphGPS and SAT that we only modify the self-attention module to DAG attention module. And we do not use any PE as well as baselines because it makes very little performance improvement [Chen et al., 2022a]. For fair comparisons, we use the same hyperparameters (including training schedule, optimizer, number of layers, batch size, hidden dimension etc.) as baseline models for all of our four versions. NA. We train the transformer-based baselines (vanilla Transformer, GraphGPS and SAT) in front of DAGNN [Thost and Chen, 2021]. For SAT, we use the 3-subtree GNN extractor (GIN). For GraphGPS, we use GatedGCN as GPS-MPNN and vanilla Transformer as GPS-GlobAttn. And we implement DAG attention on those respectively. We follow the exact training settings of DAGNN [Thost and Chen, 2021]. Then we also train a sparse Gaussian process (SGP) [Snelson and Ghahramani, 2005] with a learning rate of 4e-4, a epochs number 200 as the predictive model to evaluate the performance of the latent representations. The transformer-based hyperparameters are summarized in Table 9. All other hyperparameters are the same as those of DAGNN [Thost and Chen, 2021].
Self-citation. We implement DAG attention on the vanilla Transformer, GraphGPS, SAT and NodeFormer. In general, we perform a hyperparameter search to produce the results for our model and baselines. The hyperparameters on the academic self-citation dataset are summarized in Table 9, where # Attention heads is tuned for transformer-based models. For SAT, we conduct positional encodings search on LapPE-8, RWPE-20 or None, and we use the 3-subtree GNN extractor (GIN). For GraphGPS, we use GatedGCN as GPS-MPNN and vanilla Transformer as GPS-GlobAttn. The other hyper-parameters for NodeFormer are the default parameters in their public code. In all experiments, we use the validation set to select the best hyperparameters. All our models and baselines are trained with the AdamW optimizer [Loshchilov and Hutter, 2016] and we use the cross-entropy loss on this classification task. The learning rate scheduler is the cosine scheduler [Loshchilov and Hutter, 2016]. Cora, Citeseer and Pubmed. We implement DAG attention on the vanilla Transformer, SAT and NodeFormer. Only for Pubmed, we choose the size of the receptive field  $k = 2$  (otherwise  $k = \infty$ ). And we perform a hyperparameter search to produce the results for our model and those transformer-based baselines in Table 9. We follow all other training setting and hyperparameters of NodeFormer [Wu et al., 2022]. For SAT, we use the 1-subtree GNN extractor (GCN).
In Table 10, we report the number of parameters for DAG+ models using the hyperparameters selected from Table 9.
C Model Visualization
Here, we demonstrate that the DAG transformer also provides interpretability compared to the traditional Transformer model. We trained a DAG+Transformer and a Transformer model on a self-citation graph, and compared the attention scores between the selected node and other nodes
learned by both models. Figure 3 visualizes the results. In the leftmost graph, the darker blue of the nodes represents a higher number of citations. For the selected node  $v$ , DAG attention only puts attention weights on the nodes that have citation relations with it ( $N_{k}(v)$ ), whereas classic self-attention focuses on nodes that are almost unrelated to  $v$ . More focus on these core nodes makes the DAG model less influenced by other patterns and thus leads to improved performance.
Figure 3: Attention visualization of DAG+Transformer and the Transformer on self-citation. The figure shows the attention weights of the node in the red box learned by DAG attention and the classic self-attention.
D.1 Semi-supervised Node Classification
We test our model on three citation networks Cora, Citeseer and Pubmed. We implement DAG attention on DIFFormer-a Wu et al. [2023]. Table 11 reports the testing accuracy. This show that our framework likely increases both quality and runtime here as well.
Table 11: Node classification accuracy (%).The baseline results were taken from [Wu et al., 2023].
Model	Cora	Citeseer	Pubmed
Accuracy	Time (s)	Accuracy	Time (s)	Accuracy	Time (s)
NodeFormer	83.4 ± 0.2	-	73.0 ± 0.3	-	81.5 ± 0.4	-
DIFFORMER-a	84.1 ± 0.6	14.21	75.7 ± 0.3	8.48	80.5 ± 1.2	1013.31
DAG + DIFFORMER-a	85.1 ± 0.7	10.96	76.2 ± 0.4	7.01	81.6 ± 0.5	84.15
D.2 MalNet-Tiny
Here we consider MalNet-Tiny. MalNet-Tiny [Freitas and Dong, 2021] is a subset of MalNet which consists of function call graphs (FCGs) obtained from Android APKs. This subset includes 5,000 graphs with a maximum of 5,000 nodes, originating from benign software or 4 malware types. The FCGs do not have any original node or edge features. The benchmark version of this dataset usually employs Local Degree Profile as the set of node features. And the objective is to predict the type of software solely based on the structure.
Note that we did not do any hyperparameter tuning. Further, we implemented DAG attention on top of GraphGPS by only modifying the self-attention module, switching it to DAG attention. As shown in Table 12, Our DAG+GraphGPS achieves a test accuracy of  $93.45\%$  while only requiring 20 seconds per epoch. This demonstrates the quality and efficiency of DAG attention.
E Further Related Works
Neural Networks on Directed Graphs. Most works we found focus on adapting the Laplacian: [Ma et al., 2019] construct a directed Laplacian matrix to improve the propagation model by using identities that involve the random walk matrix and its stationary distribution. [Monti et al., 2018] consider several different symmetric Laplacian matrices corresponding to local directed graph motifs.
Table 12: Graph classification on MalNet-Tiny.
Model	Accuracy (%)	Time(epoch)
GraphGPS	92.64 ± 0.78	46s
DAG+GraphGPS	93.45 ± 0.41	20s
Moreover, DiGCN [Tong et al., 2020] is based on a directed Laplacian with a PageRank matrix to learn multi-scale features and [Zhang et al., 2021] proposed MagNet, a GNN for directed graphs based on the Magnetic Laplacian. The latter has been recently integrated into Transformers in the form of a positional encoding [Geisler et al., 2023]. Note that this improves predictive performance in directed graphs, but does impact the (quadratic) complexity of self-attention.
Transformers over Trees. Observe that trees are special in that they are a particular, more restricted type of DAGs. [Sun et al., 2020] proposed TreeGen, a tree-based Transformer that combines grammar rules and tree structure using a tree reader. However, it is not directly relevant to our work as it focuses on program-specific information. In addition, there are some methods of leveraging tree paths and integrating them into the self-attention module. [Alon et al., 2018] used AST to represent a source code snippet. They considered pairwise paths between leaf nodes, representing them as sequences of the leaf and non-leaf nodes in AST. [Kim et al., 2021] used another type of path that goes from the leaf nodes up to the AST root by traversing its ancestors. [Zügner et al.] presented Code Transformer, which combines distances computed on ASTs and context of source code in the self-attention. Specifically, it utilizes four different paths to reason about ASTs relative positions. Furthermore, [Peng et al., 2022] designed a new directed tree Transformer position encoding for each node based on a two-dimensional description of tree structures. In contrast to these works, we shift our focus toward more general DAGs, use a very simple and intuitive PE, and also adapt the receptive field of the attention.