SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeyang Zhang $^{1*}$ , Xin Wang $^{1}$ , Ziwei Zhang $^{1}$ , Guangyao Shen $^{2}$ , Shiqi Shen $^{2}$ , Wenwu Zhu $^{1\dagger}$ $^{1}$ Department of Computer Science and Technology, BNRist, Tsinghua University, Wechat, Tencent zy-zhang20@mails.tsinghua.edu.cn, {xin_wang, zwzhang}@tsinghua.edu.cn, {lucasgyshen, shiqishen}@tencent.com, wwzhu@tsinghua.edu.cn
The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing to handle ubiquitous scenarios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due to the nature of the graph and the complexity of the neural architecture search process. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Specifically, we first design a disentangled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which are optimized simultaneously. Then, we estimate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-graph disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed DSGAS model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.
Graph neural architecture search (GNAS), aiming to automatically discover the optimal architecture for graph neural network (GNN) based on graph-structured data and task, has shown remarkable progress in enhancing the predictive power and saving human endeavors for various graph applications [1]. The existing GNAS methods generally follow a supervised paradigm such that they optimize the weights within architectures given a training dataset with a supervised loss (e.g., the cross entropy loss of label predictions) and estimate the architecture performance based on the validation dataset with supervision signals. For example, the label prediction accuracy is adopted for architecture ranking during the neural architecture search process [2, 3, 4]. As a result, supervised labels become indispensable for applying the existing GNAS methods.
However, ground-truth labels in reality may be extremely scarce or hardly available in many graph applications. For example, a variety of biological problems require a significant amount of human labors and time costs in clinical tests to obtain labels for supervision [5, 6, 7]. As the existing
*This work was done during the author's internship at Wechat, Tencent
GNAS approaches heavily rely on supervised labels for weight training and architecture evaluation, they will suffer from performance deterioration in unsupervised settings, failing to discover optimal architectures in the scenarios where labels are scarce or not available.
In this paper, we study unsupervised graph neural architecture search, i.e., discovering optimal GNN architectures without labels for graph-structured data, which remains unexplored in the literature. The key problem lies in two important aspects: i) discover the latent graph factors that drive the formation process of graph data [8, 9, 10, 11, 12, 13, 14]; ii) capture the underlying relations between the factors and the optimal neural architectures. For instance, a molecular graph may consist of groups of atoms as well as bonds representing different functional units [15], requiring different optimal neural architectures to make accurate predictions.
Nevertheless, solving the problem is highly non-trivial and challenging given that the hidden factors are entangled in the graph and very difficult to capture, e.g., a social network may contain several communities originating from various interests (e.g., sports, games, etc.) [16, 10], with the nodes and edges belonging to different communities mixing together. Moreover, the architectures with different functional factors are also entangled within the weight-sharing super-network [17, 18, 19], resulting in inaccurate architecture performance estimations under different hidden factors.
To tackle the challenge, we propose a novel unsupervised graph neural architecture search method, i.e., Disentangled Self-supervised Graph Neural Architecture Search  $(\mathbf{DSGAS})^3$ . Given graph data without supervised labels, our proposed DSGAS model can discover the optimal architectures capturing multiple latent factors in a self-supervised fashion. In particular, we first design a disentangled graph super-network, where multiple architectures are disentangled for simultaneous optimization w.r.t various latent factors. Then, we propose a self-supervised training with joint architecture-graph disentanglement, which disentangles architectures and graphs within a common latent space. The super-network is trained through a routing mechanism between architectures, graphs and self-supervised tasks, to obtain an accurate estimation of the architecture performance under each latent factor. Finally, we propose a contrastive search with architecture augmentations, where a novel architecture-level instance discrimination task is introduced to discover architectures with distinct capabilities of capturing various factors in a self-supervised fashion. Extensive experiments show that the proposed DSGAS model is able to significantly outperform the state-of-the-art GNAS baselines under both unsupervised and semi-supervised settings. Detailed ablation studies and analyses also demonstrate that DSGAS is able to discover effective architectures with our proposed disentangled self-supervision designs. The contributions of this paper are summarized as follows:
- We are the first to study the problem of unsupervised graph neural architecture search and propose the Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model capable of discovering the optimal architectures without labels, to the best of our knowledge.
- We introduce three novel modules, i) disentangled graph architecture super-network, ii) self-supervised training with joint architecture-graph disentanglement and iii) contrastive search with architecture augmentations, which can discover the optimal architectures capturing various graph latent factors with disentangled self-supervision.
- Extensive experiments on 11 real-world graph datasets show that our proposed method DSGAS is able to discover effective graph neural architectures without supervised labels and significantly outperform the state-of-the-art baselines in both unsupervised and semi-supervised settings.
Graph Neural Architecture Search Denote the graph space as  $\mathcal{G}$  and the label space as  $\mathcal{V}$ . A graph neural network can be denoted as a function  $f_{\alpha, w}: \mathcal{G} \to \mathcal{V}$ , which is characterized by architecture parameters  $\alpha \in \mathcal{A}$  and learnable weights  $w \in \mathcal{W}$  given an architecture space  $\mathcal{A}$  and a weight space  $\mathcal{W}$ . Graph neural architecture search (GNAS) aims at automating the design of graph neural architectures, i.e., obtaining the best-performed architectures by searching  $\alpha$ . As  $\alpha$  is usually instantiated as selecting GNN operations, e.g., GCN [20], GAT [21], GIN [22], we also call  $\alpha$  as
Disentangled Self-Supervised Graph Neural Architecture Search
Figure 1: The framework of Disentangled Self-supervised Graph Neural Architecture Search (DSGAS), including the following three key components: 1) Disentangled graph architecture supernetwork enables multiple architectures to be disentangled and optimized simultaneously in an end-to-end manner. 2) Self-supervised training with joint architecture-graph disentanglement estimates the performance of architectures under various latent factors by considering the relationship between architectures, graphs and factors. 3) Contrastive search with architecture augmentations encourages and discovers architectures with distinct capabilities of capturing factors. (Best viewed in color)
operation choices for brevity. Generally, GNAS solves the bi-level optimization problem [23]:
$$
\alpha^ {*} = \underset {\alpha \in \mathcal {A}} {\arg \min } \mathcal {L} _ {\text {v a l}} (\alpha , w ^ {*} (\alpha)), \tag {1}
$$
$$
\operatorname {s. t.} w ^ {*} (\alpha) = \underset {w \in \mathcal {W} (\alpha)} {\arg \min } \mathcal {L} _ {\text {t r a i n}} (\alpha , w), \tag {2}
$$
where  $\mathcal{L}_{\mathrm{train}}$  and  $\mathcal{L}_{\mathrm{val}}$  denotes the loss of the predictions of the architecture  $f_{\alpha, w}(\cdot)$  against supervised labels on training and validation datasets. The optimization problem can be viewed as having two objectives that Eq.(2) aims to obtain accurate architecture performance estimation, and Eq.(1) aims to search the best-performed architectures. To avoid the cost of training from scratch for each architecture, the super-network [24, 25] arises as a commonly adopted technique to obtain faster architecture performance estimation, where the architecture candidates are viewed as sub-networks of the super-network, and their weights are shared during the training process.
Unsupervised Graph Neural Architecture Search We consider the problem of unsupervised graph neural architecture where labels, which are adopted for the performance estimation and the search process in the supervised GNAS, are not accessible. The problem of unsupervised GNAS can be formulated as optimizing an architecture generator that is able to discover powerful architectures by exploiting inherent graph properties without labels, i.e.,  $\mathcal{G} \mapsto f_{\alpha, w}$  instead of  $(\mathcal{G}, \mathcal{Y}) \mapsto f_{\alpha, w}$  as done by supervised GNAS methods. Then, the discovered architectures  $f_{\alpha, w}(\cdot)$  can be utilized in downstream tasks, e.g., finetuning the weights  $w$  or the operation choices  $\alpha$ , or extra shallow classifiers for further prediction.
3 Disentangled Self-supervised Graph Neural Architecture Search
In this section, we introduce Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) to search architectures without labels, by proposing three key components: disentangled graph architecture super-network, self-supervised training with joint architecture-graph disentangle-ment, and contrastive search with architecture augmentations.
3.1 Disentangled Graph Architecture Super-Network
To discover architectures that potentially have optimal performance, we resort to guiding the search towards architectures' capabilities of capturing the inherent graph factors, which are shown important in the graph formation [8, 9]. As architectures may expert in different graph factors, we propose a
disentangled graph architecture super-network to incorporate  $K$  different architectures to be estimated and searched w.r.t factors simultaneously, where the hyperparameter  $K$  denotes the number of factors.
Disentangled Super-Network Layer For each super-network layer, we adopt  $K$  mixed operations parameterized by different  $\alpha$  to learn  $K$ -chunk graph representations:
$$
\mathbf {H} _ {k} \leftarrow \overline {{\mathrm {G N N}}} _ {\alpha_ {k}} (\mathbf {H}, \mathbf {A}), \tag {3}
$$
where  $\mathbf{A}$  is the adjacency matrix of the graph,  $\mathbf{H}$  denotes the input graph representations, and  $\overline{\mathrm{GNN}}_{\alpha_k}(\cdot)$  denotes the mixed GNN operations parameterized by  $\alpha_{k}$ . For the convenience of differentiable optimization, we adopt continuous parameterization and weight-sharing mechanism [25] to implement the mixed operations:
$$
\overline {{\mathrm {G N N}}} _ {\alpha_ {k}} (\mathbf {H}, \mathbf {A}) = \sum_ {i = 1} ^ {| \mathcal {O} |} \alpha_ {k, i} \mathrm {G N N} _ {i} (\mathbf {H}, \mathbf {A}), \tag {4}
$$
where  $|\mathcal{O}|$  is the number of GNN operation choices,  $\alpha_{k,i} = \frac{\exp(\theta_{\alpha_{k,i}})}{\sum_j\exp(\theta_{\alpha_{k,j}})}$  denotes the probability of the  $i$ -th operation for the  $k$ -th architecture  $\alpha_{k}$ , and  $\theta$  is learnable parameters.
Overall Disentangled Super-Network The overall super-network is constructed in the form of a directed acyclic graph (DAG) with an ordered sequence of disentangled super-network layers. More details about the DAG construction and the according GNN operations for each disentangled super-network layer are included in Appendix. The output of the last layer  $\mathbf{Z} = [\mathbf{H}_1, \mathbf{H}_2, \dots, \mathbf{H}_K]$  describes the various aspects of the graphs and serves as the final graph representations, which can be utilized or finetuned in downstream tasks. In this way, the architectures' operation choices  $\alpha = [\alpha_1, \alpha_2, \dots, \alpha_K]$  and weights  $w$  are incorporated in one super-network. For brevity, we use  $f_{\alpha_k, w}(\cdot)$  to denote the  $k$ -th architecture induced from the super-network. Note that the design of  $K$  operation choices alleviates the entanglement of architectures by providing more flexible choices of paths [26] in the super-network. For instance, the 'mean' operation captures structural properties while the 'max' operation captures representative elements [22], and in this case, our design can capture both of them by choosing corresponding operations to learn respective representations instead of choosing only one of them which may conflict each other.
3.2 Self-supervised Training with Joint Architecture-Graph Disentanglement
Inspired by graph self-supervised learning, we utilize graph pretext tasks to measure the architectures' capabilities of capturing latent factors. Predictive pretext tasks [27], for example, design a pseudo label generator  $s(\cdot)$ , and optimize the prediction probability  $p(s(\mathcal{G}_i)|\mathcal{G}_i)$ . However, the tasks usually take holistic views of the graphs and neglect the entanglement of the latent factors, which may lead to suboptimal performance estimation. Therefore, to disentangle the factors, we transform the probability into the expectation of multiple subtasks under different latent factors by the Bayesian formula:
$$
p \left(s \left(\mathcal {G} _ {i}\right) \mid \mathcal {G} _ {i}\right) = \mathbb {E} _ {p (k \mid \mathcal {G} _ {i})} p \left(s \left(\mathcal {G} _ {i}\right) \mid \mathcal {G} _ {i}, k\right), \tag {5}
$$
where  $p(k|\mathcal{G}_i)$  denotes the probability of latent factor  $k$  given the  $i$ -th graph instance  $\mathcal{G}_i$ , and  $p(s(\mathcal{G}_i)|\mathcal{G}_i,k)$  denotes the pretext task under  $k$ -th latent factor. An intuitive explanation of Eq. (5) is that it first infers the latent factors and then conducts factor-specific self-supervised training to capture the latent factors, which we describe in detail as follows.
Architecture-aware Latent Factor Inference Directly modeling  $p(k|\mathcal{G}_i)$  is difficult as we do not know prior what GNN encoders are suitable for inferring the latent factors. Intuitively, one solution is to get the architectures being searched involved in the inference stage. By the Bayesian formula, we factorize the probability w.r.t architecture choices:
$$
p (k | \mathcal {G} _ {i}) = \mathbb {E} _ {p \left(\alpha_ {j} \mid \mathcal {G} _ {i}\right)} p \left(k \mid \mathcal {G} _ {i}, \alpha_ {j}\right), \tag {6}
$$
4We take graph classification as an example for simplicity, while the case of node classification can be easily extended.
where  $p(\alpha_j|\mathcal{G}_i)$  is a prior distribution, and we adopt a uniform distribution for simplicity. Then we can model the probability distributions of latent factors given the graph  $\mathcal{G}_i$  by utilizing the architectures being searched:
$$
p (k | \mathcal {G} _ {i}, \alpha_ {j}) = \frac {\exp \phi (\mathbf {z} _ {i , j} | | \operatorname {E n c} (\alpha_ {j}) , \mathbf {c} _ {k})}{\sum_ {m = 1} ^ {K} \exp \phi (\mathbf {z} _ {i , j} | | \operatorname {E n c} (\alpha_ {j}), \mathbf {c} _ {m})}, \tag {7}
$$
where  $\mathbf{c}_k$  is a learnable vector to represent the  $k$ -th latent factor, and  $\mathbf{z}_{i,k} = f_{\alpha_k,w}(\mathcal{G}_i)$  denotes the graph representations output by the  $k$ -th architecture for the graph  $\mathcal{G}_i$ .  $\mathrm{Enc}(\cdot)$  denotes architecture encoding techniques to obtain embeddings of operation choices  $\alpha$  so that the structural properties and correlations of the neural architectures can be considered [28, 29].
Factor-aware Graph Self-Supervised Learning Under the  $k$ -th latent factor, we leverage the corresponding architectures  $f_{\alpha_k,w}(\cdot)$  for conducting the factor-specific pretext tasks as  $p(s(\mathcal{G}_i)|\mathcal{G}_i,k)$  to estimate the capturing capabilities of architectures under various factors. The overall objective is to maximize Eq.(5), and the loss can be calculated by
$$
\frac {1}{N} \sum_ {i} - \log \mathbb {E} _ {p (k | \mathcal {G} _ {i})} \left(p (s (\mathcal {G} _ {i}) | \mathcal {G} _ {i}, k)\right) \leq \frac {1}{N} \sum_ {i} \mathbb {E} _ {p (k | \mathcal {G} _ {i})} \left(- \log p (s (\mathcal {G} _ {i}) | \mathcal {G} _ {i}, k)\right), \tag {8}
$$
where  $N$  is the number of samples and the upper bound is obtained by Jensen's Inequality. Then we can generalize our method to other graph self-supervised tasks with specially-designed task loss functions by defining  $-\log p(s(\mathcal{G}_i)|\mathcal{G}_i,k)$  as the task loss function  $l(f_{\alpha_k,w},\mathcal{G}_i)$  for the graph  $\mathcal{G}_i$  under the  $k$ -th factor, and calculate the loss by
$$
\mathcal {L} _ {w} = \frac {1}{N} \sum_ {i} \mathbb {E} _ {p (k | \mathcal {G} _ {i})} \left(l \left(f _ {\alpha_ {k}, w}, \mathcal {G} _ {i}\right)\right). \tag {9}
$$
In this way, the disentangled architectures in Sec. 3.1 coupled with factors disentangled from graph data can be routed pairwise, and trained with factor-specific self-supervision to obtain more accurate performance estimation under each factor. Similar to [25], the super-network weights are updated with  $w = w - \lambda_w\nabla_w\mathcal{L}_w$  to obtain the weights that can represent the architectures' capabilities.
3.3 Contrastive Search with Architecture Augmentations
In this section, we focus on encouraging the disentanglement of architectures and searching architectures with distinct capabilities of capturing different factors. The main insight of our proposed search method is intuitively based on the following two observations shown in the literature: 1) As architectures similar in operation choices and topologies have similar capabilities of capturing semantics for downstream tasks [28, 30, 31], slight modifying the architecture will have a slight influence on its capability. 2) Since different GNN architectures expert in different downstream tasks [32], the architectures searched for different disentangled latent factors are expected to have dissimilar capabilities under different factors.
Contrastive Search Inspired by self-supervised contrastive learning [33, 34] that capture discriminative features by pulling similar instances together and pushing dissimilar instances away in the latent space, we propose an architecture-level instance discrimination task to encourage the architectures to capture various latent factors. The task is defined as
$$
p \left(s \left(\alpha_ {k}\right) \mid \mathcal {G} _ {i}, \alpha_ {k}\right) = \frac {\exp \phi \left(\mathbf {z} _ {i , k} , \mathbf {z} _ {i , s \left(\alpha_ {k}\right)} ^ {\prime}\right)}{\sum_ {j = 1} ^ {N} \exp \phi \left(\mathbf {z} _ {i , j} , \mathbf {z} _ {i , s \left(\alpha_ {j}\right)} ^ {\prime}\right)}, \tag {10}
$$
$$
\mathbf {z} _ {i, k} = f _ {\alpha_ {k}, w} \left(\mathcal {G} _ {i}\right), \mathbf {z} _ {i, k} ^ {\prime} = T _ {f} \left(f _ {\alpha_ {k}, w}\right) \left(\mathcal {G} _ {i}\right), \tag {11}
$$
where  $s(\alpha_k)$  is assigned to  $k$  as surrogate labels for the architecture,  $\phi(\cdot)$  calculates the similarity of two embeddings and  $T_f(\cdot)$  denotes architecture augmentations that transform an architecture to another architecture with similar capabilities capturing factors, i.e.,  $f_{\alpha_k,w} \mapsto f_{\alpha_k,w}'$ . Then the loss function can be calculated by
$$
\mathcal {L} _ {\alpha} = \sum_ {i} - \log \mathbb {E} _ {p \left(\alpha_ {k} \mid \mathcal {G} _ {i}\right)} p \left(s \left(\alpha_ {k}\right) \mid \mathcal {G} _ {i}, \alpha_ {k}\right). \tag {12}
$$
Similar to [25], the architecture parameters are updated with  $\alpha = \alpha -\lambda_{\alpha}\nabla_{\alpha}\mathcal{L}_{\alpha}$  to search architectures with better capabilities of capturing factors.
Architecture Augmentations To create various views of architectures, we design three basic architecture augmentations from the perspectives of architecture operation choices  $\alpha$ , architecture weights  $w$  and the internal embeddings  $\mathbf{H}$ :
- Operation Choice Perturbation. This augmentation randomly reshapes the distributions of the mixed operations by altering the temperature in the softmax function in Eq. (4) with
$$
\alpha_ {k, i} = \frac {\exp \left(\theta_ {\alpha_ {k , i}} / \tau\right)}{\sum_ {j} \exp \left(\theta_ {\alpha_ {k , j}} / \tau\right)}, \tag {13}
$$
where the temperature  $\tau$  is sampled from a uniform distribution  $\mathcal{U}([1 / r_1, r_1])$ , and  $r_1 \geq 1$  is a hyper-parameter controlling the perturbation degree.
- Weight Perturbation. This augmentation randomly adds Gaussian noises  $\epsilon \sim \mathcal{N}(0, \sigma^2)$  to  $r_2\%$  of the architecture weights  $w$ , where  $r_2$  controls the perturbation ratio, and  $\sigma^2$  is the standard deviation of the weights.
- Embedding Dropout. This augmentation randomly drops  $r_3\%$  of the embeddings  $\mathbf{H}$  output from the mixed operations, where  $r_3$  controls the dropout ratio.
Note that these architecture augmentations can further be randomly composed to generate mixed augmentations.
In this section, we conduct experiments on 8 real-world datasets with unsupervised settings to verify the design of our method. We also include detailed ablation studies to analyze the effectiveness of each component, and 3 real-world datasets in semi-supervised settings to show that our method can alleviate the label scarcity issues by pretraining the super-network.
Baselines We compare our method with 11 baselines from the following two different categories.
- Manually designed GNNs. We include five representative GNNs as our baselines, i.e., GCN [20], GAT [21], GIN [22], GraphSage [35], GraphConv [36] and a simple baseline MLP, which constitutes our search space. For graph-level classification tasks, we adopt global mean pooling for each layer and concatenation to obtain the graph representations for these baselines.
- Graph neural architecture search. We include representative GNAS baselines GraphNAS [3], PAS [4] and GASSO [37], where PAS is specially designed for graph-level classification tasks by searching the pooling operations, and GASSO is specially designed for node-level classification tasks by searching the graph structures simultaneously. We also include two classical NAS baselines, random search and DARTS [25]. As these baselines are not specially designed for graphs, we adopt our search space for these baselines.