dataesr/scientific-paragraphs-categorization huggingface dataset:
On the virtual screen's right side, again as in the original, we render the scene using a birds-eye-view camera which we placed at (0, 1 m, -1. 75 m) and which faces the center of the scene, where we place the dataset by default. As in the original technique, we apply translations with respect to the tablet's coordinate system, with the z-axis defined by the normal of the virtual tablet. This control is convenient when looking at either the tablet or the virtual screen's left side, as the axes of these respective cameras are aligned with the translation motion. Based on a pilot study, we allow participants to rotate the 3D scene around the center of the dataset using the tablet as a tangible device. 3D rotations are strong depth cues that allow users to align both the tablet and the dataset along an axis they can understand, and to review the 3D extruded-wireframed volumes within the 3D scene.
3D result. These examples show that, compared to view-occluding VR-HMDs, AR-HMDs allow users to better interact with additional, physical devices and we rely on an AR-HMD coupled with a tablet for volumetric selection for the same reason. 2. 3. Volumetric/Spatial Selection Many 3D selection techniques exist. We separate the selection of explicit objects from the spatial selection of subvolumes. The first group relies on predefined shapes or objects in the 3D space that users explicitly mark, where most techniques rely on raycasting [GB06, RN10, RCK * 17, BPC19] in 3D environments or tap and lassos in 2D ones [XFAT12, YEII16, CZY * 20]. The second allows users to specify 3D subspaces [AA13, BSA * 19, BYK * 21]. We focus here on unstructured datasets which do not include explicit objects but instead sample physical properties of, e.
g. , particles or volume cells, which one can filter to make selections. Jackson et al. [JLS * 13] filter vector fields based on a tangible cylinder's orientation. Hurter et al. [HRD * 19] brush paths using two VR controllers and select paths that connect both targeted points. Akers et al. [ASM * 04] filter brain white matter pathways using 3D boxes associated to Boolean operations. Yu et al. [YEII16] select regions based on the density of the field and 2D lasso input. Montano-Murillo et al. 's [MMNK * 20] hybrid VR-HMD plus tablet setup allows one to select points in cloud point datasets. The user first frames the ROI using the touch tablet and then uses the tablet as a magical lens to target at with a VR controller and raycasting. Transfer functions can also extract ROIs [KIL * 03, GXY12].
The first component is used to learn the discrete graph structure if the structural information is not available in the query graph. Then, the second component builds a surrogate model by jointly learning from the nodes' features and the response of the target model. Ideally, the surrogate model should achieve both high accuracy and high fidelity whereby accuracy measures the prediction correctness  [54, 70]  and fidelity measures the prediction agreement between the target model and the surrogate model  [34, 41]. We evaluate all our attacks on three popular inductive GNN models including GraphSAGE  [25] , Graph Attention Network (GAT)  [73] , and Graph Isomorphism Network (GIN)  [85]  with six benchmark datasets, i. e. , DBLP  [53] , Pubmed  [61] , Citeseer Full  [21] , Coauthor Physics  [62] , ACM  [76] , and Amazon Co-purchase Network for Photos  [50].
Extensive experiments demonstrate that our model stealing attacks consistently achieve strong performance in both types of attacks. For instance, when the target model is GIN trained on the Pubmed dataset and the response is embeddings, our Type I attack achieves a 0. 877 accuracy score and a 0. 906 fidelity score (see Section 5. 2). In particular, when the aforementioned target model's response is the t-SNE projection, our attack still achieves strong performance with a 0. 823 accuracy score and a 0. 846 fidelity score. Moreover, we empirically demonstrate that even without graph structure information, i. e. , Type II attacks, the adversary could still launch effective attacks, extracting high-accuracy and high-fidelity surrogate models (see  Section 5. 3). This further demonstrates the severe model stealing risks of GNN models.
In summary, we make the following contributions. • Our work is the first research effort to perform model stealing attacks against inductive GNNs. • We systematically define the threat model to characterize an adversary's background knowledge along two dimensions. Moreover, we propose six different attack scenarios based on the adversary's different background knowledge. • Extensive evaluation on three popular inductive GNN models and six benchmark graph datasets demonstrates the efficacy of our attacks. N l (v) l-hop neighborhood of v G l v subgraph induced by l-hop neighborhood of v X ∈ R n×d node feature matrix x v ∈ R d feature vector of node v H ∈ R n×b hidden state matrix h v ∈ R b hidden state of v G O /G Q training/query graph R query response ϒ ϒ ϒ ∈ R n×2 2-dimensional t-SNE projection Θ Θ Θ ∈ R n×|C| predicted posterior probability M T /M S target/surrogate GNN model 2 Background 2.
Here we consider the term "same distribution" as G O and G Q are drowned randomly from the same dataset. We do not necessarily require G O and G Q to have the same graph characteristics. Besides, the nodes in the query graph G Q does not need to be in the training graph G O. For example, both G O and G Q can be subgraphs sampled from social networks like Twitter but there is no overlap with each other. This assumption is in line with recent attacks to neural networks where an adversary uses part of a public dataset to exploit the target model  [27, 34]  and graphs are available in some domains (e. g. , social networks and molecular graphs). We further relax this assumption and demonstrate that the adversary can still launch effective attacks even without the graph structural information of G Q.
This weak assumption of adversary knowledge makes our attack more practical in real-world scenarios. For instance, the adversary may compromise a user database and acquire their profile information. However, the relationships among the users, i. e. , the graph structure, may not be revealed. We later demonstrate that the adversary can still launch high-performance model stealing attacks in this scenario (see  Section 5). Notes. Our threat model is different from the causative and evasion adversarial attacks to GNNs  [10, 14, 37, 66, 82, 83]. Those attacks allow the attackers to manipulate the training graphs in order to change the parameters of the target model, or modify the node features and/or graph structure to fool the inductive GNN models. Our attack is an instance of exploratory attacks.
Experimental Setup Datasets. We use 6 public datasets to evaluate the performance of our attack, including DBLP  [53] , Pubmed  [61] , Citeseer Full (abbreviated as Citeseer)  [21] , Coauthor Physics (abbreviated as Coauthor)  [62] , ACM  [76] , and Amazon Co-purchase Network for Photos (abbreviated as Amazon)  [50]. These datasets are widely employed as benchmark datasets to evaluate the performance of   6 ). The third part consists of the rest 50% of the nodes, functioning as the testing data for both M T and M S. This setting matches the inductive learning on evolving graphs as laid out in  [25]. Target Model (M T ). We use GIN, GAT, and GraphSAGE as our target models' architectures in our evaluation. For reproducibility purposes, we outline the details below. • GIN. We use a 3-layer GIN model with a fixed neighborhood sample size of 10 at each layer.
Each layer employs a GCN aggregator and uses 0. 5 dropout rate to prevent overfitting. Finally, we use a linear transformation layer for classification. All models use cross-entropy as the loss function, ReLU as the activation function between layers, and Adam as the optimizer with an initial learning rate of 0. 001. We train all models for 200 epochs and select the best models with the highest validation accuracy. All models above follow the design specifications outlined in the respective papers. Query Response (R). For the node embedding H, we fix the sizes to three commonly used values, i. e. , 64, 128, and 256. For the node predicted posterior probability Θ Θ Θ, the dimension sizes (i. e. , the number of classes) are dataset dependent and outlined in Table  3. For the node projection ϒ ϒ ϒ, we use t-SNE to project the node embedding H into 2-dimensional vectors, which cover most of the data visualization cases.
e. , the number of correct predictions made divided by the total number of predictions made) as our evaluation metric of theft adversary. Accuracy has been dominantly used in evaluating node classification performance of GNNs  [25, 40, 73]. Recall that the goal of reconnaissance adversary is to closely match the behavior of the target model (see Section 3), we use fidelity (i. e. , the number of predictions agreed by both M S and M T ) as the second evaluation metric of our attack  [34, 38]. Both metrics are normalized between 0 and 1. Higher scores imply better performance. Runtime Configuration. Note that we have 27 different combinations for each response in each dataset (i. e. , combinations of three target models, three responses, and three surrogate models). All the experiments in this paper are repeated 5 times.
Performance Evaluation: Type I Attacks We first summarize the accuracy of the target models for the original node classification tasks in Table  4. We can observe that all GNN models achieve good performance on all datasets, which demonstrates that jointly considering node features and graph structure are effective for classification. We then show the accuracy and fidelity of Type I attacks in Table  5. Due to space limitations, we only show the attack results when the adversary uses GraphSAGE as the surrogate model. The performance results using GIN and GAT as the surrogate models follow similar patterns and can be found in Appendix A. Accuracy. As we can see from Table  5 , Type I attacks can build surrogate models close to the target models given the response is predicted posterior probability or node embeddings (i.
e. , Type I. 1 and Type I. 2 attacks respectively). Take the Pubmed dataset as an example, the target models (i. e. , GIN, GAT, and GraphSAGE) respectively achieve 0. 924, 0. 905, and 0. 909 accuracy scores (see Table  4 ), while the surrogate models can consistently achieve at least 0. 869 Figure  2 : Heatmap of the accuracy scores of Type I attacks. We show the performance results of 9 combinations of surrogate and target models given different response information. The accuracy differences (in parenthesis) of the surrogate models to the target models are also reported. We fix the dataset to Pubmed. accuracy score (see Table  5 ). This represents an approximately 0. 04 accuracy score drop in all cases compared to the target models. We can also observe that Type I attacks can build surrogate models that offer usable accuracy even the response is a 2-dimensional t-SNE projection matrix (i.
e. , Type I. 3 attack). For instance, on the Pubmed dataset, the surrogate models achieve 0. 823, 0. 743, and 0. 844 accuracy scores respectively. This represents a 0. 162 accuracy score drop in the worst case when the target model is GAT. For the rest of the five datasets, we also observe a subtle performance drop in Type I. 3 attacks. Also, such performance drop, compared to Type I. 1 and Type I. 2 attacks, is expected since each t-SNE projection is only a 2-dimensional vector, which leads to additional information loss. Overall, our results show that all three Type I attacks can build usable surrogate models. Given our experimental configuration (i. e. , three target models and three surrogate models), there are 9 different combinations for each response type given a single dataset. Such configuration enables us to understand if the adversary can reliably steal target models in different circumstances.
Using the Pubmed dataset as an example, we summarize the attack accuracy performance in Figure  2. We can see that in general, the adversary can build accurate surrogate models given different combinations of GNN architectures for each response. The subtle performance drop only occurs when GAT is the target model and the response is a t-SNE projection, i. e. , Type I. 3 attack. Even in this case, the adversary can steal usable surrogate models with no more than a 0. 162 accuracy score drop, which indicates our attack remains effective. Our results demonstrate that the adversary does not require knowledge about the architecture of target models to conduct the attacks. To better illustrate it, take the ACM dataset as an example, we extract the embeddings of a given set of nodes from both target and surrogate models and project them into a 2dimensional space using t-SNE.
The result is shown in Figure  3. We use the triangle (cross) to denote the embeddings extracted from the target (surrogate) model and different colors to denote different classes. We find that for both target and surrogate models, the different classes' embeddings can be separated easily. It means that the surrogate model can also successfully map nodes from different classes into different space, which lead to high accuracy. Fidelity. The reconnaissance adversary's motivation is faithfully copying the behavior of the target model. We also summarize the fidelity performance of the surrogate models in Table  5. It is straightforward to see that the better accuracy performance a surrogate model can reach, the better fidelity it can achieve. For instance, on the Coauthor dataset, the surrogate models reach at least 0.
816 accuracy score while these models achieve at least 0. 817 fidelity score to the target models in all 9 cases. The surrogate models for other datasets also follow similar patterns. We then calculate the Pearson correlation coefficient between accuracy and fidelity of the surrogate models given three target models. The coefficient scores are 0. 957, 0. 988, and 0. 959 respectively. The results exemplify that the fidelity of the surrogate models to the target models is highly correlated with their accuracy performance. From Figure  3 , we can also observe that for each class, the embeddings extracted from target and surrogate models lie in the same region, which implies that the surrogate model has the ability to generate the node embeddings that is close to the one generated by the target model, which leads to high fidelity.
Besides, we compare our attacks with Attack-3 proposed by Wu et al. [78]  using the overlapping Pubmed dataset and the results are shown in Table  6. Note that we use fewer data to train the surrogate model compared to Wu et al. [78]. Given SAGE as M S and GIN, GAT, and SAGE as M T , our attack achieves 0. 903, 0. 898, and 0. 924 fidelity scores respectively while their attack only achieves 0. 818 fidelity score. The results exemplify that interacting with the target model can better facilitate the adversary to attain the reconnaissance goal. Stability. We run each combination 5 times with different graph partition seeds. It enables us to measure how widely accuracy/fidelity values are dispersed from the average value (i. e. , standard deviation). A low standard deviation indicates a low volatility.
Performance Evaluation: Type II Attacks To launch Type II attacks, the adversary first builds an adjacency matrix A Q for query graph G Q before querying the target model M T. We follow the aforementioned graph reconstruction configuration to restore A Q for query graph G Q Figure  5 : Heatmap of the accuracy scores of Type II attacks. We show the performance results of 9 combinations of surrogate and target models given different response information. The accuracy differences (in parenthesis) of the surrogate models to the target models are also reported. We fix the dataset to Pubmed. and conduct the model stealing attacks. The performance of Type II attacks is summarized in Table  7. Due to space limitations, we only show the attack results when the adversary uses GraphSAGE as the surrogate model.
The performance results using GIN and GAT as the surrogate models follow similar patterns and can be found in Appendix B. Accuracy. As see in Table  7 , given all the datasets, the adversary can launch Type II. 1/2 attacks to build surrogate models that offer accuracy on par with the target models. At the same time, we observe that the adversary can launch Type II. 3 attacks and steal usable surrogate models. Take the Pubmed dataset as an example, when the response is the t-SNE projection of query nodes, the surrogate models achieve the average accuracy score of 0. 836, 0. 739, and 0. 850 concerning different target models. This represents 0. 166 accuracy drop in the worst case (compared to the target model performance). In general, Type II. 3 attacks can achieve com-parable accuracy performance in all 6 datasets.
Take the Coauthor dataset as an example, the surrogate models reach at least 0. 793 accuracy score while these models achieve at least 0. 801 fidelity score to the target models in all cases. We also calculate the Pearson correlation coefficient between accuracy and fidelity of the surrogate models given three target models. The coefficient scores are 0. 970, 0. 991, and 0. 967 respectively. These correlation scores are similar to what we observe from Type I attack results. Efficacy of Learned Query Graph Structure. To investigate the effect of graph structure, we compare IDGL with two additional methods, i. e. , random graph construction and kNN. For the two additional methods, we set the average node degree to 24, which is the same value used to initialize IDGL. Due to space limitations, we only show the accuracy of Type II attacks on the Citeseer dataset using GAT as the target model and GraphSAGE as the surrogate model.
The accuracy and fidelity of other datasets follow similar patterns. As we can see from Figure  4 , using IDGL to reconstruct the graph structure reaches the highest attack accuracy for all responses. For instance, the accuracy score is 0. 879 when using IDGL as the reconstruction method and taking embedding as the response, while the corresponding accuracy score is only 0. 411 and 0. 737 respectively when using random graph construction or kNN as the reconstruction method. It demonstrates that an effective graph reconstruction method does benefit the final attack performance. Stability. As we can observe in Table  7 , the standard deviation values remain low in all cases. This shows that the adversary can steal the target models with statistically stable accuracy and fidelity in Type II attacks.
Performance Evaluation: Query Budget We then investigate the attack performance with respect to different query budgets, i. e. , different sizes of query graph G Q. Due to space limitations, we only show the results for the Pubmed dataset, other datasets follow similar trends. The corresponding accuracy and fidelity of Type I and II attacks are summarized in Figure  6. We observe that in general, larger query budgets lead to better accuracy and fidelity. Figure  7 : Defense: adding Gaussian noise to t-SNE projection and node embedding. We report the accuracy scores and standard deviation of the surrogate model (GraphSAGE) given all three target models under this defense mechanism (Type I. 1 and I. 3 attacks, respectively). σ is the standard deviation of the Gaussian noise. We fix the dataset to ACM.
For instance, in Type I attack, when the response is Graph-SAGE's embedding, the accuracy score increases from 0. 839 to 0. 870 when the query budget increases from 3% to 27%. However, in most of the cases, we can achieve similar performance even using only 3% of randomly sampled nodes of the original dataset (10% of the G Q used in previous experiments). The results demonstrate that the adversary can still launch effective attacks even with a low-quality query graph (less query budget and no graph structure information). Besides, when the query budget is small, the adversary can investigate the characteristics of query data (e. g. , sparsity of the learned graph) and decide if edge reconstruction is necessary to launch attacks.
A higher accuracy of the surrogate model indicates a more successful attack. We use the ACM dataset as an example and the results are shown in Figure  7. We observe that the random Gaussian noise slightly affects the accuracy performance of the surrogate model. For instance, in Figure  7a , when σ (i. e. , the standard deviation of the added noise) is greater than 7, we can observe the accuracy performance of the surrogate model starts to decrease for different target models. In contrast, if the query response from the target model is the node embedding vector, we can only observe much fewer fluctuations of the surrogate model's accuracy with increasingly stronger random Gaussian noise injected to the embedding (see Figure  7b ). To summarize, our preliminary experiment does not establish concrete evidence that adding random noise would counter our attacks.
There exist some preliminary work on model stealing attacks against GNNs  [16, 78]. However, they are only focusing on the transductive setting of GNNs, which cannot generalize to unseen data. Our attacks instead focus on a more popular and general setting of GNNs, i. e. , inductive setting. We fill the gap and understand if both theft and reconnaissance adversaries can steal inductive GNNs with high accuracy and high fidelity. Causative Attacks on GNNs. Many adversarial attacks to GNNs  [2, 14, 19, 49, 67, 68, 84, 89, 90, 97]  are causative attacks  [31]. These attacks assume that an adversary can manipulate the training dataset in order to change the parameters of the target model and influence their behavior. In this context, Zügner et al. [96]  was the first to introduce unnoticeable adversarial perturbations of the node's features and the graph structure.
Conclusion In this paper, we perform the first security risk assessment against inductive GNNs through the lens of model stealing attacks. We propose a threat model to systematically categorize an adversary's background knowledge into two dimensions, i. e. , query graph and model responses. By jointly considering the two dimensions, we summarize six attack scenarios. We then propose a general attack framework that can be applied in different scenarios. Extensive experiments on three popular inductive GNN architectures and six benchmark datasets show that our model stealing attacks can handle different types of responses and achieve strong performance. Moreover, the attacks are still effective even the adversary has no knowledge about the graph structural information. for our model stealing attack since we use inductive GNNs as the part of our surrogate models.
At the same time, our attack is in a full black-box setting. As such, we evaluate the impact of three hyperparameters on the performance of surrogate models, i. e. , hidden unit size, number of epochs, and batch size. For each hyperparameter, we only show the results on one dataset under Type I attacks given the space limitation. Other datasets follow a similar trend. Hidden Unit Size. In general, the larger the hidden units, the greater space of representation functions a graph convo-lutional layer can offer. As such, we investigate how the hidden unit size may affect the attack performance. Recall that the adversary does not know the target model's hidden unit size. They can only blindly guess the hidden unit size used by the target model. To this end, we first build a target model (i. e.