Intruding with Words: Towards Understanding Graph Injection Attacks at the Text Level

Authors: runlin  lei
yuwei  hu
yuchen  ren
zhewei  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:11:43

Abstract: Graph Neural Networks (GNNs) excel across various applications but remain vulnerable to adversarial attacks, particularly Graph Injection Attacks (GIAs), which inject malicious nodes into the original graph and pose realistic threats.
Text-attributed graphs (TAGs), where nodes are associated with textual features, are crucial due to their prevalence in real-world applications and are commonly used to evaluate these vulnerabilities.
However, existing research only focuses on embedding-level GIAs, which inject node embeddings rather than actual textual content, limiting their applicability and simplifying detection.
In this paper, we pioneer the exploration of GIAs at the text level, presenting three novel attack designs that inject textual content into the graph.
Through theoretical and empirical analysis, we demonstrate that text interpretability, a factor previously overlooked at the embedding level, plays a crucial role in attack strength. 
Among the designs we investigate, the Word-frequency-based Text-level GIA (WTGIA) is particularly notable for its balance between performance and interpretability. 
Despite the success of WTGIA, we discover that defenders can easily enhance their defenses with customized text embedding methods or large language model (LLM)--based predictors. 
These insights underscore the necessity for further research into the potential and practical significance of text-level GIAs.

Main paper: 
 introduction: 
 Graph Neural Networks (GNNs) have exhibited exceptional performance in various tasks [30, 34] . However, their vulnerability to graph adversarial attacks has been increasingly exposed. Research shows that Graph Modification Attacks (GMAs), which perturb a small portion of edges and node features of the original graph, can significantly degrade the performance of GNNs [37, 38] . Besides GMAs, Graph Injection Attacks (GIAs) generate harmful nodes and connect them to the original nodes, thereby reducing the performance of GNNs. Since adding new content to the public dataset is more practical than modifying existing content, GIAs are considered one of the most realistic graph adversarial attacks [28, 33] .Text-Attributed Graphs (TAGs), characterized by nodes with text-based features, form a crucial category among various graph types. In the early stages of GNNs, textual node feature processing is decoupled from the network design. Researchers transform text into fixed embeddings to investigate GNN designs on downstream tasks [13, 8, 27] . However, recent advancements have shifted toward a non-decoupled framework that integrates raw text into the design, significantly improving the ability to capture textual characteristics and enhance performance [9, 10] . Despite this progress in GNN design, current GIAs still focus on fixed embedding, aiming at only the injection of node embeddings rather than raw text on TAGs. This leads to several practical issues: 1) The attack setting is unrealistic. In real-world scenarios like social or citation networks, it is more practical to inject nodes with raw text rather than node embeddings. For example, to attack a citation network dataset, it would be more natural to upload fake papers rather than a batch of embeddings.2) The injected embeddings may be uninterpretable. As minor perturbations at the embedding level can lead to unpredictable semantic shifts, traditional GIAs fail to ensure that injected nodes convey understandable semantic information. 3) Increased Detectability of Attacks. Attackers usually have access only to raw text, not the processed embeddings defenders use. This discrepancy makes it challenging for attackers to mimic the embedding methods of defenders, leading to structurally abnormal embeddings. For example, embeddings based on word frequency are sparse, while those from Pre-trained Language Models (PLMs) are typically output continuous and dense. If an attacker injects dense embeddings while a defender uses a word-frequency-based model, the difference in the embedding structure makes the injected embeddings easy to detect. To address the above limitations, in this paper, we innovatively explore text-level GIAs, comprehensively examining their implementation, performance, and challenges. We explore three text-level GIAs: Vanilla Text-level GIA (VTGIA), Inversion-based Text-level GIA (ITGIA), and Word-frequency-based Text-level GIA (WTGIA), all of which successfully inject raw text into graphs and degrade the performance of GNNs, as shown in Figure 1b . Experiment results indicate that interpretability presents a significant trade-off against attack performance. ITGIA struggles to invert embeddings to interpretable text because the interpretable regions of the injected embeddings are ill-defined. VTGIA, which solely relies on Large Language Models (LLMs) for text generation, compromises attack effectiveness for its pursuit of text interpretability. WTGIA, as a balanced approach, manages to produce harmful and coherent text more effectively. However, the trade-off between performance and interpretability, coupled with poor transferability to different embedding methods, limits the performance of text-level GIAs in practice. Moreover, LLM-based predictors are shown to significantly enhance defenses against text-level GIAs, highlighting the substantial progress still needed to refine these attacks. In summary, our contributions are as follows:• To the best of our knowledge, we are the first to unveil the dynamics of text-level graph adversarial attacks on TAGs, especially for GIAs. We discuss the impracticality of embedding-based GIA in real-world applications and propose a more realistic attack setting.• We propose three effective text-level GIAs and demonstrate the trade-off between attack performance and text interpretability from both theoretical and empirical perspectives, providing insights for further refining text-level GIAs.• We reflect on the challenges of graph adversarial attacks at the text level. We discover that simple strategies at the text level can enhance defense performance against text-level GIAs, which reemphasizes the importance of exploring GIAs at the text level. 
background and preliminaries: 
 A Text-Attributed Graph (TAG) is represented as G = (V, E, {s i }), where V is the node set and E, the edge set. We denote N as the number of nodes and A ∈ {0, 1} N ×N as the adjacency matrix.For each node v i ∈ V, a sequential text feature s i is associated. We focus on semi-supervised node classification tasks, where each node is categorized into one of C classes. The node labels are denoted by y ∈ {0, . . . , C -1} N . The task is to predict test labels Y test based on G and training labels Y train .Graph Neural Networks. We use f θ to denote a GNN parameterized by θ. Generally, raw text {s i } is first embedded into a feature matrix X ∈ R N ×F using word embedding techniques, where F is the output dimension. In the rest of our paper, when referring to vanilla GNNs that take text embeddings as inputs, we represent the text-attributed graph as G = (V, E, X) and express f as f (X; A). The update process of the l-th layer of f can be formally described as:EQUATIONwhere h 0 i = X i , N i is the neighborhood set of node v i , AGGR is the aggregation function, and f l is a message-passing layer that takes the features of v i and its neighbors as inputs.Graph Injection Attacks and Related Works. Graph injection attacks aim to create a perturbed graph G ′ = (V ′ , E ′ , X ′ ) to disrupt the accuracy of a GNN on target nodes. Formally, denote the number of injected nodes as N inj , and the number of nodes in the poisoned graph G ′ as N ′ . Then, the node set of G ′ can be formulated as V ′ = V ∪ V inj , and:EQUATIONwhere X inj ∈ R Ninj×F , A inj ∈ {0, 1} N ×Ninj , and O ∈ {0, 1} Ninj×Ninj .For the design of injected node features, one line of research focuses on statistical features to ensure unnoticeability, sacrificing attack strength by excluding learnable processes for features [25, 4] . In parallel, other works target learnable injected embeddings. Wang et al. [29] pioneer injecting fake nodes with binary features to degrade GNN performance. AFGSM [28] introduces an approximation strategy to linearize the surrogate model, efficiently solving the objective function.Starting from KDDCUP 2020, research shifts towards injecting continuous node embeddings under the constraints in Equation (3) [33, 36, 2] . Denote the budget of injected nodes as ∆, the maximum and minimum elements in the original feature matrix X as x max and x min , and the maximum degree allowed for each injected node as b. The objective of these GIAs can be formulated as:EQUATIONwhere V T is the target node set, d u is the degree for node u, and X 1 ≤ e X 2 indicates each element of X 1 is less than or equal to the corresponding element in X 2 . Among continuous strategies, TDGIA [36] generates node features by optimizing a feature smoothness objective function. Chen et al. [2] enhance homophily unnoticeability by adding a regularization term to the objective function. These approaches show better performance while maintaining unnoticeability at the embedding level.It is worth noting that the evaluation datasets for the aforementioned GIAs are primarily TAGs [33] , e.g., such as Cora and CiteSeer. However, all the studies remain at the embedding level. In our experiments, we adhere to the structural budget detailed in Equation ( 3), omitting embedding-level constraints as we pioneer GIAs at the text level. More related works are provided in Appendix D.Evaluation Protocol. In this paper, we explore GIAs in an inductive, evasion setting, following the framework established by [33] to ensure closeness to practical scenarios. Attackers have access to the entire graph and associated labels, except for the ground-truth labels of the test set and the victim models of the defenders. l Defenders train their models on clean, unperturbed training data and then use trained models to classify unseen test data.General Experiments Set-up. We conduct experiments on three TAGs, Cora [17] , CiteSeer [6] and PubMed [23] , which are commonly used in evaluating GIAs. The dataset statistics and attack budgets are provided in Table 5 and Table 7 in the Appendix. We adopt full splits in [33] for each dataset, which uses 60% nodes for training, 10% nodes for validation, and 30% for testing. For word embeddings techniques, we include Bag-of-Words (BoW) and GTR [19] , a T5-based pretrained transformer that generates continuous embeddings of unit norm. The default BoW method constructs a vocabulary using raw text from the original dataset based on word frequency, with common stop-words excluded. The embedding dimension of BoW is set to 500. In the experiments, we use a 3-layer vanilla GCN or a 3-layer homophily-based defender EGNNGuard following [2] . EGNNGuard defends homophily-noticeable attacks by cutting off message-passing between nodes below the specified similarity threshold. The hyperparameters and training details are provided in Appendix H.1. We repeat each experiment 10 times and report the average performance and the standard deviation. Unless otherwise specified, we bold the best (the lowest) results. 
text-level gias: interpretability matters: 
 Although GIAs are considered realistic graph attacks, traditional methods focus mainly on the embedding level, which can be overly idealized. Typically, attackers only have access to public raw data, such as paper content and citation networks, on platforms like arXiv, and defenders process this data more personally. Since attackers struggle to access embeddings trained by defenders, injecting nodes with suitable embeddings is challenging. Conversely, injecting new raw data into the public dataset is more realistic for attackers. This leads us to an important question: Can we generate coherent text for injection that ensures the effectiveness of attacks while maintaining interpretability?Embedding-level GIAs have demonstrated strong attack performance against GNNs [36, 2] . To achieve text-level attacks, a straightforward idea is to use inversion methods to convert embeddings back into text. Recent advancements in text inversion methods, such as Vec2Text [18] , have enabled text recovery from PLM embeddings with a 66% success rate for examples averaging 16 tokens. This breakthrough opens new possibilities for achieving text-level GIA that is comparable in performance to those at the embedding level, which we term Inversion-based Text-level GIA (ITGIA).Implementation of ITGIA. ITGIA consists of two steps: generating poisoning and invertible embeddings, and then converting them into text. Since the effectiveness of current inversion methods has only been validated on normal text embeddings, it is crucial to ensure that generated embeddings lie within an interpretable and feasible region. However, defining these interpretable regions precisely is challenging. The constraint specified in Equation (3) limits generated embeddings to a cubic space, which may be overly broad for feasible embeddings. For example, PLMs typically produce embeddings located on a spherical surface with an L2-norm of 1, representing a significantly smaller region. To meet this fundamental requirement, we employ Projected Gradient Descent (PGD) [16] , projecting each injected embedding onto the unit sphere after every feature update. To further address the challenge of defining interpretable regions, we incorporate Harmonious Adversarial Objective (HAO) [2] into the GIA objective function. Optimizing over HAO increases the similarity between injected embeddings and those of normal text from original datasets, thereby bringing the injected embeddings closer to interpretable regions. Details of ITGIA and HAO are provided in Appendix I.1 and H.1. Once the embeddings are obtained, we utilize the GTR inversion model in [18] with a 20-step correction process to revert embeddings back into text.Analysis of Performance and Interpretability. Following [2] , we use sequential injection frameworks, including the sequential variants of SeqGIA (Random injection), TDGIA, ATDGIA, MetaGIA, and AGIA as the embedding-level backbones. The results of ITGIA are displayed in Table 1 . We can see the embeddings after inversion deviate significantly from the injected embeddings, as indicated by the low cosine similarity. This suggests substantial information loss during inversion, leading to poor attack performance. Figure 6 illustrates the underlying reason: although injected embeddings meet basic norm constraints, they struggle to fall within the much smaller interpretable embedding region, hindering accurate inversion. Although incorporating HAO enhances the inversion accuracy and improves attack performance, the improvement comes at the expense of embedding-level attack performance [2] . As a result, increasing HAO weights does not consistently yield better results, as shown in Figure 2 . Furthermore, the generated text remains incoherent with high perplexity, as demonstrated in Appendix J. This renders real-world applications impractical and highlights the challenges faced by continuous embedding-level GIAs. Although ITGIA manages to generate harmful text, interpretability remains challenging to address.To ensure that the generated text is both interpretable and deceptive, a direct approach is to utilize an LLM for text generation and guide it to produce poisoning content through carefully designed prompts. Based on experiences with embedding-level graph adversarial attacks, when the features of injected nodes exhibit heterophily, confusion, or irrelevance, the classification performance of their neighboring nodes is negatively affected [35] . Thus, we attempt to incorporate prior knowledge used in traditional GIAs and create prompts from three different angles:Heterophily Prompt. This prompt involves sampling the text of nodes from the original dataset to generate heterophilic (dissimilar) content. Specifically, we first conduct embedding-level GIAs to obtain a perturbed graph. Next, we sample the content from the neighbors of injected nodes. Finally, we design a prompt that requests the generated content to be dissimilar to their neighboring nodes, exploiting the weakness of GNNs in handling heterophily.Random Prompt. This prompt generates node text that is entirely unrelated to the original categories of the graph. For example, in the Cora dataset, the original categories are about machine learning topics. In this prompt, we generate papers in categories like sports, art, and history, which are dissimilar to all nodes in the original graph.Mixing Prompt. This prompt generates node text that may be classified into multiple or even all categories. Using Cora as an example, a potential paper title could be: "Unified Approach for Solving Complex Problems using Integrated Rule Learning, Neural Networks, and Probabilistic Methods".After defining the prompts, we propose a simple Vanilla Text-level GIA (VTGIA). This method first generates text for the injected nodes based on the prompt via LLMs and subsequently optimizes the graph structure. We use GPT-3.5-1106 [20] as the LLM backbone for text generation, and the five sequential injection backbones same as ITGIA. The pseudo-code of VTGIA, prompts, and generated examples are provided in Appendix I.2, H.2, and J. The results are shown in Table 2 . Although the generated content exhibits adversarial effects, directly producing harmful text via prompts is not sufficiently effective and is far from the optimal results achievable by attacks at the embedding level. Based on previous analysis, to achieve an interpretable and effective GIA, the following conditions should be met: 1) Guidance from Embedding-level GIA to ensure effectiveness. 2) Utilization of LLMs to guarantee interpretability. 3) A well-defined LLM task formulation to minimize information loss during the embedding-text-embedding conversion process.We address these conditions by employing word-frequency-based embeddings, specifically binary Bag-of-Words (BoW) embeddings, which offer several advantages: 1) Clear physical meaning. In binary BoW embeddings, a '1' indicates the presence of a word, while a '0' indicates its absence.2) Clear task formulation. The task involves generating text containing specified words while excluding prohibited words, which can be effectively done via LLMs. 3) Controllable embedding-textembedding process. As long as the text includes specified words and excludes prohibited words, the embeddings can be exactly inverted. Building on these advantages, we design the Word-frequencybased Text-level GIA (WTGIA), leveraging BoW embeddings and the generative capabilities of LLMs. Its implementation consists of three steps: 1) Obtain binary injected embeddings. 2) Formulate embedding to text tasks for LLMs. 3) Perform multi-round corrections for LLMs. 
row-wise constrained fgsm: unnoticeable and effective at the embedding level: 
 Implementaion of Row-wise Constrainted FGSM. To generate binary embeddings, we adopt a row-wise constrained Fast Gradient Sign Method (FGSM) algorithm based on [28] . Specifically, we first obtain vocabulary based on BoW on the original dataset. Next, we set the injected embedding X inj to all zeros. We then set a sparsity constraint S on injected embeddings, that each embedding can retain at most F ′ = ⌊S * F ⌋ non-zero entries, where F is the dimension of features. In each epoch, we define the flippable set as the entries in X inj that satisfy the sparsity constraint. Based on the element-wise gradient of the loss concerning X inj , we flip the most significant B entries from the flippable set, where B is the batch size. The process stops when all rows run out of word budgets. 2Note that the sparsity budget S is a hyper-parameter that limits the use of at most F ′ words from the predefined vocabulary in a single text. Intuitively, a higher number of used words intensifies the embedding-level attack but can compromise text-level interpretability, as it becomes harder to integrate more specified words naturally in the generated text. Based on the row-wise constraints, we can formally establish the relationship between embedding-level performance and unnoticeability and text-level interpretability.Definition 1 (Single node GIAs towards one-hot embedding). For a target node v t with embeddingx t = [ k 1, • • • , 1, 0, • • • , 0] ∈ {0, 1} F , k ≪ F, the embedding indicates that the node uses k words from a predefined vocabulary, while F -k words are not used. We define the set containing the specified k words as set W u and the set containing the F -k prohibited words as set W n . We assume words in W u are related to their corresponding class, whereas words in W n are associated with other classes. To attack node v t , an adversarial node v i with embedding x i ∈ {0, 1} F is injected. • Performance: An attack is more effective against node v t if it uses more words w n ∈ W n . The intuition is that mixing words from other classes makes x t harder to be correctly classified.• Unnoticeability: An attack is less detectable if the similarity between x t and x i is higher.• Interpretability: An attack is more interpretable if fewer words are required to use in x i , which means x i • 1 is smaller. The intuition is that generating content with fewer specified words is easier, which fits in practical scenarios involving limited length. Theorem 1. In the setting outlined in Definition 1, assume we apply a cosine similarity constraint with a threshold c ∈ (0, 1) for unnoticeability. Specifically, this constraint requires that the cosine similarity between x t and x i satisfies xt•xi ∥xt∥∥xi∥ > c. Let a denotes the number of words used by x i from the set W u , and b denotes the number of words used by x i from W n . If the budget is m words at most to ensure interpretability, then the maximum value of b is max(b) = max ⌊(m -c √ mk⌋, 0 .The proof and detailed illustration for intuitions are provided in Appendix F. Theorem 1 reveals that as long as m ≥ k, we can find a positive value for b (disregarding the flooring operation for simplicity), effectively introducing harmful information to the target node. Practically, given the typically low value of c, a destructive x i can be easily identified with a relatively large m at the embedding level.In Figure 3 , we present the average and best performance of Row-wise Constrained FGSM coupled with five sequential injection methods used in ITGIA and VTGIA against GCN and EGNNGuard.As the sparsity budget increases, the performance of EGNNGuard declines sharply, confirming that enhancements in performance are possible under the unnoticeability constraint. However, achieving significant performance degradation requires sacrificing interpretability. For example, to achieve a 10% performance drop in Cora and CiteSeer, FGSM attacks need a budget close to or exceed the maximum sparsity level of all texts belonging to the original datasets. The subsequent subsection will explore the consequences of sacrificing text-level interpretability while completing WTGIA.Task Formulation of LLMs. After obtaining the binary injected embeddings, we can obtain the Specified words included and the Prohibited words excluded in each text based on the BoW vocabulary. Subsequently, we formulate the task for LLMs as a "word-to-text" generation problem. We request LLMs to generate text under a specified length limit using specified words, optionally within a given topic based on the origin dataset. 3 Handling prohibited words is more complex. Given the sparsity of BoW embeddings, the number of prohibited words is relatively large. Directly requesting the absence of prohibited words through the prompt would be challenging for LLMs to understand. So, for closed-source LLMs like GPT, we only constrain the specified words in the prompt. For open-source LLMs, we mask their corresponding entries during the output process.Masking Prohibited Words. Denote t i+1 as the next token at time i. The standard generative process for LLM can be formulated as:t i+1 = argmax (SoftMax (logits(t i+1 | t i , t i-1 , . . .))) .where logits(t i+1 | t i , t i-1 , . . .) are the unnormalized log probabilities of the potential next token given the previous tokens. To enforce constraints on avoiding prohibited words, we introduce a masking vector M such that: M [j] = 0 if token j is prohibited, and 1 otherwise. The generative process is then modified by applying this mask to the logits before the SoftMax operation:t i+1 = argmax (SoftMax (logits (t i+1 | t i , t i-1 , . . .) ⊙ M )), where ⊙ represents the element-wise multiplication of the logits by the mask. This mask effectively removes words from outputs by making their probabilities negligible.Multi-Round Correction. After obtaining the output from the LLM, we calculate the use rate Number of Specified Words used Number of Specified words in the generated text and correct any omissions in their usage. Through multiple rounds of dialogue, we select the text with the highest use rate as the final output for LLMs.Main Results. We utilized GPT3. 5-turbo-1106 [20] as the close-source LLM and Llama3-8b [26] as the open-source LLM to implement the WTGIA. Based on Row-wise FGSM in Figure 3 , we combine WTGIA with the five injection strategies and report the average results. The specific prompt details are in Appendix H.2. The length limit is specified in Table 6 . The variants that use prompts that specify the generated content must belong to the original class set are marked as "-T." The results are displayed in Figure 4 . It can be seen that • Desirable Performance: WTGIA achieves comparable attack performance to Row-wise FGSM, demonstrating strong embedding-to-text capabilities.• Trade-off: Variants with topic specification perform worse than those without, sacrificing attack strength for better text-level unnoticeability.• Masking Helps: Llama-WM and GPT that do not mask prohibited words generally perform worse than Llama.Complete experiments and full results are given in Appendix K and Appendix L. Overall, WTGIA effectively replicates the performance of FGSM at the embedding level, achieving both effectiveness and interoperability. New Trade-offs. However, we fail to further enhance performance at the text level by increasing the sparsity budget, similar to FGSM at the embedding level. As shown in Figure 5 , with the increasing sparsity budget, the use rate keeps decreasing, representing a trade-off to maintain the interpretability of the generated content. In Cora and CiteSeer, the text-level attacks perform best when sparsity is around 10%. In PubMed, GPT-Topic even fails to generate meaningful text with a sparsity budget S ≥ 15%. These observations confirm that in text-level attacks, interpretability represents an additional trade-off that is overlooked in embedding-level attacks. Enhancing or evaluating performance solely at the embedding level fails to provide a practical understanding of GIAs. 5 New Challenges for Text-level GIAs 
transferbility to different embeddings: 
 In practical scenarios, defenders can employ any text embedding techniques they want to process the text once an attacker completes the injection. However, embedding-level attacks implicitly assume that defenders will use the same embeddings as the attackers during evaluation, potentially benefiting the attackers. In fact, it is crucial to fully consider the transferability of a text attack across different text embedding technologies for text-level attacks.In Table 3 , we present the results of ITGIA and WTGIA when transferred to different embeddings on the Cora dataset. When transferred to GTR embeddings, WTGIA does not exhibit a significant performance advantage over ITGIA. The performance of ITGIA when transferred to BoW embeddings is even poorer. Since the injected text is highly uninterpretable and deviates significantly from the original dataset, the BoW method inherently filters out uncommon terms, rendering the injected embeddings particularly sparse and ineffective. These limitations in embedding transferability underscore the research gaps between embedding-and text-level GIAs. Since attacks are text-level, the defenders can directly utilize LLMs as predictors for node classification tasks without necessarily employing GNNs. Taking the LLMs-as-Predictor from [3] as an example. The method feeds the formulation of the node classification task, texts of target nodes, and optionally, the sampled texts of their neighbors in the prompt. It then utilizes LLMs to predict the labels of target nodes directly. Remarkably, this method demonstrates superior performance on clean datasets. We now examine the performance of WTGIA against LLMs-as-predictor from [3] . For WTGIA, we select the strongest variant in Figure 4 for each dataset as the attacker. In the zero-shot setting, only the text of target nodes is fed to LLMs. In the few-shot setting, the text of labeled nodes with their labels is fed to LLMs as examples. On Cora and CiteSeer, we adopt the whole test set for evaluation. For PubMed, we sample 1000 nodes from the test set for evaluation.The results are presented in Table 4 . Remarkably, on the PubMed dataset, the baseline without the use of neighborhood information ("Clean (w/o Nei)") achieves outstanding performance, which is consistent with observations in [3] . Therefore, even without utilizing any neighborhood information, LLM-based predictors can achieve high accuracy in node classification, meaning defenders can evade the influence of injected nodes. For other datasets, Cora and CiteSeer, while incorporating neighborhood information enhances the performance of LLMs-as-Predictors, the "Clean (w/o Nei.)" baseline still represents a practical upper limit for the GIAs. Even if attackers manage to degrade the performance below this baseline, defenders can turn to this neighbor-free method as a secure fallback. Consequently, the flexibility of defenders should be fully considered when evaluating GIAs at the text level in practical scenarios. 
conclusion: 
 In this paper, we explore the design of GIAs at the text level. We address the limitations of embeddinglevel GIAs on TAGs by extending them to the text level, which better aligns with real-world scenarios.We present three types of text-level GIA designs: ITGIA, VTGIA, and WTGIA. Our theoretical and empirical analysis reveals a trade-off between attack performance and the interpretability of the injected text. Among these methods, WTGIA achieves the best balance between performance and interpretability. Additionally, our findings indicate that defenders can effectively counter these attacks using different text embedding techniques or LLM-based predictors, highlighting the complex and challenging nature of graph adversarial attacks in practical applications. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We have included resources information in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this paper, we potentially aid in developing more advanced and sophisticated attacks through understanding text-level vulnerabilities and providing significant positive impacts by enhancing the design of robust defense mechanisms. By pioneering the exploration of text-level GIAs, we enable the creation of more effective countermeasures to protect against these advanced threats, thereby improving the overall security and resilience of systems that rely on GNNs. This dual impact highlights the importance of our work in driving both offensive and defensive advancements, ultimately contributing to safer and more secure AI applications.GPT-turbo: https://openai.com/. In this paper, we use the GPT- All our experiments are conducted on a machine with an NVIDIA A100-SXM4 (80GB memory), Intel Xeon CPU (2.30 GHz), and 512GB of RAM.Graph Injection Attacks. Graph Injection Attacks aim to decrease the performance of GNNs by injecting suspicious nodes into the original graph. The newly injected nodes are allowed to process fake features and form connections with existing nodes on the graph. Wang et al. [29] first attempt to inject fake nodes to degrade the performance of GNNs. NIPA [25] applies reinforcement learning to make graph injection decisions. AFGSM [28] uses an approximation strategy to linear the model and solve the objective function efficiently. TDGIA [36] selects edges for newly added edges based on a topological defective edge selection strategy. It then generates node features by optimizing a proposed feature smooth objective function. Chen et al. [2] add a regularization term to the existing objective function for GIA to improve homophily unnoticeability. GANI [4] employs a statistics-based approach to calculate node features, ensuring the generated features remain unnoticeable but leading to a significant decline in performance. All the GIAs mentioned are designed at the embedding level. Despite some constraints being applied to the injected features, the semantic coherence and logical consistency of generated features at the text level remain unexplored.Graph Modification Attacks. Graph Modification Attacks aim to decrease the performance of GNNs by modifying the graph structure or node features of the original graphs. Nettack [37] adopts a greedy strategy to select edge and node feature perturbations with unnoticeability constraints. Zügner et al. [38] treat the graph structure as a hyperparameter and use meta-gradients to solve the bilevel poisoning attack objective. Xu et al. [31] use Projected Gradient Descent (PGD) to find the best perturbation matrix. The above algorithms significantly degrade the performance of GNNs, yet all require O(N 2 ) space complexity. To address the scalability issues, Geisler et al. [5] propose PRBCD and GRBCD using Randomized Block Coordinate Descent. The algorithms are of O(|E|) time and space complexity and can be scaled up to datasets containing more than 1 million nodes.Raw Text learning on TAGs. With the development of LLMs, works combining them with GNNs have continuously emerged, showing outstanding performance in various graph-related tasks on TAGs. TAPE [9] uses LLMs as a feature enhancer to obtain high-quality node features. In Instruct-GLM [32] , the authors propose scalable prompts based on natural language instructions that describe the topological structure of a graph. Guo et al. [7] analyze the ability of LLMs to understand graph data and solve graph-related tasks. Chen et al. [3] conduct a benchmark evaluating the performance of LLMs as the feature enhancer and the predictor, respectively. More related works about LLMs in Graph could be referred to [15, 24, 14] . Although LLMs have seen numerous applications in graph tasks, works exploring the robustness of GNNs are still sparse.Although Graph Modification Attacks are powerful attacks, recent works point out that they do not align well with real-world attack scenarios [33] , and they are theoretically inferior to GIAs in attack performance [2] . Moreover, while raw texts are proven to be helpful in improving the performance of graph tasks, works exploring the robustness of GNNs at the text level are still sparse.Dataset information is summarized in Table 5 and Table 6 . The budget of GIAs is summarized in Table 7 . F Further Illustration and Proof to Theorem 1The formulation for Theorem 1 relies on the below intuitions.• Performance: An attack is more effective against node v t if it uses more words w n ∈ W n .• Unnoticeability: An attack is less detectable if the similarity between x t and x i is higher.• Interpretability: An attack is more interpretable if fewer words are required to use, where the x t • 1 is smaller.In the intuition behind performance, we actually believe that words beneficial to the classification of the target node are located in W u , while words detrimental to the classification of the target node are located in W n . However, strictly speaking, not all words in W n have a negative impact on the classification of the target node. Additionally, when a certain amount of classification words is introduced, the negative impact becomes marginal and does not consistently increase. Therefore, using more words w n ∈ W n does not linearly correspond to higher performance.However, this does not affect the insight provided by Theorem 1. Since k ≪ F , the continuous increase in the usage of w n ∈ W n within a certain range is significant. Since we do not consider situations where m ≫ k, we indeed focus on "uses more words w n ∈ W n in a certain range" intrinsically. Thus, under this premise, the theorem's description of performance remains applicable.Proof. The problem is formulated as:EQUATIONEQUATIONFrom inequality (6) , we obtain:a 2 -c 2 ka -c 2 kb ≥ 0,which leads to:a ≥ c ck + k(4b + c 2 k) 2 or a ≤ c ck -k(4b + c 2 k) 2 < 0.Since a is positive, the only feasible solution is:a ≥ c ck + k(4b + c 2 k) 2 .By substituting this into equation ( 5), we obtain:b ≤ m -a ≤ m - c ck + k(4b + c 2 k) 2 .Solving the resulting inequality with respect to b, we find:b ≤ m -c √ mk.Hence, the solution to the maximization problem is:max(b) = m -c √ mk .The illustration of interpretable regions is provided in Figure 6 .Feasible region (2) In addition to the methods mentioned in the main body of the paper, we also considered using the Riemannian Gradient Descent as a way to obtain the embedding for ITGIA. This method performs gradient descent in the tangent space of the sphere to ensure the resulting embedding's normality.Formally, to solve the optimization problemEQUATIONusing Riemannian gradient descent, we follow these steps:Initialization: Start with an initial point x 0 such that ∥x 0 ∥ = 1.Gradient Calculation: At each iteration k, compute the Euclidean gradient ∇f (x k ).Projection onto Tangent Space: Project the Euclidean gradient onto the tangent space of the sphere atEQUATIONwhere ∇ R f (x k ) is the Riemannian gradient.Step: Move in the direction of the Riemannian gradient and re-project onto the sphere:EQUATIONwhere α k is the step size and Proj S (y) = y ∥y∥ ensures that the updated point lies on the sphere. Iteration: Repeat steps 2-4 until convergence.The PGD method we adopted can be regarded as the variant of Riemannian Gradient Descent without the "Projection onto Tangent Space" step. Although the Riemannian Gradient Descent has better theoretical properties [1] , since we did not observe significant advantages over PGD in our experiments, we opted for the simplicity of using PGD, as discussed in the main body of our paper.The co-occurrence constraint in [28] requires that columns that do not appear together in the original dataset must not appear together in the injected embeddings either.Let X be a binary feature matrix with dimensions N × F , where N represents the number of samples and F represents the number of features. Each element x ij in the matrix X is defined as:x ij = 1 if word j is present in texti, 0 otherwise.The co-occurrence matrix C is then an F × F matrix where each element c ij , representing the pair of words i and j, is computed as follows: C = (X T X) > 0Here, X T is the transpose of X, and the multiplication X T X results in a matrix where each element c ′ ij is the sum of the products of corresponding elements of X:c ij = n k=1x ki x kjThe value c ij represents the number of texts where both words i and j are present. The physical meaning of c ij > 1 is that features i and j co-occur in at least one text, indicating a possible association or dependency between them within the dataset. A co-occurrence-constrained FGSM only allows flips that satisfy if word i and word j appear in the injected embeddings, they must havec ij > 1.We examine the sparsity of C in Table 8 . We can see that almost every two words have co-occurred before, making the constraints meaningless.Attacks. For the injected embedding features, we use PGD [33, 2] with a learning rate of 0.01, and the training epoch is 500. An early stop of 100 epochs according to the accuracy of the surrogate model on the target nodes. For TDGIA and ATDGIA, we use the method of [36] to generate injected embeddings. The weights k 1 and k 2 are set 0.9 and 0.1 by default. The sequential step is set as 1.0 for MetaGIA to save computational costs, while 0.20 for methods. For Binary FGSM, the batch size is set to as B = 1 in Cora and CiteSeer, and B = 50 for PubMed. For the hyperparameters of SeqGIA, MetaGIA, and SeqAGIA, we directly follow the setting as in [2] . A simplified description of the Sequential injection framework is provided in Appendix 1, and the FGSM function is provided in Appendix 2.Harmonious Adversarial Objective. The HAO constraint can be formulated as:L = L pred + γL HAO .Here, L pred represents the cross-entropy loss on target nodes, calculated between the current predictions of the surrogate model and the original predictions made on the clean dataset by the same model. The loss term L HAO = sim(X ′ inj , (X ′ A ′ ) inj ), where sim returns the cosine similarity between injected node features and propagated injected node features. We set γ = 10 in SeqGIA, SeqAGIA and MetaGIA, γ = 1 in TDGIA, ATDGIA and set γ = 1e -6 in FSGM, where we empirically find a balance in the attack performance and unnoticeability to escape from homophily-based defender.Note that in Figure 2 , γ is multiplyed by the weights as the final coefficient. Denote γ ′ = γ * w, where w is the weight listed in the x-axis of the figure, we apply L = L pred + γ ′ L HAO as the final loss function.Defense. For both GCN and EGNNGuard, we set the number of layers as 3 and use a hidden dimension of 64. We set the dropout rate as 0.5 and adopt Adam optimizer [12] with a learning rate of 0.01. The number of training epochs is 400. We adopt the early stop strategy with a patience of 100 epochs. The threshold is set as 0.1 following [2] , which we find powerful enough in detecting noticeable injections.Others. The temperature of the GPT-turbo is set as 0.7. In ITGIA, the beam-width is set as 2. In WTGIA, the max-token for gpt-turbo and Llama3 is set to 500 for Cora and CiteSeer and 550 for PubMed. We set the the number of dialogs in multi-round corrections as 3. We set the max-token to avoid over-length generation. Based on the typical ratio of 0.75:1 between the number of words and the number of tokens, we proportionally scale this relationship using the data from Table 6 to obtain the specific numbers for this limitation.Table 9 provides our prompt designs for different prompt strategies of vanilla text-based GIAs. The <adj/example_paper_content> is comprised of paper id, title, and abstract of adjacent/example nodes, and ... denotes fillable blanks for the model to fill in. The <LMIN> and <LMAX> correspond to the length limit for each dataset as specified in Table 6 . Generally, our experiments find that the current instructions allow the LLM to produce output that conforms well to the expected format without significant deviations. The generated content should be able to be classified into any type of paper.Length limit: Min: <LMIN> words, Max: <LMAX> words. Title: ..., Abstract: ...Task: Paper Generation.There are <num_classes> types of paper, which are <category_names>. Generate a title and an abstract for paper <inj_node> that does not belong to any one of the paper types. For example, the paper could belong to ['history', 'art', 'philosophy', 'sports', 'music'] and types like that. Length limit: Min: <LMIN> words, Max: <LMAX> words. Title: ..., Abstract: ... The pseudo-code for sequential node injection strategy is displayed in Algorithm 1. A simple illustration of the feature update function of SeqGIA, MetaGIA, TDGIA, and SeqAGIA is given in Algorithm 3. The pseudo-code for Binary FGSM feature update functions is displayed in Algorithm 2.For FGSM binary attacks, we implement the structure updates in the sequential node injection manner, as it is shown more powerful by previous works [28, 2] . The initialization of X inj is set as X inj = 0 Llama/GPT Generate a title and an abstract for an academic article. Ensure the generated content explicitly contains the following words: <use_words>. These words should appear as specified, without using synonyms, plural forms, or other variants. Length limit: <max_words> words.Output the TITLE and ABSTRACT without explanation. TITLE:... ABSTRACT:...There are <num_classes> types of paper, which are <category_names>. Generate a title and an abstract for paper belongs to one of the given categories.Ensure the generated content explicitly contains the following words: <use_words>. These words should appear as specified, without using synonyms, plural forms, or other variants. Length limit: <max_words> words.Output the TITLE and ABSTRACT without explanation. TITLE:... ABSTRACT:... andOutput :Attacked Graph G ′ = (V ′ , E ′ , X ′ ) Parameter :Sequential step st, Normalization flag N orm; 1 V ′ ← V; E ′ ← E; X ′ ← X 2 n inj ← 0 3 Y orig = f (X, A) 4 while n inj < N inj do 5 n ← min(N inj -N Total , ⌊N inj * st⌋) 6 Initialize V inj , E inj , X inj // Initialization of the current epoch 7 V ′ ← V ′ ∪ V inj ; E ′ ← E ′ ∪ E inj 8 V inj , E inj ← H(X, X inj , V ′ , E ′ , f, Y orig ) // Update Structure 9 V ′ ← V ′ ∪ V inj ; E ′ ← E ′ ∪ E inj 10 X inj ← F(X, X inj , V ′ , E ′ , f, Y orig , N orm) // Update Features 11 X ′ = X ′ ∪ X inj 12 n inj ← n inj + n 13 return Attacked Graph G ′ = (V ′ , E ′ , X ′ )for epoch ← 0 to e do 2 X cat = X ∪ X inj 3 X inj ← Proj X inj -η∇ Xinj L ch f (X cat , A ′ ), Y orig //< F ′ do 3 X cat ← X ∪ X inj 4 Z ← f (X cat , A ′ ) 5 ∇L ← ∇ Xinj L(Z, Y orig ,EQUATIONUpdate the flip counter for row r 15 return X injThe pseudo-code for vanilla text-level GIAs is displayed in Algorithm 4, which is implemented in a sequential manner following SeqGIA. Note that the feature is not updated within the "while loop" since gradient information is infeasible.We calculate perplexity based on GPT2 [21] following https://huggingface.co/docs/ transformers/perplexity. Note that due to the model capacity constraint, the output of IT-GIA is truncated to 32 tokens. So, we truncate all texts into 32 tokens before calculating Perplexity to make them comparable to each other. We present the results of average perplexity, and use rate in WTGIA in Table 11 , Table 12 , Table 13 and Table 14 .In the experiments, VTGIA consistently achieves the lowest perplexity scores, indicating superior performance, followed by the WTGIA model. Conversely, ITGIA exhibits the highest perplexity, rendering it the least interpretable. Notably, even the variant incorporating -HAO fails to enhance interpretability corrsponding to normal texts. Furthermore, our findings suggest a positive correlation between model use rate and perplexity. Specifically, a lower use rate corresponds to reduced perplexity, thereby enhancing interpretability. This observation aligns with the hypothesis that as the use rate decreases, LLMs preferentially select words from a specified set that are more interpretable, rather than striving to utilize all specified words. Output :Attacked Graph G ′ = (V ′ , E ′ , {s i } ′ ) Parameter :Sequential step st, prompt type p 1 V ′ ← V; E ′ ← E 2 if p == Heterophily prompt then /* Integrate attacked graph into prompts */ 3 V ′ , E ′ ← Embedding-based GIAs 4 {s i } inj ← GenerateText(N inj , , G ′ , V ′ , E ′ ) 5 else 6 {s i } inj ← GenerateText(N inj , , G ′ ) 7 {s i } ′ = {s i } ∪ {s i } inj 8 X ′ ← TextEmbedding({s i } inj ) 9 N Total ← 0 /* Graph structure refinement */ 10 while N Total < N inj do 11 n ← min(N inj -N Total , ⌊N inj * st⌋) 12 V ′ , E ′ ← AdaptiveInjection(V ′ , E ′ , X ′ ) 13 N Total ← N Total + n 14 15 return Attacked Graph G ′ = (V ′ , E ′ , {s i } ′ )We show the generated examples on the Cora dataset for VTGIA, one for each prompt type.Heterophily Prompt Example. Title: Reinforcement Learning Techniques for Rule Learning Optimization Abstract: This paper explores the application of reinforcement learning techniques to optimize the process of rule learning. Traditional rule-learning algorithms often struggle with complex datasets and high-dimensional feature spaces, leading to suboptimal rule sets. By leveraging the power of reinforcement learning, we propose a novel approach that dynamically adapts the rule learning process based on feedback from the environment. We start by formulating the rule learning problem as a Markov decision process, where the agent's actions are the selection and refinement of rules based on the input data. We introduce a reward function that incentivizes the discovery of rules with high predictive power while maintaining simplicity and interpretability. Through a series of experiments on benchmark datasets, we demonstrate the effectiveness of our approach in outperforming traditional rule learning methods. Furthermore, we investigate the convergence properties of our reinforcement learning algorithm and provide theoretical bounds on its performance. Our results show that by combining reinforcement learning with rule learning, we can achieve more accurate and concise rule sets, leading to improved generalization performance on unseen data. Overall, this paper contributes to the intersection of reinforcement learning and rule learning, opening up new possibilities for optimizing rule-based systems in various domains.Random Prompt Example. Title: Exploring the Influence of Martial Arts on Japanese History Abstract: This paper delves into the deep-rooted connection between martial arts and the historical landscape of Japan. By tracing the evolution of various martial art forms such as Judo, Kendo, and Karate, we analyze how these disciplines have not only shaped the physical combat techniques but also contributed significantly to the cultural, philosophical, and political development of Japan. Through studying the impact of samurai warriors, martial arts schools (ryu), and renowned martial artists throughout different historical periods, this paper uncovers the intricate interplay between martial arts and Japan's societal norms, values, and power dynamics. Furthermore, by exploring the ways in which martial arts have been depicted in art, literature, and popular culture over the centuries, we gain a deeper understanding of how these practices have transcended mere combat techniques to embody spiritual, ethical, and aesthetic principles. Ultimately, this paper aims to shed light on the enduring legacy of martial arts in Japanese history and the profound influence they continue to exert on various aspects of Japanese society today.Mixing Prompt Example. Title: Unified Approaches in Machine Learning and Control Systems Abstract: This paper presents a novel approach to machine learning and control systems by integrating various techniques such as probabilistic methods, neural networks, case-based reasoning, genetic algorithms, theory, reinforcement learning, and rule learning. The proposed framework leverages the strengths of each approach to address complex problems in a unified manner. It introduces a new algorithm, contextual machine learning, which incorporates adaptative history-sensitive models for source separation, making use of the temporal structure of input data. Furthermore, the paper explores the application of non-linear systems in robust control and the use of functional programming by analogy in the domain of program synthesis. Additionally, it presents a learning algorithm for Disjunctive Normal Form (DNF) under the uniform distribution, with implications for the case when certain parameters are constant. The paper also discusses the use of Gaussian noise in sigmoidal belief networks and the training of such networks using slice sampling for inference and learning. Finally, it showcases the application of radial basis function networks in predicting power system security margin. Overall, this paper provides a comprehensive overview of the diverse applications and advancements in machine learning and control systems, demonstrating the potential for creating unified approaches that transcend traditional boundaries.We show the generated examples on the Cora dataset for ITGIA in this subsection. The output is highly uninterpretable due to ill-defined interpretable regions for injected embeddings.ITGIA-TDGIA without HAO, Cos: 0.136• of the following: MC Dallas' NSA sign, and some of the relays of the past few years: Trumpet Jam's sinus • the liner notes of The MC6's "Desirty Pigs": the relayed that some of the plaque • liner notes of The MC's "The Lion's Pie" album: some relayed that the 'bumpy' fluid • the following: One of the croons of the "Petsy on the MC12" relayed: Demitab ITGIA-TDGIA with HAO, Cos: 0.742• "Sympteric" Alignment. This is one of the best examples of how to achieve high level information retrieval algorithms using proportion • "Asymptotically" by Grant. This is a key method for constructing local information representation systems that support multicluster connections • Emilia. The "Proplective information with no associativity at all" NNNN algorithms are based on Monte simple scenariosJ.4 WTGIAWe show the generated examples on the Cora dataset with sparsity budget set as "average" for WTGIA in the subsection. We choose the FGSM-TDGIA for embedding generation. We display one example for each generation type.GPT. Title: Hierarchical Recurrent Neural Networks for Visual Sequence Identification and Dimensional Improvement Abstract: In this article, we demonstrate the idea of using hierarchical recurrent neural networks for visual sequence identification, resulting in dimensional improvement. We show that this approach requires the underlying mechanisms of backpropagation through time to obtain a more powerful representation of visual sequences. Additionally, we identify the nonlinear nature of the sequence identification problem and how the use of recurrent neural networks leads to nonlinear improvement in identifying visual sequences. Furthermore, we present a novel approach to dealing with hierarchical structures in visual sequences, utilizing the power of recurrent neural networks to identify objects in a hierarchical and sample-efficient manner. This work contributes to the science of visual sequence processing and demonstrates the capability of recurrent neural networks to handle complex visual sequences, such as trees, lines, and vector objects. Abstract: This article proposes a groundbreaking idea that combines the strengths of hierarchical neural trees and recurrent back-propagation to identify and classify nonlinear sequences. By leveraging the underlying mechanisms of neural trees and incorporating the addition of recurrent feedback, we demonstrated a remarkable improvement over conventional sequence identification methodologies through the power of back-propagation. Our methodology requires the creation of a vector-based object that represents the sequence, which is then fed into a hierarchical neural line that utilizes recurrent back-propagation to identify the underlying dimensional transformations. This leads to a resulting sample of sequences that can be analyzed and visualized to obtain a deeper comprehension of the object's underlying mechanisms. To validate our idea, we conducted a comprehensive sequence identification experiment, where we utilized a sample of sequences to train and fine-tune our hierarchical neural trees. Our resulting sequences demonstrated a remarkable capability to identify and classify nonlinear sequences, out-performing other leading methodologies, and providing a visual insight into the underlying mechanisms. This breakthrough has the power to revolutionize the science of sequence identification and has far-reaching implications for fields such as signal-processing, image-recognition, and predictive analytics.Llama-Topic. Title: Hierarchical Recurrent Neural Trees for Dimensionally Reduced Sequence Identification and Power System Monitoring Abstract: This article demonstrates the effectiveness of hierarchical recurrent neural trees (HRNT) for identifying sequences of nonlinear phenomena occurring within power grids. By leveraging the power of back-propagation through trees, HRNT is capable of capturing underlying mechanisms governing the system's nonlinear interactions. Our HRNT-based sequence identification idea requires a hierarchical arrangement of recurrent neural trees, which are then utilized to identify sequences of interest along a dimensional line of power transmission. Notably, our methodology leads to a resulting improvement of up to [X]% over conventional sequence identification schemes, which has been demonstrated through a sample dataset of power system measurements. Furthermore, our HRNT-based system can effectively identify nonlinear phenomena, such as addition of harmonics, and nonlinear interactions between power system objects, leveraging the power of vector calculus to obtain a comprehensive picture. Our visualizations of the resulting sequences and underlying mechanisms have been found to be particularly insightful for power system scientists, contributing to the advancement of the science and providing a valuable addition to the scientific community.K More Experiments about WTGIA and ITGIA K.1 Transfer WTGIA to More DatasetsIn Tables 15 and Table 16 , we included the ogbn-arxiv dataset [10] , containing over 160,000 nodes, and a social network dataset, Reddit [11] , with more than 30,000 nodes, to evaluate the scalability and generalization of text-level GIAs. We injected 1,500 nodes into ogbn-arxiv, following the budget specified in [2] , and proportionally injected 500 nodes into Reddit. On both datasets, WTGIA maintains use rates above 90% and shows good attack performance when transferred to GTR. In Table 17 , Table 18 , Table 19 , Table 20 , Table 21 , and Table 22 , we include classic methods like GAT [27] and GraphSAGE [8] , as well as the EGNNGuard and Layernorm (LN) methods, which have been proven highly effective in [2] .We discover interesting new results, such as LN's effectiveness on text-level GIAs not being as strong as previously reported for embedding-level attacks, especially for WTGIA. This is because embedding-level GIAs often exhibit abnormal norms, which LN exploits for defense. However, text-level GIAs derive embeddings from real text, avoiding structural anomalies and bypassing LN's defenses. This suggests that some traditional defense methods that were particularly effective may be limited to the embedding level. The EGNNGuard method is generally effective and performs well overall. As mentioned in the main paper, attackers can use techniques like HAO to bypass EGNNGuard, but this often involves trade-offs. In Table 23 and Table 24 , we included SentenceBert all-MiniLM-L6-v2 [22] as a new embedding backbone. We find that WTGIA shows some transferability between different embeddings, but the L.1 Figure 5 for WTGIA with TopicThe results of WTGIA variants with topic constrained in the prompt are shown in Figure 7 . The injected raw texts are embedded with BoW and GTR, and then evaluated using a GCN. The last column reports the average result of GIAs in the last five columns. The lowest(strongest) result in the column is bold. Note that GPT-Topic fails to give a meaningful response.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: We conform in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We do not consider the co-occurrence constraint in[28] because we find the co-occurrence matrix is dense and ineffective in practice (see Appendix G.3).The length requirement is necessary because the task would become trivial if the text length is unlimited. However, we usually cannot arbitrarily inject long text into real-world applications. 


Decision: It seems to me that all major concerns of the reviewers have been addressed during the discussion period and that the paper merits acceptance at the NeurIPS conference. 

The reviews and the rebuttal have given rise to several interesting points and results that I encourage the authors to include in the camera-ready version. Especially, the multitude of additional experimental results that the authors added in the general author rebuttal would be a valuable addition to the paper, since the evaluation on only three datasets, seems to be insufficient to me too. I furthermore want to encourage the authors to add even more defense baseline models to their camera-ready version (as several reviewers also requested). While the new LayerNorm and EGNNGuard results provided in the rebuttal are interesting, I feel that they still do not do justice to the vast literature that exists on defense methodology to adversarial attacks on graph structure and node features. The impact of this paper could be improved if the method was benchmarked against a broader range of defense methodologies.

Reviews: review: paper_summary: paper_summary: This paper explores the vulnerability of GNNs to Graph injection Attacks (GIAs), which involve injecting malicious nodes into a graph. This paper explores GIAs at the text level, presenting three types of GIA designs that inject textual content into the graphs. The significance of text interpretability in attack effectiveness and the defense against text-level GIAs are also explored.
strength_weakness: strength_weakness: 1. In line 126–131, the paper claims that traditional methods focus mainly on the embedding level. However, some existing works (e.g., [1]) focus on the raw feature level. The authors fail to discuss why these works cannot be applied to this task.
2. Limited technical contribution. In this paper, three kinds of text-level GIA are proposed. However, I think all of them are simply combining both existing GIA techniques and language models, which are kind of incremental contributions. 
3. I think the proposed text-level GIA is similar to the traditional GIA, especially when the nodes' texts are encoded into embeddings by LMs. Therefore, I think some existing defense methods against GIA can be also applied to this task. The authors need to discuss that. 
4. The experiments only focus on small-scale datasets. More large-scale datasets need to be included.
5. Only vanilla GCN is used in the experiments, which may induce to biases to the conclusions. More advanced GNNs need to be included.



[1] Understanding and Improving Graph Injection Attack by Promoting Unnoticeability. ICLR 2022.
questions: questions: Refer to weaknesses.
limitations: limitations: A section to discuss the limitations is missing.
paper_summary: paper_summary: The paper studies GIAs, particularly focusing on text-attributed graphs (TAGs). It introduces a method for GIAs by injecting textual content directly into the graph, as opposed to the traditional method of embedding-level attacks. The authors propose three new attack designs: Vanilla Text-level GIA (VTGIA), Inversion-based Text-level GIA (ITGIA), and Word-frequency-based Text-level GIA (WTGIA). They demonstrate that text interpretability is necessary for the effectiveness of these attacks. They also show that defenses can be enhanced using customized text embedding methods or large language model (LLM)–based predictors.
strength_weakness: strength_weakness: The paper does not explore the robustness of the proposed GIAs against more sophisticated defense mechanisms beyond customized text embedding methods and LLM-based predictors. 

Although the authors mention that LLM-based defenses enhance protection against text-level GIAs, the experiments primarily focus on a few specific methods and datasets. These are limited in scope. 

The trade-offs between performance, unnoticeability, and interpretability, as discussed in Sections 4.1 and 4.2, could be examined more thoroughly. While the paper shows that increasing the sparsity budget enhances attack performance but reduces interpretability, further experiments across a wider variety of datasets could validate these findings more convincingly, especially with additional baselines. Including more traditional GIA methods or other recent advancements in graph attack techniques would also be useful.

The discussion on the practical implementation of these attacks in real-world applications could be expanded. The paper mainly provides experimental insights but lacks detailed exploration of the real-world scenarios you stake your paper around, and practical challenges like the scalability of the proposed methods to larger and more complex graphs, the computational costs associated with implementing these attacks, and potential detection mechanisms in real-world systems are not thoroughly addressed. Providing more actionable insights and guidelines for practitioners could significantly enhance the practical relevance of the research.

Lastly, while the authors discuss the potential for enhancing defenses using customized text embedding methods or LLM-based predictors, they do not explore the possibility of integrating multiple defense strategies. Combining different defense mechanisms could potentially create more robust protection schemes against text-level GIAs. 

Typos: line 296 and line 211
questions: questions: 1. Can the authors provide more details on how the sparsity budget impacts the interpretability and effectiveness of the text-level GIAs across different datasets?
2. How would this work apply to dynamic graphs where nodes and edges evolve over time?
3. Are there any challenges in scaling the proposed GIAs to larger graphs or more complex datasets?
limitations: limitations: N/A
paper_summary: paper_summary: This paper investigates an interesting topic. The authors first discuss the practicality of Graph Injection Attacks (GIA), arguing that previous approaches, which merely inject harmful embeddings, are unrealistic. It introduces three text-level GIA methods: ITGIA, VTGIA, and WTGIA, and conducts a thorough examination of their attacking performance and interpretability. The findings reveal that WTGIA excels in both attacking effectiveness and interpretability. Additionally, the paper explores the defensive capabilities of Large Language Models (LLMs) and discovers that they can effectively evade the impact of injected nodes. This underscores the importance of enhancing the performance of text-level attackers.
strength_weakness: strength_weakness: 1. The authors should provide references for the cited papers on TDGIA, ATDGIA, MetaGIA, and AGIA.

2. The authors should explain why the attack performance initially decreases and then increases as the sparsity budget increases, as shown in Figure 5.

3. The authors should clarify how the LLM optimizes the graph structure after generating the text for the injected nodes.
questions: questions: 1. For VTGIA, how does the LLM optimize the graph structure after generating the text for the injected nodes?

2. The attacking method of WTGIA is interesting. Could this idea inspire other areas of GNN?

3. In Section 4, could you clarify the embedding-text-embedding process? Specifically, in Section 3, under Analysis of Performance and Interpretability, it seems that after inversion, text is generated. What embedding results after this inversion?

4. Why do you introduce FGSM to generate binary embeddings? Is it not possible to directly transform the embeddings into binary embeddings?

5. In Figure 5, why does the attack performance initially decrease and then increase as the sparsity budget increases?
limitations: limitations: Some parts of Section 4.1 are hard to understand, I admit I didn't check the proof related to the theorem in the Appendix. I think the authors could add figures or improve their presentations to better illustrate WTGIA.
paper_summary: paper_summary: The paper studies Graph Injection Attacks on text-attributed graphs. The study presents three attack designs: Vanilla Text-level GIA (VTGIA), Inversion-based Text-level GIA (ITGIA), and Word-frequency-based Text-level GIA (WTGIA). The key contributions include demonstrating the effectiveness of text level perturbation on Graph Injection Attack and the performance of using LLMs as a defender. WTGIA shows significant attack potential while maintaining understandability. It is a timely study on the vulnerabilities of text attributes in graphs.
strength_weakness: strength_weakness: 1. The embedding-text-embedding process is not very clear to me. Additional explanation will be helpful. It is unclear how different embedding models may affect the performance of this process. 

2. The term interpretability in this paper has different semantics to the commonly used one in explainable AI. It is confusing and creates unnecessary obstacles in reading. It is better replace with understandability.
questions: questions: 1. What is the computing complexity of WTGIA? How long does it take to generate an attack in average in your experiment environment? 

2. What is the impact of different text embedding models on the attack performance?
limitations: limitations: Limitations on transferability is discussed in the paper.

Comments: title: Response to Authors
comment: Thank you for your valuable feedback and insights. We appreciate your time and effort in reviewing our work. Please feel free to reach out if you have any further questions or need additional information.
Thanks for your responses. I read all the other reviewers' comments, I have adjusted my scores and will make final decisions after discussing with other reviewers and Acs.
Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your feedback has been invaluable to us. We have provided detailed responses to the concerns and questions you raised during the rebuttal period. Your insights are crucial to enhancing the quality of our work.

With the discussion period ending within one day, we hope we have thoroughly addressed all your concerns. Should you have any further questions or suggestions, we would be delighted to continue our discussion. 

Thank you for your support and guidance.

Best regards,

Submission15853 authors
Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your feedback has been invaluable to us. We have provided detailed responses to the concerns and questions you raised during the rebuttal period. Your insights are crucial to enhancing the quality of our work.

With the discussion period ending within one day, we hope we have thoroughly addressed all your concerns. Should you have any further questions or suggestions, we would be delighted to continue our discussion. 

Thank you for your support and guidance.

Best regards,

Submission15853 authors
Thank you for your appreciation of our work. We will carefully consider your suggestions to further improve our paper.
Thanks for the detailed response. I am pleased to note that you have addressed all the concerns I raised. I appreciate your efforts in clarifying these points and I have raised the scores.
Thank you for your appreciation of our work.
We will carefully consider your suggestions to further improve our paper.
Thanks for the response and additional experimental results. These largely address my concerns.