Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks?
Authors: Margarita Bugueño
Gerard de Melo
Venue: EMNLP 2023 Findings (2023)
Keywords: Graph-Based Text Representation
Graph Neural Networks
Text Classification
TL;DR: An empirical investigation of graph representations for text classification using GNNs.

Abstract: Given the success of Graph Neural Networks (GNNs) for structure-aware machine learning, many studies have explored their use for text classification, but mostly in specific domains with limited data characteristics. Moreover, some strategies prior to GNNs relied on graph mining and classical machine learning, making it difficult to assess their effectiveness in modern settings.
This work extensively investigates graph representation methods for text classification, identifying practical implications and open challenges.
We compare different graph construction schemes using a variety of GNN architectures and setups across five datasets, encompassing short and long documents as well as unbalanced scenarios in diverse domains. 
Two Transformer-based large language models are also included to complement the study. 
The results show that i) although the effectiveness of graphs depends on the textual input features and domain, simple graph constructions perform better the longer the documents are, ii) graph representations are especially beneficial for longer documents, outperforming Transformer-based models, iii) graph methods are particularly efficient for solving the task.

Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks?
Margarita Bugueño and Gerard de Melo
Hasso Plattner Institute (HPI) / University of Potsdam
Potsdam, Germany
beteggaritabuugueee gerard. demelo Jhhpi de
Abstract
alia), the rise of Graph Neural Network (GNN) models in recent years has made it particularly appcaling to convert even unstructured data into graphs. The model can then capture relevant pat- terns while accounting for dependencies between graph nodes via message passing
Given the success of Graph Neural Networks (GNNs) for structure-awadt aware machine learning, many studies have explored their for text classification, but mostly in specific domains with limited data characteristies. Moreover, some strategies prior GNNs relied on graph mining and classical machine learning, making it difficult to assess their effectiveness in mod- ern settings. This work extensively investigates graph representation methods for text classifi- cation, identifying practical implications and open challenges. We compare different graph construction schemes using a variety GNN architectures and setups across five datasets, compassing short and long documents as well as unbalanced scenarios in diverse domains. Two Transformer-basee large language mod- els are also included to complement the study. The results show that i) although the effective- ness of graphs depends on the textual input features and domain, simple graph construc- tions perform better the longer the documents are, ii) graph representations are especially ben- eficial for longer documents, outperforming ceresfoomrr-basee models, iii) graph methods are particularly efficient at solving the task.
For text classification, numerous graph-based text representation schemes have been proposed and demonstrated the efficacy of graphs. However, most of them were designed for particular domain- specific tasks and validated only on short docu- ments using a restricted set of model architectures (Yao et al., 2019; Huang et al., 2022; Wang et al., 2023). Moreover, some of these proposals predate the introduction of GNNs and were validated using graph mining or classical machine learning models. making challenging to determine the applicabil- ity and effectiveness of graphs in broader settings (Castillo et al., 2015, 2017).
Text classification increasingly extends beyond simple topic classification tasks, encompassing real-world challenges such as noisy texts, imbal- anced labels, and much longer documents consist- ing of more than just few paragraphs. Hence, a comprehensive assessment of the merits and draw- backs of different graph representations and meth- ods in more diverse scenarios is needed.
1 Introduction
Document comprehension involves interpreting words that can alter the meaning of the text based on their placement. For example, in the sentence "the movie was boring, but was surprised by the ending", " the word but contrasts ideas. While tra- ditional vector-based text representation methods lack the ability to capture the structural informa- tion of a text effectively, graph-based representa- tion strategies explicitly seek to model relationships among different text elements (nodes) through asso- ciations between pairs of them (edges), capturing dependencies between text units and leveraging language structure.
This work presents a thorough empirical inves- tigation of previously proposed graph- wraph-based text representation methods, evaluating how graphs gen- cralize across diverse text classification tasks. We analyze their effectiveness with several GNN-based architectures and setups across five prominent text classification datasets from broad range of do- mains. Unlike previous work (Galke and Scherp, 2022), our study considers diverse datasets with both short and longer documents, as well as un- balanced classification scenarios. Additionally, we evaluate the efficacy vs. efficiency of the proposals, aspect usually neglected in previous studies. For each graph method, we conducted extensive
While such ideas have a long history (Hassan and Banea, 2006; Mihalcea and Tarau, 2004, inter

---

experiments using 3 types of convolutional layers different message-passing strategies for 12 GNN architecture variants, each using one out of 4 pre- trained word embedding techniques as node feature vector initialization. This allows us to shed light on what are the most successful choices of GNN architectures for learning from them.
lization of graph neural networks essential com- ponents within the learning framework.
2.1 Early Graph Constructions
For graph-based text representation, simple ap- proach to consider word co-occurrence within a fixed-size sliding window: Words are modeled as nodes, and two nodes are connected if the respec- tive words co-occur within a window of at most N words. Mihalcea and Tarau (2004) used such co-occurrence graphs for N E {2. .2..... 10) ranking model for keyword extraction. They found smaller N to be preferable, as the connection be- tween words further apart often weaker. Hassan and Banea (2006) used the same approach with N = along with TextRank to replace term fre- quency weights, and then conducted text classifica- tion with classic machine learning models. In most of their experiments, this scheme outperformed us- ing TF-IDF vectors. Rousseau et al. (2015) also used fixed-size sliding window graph (calling it graph-of-wordy) They cast text classification as a classification problem by applying graph mining to obtain subgraph features to train a classifier.
Our study finds that graph methods are com- petitive and particularly efficient choice for solving classification tasks. This is because GNNs can cap- ture both local and global dependencies between structural components. Therefore, they can cap- ture rich semantic relationships and dependencies that are important for the task. Additionally, un- like many sequence models, GNNs can naturally handle variable-length inputs by operating on the graph structure, without any need to map every data sample to a fixed-sized vector or truncate them at a fixed maximum sequence length. While longer doc- uments can be particularly challenging, our study finds that GNN methods hold particular promise for longer documents, aspect unexplored in prior research. However, the graph's 's effectiveness de- pends on the textual input features and domain. Based on our experimental results, we provide discussion regarding what graph construction and GNN architecture choice is preferable depending on the task to be solved. Surprisingly, although transformer-basee Language Models (LMs) yield outstanding results for the considered tasks, they of- ten have difficulties converging when dealing with short texts.
Sequence graphs are another simple scheme with edges reflecting the original order of words in the text (Castillo et al., 2015). The authors used the number of times the corresponding two words ap- pear consecutively in the text edge weights.
2.2 Linguistic Features Graphs
Other graph construction methods have been pro- posed. Mihalcea and Tarau (2004) highlighted that multiple text units and characteristics can be con- sidered as vertices depending on the application at hand. They invoked application-speciff criteria to define edges, such as lexical or semantic rela- tions. To this end, they also proposed a similarity- weighted graph for sentence extraction. Every node represents an entire sentence, while edges are de- fined by measuring their content overlap as the number shared tokens. Although this scheme can be applied in other tasks (text classification or summarization). tends to yield fairly densely con- nected graphs, making it difficult to extract local patterns and discern the content of the text.
The study structured around three research questions, which are discussed in Scction 4:
1. How does the choice of GNN architecture and setup affect the classification effecctivenes??
2. What graph construction method is most ef- fective for text classification?
3. Can graphs compete with state-of-the-art se- quence classification models?
2 Prior Work on Graphs in NLP
Previous graph-based text representation methods can be categorized into three categories based on the nature of the underlying graph structure. Early graph constructions primarily relied on co- occurrence and textual statistical patterns. Subse- quently, more advanced representations integrated linguistic features as graph components. Recently, specialized graph constructions have evolved, en- tailing intricate structures that encompass the uti-
Given that traditional work in linguistics and computational linguistics often considers tree and graph-structured formalisms as the principal way of analyzing individual sentences, these may also serve as building blocks for document-level repre- sentations (Arora et al., 2009; Joshi and Rosé, 2009,

---

inter alia). For instance, neural parsing model (Dozat and Manning. 2016; Yuan al., 2021) can infer word dependencies to obtain syntactic depen- dency trees. However, the overall graph represen- tation becomes rather sparse, as nodes share edges with only a limited number of other units.
proposes heterogeneous graph construction, in- troducing lopic nodes to enhance class-aware rep- resentation learning. However, has the same limitations as TextGCN.
Alternatielly, two inductive models have re- ported good results on traditional text classifica- tion benchmarks, but the improvement is mostly due to the combination of GNN and BERT models (Huang et al., 2022: Wang et al., 2023). Thus, these strategies are resource-intensive hard to apply long documents, and beyond the scope of our study.
2.3 Specialized Graph Constructions
Text Graph Convolutional Network (TextGCN; Yao al. 2019) one of the first approaches to include a Graph Convolutional Neural Network (GCN) as a classification method. TexIGCN pro- poses a hetcrogencous graph construction using words and documents as nodes. However, this means that new documents cannot be processed without re-training. It employs Point-wise Mutual Information (PMI) similarity
Since Zhang et al. (2020) outperform Textlevel- GCN despite using the same graph construction, it is clear that the graph construction method and the way patterns are extracted from it are closely lated. Hence, in-depth study analyzing multiple f'actors in controlled setting is necessary.
edge weight- ing function for word pairs and TF-IDF for word- in-document edges. Other proposals also sug- gested integrating heterogeneous contextual infor- mation such as TensorGCN (Liu et al., 2020), Het- eGCN (Ragesh et al., 2021), and HyperGAT (Ding et al., 2020). However, such approaches are fairly resource-intensive.
In terms of broader empirical comparisons, one previous study also conducted a comparative analy- sis of different approaches for text classification to evaluate the necessity of text-graphs. The authors compared multiple Bag of Words (BoW), sequence. and graph models (Galke and Scherp, 2022), argu- ing that multi-layer perceptron enhanced with BoW is strong baseline for text classification. Nevertheless, the authors limited their analysis to standard data collections with only short texts. contrast, with the aim to study how graphs perform in more challenging scenarios, our study considers broader range of domains including much longer documents and unbalanced classification contexts. In addition, we assess the balance between the ef- fectiveness and efficiency of the proposals, a facet typically overlooked in prior research.
TextLLeveGGCN (Huang et al., 2019a) creates one graph per input text. The proposal defines every word as a node, which can be duplicated a word appears more than once in text. Edges are defined for word nodes within sliding window using PMI edge weights. Despite promising results, the exper- iments were limited very short documents.
GraphIE (Qian et al., 2019) uses a homogeneous scheme based on co-reference, integrating a GCN with an RNN encoder- deloder architecture for tag- ging and information extraction tasks. Nodes can be defined as words entire sentences, connected via co-reference and identical mention edges, to account for non-local and non-sequential ties. A downside of this that prior domain knowledge required to establish the edge types.
3 Comparing Graph-Based Text Representations
To study the merits of prominent graph- based text representation strategies, we conducted comprehen- sive experiments on five well-known text classifi- cation datasets. For each task, we compare differ- ent graph construction schemes using variety of GNN models to separate the effect of the graph con- struction strategy from that of the pesssag--passing technique in the model.
Some studies have brought back the classic co- occurrence graph construction methods, but using different message passing function based on Gated Recurrent Units (Li al., 2015; Cho et al., 2014) for updating node feature vectors (Zhang et al., 2020).
3.1 Methods
MPAD (Nikolentzos et al., 2020) included an extra master node connected to every other node in the graph. Therefore, the network is densely con- nected, and the structural information is vague dur- ing message passing. Text-MGNN (Gu et al., 2023)
3.1.1 Graph-Based Text Representation
Among the studied techniques, there are some graph construction methods that follow an intuitive construction process. They are based solely on sim-

---

ple relationships between pairs of nodes and only consider basic co-occurrence statistics if needed. Thus, they do not require a deep understanding of the semantic structure. In the following, we re- fer to these sorts of networks as Intuitive Graphs. Figure illustrates how they work.
'ransformer-basedd LMs: We also include BERT (Devlin et al., 2018) and Longformer (Beltagy al., 2020) Transformers as powerful masked language model-based encoders. While BERT has a maxi- mum input length of 512 tokens, the Longformer extends this limit via a modified attention mech- anism that scales linearly with sequence length. The latter trait is desirable when comparing LMs to graphs, which use the complete source text. Please note that Transformer-basee LMs are in- cluded merely as an informative point of reference for comparison.
Window-based: Following Hassan and Banea (2006), given an input text, a term has not been previously seen, then a node added to the graph, and an undirected edge is induced between two nodes if they are two consecutive terms in the text. Window-based extended: As for the above con- struction, but with a window size of three. With this, each word will ultimately be tied to the two previous terms and the two subsequent ones. Sequence-weighted This strategy (Castillo et al.. 2015) defines a directed graph with nodes for words and edges that represent that the two corresponding lexical units appear together in the text sequence and follow the order in which they appear. Addi- tional edge weights capture the number of times that two words appear together, which is intended reflect the strength of their relationship.
3.2 Datasets
The literature review reveals that many graph-based text representation methods have been evaluated on different datasets. Most of the time, the proposals were each introduced for a specific task domain and validated on text with very restricted character- istics, such a limited vocabulary and average document length of up to 221 words (Hassan and Banea, 2006; Yao al., 2019). Hence, it unclear how well these approaches can generalize to other kinds of data in different domains and be applied to longer documents.
Sequence simplified: Inspired by the above, sim- plified version omits the edge weights. Thus, the effect the edge importance function over the pure graph structure can be studied in isolation.
We assess the generalizability of graph strategies in text classification, including sentiment analysis, topic classification, and hyperpartisan news detec- tion, across balanced and unbalanced scenarios, including longer documents. We utilize five pub- licly available datasets (see Table 1), with further details provided in Appendix A.
A more sophisticated graph-based text represen- tation strategy requiring a more elaborate graph construction process is also considered.
Ce:: Every word appearing in a text is treated as a node, and edges are defined between adjacent words in a fixed-size window. Unlike the above Intuitive Graphs, TextLeveIGCN (Huang et al., 2019b) considers each word token occurrence separate node, i.e., it allows multiple nodes if the corresponding term occurs more than once in the text. Therefore, the specific in-context meaning can be determined by the influence of weighted information from its neighbors. The authors further employed PMI as an edge weighting function for the word associations, as in Yao et al. (2019).
App Reviews (Grano et al., 2017) - English user reviews of Android applications for fine-graid fine-grained sentiment analysis in an imbalanced setting.
DBpedia (Zhang et al., 2015) A dataset for topic classification consisting Wikipedia articles based on DBpedia 2014 classes (Lehmann et al., 2015). IMDB (Maas et al., 2011) I) - Movie reviews from the Internet Movic Database for binary sentiment classification.
BBC News (Greene and Cunningham, 2006) - A topic classification dataset! consisting of 2,225 En- glish documents from the BBC News website. Hyperpartisan News Detection (HND) (Kiesel et al., 2018) - A collection of 645 news articles? labeled according to whether it shows blind or un- reasoned allegiance to one party or entity. The dataset exhibits minor class imbalance.
3.1.2 Mainstream Text Representations
We further considered several mainstream repre- sentation schemes. allowing us to better understand how the graph approaches fare in comparison. Bag of Words (BoW): Given a vocabulary of known words, this strategy uses vectors of term frequencies, discarding any information about the order of words in the text.
'http://derekkreenne htttss///ammoo.rrg////rrordd
con/bbc/ httpt//rreeeeeccmbb/b//e

---

working
working
working
working The
The
The
The
Statt
Gooner
Saart
Start
Sart
Start
Sopner
money
you
money
money
money
money
partt
you
you
working
have
Will
have
ail
have
have
will
sooner
(a) Window
(b) WindoWext-
(c) Sequence
(d) Sequencceimp
(c) Textl eveIGNN.
Figure I: Graph Construction Methods. Given the input text "Start working! The sooner you start working, the sooner you will have money", the five co-occurrence graph representations studied are shown. From left right: window-based graph, window-hased graph extended (new edges are shown as dashed in blue), sequence-weighted sequence simplified omitting edge weights, and TextLeveIGCN (edge weights shown for first and last node, in blue).
Dataset
ADL
K
IR
>512
>1,024
3.3.2
Model Settings
App Reviews
14
5
1:8
0%
0%
DBpedia
51
1:1
0%
0%
Graph Neural Networks. For GNN experiments on Intuitive Graphs, we vary the number of hidden layers from 1 to 4 and vary the dimensionality of node representations in {16, 32, 61). We applied Dropout after every convolutional layer with a re- tention probability of 0.8 and used average pooling for node-level aggregation. The final representation is fed into softmax classifier.
IMDB
283
2
1:1
12%
1.4%
BBC News
438
5
28.5%
1.6%
HND
912
2
1:2
63.3%
29.8%
Table 1: Statistics of datasets. This includes the aver- age document length (ADL), the number of classes (K), the imbalance rate between the minority and majority classes (IR), and the proportion of long documents.
We compared three types of graph convolu- tional neural layers: (i) the traditional one (GCN; Kipf and Welling 2016), (ii) using graph isomor- phism operator (GIN; Xu et al. 2018), which has shown improved structural discriminative power compared to GCNs, and (iii) including a graph attentional operator (GAT; Velickovic et al. 2017) with 4 attention heads. Our experiments were based on Py Torch Geometric (see Appendix E).
3.3 Experimental Setup
3.3.1 Data Preparation
A fixed-size data partition was taken from each dataset to conduct a fair comparative analysis among the methods. Thus, a training and test split was defined, consisting of 7,000 and 3,000 samples, respectively. For those datasets that did not have that many examples, i.e., BBC News and HND, 80% of the samples were used for training and the remaining 20% for testing. For all datasets, we randomly reserve 10% of the samples from the training set for building the validation set.
For TextLeveIGCN, we used default parameter settings as in the original implementation, varying the window size (n-gram parameter) from 1 to 4.
Four different node vector initialization strate- gies were also compared. We considered GloVe Wiki-Gigaworr 300-dim. embeddings (Pennington et al., 2014), Word2 Google News 300-dim. embeddings (Mikolov et al., 2013), static BERT pre-trained embeddings (encoding each token in- dependently and averaging for split terms), and contextualized BERT embeddings. The latter in- volves encoding the entire input text using BERT and using token embeddings from the 12th layer.
Since each graph node represents word of the input text, a consistent text normalization scheme is needed: We applied lowercase conversion, punc- tuation mark and stop word removal, as well as eliminating any other non-ASCII characters.
Note that our TexILLeveIGCN experiments are conducted using the official implementation', which incorporates additional preprocessing. This includes removing tokens with fewer than three characters, limiting document lengths to 350 terms, climinating words with frequeney less than 5, applying lemmatization, well as applying expan- sion rules to remove English contractions.
Bag of Words Baseline. We employed cut-off for building the BoW vocabulary by eliminating terms with a document frequency higher than 99% or lower than 0.5%. Once the BoW representations are obtained, a Multi- Layer Perceptron model with

---

one hidden layer is trained for text classification (BoW MLP). We varied the number of hidden units in (32, 61, 128, 256} and applied Dropout right be- fore the final classification layer, as done for GNNs.
and BBC News, compared other convolutional layers. Due its attention mechanism, GAT can identify those nodes that are relevant for the final prediction. GAT models also proved to be more ro- bust to variations in parameters such as the number of layers and the hidden units (Appendix B).
All GNNs and BoW MLP used batch size of 64 samples and were trained for a maximum of 100 epochs using Adam optimization (Kingma and Ba, 2014) with an initial learning rate of 10-4. The training was stopped if the validation macro- averaged Fl score did not improve for ten consecu- tive epochs. Only for HND, the patience was 20.
However, for imbalanced classification with very short texts (as on App Reviews), GAT is not as ef- fective. In such settings, the graphs have very few nodes, and the attention heads appear to fail to identify the most pertinent ones. Similarly, GAT struggled on HND: Although HND contains ex- tremely long documents and thus there are suffi- cient elements exploit, many of the tokens are HTML and PHP markers, or similar source arti- facts. Thus, much of the input insignificant for the task and the attention heads fail to identify rel- evant nodes. GIN proves to be the best choice for such cases, exploiting the graph structural in- formation for superior discriminative power over traditional GCNs (Xu et al., 2018). While GCNs use simple averages of neighboring node represen- tations, GIN defines weighted average by learning to determine the importance of node compared to its neighboring nodes (e-value), which is then fed into an MLP. Thus, GIN can distinguish node neighborhoods, discerning structural information among graph classes. Since our document graphs are based on word co-occurrence, GIN can exploit structural regularities and identify recurrent associ- ations between specific words, which can be crucial for predicting the correct graph-level label.
Transformer-basedd Baselines. We fully fine- tuned BERT-base uncased, including a Dropout layer right after with a retention probability of 80%, and a final dense layer for conducting the text classification. During training, the batch size and learning rate were set to 32 and 10 respectively. The maximum number of epochs was 10, and the followed
patience was 5. The same procedure for Longformer-basee However, given the com- plexity of the model (148 M trainable parameters) and computing resource constraints, the maximum sequence length was set to 1,024 tokens, and the batch size was set to 16.
General Setup. The objective function of each model was to minimize the cross-entropy loss. Sup- plementary experimental details provided in Appendix A, Appendix C, and Appendix E. For reproducibility, we release our code on https: hww.:b////o/uuuuua/rueeaa /GRCCGGGN
4 Results and Analysis
Node Feature Initialization. A noteworthy find- ing is that the best results were mostly obtained with non-BERT initializations. Well-known static word embeddings with much lower dimension- ality appear to yield better results than BERT em- beddings. This the case for App Reviews and IMDB using Word2 Vec, and BBC News using GloVe. Similarly, when using TextLeVeIGGNN as an elaborated graph construction, Word2 obtained better results than BERT initialization for some tasks. Moreover, 1-gram graph construction is sufficient for medium and long text classification when using such an initialization strategy. How- ever, denser graphs are required for short texts.
Table 2 and Table 3 show the best architecture and setup for cach dataset employing Intuitive Graphs and Textl evelGGN, respectively. The results cor- respond to the average obtained from 10 indepen- dent runs. As some datasets exhibit class imbal- ance, each table reports the accuracy and the macro- averaged Fi-score. The best results are reported in bold, while a star mark used to indicate the best architecture across the entire dataset. For a full comparison, see Appendix B and Appendix C. A comparison with baselines such as BERT is given in Table 4.
4.1 How do GNN Architecture and Setup Affect the Classification Effectiveness?
Convolutional layers. The results indicate that the optimal number of convolutional layers is task- dependent, with 1 or 2 layers favored for tasks centered on local patterns and more layers nec-
GNN Message Passing. Table 2 shows GAT as the most effective strategy for DBpedia, IMDB,
com

---

Window
Windowext
Sequenee
SequencCaimp
Dataset
Emb.
L-Conv
#U
Acc
Fl-ma
Acc
FI-ma
Acc
F1-ma
F1-ma
64.7
31.0
63.6
33.9
26.4
65.3
29.1
App Reviews
Word2Vee
3-GIN
62.0
34.9
35.0
31.0
x63.7
*35.7
61.1
35.1
62.4
35.4
33.0
62.5
34.8
+97.5
*97.4
97.3
97.3
97.3
97.2
97.3
97.3
DBpedia
BERT-C
1-GAT
32
97.2
97.2
97.3
97.2
97.0
96.9
97.0
97.0
64
97.1
97.1
97.1
97.1
96.7
96.7
97.0
96.9
16
87.3
87.3
87.3
87.3
87.7
87.7
+87.9
+87.9
IMDB
Word2Vee
1-GAT
32
87.3
87.3
86,9
86.9
87.5
87.5
87.5
87.5
64
87.4
87.3
86.7
86,6
87.2
87.2
87.4
87.4
16
97.8
97.7
+8.0
+98.0
97.8
97.8
97.4
97.3
BBC News
GloVe
4-GAT
32
97.8
97.7
97.6
97.6
97.8
97.7
97.4
97.3
64
97.8
97.7
+98.0
+98.0
97.6
97.5
97.2
97.1
77.6
76.8
75.2
73.9
56.6
36.1
77.4
76.5
HND
BERT
2-GIN
75.3
73.6
77.4
76.8
56.6
36.1
78.3
77.6
77.1
76.5
76.9
75.8
56.6
36.1
+79.1
+78.5
Table 2: Best-performing GNN for Intuitive Graphs. The node feature initialization (Emb.) and architecture details are reported. L-Conv and #U stand for the hidden convolutional layer and units, respectively. The results report the average obtained from 10 independent runs. Full comparison in Appendix B.
-gram
Z-gram
3-gram
4-gram
Dataset
Emb.
Acc
FI-ma
Acc
F1-ma
Acc
FI-ma
Acc
F1-ma
App Reviews
Word2Vec
66.6
34.7
64.7
35.2
+64.5
+35.8
64.3
35.5
DBpedia
BERT
95.7
95.7
+96.1
+96.0
95.9
96.0
96.0
IMDB
Word2Vec
+86.8
+86.8
86.5
86.4
86.2
86.2
BBC News
BERT
97.0
97.0
97.2
97.2
+97.3
+97.3
97.0
HND
Word2Vec
*75.7
+73.4
71.6
67.9
72.2
69.8
70.4
67.0
Table 3: Best-performing eee.GNN Results for the best node feature initialization (Emb.). The results report the average obtained from 10 independent runs. Full comparison Appendix B.
essary for tasks requiring broader information. The contextual understanding, whether local or global, is also influenced by the document length. For instance, to comprehensively grasp the docu- ment's s sentiment, a sentence-level analysis is vital, whereas the document comprises only one or two sentences, wider document-level view is prefer- able. This is shown in Table 2 and Table 5, where using 3 layers produced the best App Reviews re- sults.
weighted version, which fails to learn the task. This may be due to noisy tokens such HTML. tags that may occur numerous times. When omitting edge weights, the model may be less affected by such noise.
Regarding the window-based graph construction, the extended version does not show a significant improvement over the base version with N = 2. This is because a higher N increases the average degree of the graph, making it difficult to extract local patterns and discern the content of the text. Hence, Window mostly outperformed Windowext.
4.2 What Graph Construction Method is Most Effective for Text Classification?
Overall, the window-based construction is rec- ommended when the classification task is as simple as topic recognition. This allows a faster and more direct identification of the input document's vo- cabulary, as each token accesses both its left and right context immediately and can identify recur- rent words. Moreover, quick vocabulary explo- ration is achieved as N grows.
Intuitive Graphs. The sequence construction in general shows worse performance than its simpli- fied version, which indicates that the use of discrete weights in the edges does not provide relevant infor- mation for datasets such as App Reviews, DBpedia, and IMDB. BBC News appears to be an excep- tion: Since news articles tend to reiterate key facts in the news multiple times, exact co-occurrences of word pairs appear be frequent and might be meaningful. Despite also consisting of news ar- ticles, HND behaves similarly to other datasets in that Sequencesimp significantly outperforms the
In contrast, for tasks such as sentiment analysis or identifying writing styles and biases in a given article, a detailed analysis of the term order is nec- essary. In this case, a sequence- construction seems preferable. Although directed graphs may be

---

Dataset
Model
Node Init.
Fl-ma
Exec. Time
#Params
BoW MLP
64.7
=2.7=0.7
104.4
10.3 K
BERT
62.0 1.2
16,9
1.89188
108
App Reviews
Longformer
63.5
37.6
5,552.2
148
Textl lvellGeN
Word2
164.5
35.8 1.0
546.4
561 K
Sequencesimp
Word2
63.7
35.7 1.3
168.8
16.3 K
BoW MLP
9..
91.3 = 0.2
24.5
52.4
BERT
98.3 - 0.1
98.3 0.1
2,201.2
108 M
DBPedia
Longformer
98.1 0.2
18.1 0.2
5,451.9
148 M
TexiG_V
BERT
96.1 0.1
96.0 = 0.2
426.8
4.8 M
Window
BERT-C
97.5 + 0.1
97.4 0.1
384.3
50.3 K
BoW MLP
83.7 0.2
83.7 0.2
40.8
192 K
BERT
88.4 0.7
18.4 0.8
1,640.1
108 M
IMDB
Longformer
90.5 + 0.6
90,5 0.6
4,645.4
148 M
TexeLGN
Word2Vec Word2Vec
86.8 0.2
86.8 0.3
1,022.3
10.9 M
Sequenccsimp
87.9 0.1
87.9 0.1
473.6
19.5 K
BoW MLP
97.9 0.1
97
8.4
329 K
BERT
97.8
97.7 0.3
398.9
108 M
BBC News
Longformer
98.2 0.3
98.2 0.3
1,470.5
148 M
EveIGGNN
BERT
97.3 0.4
97.3 0.4
684.2
M
Windowext
GloVe
98.00
18800 0.3
170.6
32.6 K
BoW MLP
75.6
74.5
K
BERT
72.6 2.9
70.6 4.5
346.1
108
HND
Longformer
77.22 3.8
t 75.5 6.1
475.1
148
TexILeveIGGN
Word2Vee
75.7 + 2.6
73.4 3.5
426.8
3.2 M
Sequencestmp
BERT
79.1 1.1
1.1
116.3
66.1 K
Table 4: General performance. The average results 10 runs for graph models and sequential baselines reported (see Appendix C). For each model, the average total execution time as well as the number of trainable parameters (#Params) are specified. The best result is shown in bold, while the second best is marked with a symbol. Please note that the graph construction step is included in the execution time calculation (see Appendix D).
limited a left-to-right construction, GNNs spread the node information between neighbors and thus exploit structural and linguistic textual features, as local and global contexts of the document.
get competitive results. Given that App Reviews documents are extremely short, the local context in the text is insignificant and exploring the global context through denser 3-gram graphs is required.
TextLevelGCN. Table 3 shows that TextLevel- GCN is the best-performing graph-based model for App Reviews, implying that the task benefits from edge weights, but that they should be soft values for a smoother learning curve. Otherwise, it is prefer- able to omit them by employing Sequenccsimp construction. Nonetheless, TextLevelGCN under- performs Intuitive Graphs on all other datasets, even when processing medium-length documents.
4.3 Can Graphs Compete with State-Of-The-Art Sequence Models?
Although graphs do not attain the results of Transformer-basel ones for short and medium- length document classification, Intuitive Graphs perform better the longer the documents are. Graph representations are designed to harness the text's structure, and as such, their performance ex- pected to excel in longer documents as there is more information and structural patterns to exploit.
As in Table for TextLeveIGCN there is con- nection between the classification task and node feature initialization. Topic classification tasks ob- tained better results when employing BERT for 2-gram and 3-gram setups. Since vocabulary explo- ration is relevant to solve the task, an extended left-right context graph construction is benefi- cial. Likewise, since BERT embeddings are high- dimensional vectors, they are more valuable than other strategies. In turn, the best results for senti- ment analysis and detection biased writing were obtained by 1-gram graphs using Word2Vec. In these cases, only 300 dimensions are sufficient to
For BBC News, Windowext has second- best accuracy at only 0.2 points behind the best- performing model, Longformer. Intuitive Graphs dominate as the best way to represent longer doc- uments (HND). For this scenario, there is a no- ticeable gap between the best and the second-best model. Therefore, graph-based document rep- resentations appear provide clear advantages when processing long texts. Note that in this task, TextLeveIGCN performs better than BERT but worse than BoW MLP. This suggests that, despite

---

100
Long former
tural information deep neural networks. Graph- based learning models are powerful and allow the extraction of complex patterns from text. How- ever, they are particularly task-sensitive and de- pend on the lexical features of the documents to be represented. Thus, special care must be taken to properly define the components of the structure (nodes, edges, and the similarity function as edge label). Despite this, the most simplistic graph con- structions can address text classification fairly well, proving competitive even in challenging scenarios such as with data imbalance and noisy documents.
Execution Time [min]
BERT
80
TextlevelGCN Intuitive Graph
60
40
20
App Reviews
DBpedia
IMBD Dataset
BBC News
HND
Figure 2: Execution time. Average execution time and shaded standard deviation. Time is shown in minutes.
An interesting finding is that when the focus of the text classification task is on the vocabulary, the global context is much more relevant than the local context of the document. Thus, the best graph struction strategies are those based extended co- occurrence windows, yielding denser graphs. On the other hand, when the foeus on understanding the document as a whole and how the various parts of the text are connected, the local context becomes much more valuable. Therefore, Window (N=2) or Sequential graphs are recommended.
its effectiveness, TextLeveIGCN loses a significant part of the input document by defining a much smaller maximum length for text documents (350 tokens). BoW MLP represents each document by considering the entire dataset's vocabulary, grant- ing access terms beyond TexILevelGCN's scope.
One of the strongest aspects of Intuitive Graphs methods is that they require much less time and compute resources than popular alternatives dur- ing training. Although extra step is required to create document graph representations, the results indicate that the total execution time, including graph creation and model execution, is not an issue. For short texts as in DBpedia, e.g., the window graph is on par with the top-performing LLM, with just a 0.8% accuracy difference and 5.7 times faster speed. Likewise, BERT beats Sequence graphs on IMDB by only 0.5% in accuracy, while being 3.5 times slower. Note that BoW MLP is not included in Figure 2, since it did not obtain good results.
5 Conclusion
We present an empirical analysis of graph represen- tations for text classification by comprehensively analyzing their effectiveness across several GNN architectures and setups. The experiments consider heterogeneous of five datasets, encompassing short and long documents. The results show that the strength of graph-based models closely tied to the textual features and the source domain of documents. Thus, the choice of nodes and edges is found to be crucial. Despite this, Intuitive Graphs are shown to be a strong option, reaching com- petitive results across all considered tasks, espe- cially for longer documents, exceeding those of BERT and Longformer. Additionally, we observed that pre-trained static word embeddings, instead of BERT vectors, allow reaching outstanding results on some tasks.
In contrast, since BERT and Longformer are highly complex models in terms of the number of learnable parameters, higher execution time than for graph-based models is expected. Interestingly, shorter documents, such as those in App Reviews and DBpedia, take even longer than medium-length documents. This suggests that the models require several iterations to converge on these particular tasks. Beyond this, note the abrupt decrease in the execution time for the BBC and HND datasets is because they have a small number of samples. Therefore, the total runtime is much shorter com- pared to the others. See Appendix D for more details on the runtime and resource utilization.
We are enthusiastic about extending our study to further tasks in future work. To this end, we are releasing our code GitHub and hope that it grow to become community resource. Ad- ditionally, we will expand this study by exploring approaches for learning the graph structure elim- inate the need for picking design manually, being less Nomain-dependeen
4.4 Discussion
The results show that braph- document repre- sentation holds promise as a way of providing struc-
htt //gi thub. com/ ConnBugguearr/GRTc_.c

---

Limitations
References
Shilpa Arora, Mahesh Joshi, and Carolyn Rosé. 2009. Identifying types of claims in online customer views. In Proceedings of Human Language Tech- nologies: The 2009 Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics, Companion Volume: Short Papers, pages 37-40.
While this study successfully shows the impact and potential of graphs for document representation, there are some points to keep in mind.
First, despite all the judgments and conclusions presented being supported by the results of the experiments, they were based on graph neural net- work models trained on particular sub-ppatitions, as stated in Section 3, so as to allow a fairer comparison between models. However, this means that the results reported here are not directly com- parable with those reported in the literature. To assess how the models are positioned with regard the state-of-the-arr in the different tasks, it is advisable train on the original training partitions and thus learn from all the available data.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long- document transformer. arXiv preprint 2rXiv:2004.05/50.
Esteban Castillo, Ofelia Cervantes, Darnes Vilari, and B David. 2015. Author verification using a graph- based representation, International Journal of Com- puter Applications, (3)..).
Esteban Castillo, Ofelia Cervantes, and Darnes Vilar- ino. 2017. Text analysis using different graph-based representations. Computación y Sistemas, (4)::581 599.
It is also important to note that our study ana- lyzes multiple text representation strategies on text classification only. Although this is one of the most important classes of NLP tasks, we cannot ensure that such graph approaches show the same behavior in other tasks. Therefore, tackling other types of problems that require a deep level of understand- ing of the local and global context of the text is an important direction for future work.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 14X.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint 1rXiv:1810.04805.
Finally, all the experiments were run on English data. As English has comparatively simple gram- mar and well-known rules for conjugations and plurals, it is possible that graph-based models may not be as effective in other languages. Analyzing this aspect would be particularly interesting for low-resource languages.
Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu. 2020, Be more with less: Hypergraph attention networks for inductive text classification. arXiv preprint arXiv:2011.008:
Timothy Dozat and Christopher Manning. 2016. Deep biaffine attention for neural dependency pars- ing, arXiv preprint arXiv:1611.01734
Ethics Statement
Lukas Galke and Ansgar Scherp. 2022. Bag-of-words vs. graph vs. sequence in text classification: Ques- tioning the necessity of text-graphs and the surpris- ing strength of a wide mlp. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 4038 4051.
This work studies fundamental questions that can be invoked in a multitude of different application contexts. Different applications entail different eth- ical considerations that need to be accounted for before deploying graph-based representations. For instance, applying a trained hyperpartisan news de- tection model in an automated manner bears the risk of false positives, where legitimate articles get flagged merely for a choice of words that happens to share some resemblance with words occurring in hyperpartisan posts. For sentiment classification, Mohammad (2022) provides an extensive discus- sion of important concerns. Hence, ethical risks need to be considered depending on the relevant target use case.
Giovanni Grano, Andrea Di Sorbo. Francesco Mercaldo, Corrado A Visaggio. Gerardo Canfora, and Sebas- tiano Panichella. 2017. Android apps and user feed- back: a dataset for software evolution and quality improvement. In Proceedings of the 2nd ACM SIG- SOFT international workshop on app market analyt- ics, pages 8-11.
Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kemnel document clustering. In Proceedings of the r3 international conference on Machine learning. pages 377-384.

---

Yongchun Gu. Yi Wang. Heng-Ru Ru Zhang, Jiao Wu, and Xingquan Gu. 2023. Enhancing text classification by graph neural networks with multi-granular topic- aware graph. IEEE Access, 11:20169-20183.
Sören Auer, et al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neu- ral networks. arXiv preprint arXiv:1511.05491.
Samer Hassan and Carmen Banea. 2006. Random-wall term weighting for improved text classification. In Proceedings of TextGraphs: the First Workshop on Graph Based Methods for Natural Language Pro- cessing, pages 53-60, New York City, Association for Computational Linguistics.
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping L.v. 2020. Tensor graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8409-84 84166
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019a. Text level graph neural network for text classification. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Procexsing Hong Kong,
Andrew L. Maas, Raymond E. Daly, Peter T. Pham,
Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland. Oregon, USA. Association for Computational Lin- guistics.
(EMNLP-IJCNLE) pages China. Association for Computational Linguisties.
Lianzhe Huang. Dehong Ma. Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019b. Text level graph neural network for text classification. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3444-3450, Hong Kong, China, Association for Computational Linguisties.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404411,, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Yen-Hao Huang, Yi-Hsin Chen, and Yi-Shin Chen. 2022. Contexting: Granting document-wist con- textual embeddings to graph neural networks for in- ductive text classification. Proceedings of the 29th International Conference on Computational Linguis- rics, pages 1163-1168.
Saif M. Mohammad. 2022. Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. Com- putational Linguisties, 48(2):239-278.
Mahesh Joshi and Carolyn Rosé. 2009. Generalizing dependency features for opinion mining. In Pro- ceedings of the ACL-IJCNLP 2009 conference short papers, pages 313-316.
Giannis Nikolentzos, Antoine Tixier, and Michalis Vazirgiannis. 2020. Message passing attention net- works for document understanding. Proceedings of the AAAI Conference on Artificial Intelligence, (115)):844-855..
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Tmm- manuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. Semeval- 2019 task 4: Hyperpartisan news detection. In Pro- ceedings of the 13th International Workshop on Se- mantic Evaluation, pages 829-839.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1553.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, David Corney, Payam Adineh, Benno Stein, and Martin Potthast. 2018. Data for pan at semeval 2019 task 4: Hyperpartisan news de- tection. November. type: dataset.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. GraphIE: A graph-based framework for information extraction. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long and Short Papers), pages 751-761, Min- neapolis, Minnesota. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14X2:6441,.690..
Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint 17Xiv:1609.02907
Rahul Ragesh, Sundararajan Sellamanickam, Arun Iyer. Ramakrishna Bairi, and Vijay Lingam. 2021. Het- egen: heterogeneous graph convolutional networks for text classification. In Proceedings of the 14th ACM international conference on web search and data mining, pages 860-868.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas. Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,

---

François Rousseau, Emmanouil Kiagias, and Michalis Vazirgiannis. 2015. Text categorization as a graph classification problem. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 1702-1712, Beijing, China. As- sociation for Computational Linguistics.
DBpedia. For topic classification, the DBPedia ontology classification dataset (Zhang al., 2015) was constructed by picking 14 non-overlapping classes from DBpedia 2014 (Lehmann et al., 2015). For each category, the authors randomly chose 40,000 40,000 Wikipedia articles as training samples and 5,000 samples for testing. Every article contains the title, content, and class label. Although the original DBpedia is multilingual knowledge base, this dataset only contains English data.
Petar Velickovic, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. stat, 1050:20.
Yizhao Wang. Chenxi Wang. Jieyu Zhan, Wenjun Ma, and Yuncheng Jiang. 2023. Text feg: Fusing con- textual information via graph learning for text clas- sification. Expert Systems with Applications, page 119658.
IMDB. English language movie reviews from the Internet Movie Database for binary sentiment clas- sification (Maas al., 2011). The dataset com- posed of 25,000 reviews for training and 25,000 for testing, with balanced numbers of positive and negative reviews.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018 How powerful are graph neural net- works? arXiv preprint arXiv: 1.Xiv:/810000866
BBC News. This is publicly available dataset consisting of 2,225 English documents from the BBC News website (Greene and Cunningham, 2006). The articles correspond to stories from 2004-2005 in the areas of business, entertainment, politics, sport, and technology. The dataset exhibits minor class imbalance, with sports being the ma- jority class with 511 articles, while entertainment the smallest one with 386 samples.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, 33(00);7770-7377.
Hao Yuan, Fan Yang. Mengnan Du Shuiwang Ji, and Xia Hu. 2021. Towards structured nlp interpretation via graph explainers. Applied Al Letters. 2(4):e58.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015, Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Hyperpartisan News Detection (HND). A dataset for binary news classification (Kiesel al., 2018). Although comprises two parts, byarti- cle and bypublisher, this study only uses the first one. The dataset has 645 English samples labeled through crowdsourcing, with 238 (37%) labeled as hyperpartisan and 407 (63%) as not hyperpartisan. The challenge of this task is to detect the hyper- partisan language, which may be distinguishable from regular news at the levels of style, syntax, semantics, and pragmatics (Kiesel et al., 2019).
Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang. 2020. Every document owns its structure: Inductive text classification via graph neural networks. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 334-339, Online. Association for Computational Linguistics.
A Dataset Deseriptions
We provide detailed description of the datasets used for our text classification experiments. All of them were labeled by experts and validated by the community.
B Word Embeddings for Node Initialization
In the following, we provide further more detailed investigations pertaining to the choice of word em- beddings to initialize node representations.
App Reviews. The dataset is a collection of 288,065 English user reviews of Android applica- tions from 23 different app categories (Grano et al., 2017). The goal of the dataset is the fine-graine fine-grained sentiment analysis in an imbalanced setting, where 60.5% of the total samples correspond to 4-star reviews. Each example includes the name of the software application package, the comment, the date when the user posted the evaluation, and the rating provided.
B.1 Intuitive Graphs
We include the results reported by the GNN models trained on the different datasets using four different node feature initialization strategies.
'BC full text: http:///deekggeenee wwwpr/eeeeee..oo..beeeeeeeeeee com/bbc/ Ptttps: / /ttp:/teeooooo////////////../ org org/HNDrecord

---

The results are shown from Table 5 Table and include BERT pre-trained word embeddings (BERT), contextualized BERT (BERT-C), GloVe, and Word2Vec.
D Runtime & Resource Utilization
Table 12 presents additional information concern- ing the execution time for graph models. The aver- age total execution time is broken down into graph representation generation time and GNN running time.
Each table presents the accuracy and macro aver- aged Fl-score as averages over 10 runs. Note that the underlined embedding strategy is the one that attained the best performance, as shown Table and Table 3.
All the experiments conducted in this study were run on an NVIDIA RTX A6000 with 48GB VRAM.
To complement the results reported in Table 4, we measured the GPU utilization (%) and GPU memory usage (%) for each of the models. We also measured these metrics for each graph construction when applied to each of the datasets to find out how the strategies behave when scaling to longer documents. We tracked model performance by using Weights Biases (W&B) platform. We reran all the models using the same batch size for a fair comparison.
B.2 TextLeveIGCN
As discussed in Section 3.1, one of the main con- tributions of TextLeveIGGN is that allows du- plicate nodes when term occurs more than once in the input text. Therefore, it takes care of poly- semy. Hence, using the message-passee parseeggssnnnggg function, the model can infer the proper meaning of the to- ken given its local context. Given this peculiarity. we exclude contextualized BERT (BERT-C) node feature initialization strategy. Thus, the per- formance of TexILevelGCN was analyzed using BERT pre-trained word embeddings, GloVe, and Vord2Vec. Note that the underlined embedding strategy is the one that attained the best perfor- mance, as in Table 3. The results are presented in Table 10 and correspond to the average over 10 independent trials.
Table 13 suggests: i) The increase in GPU utiliza- tion is minimal as the document length increases. Specifically, the document length increases by one order of magnitude, GPU utilization increases by about 1.5% when employing Intuitive Graphs and 8- 6-10% for TLGCN. ii) The GPU memory allo- cated for graph strategies is constrained below 6%, representing a mere fifth of the memory con- sumed by BERT and less than a tenth of the mem- ory consumed by Longformer. This is a significant consideration when computational resources are restricted.
Transformer-based language models
order to provide results on a broader spectrum regarding the behavior of Transformer- based LMs, we performed additional experiments using the pre- trained BERT and Longformer models. The corre- sponding results are shown in Table 11.
E Libraries Used
In order to provide the reader and practitioners with the necessary details to regenerate the reported results, Table 14 presents all the libraries used to perform the experiments.
pre-trained BERT-base uncased model was in- cluded by freezing the encoder architecture and stacking final dense layer for conducting the corresponding text classification, as done for the fully fine-tuned version described in Section 3.3.2.
The same process was followed for the pre-trained Longformer-base In this case, conducted exper- iments setting a maximum sequence length of 512, and 1,024. This was done to have fair comparison regarding BERT and thus separate the effect that attention has on both approaches.
For training, we used Adam optimizer (Kingma and Ba, 2014) with an initial learning rate of 10-4 a batch size of 64 samples, 100 epochs as max- imum, and carly stopping with patience 10. Only for HND dataset, the patience was 20.
hhttps://docs.w http://ddoos..adddo...o.a www.b. wwwpwaaa//guuullee////aaaaaaauue
system-metrics

---

Window
Windowesd
Sequence
Sequencesiun
Emb.
Layers
Units
Acc
Fl-ma
Acc
Fl-ma
Acc
FI-ma
Ace
Fl-ma
16
63.4
32.4
59.8
32.4
39,9
23.4
63.2
32.4
32
61.6
34.0
58.0
31.9
57.9
26.5
60.3
33.2
64
60.3
32.0
57.9
31.9
57.1
26.6
59.3
33.4
BERT
16
63.9
29.6
60.2
31.0
59.6
23.4
64.0
27.5
34.8
32.7
59.0
22.6
33,2
60.0
33.9
57.8
32.6
57.3
25.1
60.5
33.2
16
62.2
30.2
62.4
29.8
32
61.1
32.0
60.0
32.5
57.1
31.8
58.7
31.1
56.7
BERT-C
16
29.6
62.5
28.3
60..8
24.9
63.0
26.6
3
32
60.4
32.4
60.1
32,2
57.8
31.0
60.6
32.2
59.7
31.7
60.8
32.5
56.7
31.0
16
63.2
31.4
63.4
32.1
63.4
27.3
64.5
31.0
2
32
61.2
34.0
60.8
33.8
59.5
33.4
63.3
33.1
64
59.6
32.9
60.2
33.0
58.3
34.3
61.2
33.8
GloVe
16
64,5
28.8
63.8
30.7
63.1
27.0
64.9
28.6
3
32
61.2
32.9
61.1
34.2
61.6
32,1
62.5
32.8
64
59.8
34,3
59.7
33.5
59.0
34,4
60.4
34.4
16
64.0
32.7
64,4
33.8
63.1
28.2
64.8
33.7
2
32
62.1
34.0
63.1
34.2
60.9
31.1
62.9
34.7
64
61.7
35.0
60.9
34.5
59.9
33.4
62.2
34.3
Vord2Vec
16
64.7
31.0
63.6
33.9
63.3
26.4
65.3
29.1
32
62.0
34.9
63.2
35.0
62.0
31,0
+63.7
*35.7
64
61.1
35.1
62.4
35.4
60.0
33.0
62.5
34.8
Table 5: Word embedding (Emb.) effect on App Reviews. Accuracy and macro averaged Fl-score for Intuitive Graphs using GIN convolutional neural network.
Window
Window nxe
Sequence
Sequene
Emb.
Layers
Units
Acc
F1-ma
Ace
F1-ma
Acc
F1-ma
Ace
F1-ma
16
95.9
95.8
95.8
95.8
95.9
95.8
95.8
95.7
32
95.9
95.9
95.9
95.9
95.9
95.9
96.0
95,9
64
95.8
95.8
95.9
95.9
96.0
95.9
95.9
95,9
BERT
16
95.6
95.5
95.5
95.4
95.6
95.3
95.6
95,5
2
32
95.5
95.4
95.5
95.4
95.6
95.5
95.4
95.4
64
95.3
95.3
95.2
95.1
95.3
95.3
95.3
95.3
16
+97.5
*97.4
97.3
97.3
97.3
97.2
97.3
97.3
32
97.2
97.2
97.3
97.2
97.0
96.9
97.0
97.0
64
97.1
97.1
97.1
97.1
96.7
96.7
97.0
96.9
BERT-C
16
97.4
97.3
97.3
97.3
97.3
97.3
97.3
97.3
2
32
97.2
97.2
97.3
97.3
97.0
97.0
97.2
97.2
64
97.3
97.2
97.3
97.3
97.0
97.0
97.1
97.0
95.9
95.9
95.8
95.8
96.0
95.9
95.9
96.1
96.0
96.0
95,9
96.0
96.0
95.8
96.0
95.9
95.9
GloVe
95.8
95.8
95.8
95.9
95.9
95.8
95.7
95.6
95.9
95.9
95.7
95.7
95.8
95.8
16
95.9
95.8
95.7
95.6
95.7
95.7
95.8
95.8
32
96.0
96.0
95.8
95.7
95.7
95.7
95.8
95.8
64
95.9
95.9
95.5
95.4
95.6
95.5
95.7
95.7
Word2Vec
16
95.6
95.5
95.4
95.3
95.6
95.5
95.7
95.6
2
32
95.4
95.4
95.4
95.3
95.5
95.4
95.3
95.2
64
95.4
95.3
95.3
95.3
95.4
95.4
95.5
95.4
Table 6:
Word embedding (Emb.) effect DBpedia. Accuracy macro averaged F1-score for Intuitive Graphs using GAT convolutional neural network.

---

Window
Windowess
Sequence
Sequencestun
Emb.
Layers
Units
Acc
Fl-ma
Acc
Fl-ma
Acc
FI-ma
Ace
Fl-ma
16
86.8
86.8
86.3
86.3
86.6
86.6
86.4
86.4
32
86.9
86.9
86.0
86.0
86.6
86.5
86.3
86.3
64
86.7
86.7
86.0
86.0
86.3
86.2
86.3
86.3
BERT
16
86.9
86.9
86.7
86.7
86.8
86.7
86.5
86.4
86.5
86.5
86.0
85.9
86.8
86.8
86.
86.
85.7
85.7
86.3
86.2
86.2
86.1
86.2
86.2
16
85.7
85.9
85.9
84.9
32
85.6
85.6
85.5
85.5
85.5
85.3
85.3
BERT-C
16
84.6
84.5
85.0
84.9
85.1
2
32
85.2
85.2
84.9
84,9
85.3
85.3
85.3
85.2
85.3
85.3
84.6
84.5
85.6
85.6
84.9
16
85.9
85.9
85.7
85.7
86.1
86,1
85.5
85.5
32
85..3
85.3
85.2
85.2
85.8
85.8
85.5
85.5
64
85.1
85.1
84.7
84.7
85.6
85.6
85.4
85.4
GloVe
16
85.1
85.1
84.6
84.5
86.1
86.1
86.0
86.0
2
32
84.9
84.9
83.7
83.7
85.5
85,5
85.3
85,3
64
84.7
84.7
83.7
83.6
85.2
85.1
84.7
84.6
16
87.3
87.3
87.3
87.3
87.7
87.7
*87.9
*87.9
32
87.3
87.3
86.9
86.9
87.5
87.5
87.5
87.5
64
87.4
87.3
86.7
86.6
87.2
87,2
87.4
87.4
Vord2Vec
16
87.5
87.4
87.3
87.3
87.6
87.6
87.8
87.8
2
32
86.9
86.9
87.1
87.0
87.0
87.0
87.3
87.3
64
86.7
86.7
86.1
86.1
87.2
87.2
86.6
86.6
Table 7:
Word embedding (Emb.) effect on IMDB, Accuracy and macro averaged F1-score for Intuitive Graphs using a GAT convolutional neural network.
Window
Window nxet
Sequence
Sequene
Emb.
Layers
Units
Acc
Fi-ma
Ace
Fi-ma
Acc
F1-ma
Ace
F1-ma
16
96.9
96.7
97.1
97.1
97.0
96.9
96.7
96.5
3
32
96.5
96.3
96.9
96.8
96.4
96.3
96.7
96.5
64
96.5
96.3
97.0
96.9
97.0
96.8
96.7
96.5
BERT
16
96.5
96.4
96.7
96.6
96.7
96.5
96.9
96.7
4
32
96.4
96.4
96.3
96.3
96.0
95.8
96.0
95.8
64
96.5
96.4
96.7
96.7
95.8
95.6
96.4
96,2
16
96.2
96.1
96.7
96.6
96.4
96.3
96.1
96.0
3
32
96.1
96.0
96.8
96.7
96.5
96.3
96.8
96.7
64
97.0
96.9
96.0
96.0
96.7
96.5
96.0
95.8
BERT-C
16
96.2
96.1
96.8
96.7
96.6
96.5
96.5
96.4
32
96.4
96.3
96.8
96.7
96.5
96.4
96.4
96.3
64
96.6
96.5
96.7
96.6
96.6
96.5
96.2
96.1
16
97.6
97.5
98.0
97.9
97.9
97.8
32
97.5
97.4
97.9
97.8
97.8
97.6
97.5
97.7
97.6
97.6
97.6
97.7
97.2
GloVe
97.7
97.8
97.7
97.6
97.6
97.8
97.7
16
96.9
96.8
97.5
97.4
97.3
97.2
97.1
96,9
3
32
97.1
97.0
97.1
96.9
97.1
96,9
97.5
97.3
64
97.3
97.2
96.8
96.6
97.6
97.4
97.7
97.5
Word2Vec
16
96.9
96.8
97.5
97.3
97.3
97.2
97.2
97.0
32
97.1
97.0
97.5
97.3
97.6
97.4
97.4
97.3
64
96.9
96.8
97.6
97.4
97.4
97.2
97.3
97.0
Table 8: Word embedding (Emb.) effect on BBC News. Accuracy and macro averaged F1-score for Intuitive Graphs using GAT convolutional neural network.

---

Window
Windowext
ext
Sequence
Sequencesiup
Emb.
Layers
Units
Acc
Fm-ma
Acc
FI-ma
Acc
FI-ma
Ace
FI-ma
16
77.6
76.8
75.2
73.9
56.6
36.1
77.4
76.5
32
75.3
73.6
77.4
76.8
56.6
36.1
78.3
77.6
64
77.1
76.5
76.9
75.8
56.6
36.1
*79.1
78..5
BERT
16
76.7
75.8
74.9
73.9
56.6
36.1
73.5
70.9
3
32
75.7
73.9
75.2
73.5
56.6
36.1
77.9
77.1
64
77.2
76.6
75.6
74.6
56.6
36.
77.3
76.
16
73.6
73.0
71.6
70.8
72.8
72.5
66.4
65.6
32
74.0
73.6
73.1
71.4
70.2
69.3
67.7
66.4
74.0
73.2
71.8
70.6
70.5
69.1
67.8
66.7
BERT-C
16
72.8
72.0
70.8
69.9
72.2
71.8
68.0
66.8
73.6
71.9
70.8
70.5
69.4
72.7
72.0
71.5
70.1
70.0
69.6
65.4
16
73.5
71.9
70.9
69.8
68.4
66.4
70.7
69.3
32
73.6
72.6
72.2
71.3
70.2
69.3
73.7
73.0
76.7
75.9
73.9
73.0
70.2
73.0
72.3
GloVe
16
74.3
72.9
69.1
68.0
66.9
63.1
74.3
73.5
3
32
74.7
73.5
72.3
71.5
69.7
67.8
74.7
73.7
64
73.7
73.0
74.3
73.4
70.8
70.1
75.0
74.4
16
73.3
73.2
74.0
73.4
59.1
42.7
72.3
71.7
2
32
75.0
74.7
73.0
72.0
71.0
69.4
72.6
72,0
64
73.1
72.7
75.6
74.9
66.0 0
57.8
73.5
73.2
Word2Vec
16
73.3
72.7
74.2
73.7
39.8
43.1
72.5
71.4
3
32
74,5
73.9
74.5
74.1
68.4
62.6
73.2
72.8
64
74.0
73.5
75.0
74.5
61.4
47.6
75.3
75.0
Table
Word embedding (Emb.) effect on HND. Accuracy and macro averaged Fl-score Intuitive Graphs GIN convolutional neural network.
using
I-gram
2-gram
3-gram
4-gram
Dataset
Emb.
Acc
F1-ma
Acc
F1-ma
Acc
FI-ma
Acc
F1 F1-ma
BERT
65.9
35.0
64.1
34.5
65.9
35.0
63.4
34.6
GloVe
65.7
34.4
65.9
34.2
65.5
34.1
64.2
33.9
App Reviews
Word2Vec
66.6
34.7
64.7
35,2
*64.5
*35.8
64.3
35.5
BERT
95.7
95.7
*96.1
*96.0
95.9
95.9
96.0
96.0
GloVe
95.7
95.6
95.8
95.8
95.6
95.6
95.6
95.6
DBpedia
Word2Vec
95.7
95.7
95.7
BERT
86.7
86.7
85.9
85.9
85.7
85.7
85.7
85.7
GloVe
86.2
86.2
86.4
86.4
86.7
86.4
86.4
IMDB
Word2Vec
+86.8
*86.8
86.5
86.4
86.2
86.2
BERT
97.0
97.0
97.2
97.2
+97.3
*97.3
97.0
97.0
GloVe
96.1
96.1
96.1
96.1
95.5
95.5
96.1
96.1
BBC News
Word2Vec
96.4
96.4
96.6
96.5
96.8
96.8
96.5
96.5
BERT
74.8
70.8
74.5
71.5
72.6
69.6
74.1
71.6
GloVe
73.8
71.0
69.8
67.2
72.1
70.2
73.8
72.0
HND
Word2Vec
+75.7
+73.4
71.6
67.9
72.2
69.8
70.4
67.0
Table 10: Word embeddings as TextLevelGCN node initialization. Accuracy and macro averaged F1-score are reported.

---

Dataset
Model
Length
FZ
Acc
61.3
18.4
BERT
512
62.0
36,9
App Reviews
60.4
512
64.0
36.8
Longfomer
60.4
15.1
1024
63.5
37.6
74.6
74.1
BERT
512
98.3
98.3
82.5
81.0
DBpedia
512
98.2
98.2
Longfommer
82.0
80.3
1024
98.1
98.1
68.6
67.7
BERT
512
88.4
88.4
77.9
77.7
IMDB
512
90.3
90.3
Longformer
78.2
78.0
1024
90.5
90.5
52.5
BERT
512
97.8
84.1
512
News
98.2
Longformer
85.0
1024
98.2
98.2
52.2
34.5
BERT
512
72.6
70.6
56.6
36.1
HND
512
79.0
78.1
Longformer
56.6
36.1
1024
77.2
75.5
Table 11: Maximum input length effect on perfor- mance. Accuracy and macro averaged F1-score for BERT and Longformer variants are reported as an av- erage over 10 independent executions. A checkmark in the FZ column indicates that the corresponding results were obtained by freezing the model.
Execution Time s
Strategy
Dataset
Gen
Run
Total
App Reviews
29.1
139.7
168.8
DBpedia
92.7
291.6
384.3
Intuitive
IMDB
256.5
217.1
473.6
Graph
BBC News
101.6
69.1
170.6
HND
48.8
67.4
116.3
App Reviews
71.1
475.3
$46.4
DBpedia
93.7
333.2
426.8
Textl evel
IMDB
487.4
534.8
1,022.3
GCN
BBC News
271.2
413.1
684.2
HND
193.5
233.3
426.8
Table 12: Graph execution time. Average execution
time for Intuitive Graph and TextLeveIGCN approaches. It includes graph generation time (Gen) and GNN run- ning time (Run).

---

App Reviews
DBpedia
IMDB
BBC News
HND
Method
Util.
Mem.
Util.
Mem.
Util.
Mem.
Util.
Mem.
Util.
Mem.
Window
4.74
3.67
4.53
4.79
4.81
Windowest
4.74
3.60
4.73
4.83
4.84
2.53
Sequence
4.74
2.87
4.74
3.93
4.79
3.67
4.79
2.47
4.83
Sequenccsimp
4.74
3.73
4.74
4.27
4.79
4.79
2,00
Textl evelGGNN 1-g
4.07
5.00
6.73
5.12
12.33
5.56
9.07
5.41
6.13
TextLeveIGCN 2-g
3,93
5.00
6.80
5.13
13.40
5.56
8.20
5.55
4.60
5,29
TextLeveIGGN 3-g
3.67
5.00
6.53
5.13
10.40
5.71
6.80
5.58
3.93
5.37
TexILeveIGCN 4-g
4.33
5.00
5.40
5.13
9.53
5.86
4.13
5.62
3,93
5.32
BERT
94.47
29.42
94,70
29.42
95.27
29.42
89.40
29.42
68.93
29.42
Longformer
99.27
67.86
99.27
67.86
99.60
67.86
99.80
67.86
99.40
67.86
Table 13: GPU statisties (%). GPU utilization (Util.) and GPU memory usage (Mem.) for each of the studied models. The i-g notation accompanying TextLeveIGCN stands for i-gram graph construction.
Library
Version
datasets
2,4.0
gensim
4.2.0
nltk
3.7
numpy
1,23.1
pytorch-lightning
1.7.4
scikit-learn
1.1.
torch
1.110
torch-cluster
1.6.0
corch-geometric
2110
torch-scatter
20.99
torch-sparse
0.6.15
torch-spline-con
1.2.1
torcmmetrics
0,9.3
torchvision
0.12.0
transformers
4,21.2
word2vec
0.11.1
Table 14: Libraries. Versions of Python libraries used for the experimental implementation.


Reviews:
[{"review_id": "UQuoLaP1cx", "reviewer": "Reviewer_bG3X", "summary": "", "questions": "Section 4.1 mentions GCN and GIN and says the properties of each, that GIN will be better than GCN, how should this be interpreted in a text classification task and why is GIN better than GCN in text categorization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your valuable suggestions and truly appreciate the thorough review. We address your comments and concerns as follows.\n\n#### **Introduction**\nWe will revise the relevant paragraph to enhance the clarity of our motivation. Specifically, there are  three key points we will highlight, while substantiating our claims with the appropriate references:\n1. Most previously proposed approaches were created with a focus on specific tasks within specific domains.\n2. The validation of previous graph proposals was performed on a limited range of datasets and model architectures.\n3. Earlier graph proposals were developed before Graph Neural Networks (GNNs) emerged and validated using graph mining techniques. \n\n#### **Related Work**\nTo present related work in a more logically structured manner, we will categorize the information into three subsections based on the type of graph representation proposed, as follows:\n* Early Graph Constructions: The first proposed graph constructions, based mainly on co-occurrence and statistics, and validated using graph mining. \n* Linguistic Features as Graphs: Constructions integrating lexical, semantic, or syntax information as more advanced graph representations.  \n* Specialized Graph Constructions: Recently proposed methods, including more complex constructions and integrating Graph Convolutional Neural Networks (GCNs) in the learning process.\n\n#### **How GNN architecture affects the results**\nWe agree that a finer analysis regarding the choice of the GNN architecture is essential to fully comprehend a model’s effectiveness under diverse conditions. Due to space constraints, we included experiments with different numbers of layers in Appendix B, focusing our discussion on message-passing methods and node initialization. \n\nOur results suggest that effective node feature initialization yields good results with fewer hidden dimensions in most tasks. Exceptions are observed on App Reviews and HND due to their intricacies, as outlined in Section 4.2. \nAdditionally, the number of convolutional layers is linked to the task. If the task focuses on local patterns, using 1 or 2 layers is recommended. But if the task needs broader information, more layers will be needed. Yet, understanding context (local/global) varies with document length. For instance, to comprehensively grasp the document's sentiment, a sentence-level analysis is vital, whereas if the document comprises only one or two sentences, a wider view will be required. This is shown in Table 2 and Table 5, where using 3 layers produced the best App Reviews results.\n\n#### **Question about GIN**\nAs stated in Section 4.1, given a node and its neighbors, GCN computes the average of all those representations in one undifferentiated step. Conversely, GIN's weighted summation aggregates neighbor node representations to the target node while incorporating a learned variation (epsilon). As a result, GIN distinguishes node neighborhoods, discerning structural information among graph classes, unlike GCN. \nSince our document graphs are based on word co-occurrence, this means that GIN can exploit structural patterns of the co-occurrences and thus identify recurrent associations between specific words, which can be crucial to predict the correct label of the graph. \n\n#### **Reproducibility**\nTo ensure the reproducibility of our research, we provided supplementary experimental details in Appendix A, C, and E, regarding the Python framework, further model hyperparameters, and GPU details. If preferable, we can move some of this information to Section 3.3 to support the reader's understanding. Please also note that we will release our code in a publicly available GitHub repository once the Anonymity Period ends.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis article systematically explores the performance of using Graph Neural Networks (GNN) for graph representation learning in text classification. Furthermore, it compares GNN with other mainstream text classification methods, summarizing the advantages and limitations of GNN in the field of text classification.\n2.\tThis article provides a comprehensive experimental design and a particular explanation of the experiments conducted.\n", "reasons_to_reject": "1.\tThe introduction of this article lacks a clear explanation of its motivation, and it is essential to provide necessary references to previous related works when mentioned.\n2.\tIn the second section, while many related works are presented, the lack of logical coherence results in a somewhat disorganized content.\n3.\tRegarding Section 4.1, I believe the author did not sufficiently explain how the architecture and settings of GNN impact the results of text classification. In this section, the author only presented the experimental results and the GNN used, without effectively integrating the text classification task with the chosen GNN characteristics for a comprehensive explanation.\n", "questions_to_authors": "Section 4.1 mentions GCN and GIN and says the properties of each, that GIN will be better than GCN, how should this be interpreted in a text classification task and why is GIN better than GCN in text categorization?", "paper_topic_and_main_contributions": "This article mainly explores the advantages and limitations of using graph representation learning in text classification. The article compares different graph construction methods and various graph-based text representation techniques, evaluating their performance in text classification tasks. Additionally, it compares these methods with mainstream non-graph-based text classification approaches to investigate the applicability and advantages of graph representation learning in text classification tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sjysHX2P35", "reviewer": "Reviewer_KR2u", "summary": "", "questions": "1. This paper primarily demonstrates the effectiveness of Graphs in text classification tasks, an aspect that has been discussed in various previous works.\n\n2. The experimental results in this paper are primarily based on the existing work and reproduction of TextLevelGCN. However, it remains unclear what specific contributions the authors have made in this domain. If there are any, further clarification is needed.\n\n3. Why did the author only use the basic 'bert-uncased-base' model in the experiments and did not explore other potentially superior pre-trained models for performance validation?\n\n4. The authors did not present performance evaluations when using graphs to model different text datasets. There is a concern about the potential increase in machine resource utilization when employing graphs to represent textual data, which could hinder practical usage in real-world scenarios. The authors are encouraged to provide further explanations on this matter. \n\nPlease provide an explanation for the concerns mentioned above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time you dedicated to reviewing our paper. Your comments are warmly welcomed. In the following lines, we address each of your suggestions and concerns. \n\n#### **Generalizability of the findings** \nWe agree that the investigation ought to be conducted across a wide range of scenarios to ensure the generalizability of results. Since most of the previous work only focused on topic classification on short documents (Reuters 21578, Ohsumed, 20Newsgroups), our study specifically had the goal of assessing the graph strategies on a wider range of text classification tasks: sentiment analysis, topic classification, and hyperpartisan detection in balanced and unbalanced scenarios (including much longer documents as well).\n\nSince our results show that graph effectiveness is highly dependent on textual features and domain, we are enthusiastic about extending our study to further tasks in future work. To this end, we are releasing our code on GitHub and hope that it can grow to become a community resource covering even more tasks.\n\n#### **Question regarding previous work**\nIndeed, numerous prior studies have demonstrated graphs' efficacy in the field. However, they only validated the effectiveness on a small range of tasks with only short documents. Instead, we seek to comprehensively assess the merits and drawbacks of different previously proposed document graphs and variants in more diverse scenarios. \n\nText classification extends beyond topic classification, encompassing real-world challenges like noisy texts, imbalanced labels, and much longer texts consisting of more than a few paragraphs. Our experiments for evaluating how graphs generalize to other tasks and our findings substantiate our contributions to the field. \n\n#### **Contributions**\nWe understand the reviewer's concern and are happy to outline our contributions. We will extend the corresponding sections to clarify the paper's arguments and findings.  \n1. We evaluate previously proposed graph construction methods for text classification on a broader range of tasks, domains, and datasets.\n2. We evaluate the efficacy vs. efficiency of the proposals, an aspect usually neglected in previous studies. We found that graphs can hardly outperform Transformer-based LLMs, but are particularly efficient in terms of execution time as well as resource utilization. \n3. We found that document graphs are especially beneficial for long-text classification, an aspect unexplored in prior research.\n4. We identify limitations regarding the graph construction, a nuanced process dependent on text features and data domain. Moreover, based on our experimental results, we provided a discussion regarding what graph construction and GNN architecture choice is preferable depending on the task to be solved. \n\n#### **Pre-trained models**\nAs mentioned above, the goal of our study is not to obtain state-of-the-art results on the respective datasets using the latest LLMs, but rather to consider the merits of graphs, which also require far fewer computational resources.  Hence, results with BERT-base are provided merely as an informative point of reference for comparison.  We further included Longformer-base as a pre-trained language model that can cope with longer sequences of text as well. This aspect was desirable when comparing LLMs to graphs, which use the complete source text.\n\n####  **Performance Evaluation**\nWe have measured the GPU utilization and GPU memory usage for each of the models presented in Table 4 to complement the already reported performance metrics. We also measured these metrics for each graph construction when applied to each of the datasets to find out how the strategies behave when scaling to longer documents (*we rerun all the models using the same batch size for a fair comparison*). The results suggest:\n* The increase in GPU utilization is minimal as the document length increases. Specifically, as the document length increases by one order of magnitude, GPU utilization increases by about 1.5% when employing Intuitive graphs and 8-10% for TLGCN. \n* The GPU usage of graph strategies is limited to only 5% (Intuitive graphs) and 14% (TLGCN) of that used by Transformer-based language models, an important aspect when computational resources are limited.\n\nWe would be pleased to integrate the analysis of our newly obtained results into the discussion of runtime in Section 4.3. \n\n#### **References**\nWe will include three recent articles in our literature review section.\n\n[1] is a heterogeneous graph construction, introducing topic nodes to enhance class-aware representation learning. However, it has the same limitations as TextGCN (high computing resource consumption and transductive learning). Hence, learning for new documents is not possible without re-training the model). \nAlternatively, [2,3] are inductive models that have reported good results on traditional text classification benchmarks, but the improvement is mostly due to the combination of GNN and BERT models. Thus, these strategies are resource-intensive, hard to apply to long documents, and beyond the scope of our study.\n \n- [1] Gu, Y., Wang, Y., Zhang, H. R., Wu, J., & Gu, X. (2023). Enhancing Text Classification by Graph Neural Networks With Multi-Granular Topic-Aware Graph. IEEE Access, 11, 20169-20183.\n- [2] Huang, Y. H., Chen, Y. H., & Chen, Y. S. (2022, October). ConTextING: Granting Document-Wise Contextual Embeddings to Graph Neural Networks for Inductive Text Classification. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 1163-1168). \n- [3] Wang, Y., Wang, C., Zhan, J., Ma, W., & Jiang, Y. (2023). Text FCG: Fusing contextual information via graph learning for text classification. Expert Systems with Applications, 119658.\n\n#### **Reproducibility**\nTo ensure the reproducibility of our research, we have provided supplementary experiment details in Appendix A, C, and E, regarding the Python framework, further model hyperparameters, and GPU details. If preferable, we can move some of this information to Section 3.3 to support the reader's understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results indicate several key findings: Firstly, the effectiveness of graphs depends on the textual input features and domain, with simple graph constructions performing better as the documents become longer. Secondly, graph representations show particular benefits for longer documents, outperforming Transformer-based models. Thirdly, graph methods demonstrate notable efficiency in addressing the text classification task.", "reasons_to_reject": "The author may need to further consider potential limitations, such as the specific selection of datasets and the need for further investigation into the generalizability of the findings across different text classification tasks and domains. \n\nPlease refer to the questions for the authors.", "questions_to_authors": "1. This paper primarily demonstrates the effectiveness of Graphs in text classification tasks, an aspect that has been discussed in various previous works.\n\n2. The experimental results in this paper are primarily based on the existing work and reproduction of TextLevelGCN. However, it remains unclear what specific contributions the authors have made in this domain. If there are any, further clarification is needed.\n\n3. Why did the author only use the basic 'bert-uncased-base' model in the experiments and did not explore other potentially superior pre-trained models for performance validation?\n\n4. The authors did not present performance evaluations when using graphs to model different text datasets. There is a concern about the potential increase in machine resource utilization when employing graphs to represent textual data, which could hinder practical usage in real-world scenarios. The authors are encouraged to provide further explanations on this matter. \n\nPlease provide an explanation for the concerns mentioned above.", "paper_topic_and_main_contributions": "The paper highlights the exploration of Graph Neural Networks (GNNs) in the context of text classification, considering their demonstrated success in machine learning. The researchers extensively investigated various graph representation methods and text representation methods, revealing practical implications and identifying open challenges in this area. The study compares different text classification schemes across multiple datasets, considering varying document lengths. Additionally, the inclusion of two Transformer-based large language models complements the analysis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The references cited in this paper are not up-to-date, with only two references from the year 2022 being the most recent. If possible, the authors are advised to conduct further research on more recent studies related to the relevant task, especially the latest research concerning text classification."}}, {"review_id": "yHH6RaM7pw", "reviewer": "Reviewer_LsjL", "summary": "", "questions": "- The authors do not mention if they include the graph construction step to calculate the execution time.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your thoughtful assessment and feedback, which is very valuable for improving our work. We address all the points detailed in your review as follows. \n\n#### **Performance Gains**\n\nThanks for sharing your concern with us. We have already provided details regarding the performance of the corresponding models in Table 4. We recently also measured the GPU utilization of the studied models. There are two important points to be made: \n1. Despite a performance gap between graph strategies and LLMs when dealing with short texts as in App Reviews, this gap becomes narrower as documents become longer. Graph representations aim to exploit the structure of a given text, so they are expected to perform better on longer documents, since there is more information and structural patterns to exploit.\n2. Even when an additional step is needed to create the graph representation of a document, our experiments indicate that the total execution time (graph creation step + model running) is not a concern. For example, for short texts as in DBpedia, the window graph is competitive with the best performing LLM, showing a difference of only 0.8% of accuracy and being 5.7 times faster. Similarly, BERT beats Sequence graph in IMDB by only 0.5% in accuracy, while being 3.5 times slower. \n\nWe will be glad to extend this analysis in our paper to showcase the behavior of graphs. \n\n#### **BERT initialization** \nSince both models use pre-trained BERT embeddings as initialization, we can note that the differences are due to how both models operate.\n\nThe GNN models considered in our study are simple in terms of architecture and number of hyperparameters when compared to BERT. Therefore, they tend to fail when aggregating information from neighborhoods described by higher-dimensional vectors (BERT/BERT-C with 768 dim.). Instead, simpler vectors such as Word2Vec or Glove (300 dim.) allow GNNs to better share and extract information between different neighborhoods via message passing, being able to exploit the local context as well as the global context of the represented documents.\n\nIn contrast, BERT can process more specialized vectors, as it has a much larger number of parameters, more neural layers, and a quadratic attention module. Having a quadratic attention module is comparable to having a fully connected graph model, being able to attend to every single pair of elements of the input text. \n\n#### **Why is structural information useful?**\nDocument comprehension involves interpreting words that can alter the meaning of the text based on their placement. For example, in \"*The movie was boring, but I was surprised by the ending*\", the word \"*but*\" contrasts ideas. Graphs can capture this complexity, representing associations and dependencies among text units, leveraging language structure. Unlike traditional models, GNNs are capable of capturing information over longer distances through message passing, in considerably fewer steps, as Figure 2 shows. \n\nIt is important to emphasize that, although graph models are competitive for solving text classification tasks in different domains, they are especially beneficial for sentiment analysis and hyperpartisan detection, benefitting from local context exploration. \n\nWe will extend the Introduction to clarify the motivation for using graph representations for text classification. Thank you for bringing this to our attention. \n\n#### **Question about Execution Time**\nThe graph construction step is included in our execution time calculation. Graph creation and model runtime details are provided in Appendix D, Table 12. \n\nIn Section 4.3, we suggested referring to Appendix D for runtime details, but we will also include this information in Table 4 to enhance clarity. We appreciate your observation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Comprehensive analysis: The authors perform an extensive analysis by considering three types of convolutional layers as different message-passing strategies for 12 Graph Neural Network (GNN) architectures. \n\n- GNN applicability in non-graph datasets is a growing body of research and this work furnishes multiple baselines highlighting the potential of these models as effective alternatives to traditional methods.\n\n", "reasons_to_reject": "- Graph based methods for text classification introduce an extra pre-processing step to convert sequential data to graphs. I don't see a lot of performance gains given the overhead (except in long pieces of text)\n\n- General performance (table 4) indicates that graph models initialized with BERT embeddings almost always perform worse as compared to BERT, What is the intuition behind this?\n\n- The authors don't provide a clear reasoning as to why structural information could be useful for text classification of datasets mentioned. For instance, how would the graph structure capture sentiment (IMDB) in a better manner as compared to traditional NLP approaches mentioned\n\n", "questions_to_authors": "- The authors do not mention if they include the graph construction step to calculate the execution time.", "paper_topic_and_main_contributions": "The authors investigate the usage of graph representation methods for text classification and empirically show that graph methods are efficient and effective for solving text classification tasks, highlighting their potential as a valuable alternative to classical machine learning and graph mining approaches in modern settings. They perform an extensive analysis by using 3 types of convolutional layers as different message-passing strategies for 12 GNN architectures. Some of these GNNs also use pre-trained word embeddings as node features. The authors also present  2 transformer-based large language models for complementing the study and providing a comparison with traditional GNN approaches. The authors note that simple graph constructions perform better as document length increases, demonstrating that graph representations are particularly beneficial for longer documents compared to Transformer-based models.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the use of Graph Neural Networks (GNNs) in text classification and conducts a thorough analysis of different graph representation and text representation methods. The study compares the performance of GNNs with mainstream text classification methods and addresses questions related to their efficiency and effectiveness for text classification tasks. The paper's strengths lie in its comprehensive experimental design, extensive analysis, and clear presentation. However, there are some concerns about the motivation, the original contributions, and the lack of explanation regarding the choice of pre-trained models.

One of the paper's significant strengths is its comprehensive analysis of GNNs for text classification, covering various GNN architectures and message-passing strategies. This thorough examination provides valuable insights into the applicability of GNNs in this domain. Additionally, the study's systematic comparison with other text classification methods enhances its overall quality.

However, the paper faces some challenges. First, the motivation for the research and its original contributions should be clarified further. The paper appears to build upon existing works without explicitly stating its unique contributions to the field. Second, the choice of using the 'bert-uncased-base' model in experiments without exploring potentially superior pre-trained models needs justification. Third, while the study demonstrates the efficiency and effectiveness of GNNs for text classification, there is a need for more extensive performance evaluations on different text datasets to assess potential resource utilization issues. Lastly, the paper should provide a clearer rationale for why structural information captured by GNNs is beneficial for specific text classification tasks, such as sentiment analysis.
Decision: Accept-Findings