## 3 Methods
### Graph Construction
As previously mentioned, in this work we use BERT and ParsBERT frameworks to obtain initial representations of documents/words; then, as we will explain below, we use GNN structures to turn these representations into an even richer one by capturing the underlying graph structure associated with the data. Once the text graph is constructed, it is fed to a two-layer GCN. Each GCN layer performs message passing among the nodes based upon their neighborhood relationships; this allows for the integration of information from larger neighborhoods. We review the message passing step more in detail. As previously mentioned, the initial representation vectors made by BERT and ParsBERT are given as the initial features of the nodes to the input of the graph convolutional neural network (GCN). The information of these nodes is conveyed by the process of messages passing through graph neural networks in such a way that each node in the graph computes a message for each of its neighbors. Messages are in fact a function of nodes, neighbors, and the edges between them. Messages are sent and each node aggregates the messages it receives using a function such as sum or average. After receiving messages, each node updates its attributes as a function of its current attributes and the aggregated messages. The basic GNN message passing formula is defined as follows:  
\[h_{u}^{(k)}=\sigma(W_{self}^{(k)}h_{u}^{(k-1)}+W_{neigh}^{(k)}\sum_{v\in N(u)}h _{v}^{(k-1)}+b^{(k)}) \tag{2}\]where \(W_{self}^{(k)},W_{neigh}^{(k)}eR^{d^{(k)}*d^{(k-1)}}\) are trainable parameter matrices and \(\sigma\) denotes an element-wise non-linearity (e.g., ReLU). The bias term b \(\varepsilon\)\(R^{d^{(k)}}\) is also the bias term. The final feature vectors obtained for each text data node are considered as the output of the GCN, which are passed through a SoftMax classifier for final prediction. By jointly training the BERT and GCN modules, we leverage the advantages of both pre-trained models and graph models. We now address the ideas behind GCNs and how they help us in our text classification. GCNs are designed to efficiently capture semantic relationships and rich dependencies among the nodes of a graph, enabling a better understanding and representation of text content. Text classification often requires considering textual information adjacent to words or phrases for accurate predictions. GCNs can collect information from neighboring nodes in the graph, efficiently gathering and disseminating contextual information. This capability allows GCNs to take advantage of the local context of each node and make informed decisions regarding text classification problems. On the other hand, one of the challenges of text classification is addressing inputs of different lengths, such as sentences and phrases with varying numbers of words. GCNs can naturally handle variable-length inputs using the graph structure. By leveraging the graph structure, GCNs can depict relationships and dependencies between words or sentences, which provides a more robust and flexible approach for text classification. Additionally, GCNs excel in modeling global dependencies in a graph. In text classification, global dependencies refer to dependencies that encompass the entire dataset and the text. By propagating information throughout the graph, GCNs can capture these global dependencies, enabling a comprehensive understanding of textual data and their classification. According to our obtained results, the performance of most of our combinations of models improved, both for balanced data and unbalanced data, when using graph neural networks, and in particular GCNs. This improvement can be attributed to the advantages of using the graph neural networks mentioned above, as they enhance the quality of the representations and the robustness of text classification models.