Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Authors: Huang Liang
Benedict Lee
Daniel Hui Loong Ng
Kelin Xia
Venue: ICML 2025 poster (2025)
Keywords: Topological Deep Learning
Representation Learning
Message-Passing Mechanism
Primary area: general_machine_learning->representation_learning

Abstract: Text classification is a fundamental task in Natural Language Processing (NLP). Short text classification has recently captured much attention due to its increased amount from various sources with limited labels and its inherent challenges for its sparsity in words and semantics. Recent studies have adopted self-supervised contrastive learning across different representations to improve performance. However, most of the current models face several challenges. Firstly, the augmentation step might not be able to generate positive and negative samples that are semantically similar and dissimilar to the anchor respectively. Secondly, the text data could be enhanced with external auxiliary information that might introduce noise to the sparse text data. In addition, they are limited in capturing higher-order information such as group-wise interactions. In this work, we propose a novel document simplicial complex construction based on text data for a higher-order message-passing mechanism. We enhance the short text classification performance by contrasting the structural representation with the sequential representation generated by the transformer mechanism for improved outcomes and mitigated issues. The proposed framework, Contrastive Learning with Simplicial Convolutional Networks (C-SCN), leverages the expressive power of graph neural networks, models higher-order information beyond pair-wise relations and enriches features through contrastive learning. Experimental results on four benchmark datasets demonstrate the capability of C-SCN to outperform existing models in analysing sequential and complex short-text data.

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Liang Huang 12
Benedict Lee? Daniel Hui Loong Ng²
Kelin Xia
Abstract
1. Introduction
Text classification is a fundamental task in Natural Lan- guage Processing (NLP). It involves analysing the content of texts and determining which predefined category they belong based on their representation. Unlike longer texts, short texts have recently captured much attention, with an increase in the number appearing in various sources, such as social media, search snippets, and news feeds. However, these short texts with a few words pose challenges to the current models in generating effective representations and are not usually labelled in real-worls cases (Linmei et al., 2019). Supervised learning short-text classification has gained significant attention and has been applied to different tasks for web reviews (Pang & Lee, 2004), news feeds (Yao et al., 2019), and medical information (Liu et al., 2020). On the other hand, data labelling has been expensive, labour- intensive and time-consuming. Few-shot learning has been popular with low resources required by training on a few labelled samples, either with or without pre-training. Fur- thermore, graph models have been widely used to capture complex relationships between text data's structural, seman- tic, and syntactic meanings. To address the label scarcity issue, contrastive learning has been adopted to enhance per- formance. Many researchers (Sun al., 2022; Wen & Fang, 2023; Liu et al., 2024) have explored the effectiveness of combining graph models and contrastive learning within the scope of few-shot learning.
Text classification is fundamental task in Natural Language Processing (NLP). Short text classifi- cation has recently captured much attention due to its increased amount from various sources with limited labels and its inherent challenges for its sparsity in words and semantics. Recent studies have adopted self-supervised contrastive learning across different representations to improve per- formance. However, most of the current models face several challenges. Firstly, the augmentation step might not be able to generate positive and negative samples that semantically similar and dissimilar to the anchor respectively. Secondly, the text data could be enhanced with external aux- iliary information that might introduce noise to the sparse text data. In addition, they are lim- ited in capturing higher-order information such as group-wise interactions. In this work, we pro- pose novel document simplicial complex con- struction based on text data for oigher-order message-passing mechanism. We enhance the short text classification performance by contrast- ing the structural representation with the sequen- tial representation generated by the transformer mechanism for improved outcomes and mitigated issues. The proposed framework, Contrastive Learning with Simplicial Convolutional Networks (C-SCN), leverages the expressive power of graph neural networks, models higher-order information beyond pair-wise relations and enriches features through contrastive learning. Experimental results on four benchmark datasets demonstrate the capa- bility of C-SCN outperform existing models in analysing sequential and complex short-text data.
Although they have achieved successful outcomes, some limitations and challenges still exist. Firstly, the data aug- menting step and the negative sampling step of contrastive learning might distort the semantic meaning and introduce unnecessary noise. For example, removing graph com- ponents is adopted as data augmentation strategy, but it might disrupt the text's original meaning, An instance from the Movie Review (MR) dataset (Pang & Lee, 2005): "there's not enough to sustain the comedy" while remov- ing the word "not" reversely changes the meaning this short sentence. Furthermore, negative sampling of texts with different syntaxes while similar semantics might be designed to be pushed away from each other. Secondly, some auxiliary information such as entities, latent topics, and part-of-speech (POS) tags (such as nouns and verbs)
'Nanyang Technological University, Singapore "HP Inc, Singapore. Correspondenee to: Liang Huang Kelin Xia <xiakelin edu......sg......
<liang013 3 1iinno013e..t...uuu........ edu...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
might be added to graph models for language understanding and enriching the limited available local context. However, this step might introduce misinformation, such pulling documents that express opposite semantics but similar top- ics closer. Lastly, graph models are mathematically limited in modelling higher-order features, such as group-wise in- teractions among a few nodes and edges expressed in terms of phrases. For example, the short sentence "It what is" uses repetition emphasise the acceptance of the status quo. At the same time, graph models with only nodes and edges learn pairwise interaction. They need to extend the number of layers in order for words to incorporate the meaning of other words further apart. Group-wise phrase "it is" needs to be linked with "what" to model such repetition.
compared with other models and ablation studies. Finally, Section 6 concludes the work and proposes future direc- tions.
2. Literature Review
Graph neural networks (GNN) are powerful deep learning models to model representations of structural data (Scarselli et al., 2009). Through a message-passingg mechanism, fea- tures of nodes and edges are aggregated in the neighbour- hood formed by components in the local document. Texts could be used to construct different types of graphs, such as heterogeneous graphs (Yao et al., 2019), knowledge graphs (Ye et al., 2019), dynamic graphs (Chen et al., 2020), and hypergraphs (Ding et al., 2020). Early graph neural net- works, such as Graph Convolutional Networks (GCN) (Kipf & Welling, 2017), Graph Isomorphism Networks (GIN) (Xu et al., 2019) and Graph Attention Networks (GAT) (Veličković et al., 2018), are integrated with the text graphs for improved results. Recently, model fusion has been adopted by (Lin et al., 2021) to jointly train the transformer model BERT with graph models with text data. On the other hand, graph models are limited in modelling higher-order information in the group-wise form.
To address the challenges mentioned above, novel model combining higher-orde features with contrastive learning is proposed in this paper called Contrastive Learning with Simplicial Convolutional Networks (C-SCN) for short-text classification tasks. Specifically, SCN adopts simplicial complex to robustly model richer and more complex in- formation for better document understanding. Document simplicial complexes are firstly constructed based on text data, and respective features are defined for simplexes of different complexities. We further integrate the features into an inductive cassage-passingg mechanism, considering long-range structural information for individual document simplicial complexes. Furthermore, contrastive learning is embraced to compare the structural representation from SCN and sequential representation from the transformer model so that the power of both sides can be combined for better performance in few-shot learning setting.
Contrastive learning, unlike traditional supervised learning, where label is required for training, is a self-supervised technique where augmented views of the same object are used to train a model which could gather positive samples closer and negative samples further apart (Jaiswal et al., 2021). With downstream tasks, contrastive learning has been actively applied in short-text classification scenarios with few-shot settings. (Sun et al., 2022) integrates the het- erogeneous graph attention mechanism with neighbouring contrastive learning to enrich the terms beyond the docu- ment and extend the relations among documents; (Wen & Fang, 2023) pre-trains text and graph encoders followed by few-shot and zero-shot fine-tuning process; (Liu et al., 2024) innovates in augmented view of graph features through the singular value decomposition (SVD) of the feature matrix and in assigning weak labels to document through k-means clustering. On the other hand, these current methods require a large amount of resources in preprocessing in the form of pre-training or enriching text with additional information, such as entity recognition and POS tagging processes. The augmented view in contrastive learning might also introduce unnecessary noise and misleading information.
Our work's 's key contributions are as follows. pose the construction of simplicial complexes, an important concept in algebraic topology, based on text data for the first time to the best of our knowledge and define features on 0-simplexes, I-simplexes and 2-simplexes in the context of short-text classification in the message-passingg mecha- nism of SCN. Secondly, we extend SCN with contrastive learning, such as C-SCN, where the power of sequential representation from the transformer model is integrated to solve the existing limitations and challenges. Lastly, the ex- periment with C-SCN in benchmark short text classification tasks demonstrates better results than competitive baseline models in the few-shot setting.
Firstly, we pro-
The remainder of the paper is organised as follows: Sec- tion 2 reviews the literature on graph neural networks, con- trastive learning, and neural networks on simplexes. Section 3 outlines the proposed model structure for message passing on higher-order structures and contextualises the methods in text classification. Section 4 introduces four short text classification task datasets from various domains used in the experiments. Section 5 presents the performance metrics
Topological deep learning combines the techniques from deep learning and topological tools that structure data man- ifolds (Zia et al., 2024). Topological representations, in- cluding cell complexes (Hajij et al., 2020; Giusti et al., 2023; Bodnar et al., 2022), simplicial complexes (Bodnar, 2022; Schaub al., 2022), combinatorial complexes (Hajij et al., 2023), sheaves (Hansen & Ghrist, 2019) and hyper-
2

---

Contrastive Learning with Simplicial Convolutional Networks for Sext Classification
graphs (Feng al., 2018; Bai et al., 2021), model not only pair-wise interactions that are present on graph, but also higher-order interactions among three elements more. Algebraic copology-bssed methods have achieved notewor- thy results in protein analysis (Xia & Wei, 2014; Sverris- son et al., 2021; Wee & Xia, 2022), virus analysis (Chen et al., 2022), drug design (Cang & Wei, 2017) and mate- rial property classification (Reiser al., 2022: Townsend al., 2020), where topologiaal representations demonstrate their robustness against deformation and noise. Extending from algebraic topology- -ooogy methods, TDL employs the message-passing mechanism on higher-order components, where the communication of information has been propa- gated through any neighbourhood relations (Roddenberry et al., 2021; Bodnar, 2022; Hajij et al., 2023). However, there is lack of studies on non-time-series sequential anal- ysis with TDL on the text data, and we aim to explore its usage in the new field.
Self-loop 1-simplex
query
11nq
clauses
two
1
should
Backward 1-simplex
use
Forward
1 1-simplex
where
Figure I. One document simplicial complex constructed for doc- ument example from the Snippets dataset (Phan et al., 2008) with different types of flow directions. Words and punctuation are to- kenised into individual 0-simplexes (nodes). I-simplexex (edges) are formed if O-simplexes are next to each other with directions in chronological order. Lastly, 2-simplexes (triangles) are con- structed if the three words form a "filled" triangle. Three types of -simplexes are illustrated: (1) Forward I-simplexes are the ones following the chronological order which points to the word that first appears in the text; (2) Backward I-simplexes are 1 --simplexes pointing the word which is used before and referenced again; (3) Self-loop 1-simplexes are formed when 1-simplexes connect the same word.
3. Methods
3.1. Simplicial Convolutional Networks (SCN)
We first provide the necessary details related to constructing document simplicial complexes, followed by the message- passing mechanism on the higher-order structures.
Definition. (Abstract Simplicial Complex) An abstract sim- plicial complex is a family of sets K. that satisfies the con- dition: for any set a E K. every non-empty finite subset o' S a must also be in K.. Each element of IC is called a simplex. set a is referred to as a k-simplex if its cardi- nality [o| - k + 1, denoted as σg. All (k- 1)-simplexes, - are faces of 0 if they are subsets of dR, while all (Ic + simipppeex,s Ts+1, are colaces of 0% if they contain 0 as one of their faces.
loop is not considered part of the 2-simplex formation since we consider unique O-simplexes appearing in texts. One target I-simplex could be part of multiple 2-simplexes. The I-simplex and 2-simplex embeddings will be initialised ran- domly, and all embedding matrices are optimised during training.
The message-passing mechanism leverages the connectivity information in simplicial complexes. For simplex ds, we denote its boundary adjacent simplexes B(or) as the set of sever-dimensional simplexes O--1 on the boundary of ok, its co- co-boundary adjacent simplexes C(ax)) as the set of nigher-dimensional simplexes 0k+1 with dj on their bound- aries, its lower adjacent simplexes N (ork as those with the same dimension 0 that share lower-dimensions simplex aa-1 on their boundary, and its upper adjacent simplexes Nf as those with the same dimension as o that are on the boundary of the same dimensioam simplex 0x+1 (Bodnar et al., 2021).
Denote Kr the set of k-simplexes for K. Ko will be referred to as the set of O-simplexes (nodes). Kc refers to the set of 1-simplexes (edges) and KC refers to the set of 2-simplexes ("filled" triangles). For the text classification task, we con- struet the document resentations of O-simplexes, 1-simplexes, and 2-simplexes, as shown in Figure 1. We embrace the bag-of word model (Harris, 1954) and treat each word and punctuation as dis- tinct O-simplexes initialised from GloVe embeddings (Pen- nington et al., 2014). The three types of direction of simplexes follow the sequential order of the tokens in each text as shown in Figure 1. 2-simplexes are formed when any three words form "filled" triangle. We differentiate their nine identities by the neighbouring I-simplexes for the 2-simplexes to be formed. An example of the I-simplex est is shown in Figure 2 where the types of 2-simplex formed are determined by the I-simplex between 8 and o and the l -iiplexx between t and o. It is to be noted that the self-
simplicial complex with initial rep-
The hidden representation of simplexes will be initialised with vectors Xx; for all σ E Ka, k (0, 1,... ,..,..KK, and we set K = 2. That means at the initial state, the layer represen- tation for simplex d is (0) Xx, for k E 1. K}. The messages are then aggregated according the neigh- bourhood in which the simplexes sit: at the state e+1 1 and for the target s-mmple oge the message function ((e1)) col- lects information from neighbouring simplexes of the same
k
3

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Similarly to the GNN framework, the update function UPDATE(++1) will synchronise the representation of the k-simplex to the new state, shown in Equation (3), and we adopt the Gated Recurrent Unit (GRU) for the text clas- sification task.
h ((-11) UPDATE, (++1) (h20 m m((1)) ON
(3)
(++1) GRU & (11 mac ()
(4)
Lastly, the readout function READOUT will obtain the rep- resentation for the document simplicial complex by pooling k-simplexes' features of the final state L in Equation (5). A global self-attention mechanism (Lin et al., 2017) is specifi- cally applied for text data, summarising the O-simplexes and I-simplexes. For the final layer representation h of the doc- ument simplicial complex with O-simplexes do E Ku and I-simplexes a E K1, its individual simplex attention score Oz is derived with two multi-layer perceptions without bias denoted by W, and W2. The final simplex representation for the document simplicial complex, h is hence the sum- mation of the attention score multiplied by the respective final simplex features Ni for l E {00, (0,1). 1}.
Figure 2. 2-simplexes types for a I-simplex with source O-simplex and target O-simplex I. For the I-simplex eas with source 0- simplex 8, a target O-simplex and the defined direction from s f, nine types of 2-simplexes could define the information flow through the 2-simplex. For O-simplex o that forms a 2-simplex with the target I --iimlex eat, there exist three types of 1-simplexes between 0 0-simplex and O-simplex s, as well as between O-simplex o and 0-simplex t: into, out of or bidirectional, resulting in nine types of 2-simplex with respect to the I-simplex ext-
= (N(O))) those of one dimension lower ah-1 E B((rk) and those of one dimension higher xk+1 E C(Tx) as illustrated in Equa- tion (1). In the context of text classification, we adopt the message function as a multi-layer perception (MLP) for both O-simplexes do E KCo and 1-simplex σ1 E K1 updates. The message passing set collect information from neigh- bouring simplexes and co-boundary adjacent simplexes. For the aggregation of all the components in the document sim- plicial complex, we adopt row summation as illustrated in Equation (2).
dimension of E ((x)) where M(ox)) where.V(os)
h READOUT (1A4 10, ,K])
(5)
Σ OIEKRs
ao lotn
(6)
oooKo
exp
(tanh(W (1 W2)
Cha Eni EKo exp (tanh(W,h)) ) W2
where h refers to the final collective representation for all dy E Kyp. Finally, a linear layer with a softmax classifier will transform the results to the same dimension as the label set and make predictions. A summary of the proposed SCN framework is illustrated in Figure 3. With the input sentence "a thriller without lot of thrills.", a simplicial complex could be constructed with the following components. For O-simplexes, have matches U1: "a", t2: "thriller", uy: "without", u4: "lot", Vy: "of", vs: "thrills", and uy: "., For 1-simplexes, c1, e2, 61, Es. C,, E7 are forward edges connect- ing two words, and e3 is backward edge hetween "without" and "a". Lastly, a 2-simplex T is a type-4 triangle formed by the 1-simplexes connecting among the words "a", "thriller" and "without".
For individual 0k E Krx and & E (0, 10. K}.
(f+1) mo
AGG ((N())) (M(( (1 112 )2
a 02-1EB(OL) scT1EC((N)
(1)
Σ (E ((t16(()))(((o))
① (MLP((t)) (n + h000 (l2).))
Assuming message functions are fully connected neural networks, the SCN could be evaluated with three compo- nents: feature transformation in neural networks, neigh- bourhood aggregation and non-linear activation. Assum- ing that all the layers are of the same size F and the em- bedding size is fixed with F for 0-simplexes, 1-simplexes
where for k = O, we do not consider simplex of dimension (n 1), 1/20 refers the simplex ax's 's feature at the state (6) is set to a zero vector if C(ax) n C(ar)) is empty. MLP((+1) refers to trainable multi-layer perception at the state f 2+1
4

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Simplex Feature Definition
Message Passing Mechanism
Readout
Simplicial Complex
thriller without a lot of thrills.
Ossimplex Embedding
1-simplex Embedding
2-simplex Embedding
Representation for
I-simplex Update
C5
02
C4
MLP
y
e
3
e5
0-simplex Update
63.
2/2
2
13
e6
e7
Input Simplicial Complex
K
Layer I.....
Figure 3. Message-passing mechanism in Simplicial Convolutional Networks (SCN) up two-dimension. The above figure illustrates an example of a simplicial complex for seven-token sentence with a message-passing mechanism that collects neighhouring information from the dimension, dimension lower and dimension higher. The input simplicial complex K. consists of O-simplexes "1,-.", VT. I-simplexes "1, ,e and the 2-simplex т. Pre-defined and trainable features of different simplexes are used as input on the left-hand side. SCN leverages neighbouring, boundary and co-boundary simplexes carry feature information and update the target O-simplexes and 1-simplexes separately in different layer states. Finally, the features of different simplexes are read out for downstream tasks.
and 2-simplexes, the features are initialised from all three kinds of simplexes and the dense matrix multiplication takes
Z, (BER )), Z. MLP eSCN
O[Ko]F=2 - [K2] F2 + ( = ([(K)]F2) The aggre- gation and update step will take O( F2 + KC₂ F2) for O-simplex and I-simplex updates. Non-linear activation is an element-wise function which will take O(Ko + [)- As a result, over L layers, the final time complexity is O(Ikol + |C,] Kr, F2 + F2 2 + (K] O(I)F
(8)
The constrastive loss derived by the cross-entropy loss (CE) between the normalised (norm) text encoder output and the normalised SCN encoder.
Lel CE (Z:). norm Z.))
3.2. SCN with Contrastive Learning
At the same time, include the training objective against the ground-truth label y, which is linear interpolation of the text encoder and SCN encoder after transformation to the same dimension as the label space ()) inspired by (Lin et al., 2021).
To alleviate the abovementioned challenges with contrastive learning, we adopt dual-encoder framework inspired by (Wen & Fang. 2023) where we generate text representations from transformer blocks and graph representations from SCN in parallel; hence, the training process could optimise the contrastive learning and classification task as shown in Figure 4.
Z = (offtmxx (2,) + softmax (2:)
(10)
We employ the BERT model (Devlin et al., 2019) the text encoder and SCN encoder that digest the document data Xdoc We denote Z,, 7, E RR7 the text encoder and SCN encoder output. MLP(.) is linear layer that processes the output to the target space's 's dimension 7,
Ltabel = CE(Z,y)
(11)
The final loss function is the integration of the contrastive loss with the classification loss.
Ltabel + L.s
(12)
where r is control parameter.
5

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Z
Contrastive
Transformer Encoder
Learning
Eet
thriller thrills
Final Loss
SCN
Label
Encoder
Clussification
Labee
Figure 4. The Contrastive Learning with SCN (C-SCN) framework. The transformer encoder and SCN encoder will generate text representation Z, and document simplicial complex representation Z. respectively. The learned features will be used for contrastive learning by comparing themselves. Meanwhile, the two representations will contribute to the classification task with equal weights. As result, the final loss L includes the contrastive loss La and Ltabel-
short text classification framework ((Sun et al., 2022; Wen & Fang, 2023; Liu et al., 2024)), from each category, 20 samples are selected randomly to form the train set, another 20 samples are selected randomly form the validation set, and the rest included in the unseen test set.
Table I. Summary statistics for text Datasets.
Dataset
Twitter
MR
Snippets
StackOverflow
# Doc
10,000
10,662
12.340
20,000
Train
40
40
160
400
Train ratio
0.40%
0.38%
1.30%
2%
Tokens
12,229
18,337
29,422
11,161
Avg. Length
9.3
20.4
18.0
9.3
Traditional Language Models: TF-IDF (Rajaraman & Ull- man, 2011) refers to the term reeuueecy-inverse document frequency, and it measures the importance of word tokens to the document. The features generated are passed a sup- port vector machine (SVM) (Crammer Singer, 2002) for the classification task. LDA (Blei et al., 2003) refers La- tent Dirichlet Allocation and extracts latent topics from the text through probabilistic models. The features are trained with SVM for short-text classification. PTE (Tang et al., 201 5) refers to Predictive Text Embedding, which utilises heterogeneous text networks for embeddings.
Class
2
2
8
20
Avg. # I-simplexes
21.87
37.79
30.23
17.24
Avg. 2-simplexes
0.24
0.74
3.24
0.21
4. Experiments
4.1. Datasets
The experiments are conducted on four datasets for short text classification tasks. The datasets are briefly introduced below, and a summary table is reported Table We adopt the same data preprocessing techniques as (Wang al., 2017) with slight modifications include punctua- tion, keep the hashtag messages and add self-connection. Twitter (Bird et al., 2009) is a binary classification dataset for sentiments "positive" and "negative" collected by Nat- ural Language Toolkit. MR (Pang & Lee, 2005) contains movie review documents from Rotten Tomato with binary sentiment categories. Snippets (Phan al., 2008) contains Google web search text data with eight categories: "busi- ness", "computer", "health", "sports". "culture and art", "education and science", ""ngineering" "and "politics and so- ciety". StackOverflow (Hamner et al., 2012) contains ques- tion text from StackOverflow, and we choose the samples as (Xu et al., 2017) for 20,000 questions with 20 categories.
Machine Learning Models: CNN (Kim, 2014) refers to Con- volutional Neural Networks with pre-trained Glo Ve word embeddings (Pennington et al., 2014). LSTM (Liu et al., 2016), which refers to Long- Short Term Memory, is trained GloVe embeddings. BERT (Devlin et al., 2019), which refers to the Bidirectional encoder representations from transformers and its modified version RoBERTa (Zhuang et al., 2021) leverages pre-training through suff-supereiee learning and could be fine-tuned to specific downstream tasks.
Graph-based Language Models: TL-GNN (Huang et al., 2019) refers to text-level GNN, which adopts small win- dows for texts to focus on local features. TextGCN (Yao et al., 2019), Text Graph Convolutional Network, constructs individual text graphs with document nodes based on word co-occurrences and word-document relations. TextING (Zhang al., 2020) adopts individual text graphs and in- ductively trains the model. HyperGAT (Ding et al., 2020), Hypergraph Attention Networks enhances the expressive power graphs on text classification by including high-
4.2. Baseline Models
We compare C-SCN with other various types of benchmark language models for short-text classification as reported by (Liu al., 2024). Following with few-shot setting for
6

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
order information and reducing computational resources needed for training. STGCN (Ye al., 2020), which refers to the short text graph convolutional network, integrates BERT and the bidirectional LSTM in graph models to en- hance performance on short texts. DADGNN (Liu et al., 2021), Deep Attention Diffusion Graph Neural Networks, applies attention diffusion and decoupling techniques target- ing some limitations of GNN such as oversmoothing and restricted receptive field.
Algorithm Algorithm Pseudo Code for C-SCN.
Input: Text data with words, punctuations and label as shown in Figure 3.
Gruph-based Models with external knowledge beyond docu- ments or Contrastive Learning: STCKA (Chen et al., 2019) refers to Short Text Classification with Knowledge-powered Attention, which utilises attention mechanisms and entity conceptualisation to enhance text features. HGAT (Linmei al., 2019) is known Heterogeneous Graph Attention Networks, and its enhanced version incorporates topic and entity beyond the texts for enriched graphs. SHINE (Wang et al., 2021) is a hierarchical heterogeneous graph represen- tation learning method for short text classification which executes entity and POS tagging for various types of node features. NC-HGAT (Sun et al., 2022) integrates HGAT with neighbouring contrastive learning. GIFT (Liu al., 2024) is the graph contrastive learning for short text classifi- cation that employs SVD and k-means clustering methods in contrastive learning.
Simplicial Complex Construction
Tokenised words from the document data Xdoc Tokenised unique O-simplex do E Ko; 1-simplex indices following the chronological order of to- kens σ1 E K1;
1-simplex features tokenised to one of the types: forward, backward, self-loop;
2 2-mimp features tokenised by the flow directions of com- ponents; / Add higher-order simplexes if needed.
Model Construction
Parameters:
Embedding matrices for O-simplexes, 1-simplexes and 2- simplexes: Eu, & and Eg;
4.3. Implementation Details
Number of layers: L;
(eessageppsssiggg mechanism for O-simplexes and 1- simplexes following Equation 2 and Equation 3: MP,, MP1;
We include the pseudo-code for C-SCN to enhance the re- producibility in Algorithm 1.
The 0-simplex embeddings are initialised with GloVe em- beddings (Pennington et al., 2014). The embedding matrices for 1-simplexes and 2-simplexes are randomly initialised and optimised to size 128. The learning rate is 1 x 10-4, and the batch size is 128. A dropout rate of 50% is implemented to reduce the complexity of the model and prevent overfit- ting problems. The model is trained with the Py PyTorch Geo- metric' package for 100 epochs with early stopping where the validation loss does not improve for ten epochs. The best weights are obtained from the model with the best vali- dation accuracy. Cross-entropy loss is used with an Adam optimiser. The experiments are conducted ten times with NVIDIA RTX A6000 with 48GB of memory. We compare the results with strong baseline models with ten iterations of different training, validation and test sets. The average test accuracies and F1 scores are used for comparison.
Attention mechanism for O-simplexes and I-simplexes fol- lowing Equation 5: Attno,, Attn;;
Activation function: &; Transformer model: Trans,; Linear layers that process the output of the transformer and the SCN to the label space: MLP,, MLP..
Initialise features: h. Eo(g)) Voo Koi h
Vo K1; h = Eg Vog K1.
for = 1 to L 1 do
h MP, (h? ((-1) h(t-1)3
hoc MPo (t-1) hon ((-1))
(T) Σe, Attn; (h!?) Eoock, Attno(ba (b)
hk
Z, MLP, MLP,(Trans (Trans, (Xtoo)): Z, = MLP, (D)
7 2 (softmax(Z, + softmax 72)));
return 9 = Z, Z., Zz.
Unlike many graph models, simplicial-complex cowaall--oplleppeeeeee -aaed SCN inductive and models message passing among neighbours of different complexities. The message-passinn passing mechanism is simplified to mparse-matrix multiplication without explicit
readhhedss...................... ht//an//aates//iddee html

---

Contrastive Learning with Simplicial Convolutional Networks for Sexrt-Text Classification
source and target O-simplex materialisation. This means that the adjacency matrices will not explicitly record the relation- ship among all the O-simplexes. Similarly to sparse matrix expression, the 1-simplex index records the relationship be- tween source and target O-simplexes. The SCN model could digest unseen documents in the test set with its structural information and 0-simplex attributes.
Twitter
MR
Yor (orPeeeeeeeee (iut
1128
Contrel paroroosr
Conrnd pormnstor 1
5. Results and Discussion
Soippets
StackOverfoon
5.1. Results
Far Poyferrrreeee Pig
The experiment results are reported in Table 2. Compared with other competitive models, C-SCN has achieved the best test accuracies and F1 scores, indicating the model's ss ability to capture sentiments and sequential information in text documents.
Caums jorumstir
We attribute the better performance the following anal- ysis. Firstly, we adopt SCN, higher-order framework extending the expressive power of GNN. Features assigned to O-simplexes, I-simplexes and 2-simplexes could better represent the sentence structure and are generalised well across different contexts. The involvement of I-simplexes and 2-simplexes in the pressage-passing mechanism also expands the receptive fields of individual O-simplexes where long-range information can be transmitted through shallow neural network layers, thereby enhancing the impact of O- simplexes on the entire document. The self-attentive readout function connects 0-simplexes and I-simplexes, creating ex- pressive document-level summaries. This has promoted the SCN to perform the best in the benchmark datasets among the graph-based models without external information or contrastive learning. Secondly, the contrastive learning framework allows C-SCN to capture both structural and textual information in the few-shot setting. Both structural representation and sequential representation are treated as augmented views of each other. This has contributed to pre- venting helpful information from being removed, avoiding introducing noise or external information and combining the capabilities of both models.
Figure 5. Hyperparameter v7 sensitivity across different datasets.
meaning, explaining the difference between NC-HGAT and GIFT. Furthermore, introducing the global network within the small train set where the connectivity or clustering effect is explored might not be significant. This could explain why our model could outperform SHINE and GIFT.
5.2. Ablation Studies
Ablation studies are conducted to remove individual compo- nents to verify the capability of higher-order simplexes and contrastive learning in enhancing text understanding. The results are reported in Table 3. It observed that removing contrastive learning deteriorates the results for both SCN and BERT. Regarding higher-order simplexes, the removal of any component might deprecate the test accuracies and Fl scores across all datasets. Moreover, the inclusion of I-simplexes followed by the inclusion of 2-simplexes im- proves the results respectively, highlighting the importance of higher-order simplicial complexes in document under- standing.
In addition, we see that the large language models, such as BERT and RoBERTa, which leverage numerous pre- training, are not performing favourably with a few available labels. In contrast, graph-based models with external aux- iliary knowledge or contrastive learning, including HGAT, SHINE, NC-HGAT, and GIFT, could achieve competitive results. External auxiliary knowledge, such as entity recog- nition and POS tagging, might help enrich the semantic and syntactic meaning of the original text. Still, it might be in- troducing extra noise and unnecessary messages to the text data, as shown in the deterioration of results from STGCN to HGAT. Furthermore, contrastive learning with perturbation of the graphs might inject misinformation about the text's
The hyperparameter sensitivity of r is investigated across different datasets, and the results are visualised in Figure 5. The control parameter indicates the weights of contrastive loss in the model training process. We could observe that there are various types of impact on test performance. In general, the test performance varies between the value (no contrastive loss) and I (higher weight of contrastive loss), while `| results in lower performance compared to the case of no contrastive loss. One explanation for such variation could be the need to balance the focus between achieving the correct label and synchronising model weights between SCN and the transformer model. In our experi-
8

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Table Results of accuracy (% and test FI- -core (%) for short text classification where the best results hased on 95% confidence in the pairwise t-tests are in bold, and the second-bed best results are underlined.
Dataset
Twitter
MR
Snippets
StackOverflw
Metrics
F1
Acc
F1
Acc
F1
Acc
F1
Acc
TF-IDF
53.62
52.46
54.29
48.13
64.70
59.17
59.19
59.06
LDA
54.34
53.97
54.40
48.39
62.54
56.4
60.19
59.52
PTE
54.24
53.17
55.02
52.62
63.10
59.11
62.56
61.32
CNN
57.29
56.02
59.06
59.01
77.09
69.28
63.75
61.21
LSTM
60.28
60.22
60.89
60.70
75.89
67.72
61.62
60.49
BERT
54.92
51.16
51.69
50.65
79.31
78.47
66.94
67.26
RoBERTa
56.02
52.29
52.55
51.30
79.55
79.02
69.91
70.35
TL-GNN
59.02
54.56
59.22
59.36
70.25
63.29
62.09
61.91
TexIGEN
60.15
59.82
59.12
58.98
77.82
71.95
67.02
66.51
TextING
59.62
59.22
58.89
58.76
71.10
70.65
65.37
64.63
HyperGAT
59.15
55.19
58.65
58.62
70.89
63,42
63.25
62.10
DADGNN
59.51
55.32
58.92
58.86
71.65
70.66
66.26
65.10
STCKA
57.56
57.02
53.25
51.19
68.96
61.27
59.72
59.65
STGCN
64.33
64.29
58.25
58.22
70.01
69.93
69.23
69.10
HGAT
63.21
57.02
62.75
62.36
82.36
74.44
67.35
66.92
SHINE
72.54
72.19
64.58
63.89
82.39
81.62
73.05
72.73
NC-HGAT
63.76
62.94
62.46
62.14
82,42
74.62
67.59
67.02
GIFT
73.16
73.16
65.21
65.16
83.73
82.35
83.07
82.94
SCN
66.13
67.25
61.15
61.93
76.13
75.66
76.85
74.04
C-SCN
75.61
76.09
69.46
69.87
84.97
85.56
84.15
83.87
Table Results of accuracy for ablation studies. "C-SCN- O-simplex" means the 1-simplexes and 2-simplexes are both removed in the model, whereas "C-SCN- I-simplex" refers to the removal of 2-simplexes from the model.
Dataset
Twitter
MR
Snippets
StackOverflow
Metrics
F1
Acc
F1
Acc
F1
Acc
F1
Acc
BERT 54.92
51.16
51.69
50.65
79.31
78.47
66.94
67.26
SCN
66.13
67.25
61.15
61.93
76.13
75.66
76.85
74.04
C-SCN O-simplex
74.50
74.78
67.48
68.27
84.58
85.13
82.79
82.36
C-SCN 1-simplex
74.91
75.41
68.54
68.77
84.75
85.32
83.08
82.58
C-SCN
75.61
76.09
69.46
69.87
84.97
85.56
84.15
83.87
ments, grid search is conducted for the best performance for the best y) values.
leverages advantages from both structural and sequential representation, learns long-range information and enhances textual understanding with contextualised I-simplexes and 2- 2-simplexes during training.
6. Conclusion
In the future, we would like to explore the interpretability of higher-order simplexes and their roles in text understanding. The impact of the number of I-simplexes and 2-simplexes on the performance C-SCN is also worth attention, and it could be more inspected within the context of longer documents. Leveraging SCN's expressiveness sequential analysis could have more applications in other fields, such as recommender systems and process mining.
In conclusion, we propose Contrastive Learning with Simpli- cial Convolutional Networks (C-SCN), which incorporates higher-order information for sequence analysis and is ap- plied in short text classification tasks. The model constructs document simplicial complexes and incorporates the higher- order simplexes' message passing with self-attention read- out. Furthermore, integrate the transformer model to generate augmented views in the contrastive learning frame- work. Extensive experiments that simulate the lack of la- bel situation in a few-shot setting indicate that our model

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Impact Statement
on Artificial Intelligence and Thirty-First Innovative Ap- plications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, 1AAA'1//AАA 19/11A991111AAAA 19. AAAI Press, 2019. Chen, J., Qiu, Y., Wang, R., and Wei, G.-W. Persistent laplacian projected omicron ba. 4 and ba. 5 to become new dominating variants. Computers in Biology and Medicine, 151: 2022.
This paper introduces novel algorithms to enhance the per- formance of short-text classification tasks by leveraging higher-order information modelling and contrastive learn- ing. The primary objective is to contribute to the ongoing advancements in the field of Natural Language Processing. Such tools should be used with caution to ensure they re- main beneficial to society and uphold data privacy.
Acknowledgement
Chen, Y., Wu, L., and Zaki, M. Iterative deep graph learn- ing for graph neural networks: Better and robust node embeddings. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 19314 19326. Curran Associates, Inc., 2020.
This work was supported in part by (i) the Singapore Min- istry of Education Academic Research fund Tier grant RG16/23, Tier 2 grants MOE-T2EP20120 20-0010 and MOE- T2EP20221-0000 and (ii) the Singapore Economic Devel- opment Board and HP Inc. Singapore through the Industrial Postgraduate Programme with Nanyang Technological Uni- versity.
Crammer, K. and Singer, Y. On the algorithmic implementa- tion of multiclass kernel-based vector machines. J. Mach. Learn. Res., 2:265-292, mar 2002.
References
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, pp. 4171-41888 Association for Computational Linguistics, 2019.
Bai, S., Zhang, F., and Torr, P. H. Hypergraph convolu- tion and hypergraph attention. Pattern Recognition, 110: 107637, 2021. ISSN 0031-3203.
Bird, S., Loper, E., and Klein, E. Natural Language Pro- cessing with Python. O' O'Reilly Media Inc., 2009.
Blei, D. M., Ng. A. Y., and Jordan, M. I. Latent dirichlet allocation. J. Mach. Learn. Res., 1022 mar 2003.
Ding, K., Wang, J., J., Li, D., and Liu, H. Be more with less: Hypergraph attention networks for inductive text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4927-4936 2020.
Bodnar, C. Topological Deep Learning: Graphs, Complexes, Sheaves. PhD thesis, Apollo University of Cambridge Repository, 2022.
Feng, Y., You, H., Zhang, Z., Ji, R., and Gao, Y. Hypergraph neural networks. AAAI 2019, 2018
Bodnar, C.. Frasca, F., Otter, N., Wang, Y., Liò, P., Montúfar, G., and Bronstein, M. Weisfeiler and lehman go cellular: Cw networks. In Advances in Neural Information Pro- cessing Systems 34 35th Conference Neural Informa- tion Processing Systems, NeurlPS 2021, pp. 2625-2640. Neural information processing systems foundation, 2021.
Giusti, L., Battiloro, C., Testa, L., Di Lorenzo, P., Sardel- litti, S., and Barbarossa, S. Cell attention networks. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2023.
Bodnar, C., Giovanni, F. D., Chamberlain, B. P., Lio, P., and Bronstein, M. M. Neural sheaf diffusion: topological perspective on heterophily and oversmoothing GNNs. In Advances in Neural Information Processing Systems, 2022.
Hajij, M., Istvan, K., and Zamzmi, G. Cell complex neural networks. In TDA & Beyond, 2020.
Hajij. M., Zamzmi, G., Papamarkou, T., Miolane, N., Guzmán-Sáenz. A., Ramamurthy, K. N., Birdal, T., Dey, T. K., Mukherjee, S., Samaga, S. N., Livesay, N., Wal-
Cang. Z. and Wei, G.-W. Topologynet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions. PLoS Computational Biology, 130))xe00 2017.
ters, R.. Rosen, P., and Schaub, M. T. Topological deep learning: Going beyond graph data, 2023.
Hamner, B., Fullerton, D., Montrose, K., Chernoff, R., and Cole, W. Predict closed questions on stack overflow. 2012. URL http:///ggggg..c//mommtttttoaaaa o
Chen, J., Hu, Y., Liu, J., Xiao, Y., and Jiang, H. Deep short text classification with knowledge powered atten- tion. In Proceedings of the Thirty-Third AAAI Conference
10

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Hansen, J. and Ghrist, R. Toward a spectral theory of cellular sheaves. Journal of Applied and Computational Topology, 3(3-4):3153358, 2019.
Liu, X., You, X., Zhang. X., Wu, J., and Lv, P. Tensor graph convolutional networks for text classification. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pp. 1409-8416 AAAI Press, 2020.
Harris, Z. S. Distributional structure. WORD, 10(2-3): 146-162, 1954.
Liu, Y., Guan, R., Giunchiglia, F., Liang, Y., and Feng. X.
Deep attention diffusion graph neural networks for text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 8 8142-8152. Association for Computational Linguistics, 2021.
Huang, L., Ma, D., Li, S., Zhang, X., and Wang, H. Text level graph neural network for text classification. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- fional Joint Conference on Natural Language Processing COMNLP-IJCNLP) pp. 3444-3450. Association for Com- putational Linguistics, 2019.
Liu, Y., Huang, L., Giunchiglia, F., Feng, X., and Guan, R. Improved graph contrastive learning for short text classification. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):187 8776-18724, Mar. 2024.
Jaiswal, A., Babu, A. R., Zadeh, M. Z.,, Banerjee, D., and Makedon, F. A survey on contrastive self-supervised learning. Technologies, 9(), 2021.
Pang, B. and Lee, L. A sentimental education: Sentiment analysis using subjectivity summarization based on mini- mum cuts. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pp. 271-278. Association for Computational Linguistics, 2004.
Kim, Y. Convolutional neural networks for sentence classifi- cation. In Proceedings of the 2014 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 1746-1751. Association for Computational Linguis- tics, October 2014.
Pang, B. and Lee, L. Seeing stars: Exploiting class relation- ships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Pennington, J., Socher, R., and Manning, C. D. Glooe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
Lin, Y., Meng, Y., Sun, X., Han, Q., Kuang, K., Li, and Wu, F. BerGCN: Transductive text classification by com- bining GNN and BERT. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 1456-1462, Association for Computational Linguistics, 2021.
Phan, X.-H., Nguyen, L.-M., and Horiguchi, S. Learning to classify short and sparse text & web with hidden top- ics from large-scale data collections. In Proceedings of the 17th International Conference on World Wide Web, WWW '08 pp. 91-100, New York, NY, USA, 2008 As- sociation for Computing Machinery.
Lin, Z., Feng, M., dos Santos, C. N., Yu, M., Xiang, B., Zhou, B., and Bengio. Y. A structured self-attentive sentence embedding. In International Conference Learning Representations, 2017.
Rajaraman, A. and Ullman, J. D. Data Mining, pp. 1-17.
Cambridge University Press, 2011.
Reiser, P., Neubert, M., Eberhard, A., Torresi, L., Zhou, C., Shao, C., Metni, H., van Hoesel, C., Schopmans, H., Sommer, T., and Friederich, P. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):33, 2022.
Linmei, H., Yang, T., Shi, C., Ji, H., and Li, X. Heteroge- neous graph attention networks for semi- supervisd short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4821-4830, Hong Kong, China, November 2019. Association for Computational Linguistics.
Roddenberry, T. M., Glaze, N., and Segarra, S. Princi- pled simplicial neural networks for trajectory prediction. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 9020-9029, PMLR, 2021.
Liu, P., Qiu, X., and Huang, X. Recurrent neural network for text classification with multi-task learning. In Proceed- ings of the Twenty-Fip Fifth International Joint Conference on Artificial Intelligence, IJCAT'16, 16, pp. 2873-2879. AAAI Press, 2016.
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. The graph neural network model. JEEE Transactions on Neural Networks, 20(1):61-80, 2009.

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
Schaub, M. T., Seby, J.-B., Frantzen, F., Roddenberry, T. M., Zhu, Y., and Segarra, S. Signal processing on simplicial complexes. In Higher-Order Systems. Understanding Complex Systems. pp. 285-309, Springer, Cham, 2022.
Xia, K. and Wei, G.-W. Persistent homology analysis of protein structure, flexibility, and folding. International Journal for Numerical Methods in Biomedical Engineer- ing. 30(8):814 (018):814-844 2014.
Sun, Z., Harit, A., Cristea, A. I., Yu, J., Shi, L.,, and Al Moubayed. N. Contrastive learning with heteroge- neous graph attention networks short text classifica- tion. In 2022 International Joint Conference Neural Networks (IJCNN), pp. 1-6, 2022.
Xu, J., Xu, B., Wang. P., Zheng, S., Tian, G., and Zhao, J. Self-taught convolutional neural networks for short text clustering. Neural networks the official journal of the International Neural Network Sociery, 88:22-31, 2017.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Sverrisson, F., Feydy, J., Correia, B. E., and Bronstein, M. M. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15272-15281, 2021.
Yao, L., Mao, C., and Luo, Y. Graph convolutional networks for text classification, In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligncee AAAI Press, 2019.
Tang, J., Qu, M., and Mei, Q. Pte: Predictive text em- bedding through large-scale heterogeneous text networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 11, pp. 1165-1174, New York, NY, USA, 2015. Association for Computing Machinery.
Ye, R., Li, X., Fang. Y., Zang, H., and Wang, M. A vec- torized relational graph convolutional network for multi- relational network alignment. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp.
Townsend, J., Micucci, C. P., Hymel, J. H., et al. Represen- tation of molecular structures with persistent homology for machine learning applications in chemistry. Nature Communications, 1 (113:3230, 2020.
AAAI Press, 2019.
Ye, Z., Jiang, G., Liu, Y., Li, Z., and Yuan, J. Document and word representations generated by graph convolutional network and bert for short text classification. In European Conference on Artificial Intelligence, 2020.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. Interna- tional Conference on Learning Representations, 2018.
Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., and Wang, Every document owns its structure: Inductive text classification via graph neural networks. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 334-339 Association for Computational Linguistics, 2020.
Wang, J., Wang. Z., Zhang, D., and Yan, J. Combining knowledge with deep convolutional neural networks for short text classification. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelli- gence, IJCAI-17, pp. 2921, 2017.
Zhuang, L., Wayne, I.., Ya, S., and Jun, Z. A robustly optimized BERT pre-training approach with post-training, In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218-1227, Huhhot, China, August 2021. Chinese Information Processing Society of China.
Wang, Y., Wang, S., Yao, Q., and Dou, D. Hierarchical het- erogeneous graph representation learning for short text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3091-3101, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics.
Zia, A., Khamis, A., Nichols, J., et al. Topological deep learning: a review of an emerging paradigm. Artifi- cial Intelligence Review, 57:77, 2024. doi: 10.1007/ 00462-024-10710--1
Wee, J. and Xia, K. Persistent spectral based ensemble learning (PerSpect-EL) for protein-proted binding affin- ity prediction. Briefings in Bioinformatics, 23(2):bbac024, 02 2022.
Wen, Z. and Fang, Y. Augmenting low-resource text classi- fication with -raph-grounded pre-training and prompting. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 506-516, New York, NY, USA, 2023. Association for Computing Machinery.
12

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
A. Efficiency Studies
order to study computational efficiency with the inclusion of higher-order objects, we compute the number of trainable parameters, shown in Table 4.
Table Number of trainable parameters.
Twitter
MR
Snippets
StackOverflow
SCN 0-simplex
3,654,154
6,120,154
9,446,428
3,969,676
SCN 1-immllex
3,671,050
1,730,500
9,463,324
3,986,572
SCN
3,672,202
6,138,202
9,464.476
3,987,724
C-SCN - 0-simplex
113,236,620
115,702,620
119,029,666 229
113,554,464
C-SCN - 1-simplex
113,253,516
115,719,516
119,046,564
113,571,360
C-SCN
113,255,568
115,720,668
119,047,716
113,572,512
The time complete training and evaluation after iterations in seconds also included for analysis, shown in Table 5.
Table 5. Time complete training and evaluation after iterations.
Twitter
MR
Snippets
StackOverflovr
SCN O-simplex
639
730
1,009
1,448
SCN 1-simplex
679
743
1,235
1,728
SCN
750
798
1,279
1,956
C-SČN O-simplex
1,015
1,033
1,943
3,080
C-SCN 1-simplex
1,025
1,152
1,970
3,191
C-SCN
1,040
1,197
2,007
3,384
is observed that when adding I-simplexes and 2-simplexes to SCN step-by-step, the average number of trainable parameters increases by 0.18%, and the time increases by 10.55% on average. For C-SCN, the number trainable parameters increases by less than 0.1% on average, and the time for training increases by 3.32% on average. The results demonstrate the computational efficiency our model involving higher-order complexes in representation learning.
B. Additional Results for Ablation Studies
Instead of fixed GloVe embedding for word nodes, we compare the results with contextual embeddings (Cont. Emb.) from the BERT model in Table 4.
observed that C-SCN with fixed embeddings achieves better results than the one with contextual embeddings. One explanation could be the limited number of higher-order objects formed with contextual embeddings. O-simplexes (nodes), which refer to the same word, will not be the same O-simplex different locations in the document with contextual embeddings. This will lead 2-simplexes formed in the document since one O-simplex will not be connected again within the text, limiting the expressiveness of structural representations of the higher-order objects.
Table Results test accuracy to compare with the separate contrastive loss.
Dataset
Twitter
MR
Snippets
StackOverflow
Metrics
F1
Acc
F1
Acc
Fl
Acc
Fl
Acc
BERT
54.92
51.16
51.69
50.65
79.31
78.47
66.94
67.26
C-SCN Cont. Emb.
74.60
75.01
50.46
55.34
83.31
83.96
82.53
83.00
C-SCN Sep. Loss
67.54
68.35
53.61
56.79
64.69
64.8
27.01
29.78
SCN - SUM
62.3
63.21
54.99
56.23
77.06
77.05
76.02
73.73
SCN
66.13
67.25
61.15
61.93
76.13
75.66
76.85
74.04
C-SCN-- SÜM
74.0ī
74.45
55.51
58.26
77.38
77.92
78.73
77..46
C-SCN
75.61
76.09
69.46
69.87
84.97
85.56
84.15
83.87
13

---

Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification
To evaluate the effectiveness of optimising the contrastive loss and objective function together, experiments separate the two losses (Sep. Loss) also conducted. The contrastive loss is first minimised for 100 epochs without labels, and the loss against the final label is optimised with early stopping. It is observed that with limited training samples (20 samples from each category), pre- ure-training with a contrastive loss followed by supervised training does not help the model improve. In detail, the separate contrastive loss could improve BERT's performance in binary classification in Twitter and MR datasets. In contrast, it worsens the performance in multi-label classification, and the most deterioration from the StackOverflow dataset, which has 20 categories.
We adopted GRU as the UPDATE function control the amount of information from the previous step and aggregated neighbourhood information. This is achieved through the reset gate and the reset gate structure in GRU. In contrast, study the role of GRU by comparing the performance if we remove GRU as the UPDATE function and replace it with the sum aggregation (SUM). One challenge we observed without GRU was the overfitting issue on the train set across different datasets. The results deteriorated when we removed GRU from SCN and C-SCN respectively, illustrating the importance of GRU in the message- message-passing mechanism for higher-order complexes.
14


Reviews:
[{"review_id": "FFLIB4mkPf", "reviewer": "Reviewer_jGxX", "summary": "Due to limited labels and  sparsity in words and semantics, short text caught much attention. Most of the current models  adopted self-supervised contrastive learning across different representations but generate samples and external auxiliary information can not guarantee the effectiveness. And they also can not extract high-order information. The authors  proposed a novel document simplicial complex construction for a higher-order message-passing mechanism. By contrasting the structural representation with the sequential representation generated by the transformer mechanism for improved outcomes and mitigated issues, the C-SCN model outperform existing models on four benchmark datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) This paper has compared the proposed model with several baselines. They provide definitions and methodology. However, more theoretical proof is demanded.**\n\nWe would like to refer to the following sources for theoretical proof. To compare the expressiveness of graph neural networks and neural network structures that involve higher-order objects, such as simplicial complexes, the Weisfeiler-Lehman graph isomorphism test (WL test) (Weisfeiler & Lehman, 1968) is often used to compare different architectures. \n\nBodnar et al. (2021) extended the WL test with simplexes, known as Simplicial WL or SWL, by involving boundary and coboundary relations. The following theorem backs up the algorithm’s expressiveness.\n\n**Theorem 1**. SWL is strictly more powerful than WL at distinguishing non-isomorphic graphs. \n\nDue to the word limit, we would like to include detailed proof during the discussion phase if needed. The following theorem explains our motivation to adopt the framework in our model architecture.\n\n**Theorem 2**. With sufficient layers and injective aggregators, Message Passing Simplicial Networks (MPSN) is as powerful as SWL.\n\n_Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia, 2(9):12–16, 1968._\n\n_Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F Montufar, Pietro Li´o, and Michael Bronstein. Weisfeiler and Lehman go topological: Message passing simplicial networks. In Proceedings of the 38th International Conference on Machine Learning._\n\n**(W2) The authors did not provide the source code for the model.**\n\nTo enhance the reproducibility of our work, we have included the pseudo-code in the Appendix section. We would like to include the code as supplementary material when the edit function is enabled.\n\n**(W3) We would like to see the author could clarify that whether the proposed model have solved the challenges discussed at the beginning.**\n\nOur work identifies three key challenges at the beginning, and the proposed model architecture could address them individually.\n\n_Challenge 1: The data augmentation step and the negative sampling step of contrastive learning may distort the semantic meaning and introduce unnecessary noise._\n\nRemoving graph components is adopted as a data augmentation strategy; however, this approach may disrupt the original meaning of the text. An instance from the Movie Review (MR) dataset: “There's not enough to sustain the comedy” while removing the word “not” reversely changes the meaning of this short sentence.\n\nIn C-SCN, we did not create the positive and negative labels that are used in pre-training for the text data. Instead, we provided augmented views of texts by applying the structural SCN and sequential language models. This has circumvented the challenges of generating distorted semantic meanings in the positive and negative classes. \n\n_Challenge 2: Auxiliary information, such as entities, latent topics, and part-of-speech (POS) tags (e.g., nouns and verbs), may be added to graph models for language understanding and enriching the limited available local context. However, this step might introduce misinformation, such as pulling documents that express opposite semantics but share similar topics._\n\nAssigning entities, such as a film name, might pull texts that complement and dislike the film into a closer neighbourhood. By the homophily assumption, which states that nodes with similar characteristics or labels tend to be connected, the model learns that the two texts are more similar to each other compared to other texts that do not mention the entity name but share similar semantics.\n\nIn C-SCN, we did not include any auxiliary information that might mislead the model in learning neighbourhood information related to entities, latent topics, or POS tags. Instead, we leverage the pre-trained information from the sequential language model BERT in the augmenting step to enhance the contextual understanding when contrasting with SCN.\n\n_Challenge 3: Graph models are mathematically limited in modelling higher-order features, such as group-wise interactions among a few nodes and edges expressed in terms of phrases._\n\nThe short sentence “It is what it is” uses repetition to emphasise the acceptance of the status quo. At the same time, graph models with only nodes and edges learn pairwise interaction. They need to extend the number of layers in order for words to incorporate the meaning of other words further apart. Group-wise phrase “it is” needs to be linked with “what” to model such repetition.\n\nIn C-SCN, we adopted the simplicial complex to incorporate higher-order relations, which have undergone the message-passing mechanism. This enables the grouped phrase “it is” as an edge to be connected to the third node “what” through the 2-simplex (filled triangle) formed.\n\nDue to the word limit, we would like to clarify more during the comment period.", "claims_and_evidence": "It sounds convincing, however, the authors do not provide solid proof.", "methods_and_evaluation_criteria": "From the results of four benchmark dataset, it looks convincing. However, more theoretical proof is demanded.", "theoretical_claims": "This paper only provides definition and methodology. No theoretical proof.", "experimental_designs_or_analyses": "The four selected benchmark datasets are commonly used. And the results show the model's effectiveness. However, the authors did not provide the source code.", "relation_to_broader_scientific_literature": "This paper have compared the proposed model with several baselines.", "essential_references_not_discussed": "Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning[J]. arXiv preprint arXiv:2501.09214, 2025.", "other_strengths_and_weaknesses": "Strengths\n1. This work is well written and easy to follow.\n2. The authors  proposed a novel document simplicial complex construction for a higher-order message-passing mechanism.The C-SCN model outperform existing models on four benchmark datasets.\n3. The results on the evaluation datasets demonstrate a remarkable performance improvement achieved by the proposed method.\nWeaknesses\n1. This paper have compared the proposed model with several baselines. They provides definition and methodology. However, more theoretical proof is demanded. \n2. The authors did not provide the source code about the model.\n3. We would like to see the author could clarify that whether the proposed model have solved the challenges discussed at the beginning.", "other_comments_or_suggestions": "Please see the weaknesses.", "questions_for_authors": "It is an interesting topic. However, please clarify that whether the proposed model have completely solved the challenges discussed at the beginning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v8PqUogfSE", "reviewer": "Reviewer_Ntsf", "summary": "The paper proposes Contrastive Learning with Simplicial Convolutional Networks (C-SCN) for short-text classification. The method constructs document simplicial complexes to capture higher-order interactions beyond simple pairwise relationships and integrates a contrastive learning framework that leverages both structural representations from the simplicial convolutional network and sequential representations from transformer models. The authors demonstrate improvements on several benchmark datasets in few-shot learning settings.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) The paper does not adequately justify the necessity of a complex graph-based model when simple large language models (LLMs) could potentially address these issues.**\n\nWe would like to highlight the novelty of our work in pioneering the use of higher-order simplicial complexes for short text classification, offering a novel geometric perspective on text representation and classification, and advancing the theoretical understanding of higher-order simplicial complexes in text classification. This lays the groundwork for future research in geometric machine learning. Unlike black-box LLMs, our approach leverages higher-order simplicial complexes to pave the path towards interpretable insights into the relationships between short texts through the higher-order simplexes. In addition, with a significantly smaller trainable parameter set, our method is particularly effective in resource-constrained environments, providing a lightweight alternative to computationally intensive large language models (LLMs). \n\nDue to time constraints, we obtained the results from Llama3.1-8B with two datasets, as shown in the table below.\n|Model||Twitter||MR|\n|--------|--------|--------|--------|--------|\n||F1|Acc|F1|Acc|\n|BERT|54.92|51.16|51.69|50.65|\n|GPT2|67.41|67.76|50.77|53.18|\n|RoBERTa|56.02|52.29|52.55|51.3|\n|Llamma3-8B|34.06|50.3|70.13|71.67|\n|C-SCN|75.61|76.09|69.46|69.87|\n\nWe can identify that Llama3, with extensive pre-training and a large parameter count of 8B, can handle longer texts with proper English in movie reviews more effectively, as shown by its improved performance on the MR dataset. On the other hand, in the Twitter dataset, which features shorter texts and non-standard English words, Llama3 may not perform well in the few-shot setting. For example, we found that Llama 3 treats both “:(” and “:)” as negative sentiments and classifies non-proper English words, such as “thankyou” and “followback”, in the negative category, resulting in worse performance. In contrast, C-SCN leverages both structural information and contextual information to prevent the pre-trained embeddings from dominating the sentence representations and manages to handle these cases better with the same number of examples and much fewer parameters, demonstrating particular strength in domains where short text data exhibits complex relational structures, which are naturally modelled by higher-order simplicial complexes. While LLMs achieve higher overall performance in one dataset, our approach introduces a novel framework that opens new avenues for exploring geometric representations in text classification.\n\nWe will consider the impact of text lengths and grammatical correctness in future work, as short texts from online resources, including tweets and reviews, may pose challenges to short text classification tasks.\n\n**(W2) The baselines used are outdated, and the absence of comparisons with current LLM-based methods weakens the evidence supporting the proposed claims.**\nThe baseline models we selected also included those after 2020, including DADGNN (2021), SHINE (2021), NC-HGAT (2022), besides GIFT (2024). \n\n**(W3) The paper does not sufficiently explain why a graph-based model is preferred over simpler LLMs (e.g. llama 3), especially when techniques, like skip connections or graph rewiring, can mitigate issues related to high-order dependencies in GNNs.**\n\nIn addition to the comparison with LLM in the previous question, skip connections (Xu et al., 2021) and graph rewiring (Topping et al., 2022) are introduced to mitigate the over-squashing problem of GNNs and to incorporate higher-order dependencies. However, skip connections combine graph features at different scales and introduce complex interactions among layers. This may diminish the effectiveness of shallow network structures and introduce architectural complexity. Furthermore, graph rewiring may compromise the sparsity of graphs (Barbero, 2024). In contrast, C-SCN maintains the original graph structure with a shallow layer and incorporates the higher-order dependencies with contrastive learning in one pipeline of training, demonstrating its effectiveness and efficiency in the few-shot learning setting.  \n\n_Xu, K., Zhang, M., Jegelka, S., & Kawaguchi, K. (2021). Optimization of Graph Neural Networks: Implicit Acceleration via Skip Connections and Increased Depth. CoRR, abs/2105.04550._\n\n_Topping, J., Di Giovanni, F., Chamberlain, B. P., Dong, X., & Bronstein, M. M. (2022). Understanding Oversquashing and Bottlenecks on Graphs via Curvature. In International Conference on Learning Representations._\n\n_Barbero, F., Velingker, A., Saberi, A., Bronstein, M. M., & Di Giovanni, F. (2024). Locality-aware graph rewiring in GNNs. In The Twelfth International Conference on Learning Representations._\n\nDue to the word limit, we would like to clarify more during the comment period.", "claims_and_evidence": "The authors claim that their approach can better capture long-range and higher-order interactions in short texts compared to traditional graph-based models, leading to enhanced classification performance. However, the evidence provided is not fully convincing. In particular, the paper does not adequately justify the necessity of a complex graph-based model when simple large language models (LLMs) could potentially address these issues. Additionally, the baselines used are outdated, and the absence of comparisons with current LLM-based methods weakens the evidence supporting the proposed claims.", "methods_and_evaluation_criteria": "While the construction of document simplicial complexes and the integration of contrastive learning are interesting, the methods lack a compelling motivation. The paper does not sufficiently explain why a graph-based model is preferred over simpler LLMs (e.g. llama 3), especially when techniques like skip connections or graph rewiring can mitigate issues related to high-order dependencies in GNNs. Moreover, the evaluation criteria are standard for short-text classification, but the outdated baseline comparisons reduce the impact of the reported improvements.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "As discussed in the previous section, LLM baselines are needed.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "LLM-related papers.", "other_strengths_and_weaknesses": "The presentation quality can be improved.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i5Pk64Mg60", "reviewer": "Reviewer_r1wU", "summary": "This work proposes C-SCN, which combines contrastive learning with simplicial complexes in convolutional networks to capture higher-order interactions and improve short-text classification performance. Experimental results demonstrate its superiority over existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) The work identifies three issues with current methods. However, it remains unclear how the proposed methods specifically address these issues.**\n\nDue to the word limit constraints, we refer to the response to Reviewer jGxX (W3) for similar concerns. \n\n**(W2) SCN is used to capture higher-order information, but there are many alternatives, such as self-attention, jump self-attention, and etc. What are the reasons for adopting simplicial complexes over these other options?**\n\nAs mentioned in Section 3, our model adopts a self-attention mechanism as the READOUT function. It summarises the words (0-simplexes, nodes) and the connections between words (1-simplexes, edges). \n\nSelf-attention (Vaswani et al., 2017) and jump self-attention (Zhou et al., 2022) are able to process the input sequence by allowing every element in a sequence to attend to every other element, regardless of their distance in the sequence. Hence, long-range information is captured. However, the attention scores calculated between two words are still pair-wise interactions, and the higher-order information is only accounted for if self-attention layers are stacked. Although the jump self-attention algorithm introduces the mix-hop mechanism, the number of hops $j$ participating in this mechanism requires additional hyperparameter tuning. In contrast, C-SCN involves a single layer of convolutional networks and incorporates multiple higher-order objects, including 0-simplexes, 1-simplexes, and 2-simplexes, without requiring additional hyperparameters. \n\n_Vaswani et al. (2017). Attention is all you need. In NIPS._\n\n_Zhou et al. (2022). Jump Self-Attention: Capturing High-Order Statistics in Transformers. In NIPS._\n\n**(W3) The paper puts effort into introducing the message-passing mechanism. How is it integrated into convolutional networks, and what is the structure of the SCN? A figure may be a helpful illustration.**\n\nWe have included the detailed message-passing mechanism in Section 3, and Figure 3 demonstrates the model architecture.\n\n**(W4) What is the TDL mentioned in the introduction section? Please give a reference.**\n\nTopological Deep Learning (TDL) bridges the gap between topology and machine learning, offering powerful tools to model complex data structures and relationships. By incorporating topological insights, it enhances the interpretability, robustness, and performance of deep learning models across a wide range of applications. Clough et al. (2022) adopted TDL to improve image classification, object detection, and segmentation by capturing the geometric and topological structure of images and utilising persistent homology to extract shape-based features that complement traditional pixel-based methods. Shen et al. (2024) incorporated both high-order interactions and multiscale properties into a TDL architecture, specifically simplicial neural networks, where polymer molecules are represented as a series of simplicial complexes at different scales to enhance the accuracy of polymer property prediction.\n\n_J. R. Clough et al. (2022). A Topological Loss Function for Deep-Learning Based Image Segmentation Using Persistent Homology. in TPAMI._\n\n_Shen et al. (2024). Molecular Topological Deep Learning for Polymer Property Prediction. In arXiv._\n\n**(W5) Can the simplicial complexes be used in Transformer?**\n\nYes, simplicial complexes can be used in Transformers to incorporate higher-order relationships and geometric structures into the model. While traditional Transformers operate on sequences or graphs, simplicial complexes extend these structures by modelling higher-order interactions (e.g., triangles, tetrahedra) rather than just pairwise relationships. This can enhance the model's ability to capture complex dependencies in data. For example, Cellular Transformer (CT) (Ballester et al., 2024) leverages algebraic topology to form cell complexes from graph data and inputs to the transformer model, where the accuracy of graph classification is enhanced. \n\n_Ballester et al. (2024). Attending to Topological Spaces: The Cellular Transformer. In arXiv. _\n\n**(W6) In the era of LLMs, models like ChatGPT, DeepSeek, and LLaMA have already achieved strong performance in short-text classification. For closed-source LLMs, we can use APIs and Apps, while for open-source LLMs, we can fine-tune them on local machines. Why not use these LLMs for short-text classification?**\n\nDue to the word limit constraints, we refer to the response to Reviewer Ntsf (W1) for similar concerns. \n\n**(W7) In the loss function in Eq. 12, both the classification loss and contrastive loss use features generated from the BERT model. It is difficult to justify whether the proposed SCN encoder contributes, and to what degree, to the performance.**\n\nWe have included a comparison between with and without contrastive loss in the Appendix section, where the loss functions are utilised for SCN and BERT separately, and the contribution of SCN is highlighted.", "claims_and_evidence": "The work identifies three issues with current methods:\n1. The augmentation step may fail to generate positive and negative samples that are semantically similar and dissimilar to the anchor, respectively.\n2. External auxiliary information may introduce noise to the sparse text data.\n3. Limited ability to capture higher-order information, such as group-wise interactions.\n\nHowever, it remains unclear how the proposed methods specifically address these issues.\n\nSCN is used to capture higher-order information, but there are many alternatives, such as self-attention, jump self-attention, and etc. What are the reasons for adopting simplicial complexes over these other options?\n\nThe paper puts effort into introducing the message-passing mechanism. How is it integrated into convolutional networks, and what is the structure of the SCN? A figure may be a helpful illustration.", "methods_and_evaluation_criteria": "The proposed method is not well justified for the problem. The chosen benchmark datasets are appropriate for the task.", "theoretical_claims": "Theoretical claims seem fine.", "experimental_designs_or_analyses": "The experiment designs are good.", "relation_to_broader_scientific_literature": "The idea to incorporate the high-order information has been highlighted in the existing works.", "essential_references_not_discussed": "What is the TDL mentioned in the introduction section? Please give a reference.", "other_strengths_and_weaknesses": "The topic is important in the field, and the idea of incorporating higher-order information into the model design is inspiring.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Can the simplicial complexes be used in Transformer?  \n\nIn the era of LLMs, models like ChatGPT, DeepSeek, and LLaMA have already achieved strong performance in short-text classification. For closed-source LLMs, we can use APIs and Apps, while for open-source LLMs, we can fine-tune them on local machines. Why not use these LLMs for short-text classification?\n\nIn the loss function in Eq. 12, both the classification loss and contrastive loss use features generated from the BERT model. It is difficult to justify whether the proposed SCN encoder contributes, and to what degree, to the performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces C-SCN, a contrastive learning framework using simplicial complexes to capture higher-order structural information in short text classification. While the novelty is somewhat limited compared to recent advances in LLMs, the idea of leveraging topological structures for interpretability and lightweight modeling is interesting and experimentally validated. I recommend a weak accept given the unique perspective and solid empirical results, though comparisons to stronger LLM baselines would strengthen the work.
Decision: Accept (poster)