RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Background: Graph Filtering and Smoothing
Graph Signal Processing Shuman et al. ( 2013 ); Ortega et al. ( 2018 ) provides a framework to analyze and process signals defined on graphs, by extending traditional signal processing concepts and tools to the graph domain. This allows for the representation and manipulation of signals in a way that is tailored to the specific structure of the graph.
In what follows we refer to matrices in boldface uppercase and vectors in boldface lowercase.
Graph Signals
Graph signals are mappings from the set of vertices to the real numbers. A graph signal for a given graph \mathcal{G} can be represented using vector \mathbf{f}=[f(v_{1}),\ldots,f(v_{n})]^{\top} such that f:\mathcal{V}\to\mathbb{R} is a real-valued function on the vertex set. The smoothness of a signal \mathbf{f} over graph \mathcal{G} can be characterized using the Laplacian quadratic form associated with Laplacian \mathbf{L} :

\mathbf{f}^{\top}\mathbf{L}\mathbf{f}=\frac{1}{2}\sum_{i,j}\ a_{ij}(\mathbf{f}_{i}-\mathbf{f}_{j})^{2}

These signals can be high dimensional and can represent many kinds of data. In our case, signals will represent text embeddings.
Graph Filters
Smoother graph signals can be obtained by minimizing the quantity described in Formula ( 1 ). That is the goal of graph filters and the filtering is generally done from a spectral perspective. A specific class of filters that additionally has an intuitive interpretation from a vertex perspective is that of the polynomial filters. When the filter is a P -th order polynomial of the form \hat{h}(\mathbf{L})=\sum_{m=0}^{p}\theta_{m}\mathbf{L}^{m} , the filtered signal at vertex i , is a linear combination of the components of the input signal at vertices within a P -hop local neighborhood of vertex i :

\mathbf{f}^{out}_{i}=\alpha_{ii}\mathbf{f}^{in}_{i}+\sum_{j\in N(i,p)}\alpha_{ij}\mathbf{f}^{in}_{j} (2)

where N(i,p) is the P -th order neighborhood of vertex i . It is possible to then make the correspondence with a polynomial filter (from a spectral perspective) as follows:

\alpha_{ij}=\sum_{m=d_{\mathcal{G}}(i,j)}^{p}\theta_{m}(\mathbf{L}^{m})_{ij} (3)

where d_{\mathcal{G}} is the shortest distance between node i and j . Several polynomial filters have been proposed in the literature such as the ones associated with Simple Graph Convolution (SGC) Wu et al. ( 2019 ) , simple spectral Graph Convolution (S²GC) Zhu and Koniusz ( 2020 ) , approximate personalized propagation of neural predictions (APPNP) Gasteiger et al. ( 2018 ) and Decoupled Graph Convolution (DGC) Wang et al. ( 2021 ) .
Proposed Methodology: Smoothing Sentence Embeddings
In this paper, we theorize that smoothing sentence embeddings with a semantic similarity graph can help supervised and unsupervised categorization models better differentiate between the similar and dissimilar documents, leading to performance gains. A common choice for quantifying semantic similarity of text is the cosine similarity; given two sentence embedding vectors \mathbf{x}_{i},\mathbf{x}_{j}\in\mathbb{R}^{d} we have

\cos(\mathbf{x}_{i},\mathbf{x}_{j})=\frac{\mathbf{x}_{i}^{\top}\mathbf{x}_{j}}{\|\mathbf{x}_{i}\|\|\mathbf{x}_{j}\|}.

We build a k -nearest neighbors connectivity graph which we denote \mathcal{G} based on this similarity measure i.e. a graph for which each node has exactly k neighbors and whose edge weights are all equal to one. We characterize the graph \mathcal{G} using its adjacency matrix \mathbf{A} , we denote its Laplacian as \mathbf{L} . Given the adjacency matrix, a standard trick to obtain better node representations consists in adding a self-loop

\hat{\mathbf{A}}=\mathbf{A}+\lambda\mathbf{I} (4)

where \lambda is a hyperparameter controlling the number of self-loops. As such in what follows we consider the symmetrically normalized version of \hat{\mathbf{A}} , that is

\mathbf{S}=\hat{\mathbf{D}}^{-1/2}\hat{\mathbf{A}}\hat{\mathbf{D}}^{-1/2}

Now given a node embedding matrix \mathbf{X} and the previous semantic similarity graph. We consider four polynomial graph filters whose propagation rules we describe in Table 1 .
Experiments
In this section we evaluate our semantically smoothed representations obtained through four filters on two tasks, clustering and classification, with respect to the original representations obtained from SentenceBERT Reimers and Gurevych ( 2019 ) as well as two large language models baselines: BERT and RoBERTa.
Datasets and Metrics
We use eight benchmark datasets of varying sizes and number of clusters, and we report their summary statistics in Table 2 .
For the metrics, in the supervised context, we use the F1 score as the quality metric while in the unsupervised context we use the adjusted rand index (ARI) Hubert and Arabie ( 1985 ) and the adjusted mutual information (AMI) Vinh et al. ( 2009 ) .
Experimental Settings
For the classification task, we use a random stratified 64%-16%-20% train-val-test split. We also tune the hyperparameters k of the k -nn graph, order of propagation P , the parameter \lambda and the filter specfic parameters \alpha and T . For the clustering task, we use k=10 for the k -nn graph, set P=2 as the propagation order, \lambda=1 , \alpha=0.1 and T=5 . We report the averages of the metrics as well as their standard deviations over 10 runs (for the classification task, we omit standard deviation due to them being insignificant).
Clustering Results
We compare the results of the k -means algorithm (kM) applied on Sentence-BERT (we refer to it as SBERT or SB) embeddings with and without the different filtering operations. Note that instead of using kM we can use any other clustering algorithms including variants of kM such as k -means ++ Arthur and Vassilvitskii ( 2007 ) and entropy kM Chakraborty et al. ( 2020 ) . In addition to this, we add a baseline which uses an ensemble technique Ait-Saada et al. ( 2021 ) on the layer outputs of the word embedding of BERT and RoBERTa, this method improves over considering a single layer or taking the mean. We report the clustering results in Table 3 . The filtering operation systematically leads to better results on the benchmark with respect to the filterless clustering scheme on all datasets we have used. These increases are statistically significant in most cases. It also significantly beats the ensemble approach on most datasets.
Classification Results
Similar to the clustering setting, we compare results from a Logistic Regression (LR) applied on the original sentence embeddings with and without the filtering operation we introduced. We also use fine-tuned BERT and RoBERTa (2 epochs) as baselines; we use the base versions due to computational restrictions. We report the results in Table 4 . We see that this operation leads to better performances on the classification task on the majority of the datasets with respect to the filterless Sentence-BERT but this performance increase is not as pronounced as for the clustering task. We also see that the representations we learn lead to competitive results with respect to BERT and RoBERTa despite Sentence-BERT not being suited to classification.
Statistical Significance Testing
Using the Bonferroni-Dunn post-hoc mean rank test Demšar ( 2006 ) , we analyze the average ranks of the clustering and classification over the Sentence-BERT representations with and without filtering in terms of AMI and ARI, for the clustering task, as well as the F1 score for the classification task on the eight datasets. Figure 1 shows that the clustering and classification results when using the proposed semantically smoothed representations are statistically similar and that they all outperform the Sentence-BERT variant with no filtering in a statistically significant manner at a confidence level of 95%.
Conclusion
We proposed a simple yet effective empirical approach that consists in using similarity graphs in an unsupervised manner to smooth sentence embeddings obtained from pretrained models in a semantically aware manner. The systematic improvements in performance on both clustering and classification tasks on several benchmark datasets of different scales and balance underscore the effectiveness of using semantic graph smoothing to improve sentence representations.
Limitations
The main limitation of our approach is the additional computational complexity entailed by creating the k -nn graph from the data, performing the smoothing. Add to that, the hyperparameter tuning that is necessary for the classification task. While this increase is in no way prohibitive even for large datasets, a performance-speed compromise is to be considered.
Figure 1: Bonferroni-Dunn average rank test at a confidence level of 95%.; Bonferroni-Dunn average rank test at a confidence level of 95%.
The propagation rules associated with the different polynomial filters. \mathbf{H}^{(0)} is the \mathbf{X} . P is the propagation order. \alpha and T are filter-specific hyperparameters.
Summary statistics of the datasets. Balance refers to the ratio of the most frequent class over the least frequent class. Length refers to the average sentence length in the corpus.
Clustering results in terms of AMI and ARI on the eight datasets. The best results are highlighted in bold. If our best performing variant outperforms the best comparative method in a statistically significant matter (t-test at a confidence level of 95%), we highlight it in blue.
Classification results in terms of F1 score on the eight data sets.