# Comparing Vectorization Techniques, Supervised and Unsupervised Classification Methods for Scientific Publication Categorization in the UNESCO Taxonomy

Neil Villamizar, Jesús Wahrman, Minaya Villasana

► To cite this version:

Neil Villamizar, Jesús Wahrman, Minaya Villasana. Comparing Vectorization Techniques, Supervised and Unsupervised Classification Methods for Scientific Publication Categorization in the UNESCO Taxonomy. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.356-368, 10.1007/978-3-031-34111-3_30. hal-05449783

HAL Id: hal-05449783

https://inria.hal.science/hal-05449783v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Comparing Vectorization Techniques, Supervised and Unsupervised Classification Methods for Scientific Publication Categorization in the UNESCO Taxonomy

Neil Villamizar✉, Jesús Wahrman✉, and Minaya Villasana✉✉

Universidad Simón Bolívar, Caracas, Venezuela
{15-11523,15-11540,mvillasa}@usb.ve

**Abstract.** A comparison of classification strategies for scientific articles using the UNESCO taxonomy for categorization is presented. An annotated set of articles were vectorized using TF-IDF, Doc2Vec, BERT y SPECTER and it was established that among those options SPECTER provided the best separability properties using quantitative metrics as well as qualitative inspection of 2D projections using t-SNE. When pairing the best performing vectorization strategy with classical machine learning strategies for the classification task, such as multiple layer perceptron and support vector machines, comparable results are found, concluding that the choice of text representation strategy exerts a greater impact over the choice of classifier. The most problematic areas for classification were identified and a cascading classification strategy was implemented and evaluated. Unsupervised methods were also tested to consider the case when annotated data is not readily available and test their suitability. Two different unsupervised methods were used and it was determined that k-means yielded the best results when considering 3 times the number of categories as the optimal number of clusters.

**Keywords:** Classification · Natural language processing · Scientific texts · Machine learning

## 1 Introduction

A fundamental problem in natural language processing is text classification as it helps sort and categorize large text samples. The National Science Foundation reported in 2019 that global research output for peer reviewed science and engineering articles and conference papers, grew 4% annually over the last decade reaching 2.6 million in 2018 [17]. Classification of scientific documents can be a

Supported by ACALconecta project.
powerful tool for information filtering and although most scientific journal can
be categorized according to a particular taxonomy, it is also true that several
taxonomies or nomenclatures exist such as ASJC, MAG or UNESCO fields. On
the other hand, many scientific articles are published in conference proceedings
or in electronic archives with broad or no classification.

In this article we show an application and the evaluation of several techniques
in a pipeline for a real world system, where the corpus of scientific research out-
puts from Latin America are classified according to the science and engineer-
ing UNESCO field taxonomy [20] and is used within the ACALconecta system
(app.acalconecta.org). The UNESCO taxonomy has labeling for fields as well as
disciplines and sub-disciplines within these fields, providing greater specificity
to document labeling. One can establish an equivalence between the nomencla-
tures for scientific outputs, and given that the corpus available used the ASJC
taxonomy [3], a dictionary was constructed at the UNESCO field level to have
annotated documents for training. However, at the discipline (there are 248 dis-
ciplines in this taxonomy) and subdiscipline (with 2238 subdisciplines available)
level requires manual expert annotation. Therefore an evaluation of alternatives
for avoidance of manual annotation for document classification at these taxon-
omy levels is also conducted.

This article is organized as follows: On Sect. 2 the main techniques are pre-
sented as well as the work pipeline. Section 3 details the main results obtained
and the comparison between the various methods. The effect over classification
power of each element in the pipeline is established, while Sect. 4 ends with final
remarks and path forward.

## 1.1 Related Works

One of the initial text representation strategies was the bag-of-words (BoW)
approach, which uses one-hot encoding to represent each word in a document.
However, BoW has several limitations, as it fails to capture the context and
relationship between words in a document. Term Frequency-Inverse Document
Frequency (TF-IDF) [7] improves on BoW by weighing the relevance of each
word for a document in a corpus, but still lacks proximity information.

Doc2Vec [8] is a generalization of the Word2Vec [10] model that represents
complete documents and uses proximity information in training. It has shown
promising results in various NLP tasks, including document classification, sen-
timent analysis, and topic modeling.

Attention and transformer-based models have taken text representation to
the next level by incorporating position encoding and proximity information
to capture contextual information. BERT [6] is a pre-trained transformer-based
model that generates multiple representations for the same word depending on its
context. SciBERT [1] and SPECTER [4] are Transformer-based models, specif-
ically trained on scientific literature. SPECTER includes positive and negative
citation paths to capture relationships between documents.
The choice of a representation model depends on the type and amount of information available. Selecting a representation model that fits the specific task and available data is important.

# 2 Methodology

## 2.1 Data Collection and Processing

A data set of scientific papers comprised of title, abstract and publishing journal was available, following the distribution in Table 1. From the table one can see that the data is not balanced in the different categories. Classification with unbalanced data affects the performance of classifiers, so a balanced subset was constructed by random sampling on each class. In this work classifiers using the unbalanced data and a balanced subset are constructed, thus showing the effects of unbalanced data on classification.

<table><caption>**Table 1.** Data distribution in UNESCO fields, the number inside parenthesis represents the UNESCO code for that field.</caption><thead><tr><td>UNESCO field</td><td>Number of samples</td></tr></thead><tbody><tr><td>Life Sciences (24)</td><td>12488</td></tr><tr><td>Agricultural Sciences (31)</td><td>10923</td></tr><tr><td>Chemistry (23)</td><td>4957</td></tr><tr><td>Mathematics (12)</td><td>4599</td></tr><tr><td>Astronomy and Astrophysics (21)</td><td>4583</td></tr><tr><td>Earth and Space Sciences (25)</td><td>3396</td></tr><tr><td>Physics (22)</td><td>2782</td></tr><tr><td>Technological Sciences (33)</td><td>2506</td></tr><tr><td>Medical Sciences (32)</td><td>1774</td></tr></tbody></table>

The dataset was processed using the following procedures: First stopwords and punctuation marks were removed and all letters were converted to lower case. Then each word was lemmatized, where the word is replaced by its correspondent lemma, that is the canonical form, dictionary form, or citation form of a word. Finally, samples were built concatenating the title, abstract and the name of the journal for each article.

## 2.2 Text Representation

In this work four methods of text representation were used: TF-IDF, Doc2Vec, BERT and SPECTER. This is a mix between classic and some state of the art methods. TF-IDF, Doc2Vec and BERT are very well known methods for
text representation while SPECTER is specialized in representing scientific documents, based on transformers like BERT but adds a citation graph in their training process for better results, because a document citing another suggests that these documents have some similarity; it uses a loss function that induces the transformer to generate closer vectors when a document cites another and codifies that relationship on the vectors generated by SPECTER.

The representation methods were evaluated using metrics that are normally used to measure the quality of a clustering. This is based on the fact that a good representation should help distinguish between the various contexts in which documents are produced. These metrics are: Separability Index [11], Silhouette Coefficient [18], Calinski-Harabasz Score [2] and Davies-Bouldin Score [5]. It's important to note that the results of these metrics serve as a heuristic to help select an appropriate representation method. t-SNE [9] was used to project the vectors in two dimensions for visualization.

## 2.3 Supervised and Unsupervised Classification Methods

Once a text representation method has been selected, the task of classifying the texts was performed using methods from supervised learning for documents annotated at the field level. Also, unsupervised methods were employed as a way to address the classification of documents with no known category, which is the case for classification at the discipline or sub-discipline level in the UNESCO taxonomy.

For supervised learning multiple layered perceptron (MLP) [14] and support vector machines (SVM) [12] were used, with accuracy as the main performance metric and confusion matrix to analyse results. Training was done using cross-validation techniques with the two sets considered: the complete data set (see Table 1) and a balanced subset that has 1774 samples per category.

For unsupervised learning, K-Means [19] and Gaussian Mixture [16] were tested. To predict the category of a sample, the annotated category that appears the most in the cluster is the classification assigned to the samples belonging to the cluster. Results with a balanced and the unbalanced complete data set are presented.

# 3 Results and Discussion

## 3.1 Text Representation

In this subsection the methods of representation are evaluated using the validation metrics mentioned above. For these experiments a subset of 5400 samples was used in which each category has 600 instances taken randomly. The complete data set was not used because the computational cost of calculating the validation metrics was high and this suffices to conclude on the best text representation.

Two dimensional representations of the data is shown in Fig. 1 using t-SNE, where it can be seen how the vectors generated by SPECTER form well-defined
groups in the 2D projection, in contrast to TF-IDF, Doc2Vec and BERT in which
groups are very overlapping. In TF-IDF, Doc2Vec and BERT groupings are not
very separated from each other and only a few fields have well recognizable
groups, such as the field of Medicine (UNESCO field 32, colored in purple) or
the field of Mathematics (UNESCO field 12, colored in dark brown) when using
TF-IDF and BERT. Using Doc2Vec, the only recognizable group is the field of
Medicine (UNESCO field 32, colored in purple). Libraries for various methods
were imported from scikit-learn [13,15].

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Two dimensional representations using t-SNE for TF-IDF (1), Doc2Vec (2), BERT (3) and SPECTER (4). Categories are represented with different colors specified in the legend.</figcaption></figure>

Some fields such as Physics (UNESCO field 22) and Astronomy and Astro-
physics (UNESCO field 21) are quite overlapping regardless of the representation
used. Rather than a deficiency in the rendering algorithm, this may be due to the
nature of the data. For example, both Physics and Astrophysics have terms and
topics in common, so it makes sense that they are close in representation. A sit-
uation similar to this can occur between other fields, such as: life sciences (which
includes animal biology) and agricultural sciences (which includes breeding).

Table 2 summarizes the metrics that help to quantitatively investigate the
adequacy of a particular representation model. These results confirm that
SPECTER surpasses the other models in all metrics except on the Calinski-
Harabasz coefficient, where it has the second best value. Therefore, henceforth
SPECTER is used as the language representation model.
<table><caption>**Table 2.** Evaluation metrics compared for each representation method. Best values for each metric are in bold.</caption><thead><tr><th>Evaluation metrics</th><th>TF-IDF</th><th>Doc2Vec</th><th>BERT</th><th>SPECTER</th></tr></thead><tbody><tr><td>Separability Index</td><td>0.712</td><td>0.484</td><td>0.664</td><td><strong>0.751</strong></td></tr><tr><td>Silhouette Coefficient</td><td>0.003</td><td>-0.036</td><td>0.017</td><td><strong>0.035</strong></td></tr><tr><td>Calinski-Harabasz</td><td>8.721</td><td><strong>305.04</strong></td><td>141.967</td><td>170.927</td></tr><tr><td>Davies-Bouldin</td><td>16.637</td><td>5.919</td><td>6.471</td><td><strong>5.421</strong></td></tr></tbody></table>

## 3.2 Supervised Methods

Different methods of supervised learning were evaluated on the task of text classification while using SPECTER as the vectorization technique.

When working with supervised methods, the balance of the data set is important. A model trained with unbalanced data will tend to favor certain predictions, according to the distribution of the data; this can be beneficial if the distribution of the data set is the actual distribution of the data in the real world. On the other had, a model trained with balanced data avoids bias when classifying. Therefore the methods in this section will be tested with the complete data set (unbalanced) and a balanced subset with 1774 samples per category chosen randomly, and trained using cross-validation (with $k = 5$) to determine the error. When using the complete data set, MLP obtained an overall accuracy of 77% and SVM 78%. Meanwhile, using the balanced subset both methods obtained an accuracy of 76% with cross-validation.

Having these estimates, each data set was split in 80% training and 20% test data, chosen randomly. Then both methods were trained with this partition of the data set to look for any possible improvement. With this data partition over the complete data set both methods obtained an accuracy of 79% on test data, close to the expected 77% and 78%. Whereas with the balanced data set the MLP model obtained a classification accuracy of 76% and the SVM 77%, both close to the initial estimations. In Tables 3, 4 and 5 the accuracy for each class and the matrix of confusion of each method can be seen on both data sets.

Table 3 shows that the 3 categories that produce more failed predictions are: Physics (22), Astronomy and Astrophysics (21), and Technological Sciences (33).

For Technological Sciences (33) Tables 4 and 5 show that the failed predictions in this category are distributed in almost all other categories because of the amplitude of this field, that includes disciplines like: Biotechnology, Chemical Engineering, Computer Technology, and others; which can easily be confused with other fields, so the low accuracy is expected.

In the case of Physics (22) and Astronomy and Astrophysics (21), these are related fields and in some taxonomies one is a subset of the other, thus the confusion is almost expected since they share many keywords and the classification could be confusing even for a human.
<table><caption>**Table 3.** Accuracy obtained in both classification models by category when training with a balanced subset and the complete data set</caption><thead><tr><th rowspan="2">UNESCO field</th><th colspan="2">Balanced set</th><th colspan="2">Complete set</th></tr><tr><th>MLP</th><th>SVM</th><th>MLP</th><th>SVM</th></tr></thead><tbody><tr><td>Mathematics (12)</td><td>90%</td><td>88%</td><td>89%</td><td>90%</td></tr><tr><td>Physics (22)</td><td>65%</td><td>69%</td><td>60%</td><td>57%</td></tr><tr><td>Earth and Space Sciences (25)</td><td>90%</td><td>91%</td><td>85%</td><td>87%</td></tr><tr><td>Agricultural Sciences (31)</td><td>70%</td><td>71%</td><td>79%</td><td>79%</td></tr><tr><td>Astronomy and Astrophysics (21)</td><td>68%</td><td>68%</td><td>74%</td><td>74%</td></tr><tr><td>Life Sciences (24)</td><td>71%</td><td>74%</td><td>81%</td><td>82%</td></tr><tr><td>Technological Sciences (33)</td><td>67%</td><td>68%</td><td>61%</td><td>62%</td></tr><tr><td>Chemistry (23)</td><td>71%</td><td>73%</td><td>78%</td><td>78%</td></tr><tr><td>Medical Sciences (32)</td><td>94%</td><td>94%</td><td>89%</td><td>88%</td></tr></tbody></table>

<table><caption>**Table 4.** Confusion matrices for MLP (using UNESCO codes). Rows correspond to the true category while columns are the predicted classes.</caption><thead><tr><th rowspan="2"></th><th colspan="8">Trained with balanced data set</th><th colspan="8">Trained with complete data set</th></tr><tr><th>12</th><th>22</th><th>25</th><th>31</th><th>21</th><th>24</th><th>33</th><th>23</th><th>32</th><th>12</th><th>22</th><th>25</th><th>31</th><th>21</th><th>24</th><th>33</th><th>23</th><th>32</th></tr></thead><tbody><tr><th>12</th><td>321</td><td>1</td><td>4</td><td>2</td><td>12</td><td>0</td><td>13</td><td>0</td><td>2</td><td>12</td><td>848</td><td>7</td><td>7</td><td>2</td><td>27</td><td>5</td><td>17</td><td>3</td><td>4</td></tr><tr><th>22</th><td>2</td><td>213</td><td>1</td><td>1</td><td>84</td><td>0</td><td>25</td><td>27</td><td>1</td><td>22</td><td>4</td><td>322</td><td>4</td><td>12</td><td>151</td><td>2</td><td>12</td><td>48</td><td>1</td></tr><tr><th>25</th><td>4</td><td>1</td><td>323</td><td>13</td><td>5</td><td>5</td><td>2</td><td>1</td><td>1</td><td>25</td><td>10</td><td>2</td><td>550</td><td>57</td><td>4</td><td>44</td><td>3</td><td>9</td><td>0</td></tr><tr><th>31</th><td>0</td><td>0</td><td>13</td><td>252</td><td>0</td><td>57</td><td>13</td><td>16</td><td>4</td><td>31</td><td>5</td><td>3</td><td>20</td><td>1776</td><td>0</td><td>316</td><td>11</td><td>47</td><td>7</td></tr><tr><th>21</th><td>21</td><td>54</td><td>4</td><td>0</td><td>247</td><td>2</td><td>16</td><td>11</td><td>0</td><td>21</td><td>37</td><td>134</td><td>5</td><td>4</td><td>682</td><td>8</td><td>23</td><td>24</td><td>0</td></tr><tr><th>24</th><td>3</td><td>0</td><td>13</td><td>60</td><td>0</td><td>246</td><td>5</td><td>25</td><td>3</td><td>24</td><td>11</td><td>3</td><td>26</td><td>336</td><td>6</td><td>2038</td><td>10</td><td>55</td><td>13</td></tr><tr><th>33</th><td>30</td><td>20</td><td>2</td><td>19</td><td>12</td><td>7</td><td>238</td><td>25</td><td>2</td><td>33</td><td>56</td><td>20</td><td>5</td><td>48</td><td>38</td><td>15</td><td>263</td><td>54</td><td>2</td></tr><tr><th>23</th><td>0</td><td>14</td><td>5</td><td>13</td><td>10</td><td>14</td><td>40</td><td>257</td><td>2</td><td>23</td><td>4</td><td>23</td><td>4</td><td>55</td><td>26</td><td>67</td><td>27</td><td>783</td><td>2</td></tr><tr><th>32</th><td>4</td><td>1</td><td>1</td><td>6</td><td>2</td><td>8</td><td>1</td><td>2</td><td>330</td><td>32</td><td>8</td><td>2</td><td>0</td><td>11</td><td>1</td><td>23</td><td>0</td><td>5</td><td>305</td></tr></tbody></table>

<table><caption>**Table 5.** Confusion matrix for SVM (using UNESCO codes). Rows correspond to the true category while columns are the predicted classes.</caption><thead><tr><th rowspan="2"></th><th colspan="8">Trained with balanced data set</th><th colspan="8">Trained with complete data set</th></tr><tr><th>12</th><th>22</th><th>25</th><th>31</th><th>21</th><th>24</th><th>33</th><th>23</th><th>32</th><th>12</th><th>22</th><th>25</th><th>31</th><th>21</th><th>24</th><th>33</th><th>23</th><th>32</th></tr></thead><tbody><tr><th>12</th><td>326</td><td>1</td><td>3</td><td>2</td><td>7</td><td>0</td><td>13</td><td>0</td><td>3</td><td>12</td><td>843</td><td>2</td><td>10</td><td>2</td><td>32</td><td>5</td><td>19</td><td>2</td><td>5</td></tr><tr><th>22</th><td>4</td><td>245</td><td>1</td><td>1</td><td>58</td><td>0</td><td>19</td><td>26</td><td>0</td><td>22</td><td>4</td><td>286</td><td>5</td><td>9</td><td>177</td><td>2</td><td>21</td><td>51</td><td>1</td></tr><tr><th>25</th><td>2</td><td>2</td><td>331</td><td>11</td><td>2</td><td>6</td><td>1</td><td>0</td><td>0</td><td>25</td><td>7</td><td>0</td><td>595</td><td>40</td><td>2</td><td>29</td><td>2</td><td>4</td><td>0</td></tr><tr><th>31</th><td>0</td><td>0</td><td>18</td><td>258</td><td>0</td><td>49</td><td>12</td><td>13</td><td>5</td><td>31</td><td>2</td><td>1</td><td>32</td><td>1739</td><td>0</td><td>350</td><td>12</td><td>37</td><td>12</td></tr><tr><th>21</th><td>18</td><td>75</td><td>4</td><td>0</td><td>229</td><td>1</td><td>16</td><td>10</td><td>2</td><td>21</td><td>28</td><td>111</td><td>6</td><td>4</td><td>706</td><td>5</td><td>31</td><td>24</td><td>2</td></tr><tr><th>24</th><td>2</td><td>0</td><td>9</td><td>58</td><td>0</td><td>254</td><td>4</td><td>24</td><td>4</td><td>24</td><td>10</td><td>4</td><td>32</td><td>316</td><td>4</td><td>2065</td><td>11</td><td>42</td><td>14</td></tr><tr><th>33</th><td>28</td><td>24</td><td>3</td><td>20</td><td>12</td><td>3</td><td>238</td><td>25</td><td>2</td><td>33</td><td>57</td><td>14</td><td>4</td><td>47</td><td>38</td><td>9</td><td>282</td><td>46</td><td>4</td></tr><tr><th>23</th><td>1</td><td>13</td><td>6</td><td>12</td><td>8</td><td>16</td><td>40</td><td>259</td><td>0</td><td>23</td><td>4</td><td>19</td><td>8</td><td>71</td><td>29</td><td>62</td><td>32</td><td>763</td><td>3</td></tr><tr><th>32</th><td>2</td><td>1</td><td>1</td><td>5</td><td>3</td><td>7</td><td>2</td><td>2</td><td>332</td><td>32</td><td>5</td><td>2</td><td>0</td><td>9</td><td>1</td><td>23</td><td>0</td><td>2</td><td>313</td></tr></tbody></table>
When Physics and Technological Science are eliminated from the complete data set both classification methods achieve an accuracy of 83%, thus errors on those categories have a great influence of the final model accuracy. A similar experiment on the balanced data set where the three most failed categories are eliminated from the data set, the results of the cross-validation provide an accuracy of 86% for both models. Thus, for the two classes: Physics (22) and Astronomy and Astrophysics (21), a binary classifier was built to discern documents among them so that this model could be used in cascade; that is bundling those two classes into one for the first model while the second model is responsible for differentiating them. Both MLP and SVM were tested using the previously described training techniques on the cascade classification scheme, selecting the best model among the trained ones. The best results obtained for this experiment were attained with a MLP as primary classifier and an SVM as secondary classifier (to discern between Physics and Astronomy and Astrophysics). When training the main classifier, an 80% success rate was obtained, while the secondary one obtained 76%. By putting them to work together, a success rate of 76% was achieved, with 66% and 65% in Physics (22) and Astronomy and Astrophysics (21), respectively over the balanced data set, thus no real improvement was obtained.

### 3.3 Method of Representation Vs Method of Classification

Is the power of the classification in the method of representation of the texts or in the method of classification? To answer this, the classification methods were tested again with the balanced data set, using the text representation given by Doc2Vec and BERT, and tuning the hyperparameters of the models in each case. TF-IDF was discarded due to the high dimensionality of the vectors it generates.

Table 6 shows the overall accuracy of the several pairs of classifier with the different text vectorization strategies. From these results it can be seen that the representation method used exerts a great influence in the classification results compared to the choice of the classification method, since by changing the representation method the accuracy varies considerably. Not so when changing the classification method, where the accuracy between the two methods is relatively close when not the same.

<table><caption>**Table 6.** Accuracy of classifiers for several representation methods, pointing out the best result of each row in bold letters.</caption><thead><tr><th></th><th>Doc2Vec</th><th>BERT</th><th>SPECTER</th></tr></thead><tbody><tr><th>SVM</th><td>66%</td><td>71%</td><td>76%</td></tr><tr><th>MLP</th><td>67%</td><td>70%</td><td>76%</td></tr></tbody></table>
## 3.4 Unsupervised Methods

To classify texts at the discipline and sub-discipline levels with supervised methods, annotated data for the over 700 categories present would have to be available, which can be prohibitive. This section explores the idea of classification using unsupervised methods that can be applied in those cases, but using the annotated data available at the field level to obtain evaluation metrics. K-Means and Gaussian Mixture were explored and while it's not expected that these outperform the supervised methods, it serves as a preliminary study for future research involving classification at the discipline and sub-discipline level.

In the tests with K-Means the complete data set and a balanced subset are used. After training the model for 9 groups with the complete data set, an accuracy of 54.71% was obtained. Accuracy improved when augmenting the number of groups, reaching up to 65.9% for 45 groups. In the case of Gaussian Mixture, after training the model for 9 groups with the complete data set, an accuracy of 55.27% was obtained, while increasing to the same number of groups did not improve the accuracy (see Table 7 for $\delta = 0$), it is clear that $k = 18$ is favorable.

Upon inspection of the clusters generated with these methods it can be seen that there is no clear classification of the cluster samples since a big percentage of the documents can be classified into the trailing category for that cluster. Again, this could be due to the interdisciplinary nature of some research articles. A second prediction is considered to enhance the performance of the unsupervised classifiers. By defining $\delta$ as the difference in the proportions of the two main categories in a cluster, consider a positive category prediction for the trailing category if the difference between the proportions of the main category and the second to main category is lower than $\delta$. Therefore, the classifier uses both results to predict, and if one of them is the correct label it is considered as a correct prediction.

Adding the possibility of double prediction described above to the tests, an accuracy of 76.57% is achieved in the case of K-means (for $k = 45$ and $\delta = 0.20$). The inclusion for a double prediction did not benefit the predictions for Gaussian Mixture since most accuracy were below 65%. The results with the complete data set can be seen at Table 7 for various values of $\delta$.

When the methods were trained with the balanced subset, K-means obtained a 52.69% accuracy with 9 groups while Gaussian Mixture obtained an accuracy of 55.27%. When the number of groups was increased in the training of this data set, the best result was obtained with 45 groups, with a 67.63% accuracy in the case of K-means, while in the case of Gaussian Mixture the best accuracy was 61.4% (also with 45 groups), in sharp contrast with the results presented in Table 3. To obtain a comparable accuracy, a threshold of $\delta = 0.20$ must be used, implying that there is at least 20% of the data in the cluster not assigned to any of the leading categories.
<table><caption>**Table 7.** Accuracy of K-Means and Gaussian Mixture, highlighting in bold the best result of each row. Each method is trained with the complete data and with the balanced set, also the accuracy for various $\delta$ thresholds.</caption><thead><tr><th colspan="6">Trained with balanced data set</th><th colspan="6">Trained with complete data set</th></tr><tr><th>δ</th><th>k = 9</th><th>k = 18</th><th>k = 27</th><th>k = 36</th><th>k = 45</th><th>δ</th><th>k = 9</th><th>k = 18</th><th>k = 27</th><th>k = 36</th><th>k = 45</th></tr></thead><tbody><tr><th colspan="6">K-means</th><th colspan="6">K-means</th></tr><tr><td>0.00</td><td>52.69%</td><td>61.71%</td><td>60.49%</td><td>66.69%</td><td>67.63%</td><td>0.00</td><td>54.71%</td><td>63.32%</td><td>63.78%</td><td>65.18%</td><td>65.9%</td></tr><tr><td>0.05</td><td>56.29%</td><td>63.15%</td><td>65.28%</td><td>68.47%</td><td>69.25%</td><td>0.05</td><td>56.18%</td><td>65.54%</td><td>67.68%</td><td>69.63%</td><td>68.79%</td></tr><tr><td>0.10</td><td>63.9%</td><td>70.07%</td><td>70.35%</td><td>69.1%</td><td>71.98%</td><td>0.10</td><td>56.87%</td><td>67.26%</td><td>68.07%</td><td>69.4%</td><td>71.69%</td></tr><tr><td>0.15</td><td>64.03%</td><td>70.44%</td><td>69.88%</td><td>70.19%</td><td>72.29%</td><td>0.15</td><td>59.84%</td><td>70.35%</td><td>70.43%</td><td>72.54%</td><td>71.47%</td></tr><tr><td>0.20</td><td>63.9%</td><td>69.51%</td><td>72.23%</td><td>71.35%</td><td>73.76%</td><td>0.20</td><td>63.05%</td><td>72.65%</td><td>72.23%</td><td>75.03%</td><td>76.57%</td></tr><tr><th colspan="6">Gaussian Mixture</th><th colspan="6">Gaussian Mixture</th></tr><tr><td>0.00</td><td>53.94%</td><td>36.07%</td><td>23.61%</td><td>58.36%</td><td>61.4%</td><td>0.00</td><td>55.27%</td><td>62.67%</td><td>60.95%</td><td>58.45%</td><td>53.74%</td></tr><tr><td>0.05</td><td>60.39%</td><td>43.49%</td><td>23.61%</td><td>61.52%</td><td>62.77%</td><td>0.05</td><td>56.79%</td><td>64.78%</td><td>62.88%</td><td>63.81%</td><td>53.74%</td></tr><tr><td>0.10</td><td>64.53%</td><td>51.5%</td><td>23.61%</td><td>65.87%</td><td>62.77%</td><td>0.10</td><td>56.79%</td><td>68.12%</td><td>65.98%</td><td>64.98%</td><td>53.74%</td></tr><tr><td>0.15</td><td>67.85%</td><td>51.5%</td><td>23.61%</td><td>66.72%</td><td>63.18%</td><td>0.15</td><td>56.79%</td><td>69.03%</td><td>68.02%</td><td>67.2%</td><td>53.74%</td></tr><tr><td>0.20</td><td>70.76%</td><td>60.55%</td><td>23.61%</td><td>66.88%</td><td>63.18%</td><td>0.20</td><td>62.26%</td><td>74.82%</td><td>70.61%</td><td>67.6%</td><td>54.22%</td></tr></tbody></table>

## 4 Conclusions and Discussion

This article set out to evaluate solutions to author classification using their scientific products so that they can be labeled in a platform that offers a scientific directory for Latin America (ACALconecta.org). The multiple classification problem was posed as a supervised learning task but also as an unsupervised one to test the methodology for the discipline classification. The best classifier overall resulted with an accuracy of 79% on test data using SPECTER as the language representation model. The training for SPECTER was done over a corpus of scientific documents. It was shown that pairing this representation model with SVM or MLP provided equivalent accuracy results in the classification task, however SVM improved the classification of 4 UNESCO fields, while MLP resulted in better classification in 2 fields. Therefore, SVM seemed better suited for the task.

Some scientific vocabulary may be exclusive to certain areas, for example, vocabulary pertinent to certain fields in mathematics are seldom used in other areas of science (e.g., Sobolev spaces, diffeomorphism, symplectic manifold), while other words are more universal and span out to various disciplines (e.g. simulation, analysis, data, etc.). Scientific articles that used specialized vocabulary were easily classified within their respective fields. In contrast, the use of ample cross-discipline vocabulary or research that is interdisciplinary in nature (e.g. mathematical biology research or applications of machine learning techniques) can deem the classification task difficult even for a human. A proof of the widespread interdisciplinary research is the fact that out of all SCOPUS classified journals, only 33% of them are classified in one of the UNESCO areas, and although articles published in cross-discipline journals were not considered,
much of the research in the remaining journals still share common scientific vocabulary that can span two or more fields.

The interdisciplinary nature of research was made evident in our classifier's difficulty discerning between physics and astronomy and astrophysics, that are very much interrelated areas and share common terms. In MAG taxonomy those two areas are bundled together while areas such as computer science holds its own category as opposed to the UNESCO taxonomy where it is a discipline within mathematics. One could pose the question of whether classification under one taxonomy is better than another, considering it as future work.

When analyzing the confusion matrices for the models trained with balanced data, it is evident that the categories that are difficult for the models to classify correctly are physics, astronomy and astrophysics and technological sciences. In the last case, failed predictions are distributed in the various fields and this is expected due to the wide range of topics covered and the intersections that exist with other areas. In the case of MLP 84 physics labelled data points are classified as Astronomy and Astrophysics while 54 of the latter are classified as the former, similar situation occurs with SVM. If the 3 categories are eliminated from the data set, the models achieve an 86% accuracy in both classification models (MLP and SVM), thus confirming that these areas are affecting the general classification results.

Pooling together similar areas (physics and astronomy and astrophysics) and applying a cascade model such that the first model predicted amongst the 8 remaining areas after pooling and the second model discerned within the pooled category is applicable, was unsuccessful in terms of a greater prediction accuracy overall in each of the pooled categories. However, a significant improvement in accuracy was observed when trained to separate the pooled categories with respect to other areas (an 80% accuracy as opposed to the 76% for when trained using all the data), while the accuracy obtained in other fields remained similar. This could again reinforces the idea that the classification of certain areas using a different taxonomy may be beneficial.

One key aspect that has been noted both here and in other articles is the relevance of the vectorization method used. Table 6 clearly demonstrates the fact that using the same vectorization strategy both classifiers yield similar precision on test data. However, the precision varies greatly when the vectorization method is exchanged. Text representation is an active area, and new transformer based embeddings may increase the classification precision on this data set.

The availability of annotated data for a large corpus is rare, therefore the possibility of using unsupervised methods was explored for classification purposes. Our findings demonstrate that among the methods evaluated K-means showed the greatest potential for classification, requiring 45 groups for a reasonable classification with no double prediction on the complete and balanced data set. The inclusion of the double prediction improved the accuracy as expected consistently maintaining the number of groups at larger values. K-Means and Gaussian mixture benefit from training with the complete data set, but it was
more evident in the case of Gaussian Mixture that achieved good accuracy with fewer groupings.

Training a classification model with bias in the data composition can be troublesome as that bias is learned as well. However, there are instances in which the trends in the training sample mimic the underlying distribution of the data, and can provide insight to the model as well, thus this prompted to train all models for the two types of sets, balanced and complete set, obtaining two models for each strategy. In the case of supervised methods, the overall accuracy was greater in the complete data set, due to the bias induced by the data. In Table 3, clearly the over expressed categories (Life Science and agricultural sciences) in the data benefit from the unbalanced training samples, while the underrepresented categories (Medical Sciences) require a balance set in order to achieve better classification metrics.

## References

1. Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. arXiv preprint arxiv:1903.10676 (2019). https://doi.org/10.48550/ARXIV.1903.10676

2. Caliński, T., Harabasz, J.: A dendrite method for cluster analysis. Commun. Statist. Theory Methods 3(1), 1–27 (1974)

3. Cascajares, M., Alcayde, A., Garrido-Cárdenas, J., Manzano-Agugliaro, F.: The contribution of Spanish science to patents: Medicine as case of study. Int. J. Env-iron. Res. Publ. Health 17, 3638 (2020). https://doi.org/10.3390/ijerph17103638

4. Cohan, A., Feldman, S., Beltagy, I., Downey, D., Weld, D.S.: Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arxiv:2004.07180 (2020). https://doi.org/10.48550/ARXIV.2004.07180

5. Davies, D.L., Bouldin, D.W.: A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. PAMI 1(2), 224–227 (1979). https://doi.org/10.1109/TPAMI.1979.4766909

6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2018). arXiv preprint arxiv:1810.04805. https://doi.org/10.48550/ARXIV.1810.04805

7. Klabunde, R.: Daniel jurafsky/james h. martin, speech and language processing. Zeitschrift für Sprachwissenschaft 21(1), 106–108 (2002)

8. Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. arXiv preprint arxiv:1405.4053 (2014). https://doi.org/10.48550/ARXIV.1405.4053

9. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(11) (2008)

10. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arxiv:1301.3781 (2013). https://doi.org/10.48550/ARXIV.1301.3781

11. Mthembu, L., Marwala, T.: A note on the separability index. arXiv preprint arxiv:0812.1107 (2008). https://doi.org/10.48550/ARXIV.0812.1107

12. Noble, W.S.: What is a support vector machine? Nat. Biotechnol. 24(12), 1565–1567 (2006)
13. Pedregosa, F., et al.: Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. **12**, 2825–2830 (2011)

14. Popescu, M.C., Balas, V., Perescu-Popescu, L., Mastorakis, N.: Multilayer percep-
tron and neural networks. WSEAS Trans. Circuits Syst. **8** (2009)

15. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. arXiv preprint arxiv:1908.10084 (2019)

16. Reynolds, D.A.: Gaussian mixture models. Encyclopedia. Biometrics **741**, 659–663 (2009)

17. Schneegans, S., Lewis, J., Straza, T.: Informe de la unesco sobre la ciencia: La carrera contra el reloj para un desarrollo más inteligente - resumen ejecutivo (2021)

18. Shahapure, K.R., Nicholas, C.: Cluster quality analysis using silhouette score. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 747–748 (2020). https://doi.org/10.1109/DSAA49011.2020.00096

19. Singh, S., Gill, N.S.: Analysis and study of k-means clustering algorithm. Int. J. Eng. Res. Technol. **2** (2013)

20. UNESCO: Proposed international standard nomenclature for fields of science and technology (1988)