dataesr/scientific-paragraphs-categorization huggingface dataset:
Related work Multiple previous efforts have shown that unlabeled data can help to improve the classification accuracy when very few labeled samples are available. As proposed by Gabrys and Petrakieva  [9]  or Bouchachia  [10] , these methods can be grouped into three main families: (1) pre-labeling approaches, where unlabeled data are labeled with an initial classifier trained on the set of labeled objects, (2) post-labeling approaches, where clusters of the whole dataset are labeled estimating their composition in terms of labeled objects, and (3) semi-supervised approaches, which consist in dealing with both labeled and unlabeled data at the same time during the clustering process.
More recently, Raskutti et al. [14]  exposed a co-training method that does not necessary need two complementary supervised learning algorithms. The idea is to produce an alternate view of the data by performing an unsupervised classification algorithm on all the dataset (i. e. labeled and unlabeled). Then, the original view and the view built from the clustering are used to create two independent predictors for co-training. In  [15] , the authors presented a co-training approach assuming to have two views of the data. The method uses the correlation between the two views to produce extra positive and negative samples in an iterative process. Experiments were performed when only one labeled sample is available. They show that the method overcome other co-training approaches. Unlike co-training, Assemble  [16]  can build semi-supervised ensemble of any size, and does not require the domain to have multiple views.
Post-labeling methods The idea of these approaches is to first produce a clustering of the dataset and then to evaluate the purity of each cluster, by calculating their composition in terms of label. Any clustering algorithm can be used in the first step. In  [17] , a general approach based on the Expectation Maximization (EM) algorithm is proposed. Other methods  [18, 19]  work on the optimization of the purity in the clusters extracted from the dataset. They propose to use a Genetic Algorithm (GA) to iteratively refine the class membership of the unlabeled objects so that the maximum a posteriori based predicted labels of the data in the labeled dataset are in agreement with the known labels.
Semi-supervised clustering In  [20] , an empirical study of various semi-supervised learning techniques on a variety of datasets is presented. Different experiments were made to evaluate the influence of the size of the labeled and unlabeled sets, or the effect of noise in the samples. The paper concludes that the performance of the methods is heavily dependent of the field of application and the nature of the dataset. However, using labeled and unlabeled samples improves the accuracy in most of the cases. Basu et al. [21]  have defined two variants of the Kmeans algorithm, dealing with labeled data. The main idea is to used the labeled objects to guide the seeding step of the algorithm. In the first method, the seeds are calculated according to the labeled data and the algorithm is ran classically, while in the second case, the labeled objects used for the seeding stay in their initial cluster.
Another approach was exposed in  [22]. The idea is to simultaneously compute the clustering and the classification, instead of proceeding in two sequential steps. To achieve this goal, the authors define an objective function evaluating not only the classification but also the clustering ability by mixing two terms: the misclassification rate (for the supervised part) and the clustering impurity (for the unsupervised aspect). A quite similar method is presented in  [23]  and applied to real marketing datasets. Ao et al. [24]  proposed to combine supervised and unsupervised models via unconstrained probabilistic embedding. In their approach, they considered an ensemble problem in which outputs coming from models developed in the supervised and unsupervised modes are combined to improve the classification accuracy of supervised model ensemble.
For example, using different clustering methods, or the same method with different parameters can lead to the generation of different results. More recently, Yu et al. [33, 34, 35]  proposed a new approach of semi-supervised constraint ensemble clustering. The idea is to transfer expert knowledge into pairwise constraints used to make a feature selection to avoid noise degradation of the results. Then, a selection of a subset of representative clustering results is extracted and they are merged to obtain the final solution. Sampling the instances, or the attributes, can also be used to generate multiple views  [36]  of the same dataset and to cluster them individually to obtain multiple clustering results. Random projection  [37]  is also an interesting approach to generate diverse results.
Some experiments on the classification of an agricultural zone of the Nevada, based on an AVIRIS image, showed the efficiency of the estimator on real problems. Jia et al. [42]  presented a new method to deal with hyperspectral data. The idea is to cluster the training data and then, to associate spectral clusters to information classes to build a cluster-space classification. Then, each pixel is classified according to its cluster membership and the membership of the cluster to information classes. Morgan et al. [43]  proposed another approach to solve the problem of the small amount of labeled data. It consists in using a feature reduction scheme that adaptively adjusts itself to the size of the samples dataset. Some experiments are presented on hyperspectral data obtained with the HyMap sensor (Hyperspectral Mapper).
Semi-supervised methods In this section, we formally present the different approaches used in our experiments. Let X denote a set of n data objects x j ∈ X. We consider a q-class classification problem with m labeled and l unlabeled objects where m is very low and l >> m. Let L be the set of labeled objects of X: L = {(x 1 , y 1 ),. , (x m , y m )} (1) where y i ∈ {1,. , q} are the class values of the samples. Let U be the set of unlabeled objects of X: U = {(x m+1 , 0),. , (x m+l , 0)} ( 2 ) where 0 means that there is no label assigned to this object. The objective of the semi-supervised classification is to build a classifier based on the training dataset X. This classifier is a function associating one of the q classes to any object x. It can be formally defined as: y = C X (x) : y ∈ {1,.
Pre-labeling methods The first methods are called pre-labeling methods. The idea is to perform an initial classification C L on the labeled data only (L). Then, the unlabeled data are labeled according to this classification. In the static labeling (SL)  [9]  method, the unlabeled data are all labeled in one step by simply applying C L to U. A new dataset W is built as: W = {(x j , y j ) : y j = C L (x j ), x j ∈ U } (4) Then, in a second step, the final classification is computed as: y = C L∪W (x) (5) The algorithm of this method is described in Algorithm 1.
Algorithm 1 Static labeling (SL) 1: build a classification C L given the labeled dataset L 2: let W = {(x j , y j ) : y j = C L (x j ), x j ∈ U } 3: build the final classifier C L∪W Another pre-labeling approach is the dynamic labeling (DL)  [9]  method. As for the static labeling method, a classifier C L is build according to the labeled dataset. Then, instead of labeling all the objects of U , they are iteratively labeled, one sample at a time. One object x j of U is chosen and labeled according to C L. Then it is added to L and a new classifier is trained with L ∪ {x j }. The process iterates until all unlabeled samples are labeled. The algorithm is presented in Algorithm 2. The order in which the objects are chosen can be defined according to the confidence in the classification of each one.
Post-labeling methods At the opposite of pre-labeling methods that use the unlabeled objects to enhance an initial classifier, post-labeling methods first build a clustering of all the objects of the dataset, but without considering their label. Then, the unlabeled data of each cluster are labeled to the majority class of their cluster. Let K l , Algorithm 2 Dynamic labeling (DL) 1: let U = U and W = ∅ 2: build a classifier C L∪W 3: for all x j ∈ U chosen according to their class confidence do 4: W := W ∪ {(x j , t j ) : t j = C L∪W (x j )} 5: U := U {(x j , 0)} 6: end for 7: build the final classifier C L∪W l = 1,. , k, denote the clusters produced by the initial clustering on the dataset, and c lj , j = 1,. , q the number of labeled objects from class j in the cluster l: c lj = {(x i , y i ) ∈ L : (x i , y i ) ∈ K l , y i = j} (6) Cluster labeling by majority.
Semi-supervised clustering The last type of approaches, called semi-supervised clustering approaches, incorporates labeled and unlabeled data at the same time. The idea is that the clustering of the data is guided by the labeled samples. Refined clustering. The refined clustering (RC)  [9]  method consists in two steps: 1. creation of a new fully labeled dataset from both labeled and unlabeled objects (L ∪ U ); 2. application of a nearest neighbor classifier to learn the new dataset. The first step is very important and is composed of two stages: 1. creation of a new dataset, including both labeled and unlabeled data subsets and representing as much as possible information; 2. labeling of this dataset by the CLM technique described previously. To create the new dataset, a divisive approach is proposed.
Algorithm 6 Refined clustering (RC) 1: build a clustering K = {K l , l = 1. k} on X with k relatively small 2: for all K l , l = 1. k do end if 17: end for 18: apply CLM to K Seeding methods. Finally, we used in this study the seeding methods proposed by Basu et al. [21]. These methods are variants of the well-known Kmeans partitioning method. The objective of the Kmeans method is to generate a k-partitioning K = k i=1 K i of the dataset X. Each cluster K i is represented by its gravity center µ i. As previously mentioned, we consider a q-class classification problem of a dataset X = L ∪ U where L are labeled objects. The idea is to guide the Kmeans algorithm by using the labeled objects L as initial seeds. A first q-partitioning {S i } i=1. q of L is calculated by grouping the objects of L having a same label into one cluster.
Semi-supervised learning enhanced by multiple clusterings The method that we propose, called Semi-supervised learning enhanced by multiple clusterings (SLEMC), could be categorized as a post-labeling method. Indeed, it tries to improve the classification by first producing a clustering of the dataset. The clustering, computed on all the labeled and unlabeled objects, regroups the similar instances together, maximizing the intracluster similarity and the intercluster dissimilarity. If the classes are well separated in the feature space, we should be able to associate to each cluster one of the classes, using the class of the labeled samples belonging to the cluster. Unfortunately, in real world problems, classes are generally not well separated. It is then possible to have samples from different classes in one cluster, or no sample in others.
To address this issue, the proposed method uses a combination of multiple clusterings. We consider here b clustering methods {C k } 1≤k≤b. Each method is an instance of one clustering algorithm (e. g. Kmeans, EM, SOM, etc. ), having its own parameters and producing a partition K k = {K 1 k ,. , K n k } of the n objects from the dataset X composed of both labeled and unlabeled samples. Each object x i ∈ X is described by a vector of p features (a i 1 ,. , a i p ). The idea is to affect to each labeled sample (x i , y i ) ∈ L, ∀i : 1 < i < m, a new features vector: v(x i ) = a i 1 ,. , a i p , K i 1 ,. , K i b , y i (10) where K i j is the cluster affected in the j th clustering partition K j to x i , and y i is the label of the sample x i (see equation 1). Then, a predictive model C V : X → {1,.
, q} where {1,. , q} are the class values of the samples (see equation 3), can be induced from this new dataset V = {v(x i )} m i=1 , using a classical supervised learning method (e. g. Naive Bayes). The description of each unlabeled object x j of U is then extended in the same way: v (x j ) = a j 1 ,. , a j p , K j 1 ,. , K j b , 0 (11) The predictive model C V , learned previously on the labeled samples, can be applied for each unlabeled object x j ∈ U. The algorithm presenting the complete classification process is presented in algorithm 9. The computational cost of the proposed method is directly linked to the clustering algorithms and the supervised learning algorithm that are used in Algorithm 9. It is interesting to note that the clustering step can be performed offline, as the model only needs to be trained once.
Algorithm 9 Semi-supervised learning enhanced by multiple clusterings (SLEMC) 1: apply b clustering algorithms {C k } 1≤k≤b to the dataset X 2: each method C k produces a partition K k = {K 1 k ,. , K n k } of the n objects 3: for all (x i , y i ) ∈ L do 4: v(x i ) = a i 1 ,. , a i p , K i 1 ,. , K i b , y i 5: end for 6: apply a supervised learning method to produce a predictive model C V from V = {v(x i )} m i=1 7: affect the features vector v (x j ) = a j 1 ,. , a j p , K j 1 ,. , K j b , 0 to each x j ∈ U 8: use C V to label all objects of U.
Experiments To evaluate the efficiency of the SLEMC approach, three sets of experiments were carried out. First on artificial datasets (Section 4. 1) where the data and the samples are well correlated to the class information (i. e. the ideal case), then on standard UCI datasets (Section 4. 2), and finally on a real application in the domain of remote sensing image analysis (Section 4. 3). For all experiments, we compare the SLEMC method to semi-supervised methods presented in Section 3. Comparison is also performed with supervised classifiers, using only the labeled objects for the learning (i. e. the unlabeled data were ignored). Several supervised algorithms were selected, from different families of learning methods: the standard tree inducer C4. 5, Naive Bayes (NB) and a 1-nearest-neighbor (1-NN) algorithm.
To apply SLEMC, the number of clusterings to compute on the dataset has to be fixed (i. e. how many attributes will have each object in the new data space). Then, the different clustering methods have to be chosen. Four different configurations were tested, to study the importance of the number of clusterings. The four configurations are the followings: 1. Simple: one EM (Expectation-Maximization); 2. Low : one EM and one KMeans; 3. Medium: two EM and two KMeans; 4. High: c EM and c KMeans (with c the number of classes in the dataset). Note that only EM and KMeans algorithms were chosen for convenience (i. e. low complexity) but other clustering algorithms could have been used. Furthermore, as the method aim at leveraging from the class distribution in the data space, these two algorithms are particularly well suited.
The supervised method used to produce the learning model is Naive Bayes in all experiments. Moreover, we also evaluate an alternative setup for SLEMC, that keeps the description along with adding new features for each object. These configurations are referred as Simple+, Low+, Medium+ and High+. Each method was run with a number of clusters equals to the number of classes actually present in the dataset except for the High configuration where the clustering method had k clusters (k ∈ {2,. , c}, randomly chosen). For each experiment, each dataset was split into two sets, each composed of 50% of the objects. The first set was used as data with unknown label for the unsupervised learning task of the semi-supervised methods. The second set was used to select few samples considered as the available knowledge.
The rest of the second set was used to evaluate the methods (i. e. compute accuracy measure). We chose to evaluate the method when 2, 4, 8 and 16 samples per class were available. We carried out the experiments for each of these configurations (2, 4, 8 and 16 samples) for each dataset. As the number of available labeled samples is very low the performance can greatly differ depending on their selection. Consequently, the experiments were carried out 30 times and the results were averaged. At each execution, we split randomly the datasets in two sets (50% for the unsupervised learning, 50% in which we selected the available labeled samples and used the remaining objects for the evaluation). For the sake of readability, only the top 3 accuracies are presented for each experiment in result tables.
Evaluation on artificial benchmarks The first set of experiments were carried out on artificial datasets, generated by the software provided by Julia Handl  [45]. Three different datasets configurations were used: 2d-4c-noX: 4 classes described by 2 attributes; 10d-4c-noX: 4 classes described by 10 attributes; 2d-10c-noX: 10 classes described by 2 attributes. For all of them, 10 different datasets were randomly generated (no0, no1,. no9). Figure  1  shows three datasets generated for the 2d4c series. For each experiment, the accuracy of the model was calculated. Tables 1, 2 and 3 present the top 3 accuracies respectively for the series 2d-4c, 10d-4c and 2d-10c. From these tables, one can notice that the High configuration of our method outperforms all the other approaches as it appears most of the time in rank 1 for multiple datasets.
This is even more visible for the configurations where very few (2 or 4) samples are available. For example in Table  1  and Table  2 , the configuration High is -always -ranked first when 2 samples are available and is first 9 times out of 10 in Table  3. One can also notice that when the number of labeled sample increase  (8, 16)  the method that keep the description along with the new features (i. e. configurations with the "+") tend to perform better. We also computed for the three sets of artificial datasets (2d-4c-noX, 2d-10c-noX and 10d-4c-noX) the average ranking among the datasets. This is computed by summing the rank of each method for each experiment and dividing the result by the number of experiments. The Table  4  shows these results. In this table, one can observe that the configuration "High" performs particularly well as it is always in first position when the number of labelled samples available is low (2 or 4).
Evaluation on UCI datasets In this section, we compare our approach to the semi-supervised methods presented previously on various datasets of the UCI repository  [46]. The Table  5  summarizes information on each dataset. Note that it is not guarantee that the class distribution is correlated with the data clusters distribution. With such datasets, the information provided by the clustering will not give any relevant insight to enhance the model learning. The results for each dataset and for each experiment are presented in Tables  7  and 8 , where the values are the accuracy of the top 3 methods for each configuration. The number of samples used is presented in brackets at the beginning of each line. In these tables, one can observe that the proposed methods outperformed, most of the time, the supervised learning and the other semi-supervised approaches, especially when the number of samples is very low (2 or 4 samples per class).
The best results are obtained on the wine dataset, where our methods are 8 times in the top 3 for the 12 configurations. On ionosphere, anneal, heart, cardio and thyroid, they are 6 times cited for the 12 configurations. It is also important to notice that for some configurations and some datasets the difference of accuracy between our method and the other semi-supervised methods appears to be important. For example, for the configuration (2) on the dataset optidigits, our method with the High configuration gives 54. 77% of accuracy while the other semi-supervised present in the top 3 (DL and 1-NN) respectively gives 50. 85% and 48. 88%. This can also be noted on the wine dataset for the configuration (2) as our method gives 88. 40% and 83. 25% respectively for the High and the Medium configurations, while the SCEC method only obtained 79.
However, as the features added from the clustering algorithms are computed on 50% of the dataset, the descriptions are more relevant. When the number of available labeled samples increases, the description of the objects (i. e. the original features) are more meaningful, and are more precise than the rough information offered by the clusterings. As stated in the introduction, if the data space of the dataset is not correlated with the class information, using clustering is useless. This affirmation can be studied on the segment and robot datasets, where the proposed semi-supervised approach obtains worse results than 1-NN or C4. 5, regardless of the number of available samples. Among the other semi-supervised methods, the DL gave particularly good results as it is present 22 times in the top 3.
It is followed by the SL which is present 5 times. These results tend to show that the pre-labeling give better results than the other ones on these datasets. The Table  6  shows the average ranking for two sets of the UCI datasets: Set 1 {iris, wine, ionosphere, waveform, cardio, optidigts, mushroom} and Set 2 {wdbc, pima, anneal, heart, trans, robot, vehicle, thyroid, liver}. The first set contains the datasets where the classes are following the data distribution, while in the second they are not. As one can see in this table, for the first set, the proposed method provides the best results and comes first in average ranking. For the second set, the 1-NN method and DL methods provide the best results. However, the average rankings in the second set (e. g. 6. 0) are much higher than the average It means that for this set, it is more difficult to find a single method that outperforms the others.
This is mainly due to the fact that most the semi-supervised methods compared in this paper make also the assumption that the data distribution is following the classes distribution. We also carried out an experiment to study the importance of the amount of unlabeled data in our methods. We made some experiments in which we used 50% of the dataset as unlabeled data (as in the previous experiment) but also made experiments with 25% and 10% of available unlabeled data. We used.
Remote Sensing data evaluation As presented in the introduction, in the field of remote sensing classification, the problem of the low ratio between the number of features and the number of samples is really important when dealing with hyperspectral data or with very high resolution images. Indeed, in this last case, the classification is basically computed in two steps: a segmentation of the image is performed, producing a set of regions (i. e. groups of homogeneous pixels); then, these regions are characterized by many features (spectral attributes, geometrical features, spatial attributes, etc. ). Thus, we have a new dataset to classify, composed of few regions (compared to the number of pixels in the image), but characterized by more features. We present here an experiment on the classification of an extract of a very high remote sensing image of an urban area of the city of Strasbourg (France).
The input data is a pan-sharpened Quickbird© image with 4 spectral bands and a spatial resolution of 0. 7 meter, i. e. a pixel on the image represents a square of 0. 7 × 0. 7m on the ground. The image and the expert annotations used for the learning task are shown on Figure  3. After having computed a segmentation on this image using the watershed segmentation algorithm  [47, 48] , we characterized each region by the 23 following features: 2 features representing the mean and the standard deviation of the mean of all spectral channels; 2 features calculated as the mean and the standard deviation of the NDVI (Normalized Difference Vegetation Index ) of the pixels composing the region; 1 feature representing the area covered by the region; 1 feature calculated as the elongation of the shape represented by the region; Finally, the dataset generated (dataset remote sensing) was composed of 186 objects described by 15 features, divided into three classes.