dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction With the evolution of technology, several kinds of devices and applications are continuously generating large amounts of data in a fast-paced way as streams. Hence, the data stream mining area has become indispensable and ubiquitous in many real-world applications that require real-time -or near real-time-processing, e. g. , social networks, weather forecast, spam filters, and more. Unlike traditional datasets, the dynamic environment and the tremendous volume of data streams make them impossible to store or to scan multiple times  [12]. Classification is an active area of research in data stream mining field where several researchers are paying attention to develop new -or improve existingalgorithms  [14]. However, the dynamic nature of data streams has outpaced the capability of traditional classification algorithms to process data streams.
In this context, a multitude of supervised algorithms for static datasets that have been widely studied in the offline processing, and proved to be of limited effectiveness on large data, have been extended to work within a streaming framework  [3, 5, 11, 18]. Data stream mining approaches can be divided into two main types  [23] : (i) instance-incremental approaches which update the model with each instance as soon as it arrives, such as Self-Adjusting Memory kNN (samkNN)  [18] , and Hoeffding Adaptive Tree (HAT)  [4] ; and (ii) batch-incremental approaches which make no change/increment to their model until a batch is completed, e. g. , support vector machines  [10] , and batch-incremental ensemble of decision trees  [15]. Nevertheless, the high dimensionality of data complicates the classification task for some algorithms and increases their computational resources, most notably the k-Nearest Neighbors (kNN) because it needs the entire dataset to predict the labels for test instances  [23].
To cope with this issue, a promising approach is feature transformation which transforms the input features into a new set of features, containing the most relevant components, in some lower-dimensional space. In attempt to improve the performance of kNN, we incorporate a batchincremental feature transformation strategy to tackle potentially high-dimensional and possibly infinite batches of evolving data streams while ensuring effectiveness and quality of learning (e. g. accuracy). This is achieved via a new manifold technique that has attracted a lot of attention recently: Uniform Manifold Approximation and Projection (UMAP)  [21] , built upon rigorous mathematical foundations, namely Riemannian geometry. To the best of our knowledge, no incremental version of UMAP exists which makes it not applicable on large datasets.
The main contributions are summarized as follows: -Batch-Incremental UMAP : a new batch-incremental novel manifold learning technique, based on extending the UMAP algorithm to data streams. -UMAP-kNearest Neighbors (UMAP-kNN): a new batch-incremental kNN algorithm for data streams classification using UMAP. -Empirical experiments: we provide an experimental study, on various datasets, that discusses the implication of parameters on the algorithms performance; The paper is organized as follows. Section 2 reviews the prominent related work. Section 3 provides the background of UMAP, followed by the description of our approach. In Section 4 we present and discuss the results of experiments on diverse datasets. Finally, we draw our conclusions and present future directions.
This straightforward technique is computationally cheap but ineffective with data streams since it relies on the whole dataset. Therefore, some incremental versions of PCA have been developed to handle streams of data  [13, 24, 26]. Graph/Neighborhood-based techniques are leveraged in the context of dimension reduction and visualization by using the insight that similar instances in a large space should be represented by close instances in a low-dimensional space, whereas dissimilar instances should be well separated. t-distributed Stochastic Neighbor Embedding (tSNE)  [20]  is one of the most prominent DR techniques in the literature. It has been proposed to visualize high-dimensional data embedded in a lower space (typically 2 or 3 dimensions). In addition to the fact that it is computationally expensive, tSNE does not preserve distances between all instances and can affect any density-or distance-based algorithm and hence conserves more of the local structure than the global structure.
Ultimately, both techniques are essentially transductive  6  and do not learn a mapping function from the input space. Hence, they need to process all the data for each new unseen instance, which prevents them from being usable in data streams classification models. Figure  1  shows the projection of CNAE dataset (see Table  1 ) into 2-dimensions in an offline/online fashions where each color represents a label. In Figure  1a , we note that UMAP offers the most interesting visualization while separating classes (9 classes). The overlap in the new space, for instance with tSNE in Figure  1b , can potentially affect later classification task, notably distance-based algorithms, since properties like global distances and density may be lost. On the other hand, linear transformation, such as PCA, cannot discriminate between instances which prevents them from being represented in the form of clusters (Figure  1c ).
Algorithm Description A very efficient and simple scheme in supervised learning is lazy learning  [1]. Since lazy learning approaches are based on distances between every pair of instances, they unfortunately have a low performance in terms of execution time. The k-Nearest Neighbors (kNN) is a well-known lazy algorithm that does not require any work during training, so it uses the entire dataset to predict labels for test instances. However, it is impossible to store an evolving data stream which is potentially infinite -nor to scan it multiple times -due to its tremendous volume. To tackle this challenge, a basic incremental version of kNN has been proposed which uses a fixed-length window that slides through the stream and merges new arriving instances with the closest ones already in the window  [23].
Datasets We use 3 synthetic and 6 real-world datasets from different scenarios that have been thoroughly used in the literature to evaluate the performance of data streams classifiers. Table  1  presents a short description of each dataset, and further details are provided in what follows. Tweets. The dataset was created using the tweets text data generator provided by MOA  [6]  that simulates sentiment analysis on tweets, where messages can be classified depending on whether they convey positive or negative feelings. Tweets 1,2,3 produce instances of 500, 1,000 and 1,500 attributes respectively. Har. Human Activity Recognition dataset  [2]  built from several subjects performing daily living activities, such as walking, sitting, standing and laying, while wearing a waist-mounted smartphone equipped with sensors.
The sensor signals were pre-processed using noise filters and attributes were normalized. CNAE. CNAE is the national classification of economic activities dataset  [9]. Instances represent descriptions of Brazilian companies categorized into 9 classes. The original texts were pre-processed to obtain the current highly sparse data. Enron. The Enron corpus dataset is a large set of email messages that was made public during the legal investigation concerning the Enron corporation  [17]. This cleaned version of Enron consists of 1, 702 instances and 1, 000 attributes. IMDB. IMDB movie reviews dataset was proposed for sentiment analysis  [19] , where each review is encoded as a sequence of word indexes (integers). Nomao. Nomao dataset  [8]  was provided by Nomao Labs where data come from several sources on the web about places (name, address, localization, etc).
The experiments were conducted on a machine equipped with an Intel Core i5 CPU and 4 GB of RAM. All experiments were implemented and evaluated in Python by extending the Scikit-multiflow framework 8   [22]. Figure  3a  depicts the influence of the chunk size on the accuracy using UMAP-kNN with some datasets. Generally, fixing the chunk size imposes the following dilemma: choosing a small size so that we obtain an accurate reflection of the current data or choosing a large size that may increase the accuracy since more data are available. The ideal would be to use a batch with the maximum of instances to represent as possible the whole stream. In practice, the chunk size needs to be small enough to fit in the main memory otherwise the running time of the approach will increase. Since UMAP is relatively slow, we choose small chunk sizes to overcome this issue with UMAP-kNN.
Based on the obtained results, we fix the chunk size to 400 for the best trade-off accuracy-memory. We investigate the behavior of a crucial parameter that controls UMAP, number of neighbors, via the classification performance of our approach. Based on the size of the neighborhood, UMAP constructs the manifold and focuses on preserving local and global structures. Figure  3b  shows the accuracy when the number of neighbors is varied on diverse datasets. We notice that for all datasets, the accuracy is consistently the same with no large differences, e. g. Har. Since a large neighborhood leads to a slower learning process, in the following we fix the neighborhood size to 15. tSNE is a visualization technique, so we are limited to project high-dimensional data into 2 or 3 dimensions. In order to evaluate the performance of our proposal in a fair comparison against each of tSNE-kNN and PCA-kNN, we project data into 3-dimensional space.
We illustrate in Figure  4a  that UMAP-kNN makes significantly more accurate predictions beating consistently the best performing baselines (tSNE-kNN and PCA-kNN) notably with CNAE and the tweets datasets. Figure  4b  depicts the quantity of memory needed by the three algorithms which is practically the same for some datasets. Compared to kNN that uses the whole data without projection, we notice that UMAP-kNN consumes much less memory whilst sacrificing a bit in accuracy because we are removing many attributes. Figure  4c  shows that our approach is consistently faster than tSNE-kNN because tSNE computes the distances between every pair of instances to project. But PCA-kNN is a bit faster thanks to the simplicity of PCA. But with this trade-off our approach performs good on almost all datasets.
In addition to its good classification performance in comparison with competitors, the batch-incremental UMAP-kNN did a better job of preserving density by capturing both of global and local structures, as shown in Figure  1d. The fact that UMAP and kNN are both neighborhood-based methods arises as a key element in achieving a good accuracy. UMAP has not only the power of visualization but also the ability to reduce the dimensionality of data efficiently which makes it useful as pre-processing technique for machine learning. Table  2  reports the comparison of UMAP-kNN against state-of-the-art classifiers. We highlight that our approach performs better on almost all datasets. It achieves similar accuracies to UMAP-SkNN on several datasets but in terms of resources, the latter is slower because of its drift detection mechanism.
UMAP-kNN has a better performance than PCA-kNN, e. g. the Tweets datasets at the cost of being slower. We also observe the UMAP-HAT failed to overcome our approach (in terms of accuracy, memory, and time) due to the integration of a neighborhood-based technique (UMAP) to a tree structure (HAT). Figure  5  reports detailed results for Tweet 1 dataset with five output dimensions. Figure  5a  exhibits the accuracy of our approach which is consistently above competitors whilst ensuring stability for different manifolds. Figures  5b  and 5c  show that kNN-based classifiers use much less resources than the tree-based UMAP-HAT. We see that UMAP-kNN requires less time than UMAP-HAT and UMAP-SkNN to execute the stream but PCA-kNN is fastest thanks to its simplicity. Still, the gain in accuracy with UMAP-kNN is more significant.
Concluding Remarks and Future Work In this paper, we presented a novel batch-incremental approach for mining data streams using the kNN algorithm. UMAP-kNN combines the simplicity of kNN and the high performance of UMAP which is used as an internal pre-processing step to reduce the feature space of data streams. We showed that UMAP is capable of embedding efficiently data streams within a batch-incremental strategy in an extensive evaluation with well-known state-of-the-art algorithms using various datasets. We further demonstrated that the batch-incremental approach is just as effective as the offline approach in visualization and its accuracy outperforms reputed baselines while using reasonable resources. We would like to pursue our promising approach further to enhance its runtime performance by applying a fast dimension reduction before using of UMAP.
The K-means algorithm is a well-known and widely used clustering algorithm due to its simplicity and convergence properties. However, one of the drawbacks of the algorithm is its instability. This paper presents improvements to the K-means algorithm using a K-dimensional tree (Kd-tree) data structure. The proposed Kd-tree is utilized as a data structure to enhance the choice of initial centers of the clusters and to reduce the number of the nearest neighbor searches required by the algorithm. The developed framework also includes an efficient center insertion technique leading to an incremental operation that overcomes the instability problem of the K-means algorithm. The results of the proposed algorithm were compared with those obtained from the K-means algorithm, K-medoids, and K-means++ in an experiment using six different datasets.
Each node in the tree has two child nodes, while leaves are without child nodes. Data points, or references to them, are stored in leaves only. The Kd-tree uses splitting planes that are perpendicular to one of the coordinate system axes, chosen so that it goes through one of the points in the tree. There are several variants of the Kd-tree according to the stopping criterion or the applied splitting plan. The idea of using the Kd-tree for clustering was firstly introduced by Moore  [19]. It was used for estimating the parameters of a mixture of Gaussian clusters to reduce the cost of the EM-based clustering. To speed up the K-means algorithm and make it tractable for large datasets, the blacklisting algorithm  [20, 21]  was proposed. This algorithm updates data points in each cluster in bulk instead of updating each data point in each cluster separately.
The Proposed Algorithm The proposed algorithm depends on the operation of an inline construction-during the execution (not pre-prepared)-for an improved variant of the Kd-tree, which is an important feature when dealing with different datasets as the algorithm can adjust the number of levels in the Kd-tree according to the required number of clusters. In this variant, the region of each node is represented by a hyper-rectangle identified by two vectors "Hmax" and "Hmin". When creating a new node, the algorithm calculates a number of attributes for the node and uses them in the next steps to decrease the mathematical calculations required in each step. The new suggested structure includes a number of additional attributes compared to those used in the filtering and fast greedy algorithms to make the computing process faster.
Experiments The algorithm was applied to three artificial datasets: data1, data2 and data3. It was also applied to three real datasets: Iris  [28] , Crude Oil  [29] , and Vowel  [30]  The number of features, instances and clusters of each dataset is provided in Table  1. The algorithm starts by adding references to all data points in one node which is the root of the tree-Step 1. When creating any node in the tree, the algorithm calculates the density of the node "density", the number of items "number_of_items", the total sum of all items in each dimension "total_sum", and the upper and lower bounds of the node dimensions "Hmax" and "Hmin", respectively. Step 2: the algorithm finds the widest dimension of the whole node and splits the node into two leaves. If the widest dimension is "d", the algorithm then finds the two data points furthest apart in this dimension and calculates the mean of these two data points to be the splitting point of the node.
Experiments The algorithm was applied to three artificial datasets: data1, data2 and data3. It was also applied to three real datasets: Iris  [28] , Crude Oil  [29] , and Vowel  [30]  The number of features, instances and clusters of each dataset is provided in Table  1. The results of the proposed algorithm were compared to those of the K-means algorithm, K-medoid, and K-means++ (other variants of the K-means algorithm). The clustering criterion "E" (Equation (  2 )) was used to evaluate the performance of the algorithms. The smaller the value of this metric, the better the clustering results. All algorithms were executed many times (100 times). The average, minimum and maximum values of "E" were noted. E = k j=1 n j i=1 x ( j) i -c j 2 (2) where: x ( j) i -c j 2 is a chosen distance measure between a data point x i (j) in the cluster (j) and the cluster center c j.
n j is the number of data points in the cluster c j. and k is the number of clusters. The value of the distortion of the proposed algorithm was compared to that of the K-means, K-medoids, and K-means++ algorithms in Table  2. As the proposed algorithm is also intended to reduce the number of nearest neighbor searches, Table  3  lists the number of distance calculations between data points and cluster centers in each dataset for both the K-means and the proposed algorithms:  The least number of ngh queries is bolded for each dataset. As evident in the results of the experiments, the proposed clustering algorithm gives stable results in all cases and outperforms the K-means, K-medoid, and K-means++ in most cases. It yields a lower value of the distortion "E", which means better clustering solutions in most cases.
The complexity of the proposed algorithm is affected by the number of clusters. The higher the number of clusters, the greater is the time the algorithm takes. In addition, the quality of the solution is generally superior in most cases. Second, another major advantage of the proposed algorithm is that it can deal with large datasets (large numbers of data points) due to its ability to deal with data points in bulks. Third, compared to other Kd-tree-based clustering algorithms, the proposed Kd-tree structure has an inline building method. This reduces the complexity of the algorithm considerably by not running the splitting process and the K-means steps on all leaves of the pre-built tree (as in the blacklisting, filtering, and fast greedy algorithms); the proposed algorithm runs the steps on a limited number of leaves of the dynamic tree only.
In other words, in the blacklisting and filtering algorithms, a whole Kd-tree is built before running the main algorithm steps. This means a longer running time and higher complexity for the algorithm. Fourth, in the fast-greedy algorithm, despite the improvement of the tree-building process by using a predefined threshold to stop the Kd-tree splitting, it is still not a fully inline building method. This is due to that the tree must be built before running the main algorithm and it is difficult to specify the stopping threshold if there is no previous knowledge about the dataset. Fifth, another major finding is that the proposed algorithm reduces the complexity considerably. This is due to the fact that the proposed algorithm does not need to find a set of points that could act as an appropriate set for the insertion of a new center and then assigning leaves to centers, which is in contrast to the fast-greedy algorithm.
Conclusions In conclusion, this paper presented an algorithm to improve the performance of the K-means clustering algorithm by discussing a new Kd-tree-based clustering method. The proposed algorithm is used to overcome the instability and the large number of nearest neighbor searches of the K-means algorithm. The proposed algorithm was compared with the K-means algorithm, K-medoids, and K-means++ in an experiment using six different datasets. The results demonstrated that the proposed algorithm gives superior and more stable clustering solutions. Another major contribution presented is that the proposed Kd-tree structure improved the process needed to assign data points to clusters which causes the number of neighborhood searches to decrease. Future research could investigate the frameworks to reduce the number of parameters needed to be configured for the proposed algorithm considering automatic detection of the number of clusters.
It was suggested that ffCNNs mainly focus on local, texture-like features, while humans harness global shape computations (  [9, [13] [14] [15] [16] [17] ; but see  [18] ). In this context, it was shown that changing local features of an object, such as its texture or edges, leads ffCNNs to misclassify  [13, 14] , while humans can still easily classify the object based on its global shape. There are no widely accepted diagnostic tools to specifically characterize global computations in neural networks. Models are usually compared on computer vision benchmarks, such as ImageNet  [19] , or with neural responses in the visual system  [20, 21]. One drawback of these approaches is that the datasets are hard to control. Psychophysical results can be used to fill this gap and create well-controlled challenges for visual models, tailored to target specific aspects of vision  [22].
In this respect, we have shown that CapsNets generalize more similarly to humans than ffCNNs and standard recurrent networks in the context of global (un)crowding. Relatedly, CapsNets may also be more resilient to adversarial attacks  [57, 58] , generalize better to new viewpoints  [59] , and outperform ffCNNs on difficult visual tasks with scarce training data such as tumor detection  [60] , also suggesting potentially more human-like generalization than ffCNNs. One limitation in our experiments is that we explicitly taught the CapsNets which configurations to group together by selecting which groups of shapes were present during training (e. g. , only groups of identical shapes in experiment 1). Effectively, this gave the network adequate priors to produce uncrowding with the appropriate configurations (i.
e. , only identical, but not different flankers). Hence, our results show that, given adequate priors, CapsNets explain uncrowding. We have shown that ffCNNs and CNNs with lateral or top-down recurrent connections do not produce uncrowding, even when they are trained identically on groups of identical shapes and successfully learn on the training data, comparably to the CapsNets (furthermore, we showed previously that ffCNNs trained on large datasets, which are often used as general models of vision, do not show uncrowding either;  [17] ). This shows that merely training networks on groups of identical shapes is not sufficient to explain uncrowding. It is the recurrent segmentation in CapsNets that is crucial. Humans do not start from zero and therefore do not need to be trained in order to perform crowding tasks.
Methods The code to reproduce our results is available at https://github. com/adriendoerig/Capsulenetworks-as-recurrent-models-of-grouping-and-segmentation. All models were implemented in Python 3. 6, using the high-level estimator API of Tensorflow 1. 10. 0. Computations were run on a GPU (NVIDIA GeForce GTX 1070). We used the same basic network architecture in all experiments, motivated by the following rationale (Fig  6A ). After training, ideally, primary capsules detect the individual shapes present in the input image, and the secondary capsules group and segment these shapes through recurrent routing by agreement. The network can only group shapes together because it was taught during training that specific shapes form a group. To match this rationale, we set the primary capsules' receptive field sizes to roughly match the size of one shape.