dataesr/scientific-paragraphs-categorization huggingface dataset:
D. 3D helix models The design the 3D models of the helix was carried out using FreeCAD software, the free and open source version of AutoCAD. FreeCAD allows us to design 3D models of a helix and a set of helices with different packaging arrangements. Such models allow us to describe the packaging and the intersection of the helices with different planes. The model was designed in the Part workbench using the Helix tool. The radius of the spiral was set at 10 mm, the height at 150 mm and the pitch at 30 mm. To give volume to the helix, we used the sketch tool in the Design Part workbench. The work plane chosen was XZ plane and a circle was drawn. By using the sweep tool in the Part workbench, the volume was created using the circle as the profile, the helix as the trajectory and choosing the solid and frenet options.
Classification of these images is needed, as the different photographed organisms and particles can perform very different functions in the marine ecosystem. Phytoplankton organisms are autotrophic and use sunlight, nutrients and carbon dioxide to build up organic biomass, whereas other organisms called zooplankton are heterotrophic and mostly feed on phytoplankton. The abundances of phyto-and zooplankton are shaped by complex physical, biological and chemical processes. The plankton distribution is therefore highly dynamic, "patchy"  [11] [12] [13]  and varies on large spatial and temporal scales  [14] [15] [16]. As a result, the training sets required to train a classifier generally deviate from the distribution of newly observed samples (dataset shift or population drift)  [17]. This can introduce a bias into the classification, if, e.
g. , a training set from the Mediterranean sea would be used to predict the classifications for an Arctic dataset. Carrying this out can distort the abundance estimates, in some cases, to a severe extent  [18]. Therefore, machine classification is often followed by a manual validation of individual annotations to obtain high, human-level precision and a higher throughput. In the case of a high prevalence of novel classes or a strong dataset shift, the human workload is still considerable. Researchers, therefore, explored other ways to annotate marine data that are centered around efficiency  [19] [20] [21]. In 2020, we proposed MorphoCluster  [22]  as another solution for this problem. We will refer to the first implementation and evaluation of MorphoCluster as MC20. MorphoCluster is a data-driven approach to accelerate the sorting of marine particles and plankton.
It was designed especially for larger datasets with millions of objects, whereas it could be less effective for smaller datasets of only a few hundred objects. Several authors mention that MorphoCluster could be a promising approach to annotate large plankton datasets  [23] [24] [25]. Efficiency in the MorphoCluster approach is maximized using two orthogonal techniques: first, similar objects are grouped into homogeneous clusters and the user validates the objects of each cluster jointly. Second, the annotation expert adds additional visually similar objects to the previously validated clusters. The general work flow of MorphoCluster comprises representation learning, feature extraction, dimensionality reduction, clustering, the validation and growing of clusters and cluster naming, as outlined in Figure  1.
Representation Learning Feature Extraction Dimensionality Reduction Cluster Naming Figure  1. Sequence of operations in the MorphoCluster approach. The simulation is not part of the MorphoCluster application but allows us to examine different algorithmic configurations. In this work, we focus on the representation learning and the iterative annotation phase and omit the final naming step. We compare different approaches for representation learning, dimensionality reduction and clustering. In the initial representation learning  [26]  phase, a deep feature extractor is trained to project the complex morphology of an object depicted in an image into a metric vector space of limited dimensionality that facilitates the partitioning of the dataset into visually distinct and homogeneous clusters and allows the separation of different morphologies.
Finally, the identified clusters are hierarchically arranged using agglomerative clustering to group similar clusters, which can be manually merged and named. Overall, only a fraction of all images needs to be checked manually by the user, and the classification task is broken up into several more simple steps that only require the user to decide if a set of images is alike. Only in the final step is the more complex task to provide a classification for each object conducted; however, it is conducted on possibly thousands of images per decision. In this work, we compare different implementations of the representation learning phase and the iterative clustering and annotation phase of the MorphoCluster approach. The final classification step will be examined and optimized in future work. To enable our analysis, we implemented a simulation framework that allows us to compare different algorithmic choices without the need to annotate whole datasets multiple times.
Simulation of the MorphoCluster Process To enable the comparison of different algorithmic choices, we simulated the operations of a MorphoCluster user in silico. For that, we used the expert annotations provided with the dataset as gold standard, which are otherwise hidden from other parts of the process. This provided us with an approximation of the behavior of a real user. The simulation acts on previously calculated image representations and consists of three components: the clusterer implements the clustering step, and the validator and grower simulate the interaction of the user with the application. All three steps were closely modeled according to the real application used in MC20  [22]. Just as in the real application, data were passed back and forth between the components, alternating between clustering, growing and validation.
If the purity is between t gu and the pure batch threshold t gp ∈ (0, 1), t gu ≤ t gp , individual non-matching objects are removed and a linear search mode is entered. Usually, many pure batches can be added before the first impurities arise. The objects found to be similar are added to the cluster, and the others are returned to the pool of unclustered objects. As in MC20, we recalculated the cluster centroid after a number of objects were added and restarted the growing step. After validation and growing, a new iteration begins: the remaining unclustered objects are clustered and the new clusters are validated and grown. This was repeated until a maximum number of iterations or until a certain threshold of objects assigned to a cluster was met. In a real application scenario, the resulting clusters would then be arranged in a hierarchy and a user would assign meaningful names to provide a taxonomic annotation of the complete dataset.
Representation Learning In MC20  [22] , a deep learning feature extractor based on a ResNet18  [29]  classifier was trained on UVP5/EcoTaxa, which is a coarsely annotated subset of UVP5/MorphoCluster  [30]. We call such a supervised representation learning approach that is enabled through target domain labels non-blind. In this work, we additionally used ZooScanNet  [31]  as a source dataset. For model training, we used a learning rate of 1 × 10 -4 , a weight decay of 0. 003, cross-entropy loss, undersampling to up to 1000 objects per class and no sample reweighting. Details on how these values were chosen can be found in Appendix B. 1. However, since the primary purpose of MorphoCluster is the exploration and annotation of completely new data, it can also be necessary to train a feature extractor without prior label information from the same imaging modality.
This might be the case if a new imaging instrument is deployed and the recorded images are annotated for the first time. When the target dataset is sufficiently similar to a labeled source dataset, representation learning can be implemented through transfer learning  [32, 33]  by training a feature extractor supervised on labeled source data stemming from a similar imaging modality (e. g. , as suggested by Orenstein and Beijbom  [34] ). Here, we used feature extractors trained with the labeled data of the UVP5/EcoTaxa  [30]  and ZooScanNet  [31]  datasets, as well as a pretrained classifier model based on the ImageNet dataset  [35] , to calculate image representations for the remaining other plankton image datasets. If no sufficiently similar labeled data are available, unsupervised learning can be employed, where a feature extractor is trained on the target data alone with no further information apart from the images themselves  [36].
Dimensionality Reduction for Image Representations There are two processes that are time-demanding in the MorphoCluster process: the clustering and the distance calculations in the growing step. As the growing step is interactive, these calculations should be as fast as possible to avoid unnecessary waiting time for the user. The runtime of the clustering and the distance calculations strongly depend on the dimensionality of the image representations. Therefore, a small number of dimensions is essential. However, the representation still needs to encode enough information to enable accurate clustering and similarity search. Dimensionality reduction methods serve to reduce the dimensionality of image features while preserving their expressiveness. In this work, we compared different methods of dimensionality reduction to find the most suitable approach for MorphoCluster.
In MC20, a multi-stage trained projection layer was used to reduce the dimensionality of features to d = 32 as a trade-off between the amount of encoded information, the size in memory and the runtime of clustering and distance calculations  [22]. We studied three different types of dimensionality reduction in this work: • SINGLE-STAGE: The projection layer was trained together with the feature extractor in one pass; • MULTI-STAGE: First, the feature extractor model is trained as-is. Then, the projection layer is inserted and the complete model is fine-tuned; • PCA: The feature extractor is trained in the usual way and used to calculate features of full dimensionality for the images of the target dataset. The dimensionality of these features is then reduced using principal component analysis (PCA).
Clustering Algorithms for MorphoCluster MorphoCluster assumes very pure clusters in the validation step which requires a clustering algorithm that labels only the densest regions as clusters and leaves less certain objects unlabeled. Moreover, the clusterer needs to support large datasets, as typical plankton image datasets can contain millions of images. In MC20, the density-based clustering algorithm HDBSCAN*  [44]  was used  [22]. It satisfies these requirements, and the leaf cluster selection method produces fine-grained, homogeneous clusters, but the choice of its parameters is not straightforward. The prototypebased k-means, on the other hand, is a well known clustering algorithm with only a single intuitive parameter. It is used in many areas  [45]  and scales well with dataset size, can cluster even billions of samples and a wide range of implementations exist for different architectures  [46, 47].
However, like most clustering algorithms, k-means partitions the dataset, i. e. , every object is assigned to a cluster. This would render the separation of validation and growing, a core component of MorphoCluster, impossible. To exploit the advantages of k-means and still meet the requirements of MorphoCluster, we here propose two extensions that are tailored specifically for the MorphoCluster approach: shrunken k-means (S-k-means) and partially labeled k-means (PL-k-means).
Shrunken k-Means We devised shrunken k-means (S-k-means) as a a modified version of regular k-means that reduces the computed clusters to their very core, subsequent to the actual clustering step: First, regular k-means was used to partition the dataset into distinct clusters. Afterwards, the distance of every object to its respective cluster centroid was calculated. Finally, the previously calculated cluster labels were only retained for a small quantile p core of objects that are closest to their respective centroid in each calculated cluster. The remaining objects were un-assigned and returned to the pool of unclustered objects. For p core = 1. 0, shrunken k-means is identical regular k-means. The exact algorithm is given in Appendix C.
Partially Labeled k-Means Most clustering algorithms are meant to be applied once to the whole dataset. In contrast, MorphoCluster involves the repeated application of clustering interlaced with the validation and cleaning of the resulting clusters until all objects are treated  [22]. Therefore, a clustering run in a later iteration could potentially benefit from the previous annotations. Moreover, so-far unclustered objects that were missed in previous growing steps could still belong to a nearby existing cluster. We further extended the shrunken k-means clustering algorithm to incorporate the already validated clusters into the cluster formation. Here, they can guide the discovery of more meaningful clusters and allow for the late assignment of so-far unclustered objects. For that, we modified both the expectation and the maximization step of the k-means algorithm.
Nevertheless, high precision is the goal of every image annotation approach. Here, we define precision as the mean precision over all clusters measured by their object's gold standard label; • Final Number of Clusters: After a dataset has been partitioned into homogeneous clusters, these have to be given taxonomic labels. Many small clusters can lead to a high precision but are more tedious to label afterwards, negating the accelerating effect of the MorphoCluster approach. We therefore strive to minimize the final number of clusters (#Clusters) after partitioning.
Datasets We used four plankton image datasets to evaluate the different representation learning and clustering approaches. UVP5  [30]  (Figure  2a ) consists of images taken by the Underwater Vision Profiler 5 (UVP5)  [9]  in the pelagic zone of various oceans. Two labelings are available: The original labeling of the data, UVP5/EcoTaxa, contains 96k labeled images in 65 categories. UVP5/MorphoCluster is the result of the first application of MorphoCluster (MC20)  [22]  and contains 1. 2 M images in 280 categories. ZooScanNet  [31]  (Figure  2b ) consists of 1. 4 M images in 93 categories acquired with the ZooScan system [10], a customized flatbed scanner with a transparency unit for digitizing preserved wet net samples. WHOI-Plankton  [54]  comprises 330 k labeled images in 103 categories, taken in 2014 by Imaging FlowCytobot (IFCB)  [55] , a submersible imaging flow cytometer.
Kaggle (official title: PlanktonSet 1. 0)  [56]  (Figure  2c ) is part of the National Data Science Bowl competition hosted on Kaggle  [57]. It contains approximately 30 k images in 121 categories. The images were recorded using the In Situ Ichthyoplankton Imaging System (ISIIS)  [58] , a submersible shadowgraph-based imaging system that produces kilometer-long continuous images as its line scan camera moves through the water. ImageNet  [35]  is a widely used dataset to train deep learning models. Its training split consists of 1. 2 M images in 1000 categories. The images are color photographs of every-day objects  [59]  and therefore very dissimilar to the grayscale microscopy-type images in the plankton image datasets.
Results Within this section, we first compare the simulation performance to the manual experiment in MC20  [22]  (Section 3. 1). Next, results of the application of different feature extractors and methods of dimensionality reduction to multiple plankton datasets are shown in Section 3. 2. The three clustering algorithms under investigation are then compared with respect to efficiency, completeness, precision and the total number of clusters, using the simulation in Section 3. 3. Finally, the top performing representation learning and clustering approaches are combined in a final comparison (Section 3. 4). All experiments were organized using experitur [60].
Representation Learning To further evaluate the choices made in MC20 and to identify the best representation learning approach, we trained different feature extractors using the images and labels from UVP5 and ZooScanNet datasets and compared them using the NCC-F1 score (see Section 2. 4 for details). This includes supervised, transfer and unsupervised approaches. We also investigated different approaches to dimensionality reduction. In Table  1 , we provide the test NCC-F1 score for the three different methods of dimensionality reduction to d = 32 and compare them to a baseline without dimensionality reduction (FULL). One feature extractor was trained for every source dataset and the dimensionality reduction method and features are calculated and evaluated for four target plankton datasets: ZooScanNet, UVP5/EcoTaxa, WHOI and Kaggle.
Additionally, we include the results of the unsupervised training of the UVP5 and ZooScanNet models. Test NCC-F1 was calculated on the test split of the respective test dataset after dimensionality reduction. In the case of the UVP5/EcoTaxa dataset, all scores are comparatively low (below 40%), even when using the full 512 feature dimensions. This could be explained with the low sorting precision of this dataset, which is also noted in MC20  [22]. All reduced methods produce results that are nearly as good as full dimensionality. This suggests that even only 32 dimensions encode enough information to successfully classify an image.
Availability of Target Labels For any given target dataset, when trained on labeled target domain data (non-blind), the feature extractor consistently performs better than when labeled data are not available (blind). This is to be expected as the image labels contain valuable information about meaningful divisions of the representation space. Independent of the dimensionality of the features (FULL and PCA), unsupervised learning (UNSUP) on the target dataset performs slightly better than the respective best transfer learning approach (ImageNet feature extractor followed by PCA dimensionality reduction). However, we only observe a small performance gap in this experiment.
Dimensionality Reduction Methods When training data for the target instrument were not available (blind), PCA was the only unsupervised option that we explored. For transfer learning, SINGLE-STAGE and MULTI-STAGE, where the dimensionality reduction is trained using a source dataset with a different distribution, do not perform as well. On the other hand, when training data are available for the target dataset (non-blind), a trained projection layer has a slight advantage over PCA dimensionality reduction. For UVP5/EcoTaxa, SINGLE-STAGE performs slightly better, and for ZooScanNet, MULTI-STAGE has an advantage. However, both are in the same order of magnitude as PCA, and the small performance gains are offset by disproportionately more complex training.
Comparison of Clustering Algorithms In the following section, we compare Python implementations of HDBSCAN*  [61]  and the proposed shrunken k-means and partially labeled k-means clustering algorithms using the simulation framework, the UVP5/MorphoCluster and ZooScaNet datasets and the original features from MC20  [22]. We set k = 8 and m 0 = 128 for HDBSCAN* (deviating from MC20); k = 1000 and p core = 0. 01 for shrunken k-means; and p noise = 0. 1, k ∆ = 100, k 0 = 500 and p core = 0. 01 for PL-k-means. These choices are motivated in Appendix E. In Table  2 , we compare the performances of HDBSCAN*, S-k-means and PL-k-means using the UVP5/MC and ZooScanNet datasets with the original HDBSCAN* configuration with k = 1  [22]. While the original HDBSCAN* configuration yields the highest completeness, it also entails the lowest sorting efficiency and a high number of resulting clusters, which made the final naming step in MC20 very time-demanding  [22].
Comparison of Feature Extractors for Clustering In the previous experiments, we compared different feature extractor techniques using the NCC-F1 score and also assessed the performance of the three proposed clustering techniques using the MorphoCluster simulation. Now, we assess the performance of the feature extractors using the best performing clusterer (HDBSCAN*) in the MorphoCluster simulation. For the UVP5/MorphoCluster and ZooScanNet target datasets, we used the respective blind and non-blind representation learning techniques. A fully supervised feature extractor, where the training dataset is identical to the target dataset, is included to estimate an upper performance bound. Image representations were calculated for the respective target dataset and their dimensionality was reduced to d = 32 with PCA.
The respective representations were then used in the simulation of the MorphoCluster process with the previously selected configuration of HDBSCAN*. The results are summarized in Table  3. All three representation learning approaches generate an acceptable number of clusters that enables quick cluster naming in the final step. Independent of the representation learning approach and dataset, the precision is well over 95%. For both target datasets, supervised training, where labeled training data for the target domain are available, leads to the highest completeness and efficiency. This was to be expected because the feature extractor is taught the exact information that is required in the evaluation, i. e. , the correspondence of certain morphologies and labels. When the source and target dataset are the same (fully supervised), the obtained scores are an upper limit for the real performance.
Representation Learning As an explorative image annotation method, MorphoCluster is usually applied when labels for the target dataset are insufficient or not available. In this work, we analyzed different methods of representation learning when labeled data from the target domain are available (non-blind) or not available (blind). Early approaches to image annotation rely solely on handcrafted image features  [10, 66]  that require no training. More recently, classifier-based  [34, 67]  or contrastive  [62]  approaches were successfully used for the supervised deep representation learning in the field of biological oceanography. The classifier-based approach is often easier to apply in practice, as contrastive approaches tend to require larger computational resources  [39, 68, 69]  and elaborate sample mining strategies  [70].
Supervised and unsupervised representation learning approaches entail a deep learning training phase. In the case of scarce computational resources, we find that transfer learning using a generic ImageNet-trained feature-extractor is a good solution as it outperforms the plankton datasets as a source in this case. One explanation for this comparatively good performance could be that the number of categories (  1000 ) is much larger compared to the plankton datasets (65 to 121 categories) and that ImageNet is more curated and therefore contains few ambiguous images. Since the projection layer for SINGLE-STAGE and MULTI-STAGE dimensionality reduction is trained, it is conditioned on the training dataset and therefore cannot adapt to the dataset shift. Meanwhile, PCA equally retains the expressiveness of image representations, but is conditioned on the target dataset and is therefore more flexible and robust to distribution shifts, which regularly occur, even without changing the instrument.