dataesr/scientific-paragraphs-categorization huggingface dataset:
We use AdamW  [43]  as the optimizer for the pre-training step and for the two optimizers of the joint training step, and find that 30 epochs are enough to converge for both steps on any of the tabular datasets used. We implemented our method under Python 3. 7. 9 and with the PyTorch 1. 10. 0 [44]  library. The experiments were conducted on Nvidia 2080 Ti and Nvidia Quadro T1000 GPUs. The networks are initialized with random weights, and following  [6] , the results are averaged over 10 runs. The code is publicly available at https://github. com/ColinTr/TabularNCD.
B. Results Comparison to the competing methods. In Table  II , we report the performance of the 4 competing methods and on the 7 datasets for the clustering task. The results show that TabularNCD achieves higher performance than the baseline and the two unsupervised clustering methods, on all considered datasets and on all metrics. The improvements in accuracy over competing methods ranges between 1. 6% and 21. 0%. This proves that even for tabular data, useful knowledge can be extracted from already discovered classes to guide the novel class discovery process. While the baseline competitor (k-means on a projection learned on the labeled data, see Sec. IV-A) improves the performance of k-means for some datasets (notably the Census and Forest datasets), it still lags behind our method in general.
The baseline obtains a lower score than the simple k-means on only 3 datasets, which means that in some cases, the features extracted from the known classes are insufficient to discriminate novel unseen data. This is the case for the Satimage dataset, where the performance of the baseline is much lower compared to the simple k-means on the original data. To understand this phenomenon, we compare the importance of the features used to predict the known classes to the importance of the features of the unknown classes. Because the features have been standardized, we can simply look at the coefficients of a logistic regression. Fig. 5  illustrates this experiment: the 5 most important coefficients in the prediction of one of the unknown classes are more than twice as large as the addition of the coefficients of the attributes used to predict the known classes.
We also observe an important decrease in performance when removing the CE  (5) , meaning that in the case of the Satimage dataset, jointly learning a classification network with the clustering network helps guide the clustering process as intended. Finally, while it is beneficial to pre-train the encoder with SSL, the impact is small. This can be explained in part by the limited success that SSL works have had in the challenging area of tabular data. Visualization. In addition to quantitative results, we also report a qualitative analysis showing the feature space that is learned. In Fig. 6 , we visualize the evolution of the representation of all the classes during the training process of the proposed method. Fig  6(a ) corresponds to the original data, while the next figures are the data projected in the latent space.
Classes overlap in the original representation, as well as in the latent space after Self-Supervised Learning (Fig  6(b) ). However, the SSL step initialized the encoder to a reasonable representation. This means that at the beginning of the joint training, the pseudo labels defined on the unlabeled data using (6) will be accurate. During the joint training in the Figures  6(c ) and 6(d), the classes are more and more separated, showing that the proposed method is able to successfully discover novel classes using knowledge from known classes. The plot clearly shows that our model produces feature representations where samples of the same class are tightly grouped.
V. CONCLUSIONS AND FUTURE WORK In this paper, we have proposed a first solution to the Novel Class Discovery problem in the challenging environment of tabular data. We demonstrated the effectiveness of our proposed approach, TabularNCD, through extensive experiments and careful analysis on 7 public datasets against unsupervised clustering methods. The greater performance of our method has shown that it is possible to extract knowledge from already discovered classes to guide the discovery process of novel classes, which demonstrates that NCD is not only applicable to images but also on tabular data. Lastly, the original method of defining pseudo labels proposed here has proven to be reliable even in the presence of unbalanced classes. The recent advances of deep learning in tabular data are worth investigating in future work.
HYPER-PARAMETERS INFLUENCE. The method proposed here has a few hyper-parameters that must be tuned to get optimal performance. In  Fig 7,  we study some of the most important hyper-parameters. The first is the top k from (6), it represents the percentage of pairs that will be regarded as positives for each instance. It varies in the range  [1, 100]  in Fig. 7 (a), with ideal values between 6 and 16% for the two datasets considered. The performance is shaped in "steps", decreasing as top k increases. This shows that after certain thresholds the model merges some of the novel classes together, with values superior to 70% causing the model to predict a single class for all observations. Then, we vary the k neighbors in  [1, 90]. This parameter controls the number of closest neighbors considered in the data augmentation (see Sec.
III-C). Fig. 7 (b) shows under 70, all values produce good results. The Human datasets appears to have high variance, but this is simply a result of the very small range of the performance (between 0. 970 and 0. 995). Finally, we study w1 and w2 in Figs. 7(c) and 7(d) by varying them in [0, 1]. They control the trade-off between the consistency loss and loss of the classification or clustering networks (see Sec. III-D). Because the classification task is secondary to the clustering problem, w1 has less influence on performance, and good values range between 0. 3 and 0. 9 for both datasets. On the other hand, w2 is very important for the accuracy of the method, and good values range between 0. 8 and 0. 9 for the Census dataset and between 0. 4 and 0. 9 for the Human dataset.
INFLUENCE OF DATA REPRESENTATION FOR THE K-MEANS In table V, the quality of different data representations of MNIST are compared using k-means. We observe that after Self-Supervised Learning, the encoder learned a representation of the original data where the k-means algorithm performs slightly better than on the original data (+2. 8% in accuracy). And after joint training, the unknown classes are even more cleanly separated, as k-means gains +24. 3% in accuracy over the original data representation. "cosine topk" is here a percentage of the max number of pairs of instances in a batch.
k-means algorithm on different data representation of the MNIST dataset. (1) is the original data, (2) is the data projected by an encoder trained with Self-Supervised Learning (Sec. III-A), and (3) is the data projected by an encoder jointly trained by the proposed method (Sec. III-B) HYPER-PARAMETERS VALUES We optimize the most important hyper-parameters for each dataset and report their values in table VI. The constants are: batch size = 512, encoder layers sizes = [d, d, d], α = 2. 0, ssl lr = 0. 001, epochs = 30 and p m = 0. 30.
[38]-Supervised Learning, CE: Cross Entropy loss of the classification network, BCE: Binary Cross Entropy loss of the clustering network, MSE:Mean Squared Error consistency loss. The dataset is Satimage[38]. TABLE IV ABLATION STUDY OF THE PROPOSED METHOD. Method BACC (%) ACC (%) NMI ARI TabularNCD 90. 8±4. 0 91. 4±5. 0 0. 71±0. 11 0. 79±0. 07 -w/o SSL 88. 4±5. 3 88. 6±7. 0 0. 67±0. 15 0. 67±0. 10 -w/o CE 72. 0±6. 1 69. 5±6. 0 0. 44±0. 12 0. 49±0. 08 -w/o BCE 33. 3±0. 0 51. 7±0. 0 0. 00±0. 00 0. 00±0. 00 -w/o MSE 66. 7±5. 7 63. 9±4. 4 0. 44±0. 02 0. 37±0. 02 SSL:.
2020 ). The newly developed InvaCost database  (Diagne et al. 2020 ) is the rst comprehensive and standardized compilation of the economic costs associated with biological invasions worldwide. This database provides unique opportunities to thoroughly assess and understand the economic impacts of invasions. Here, we used and enriched this database with additional data to present a detailed and up-to-date, global assessment of the economic costs of invasive ants. Our analyses aimed to: (i) describe the ant species associated with the reported economic costs; (ii) describe the spatial and temporal distribution of reported costs; (iii) highlight the type of costs reported; (iv) decipher the economic sectors impacted by these costs; and (v) identify the potential geographic gaps in the cost reporting.
Data collection We used the latest version of the InvaCost database (InvaCost_3. 0; 9,823 entries; Diagne et al. 2020, https://doi. org/10. 6084/m9. gshare. 12668570 ), consisting of cost data extracted from documents obtained through standardized literature searches (i. e. , using ISI Web of Science platform, Google Scholar and the Google search engine) and opportunistic targeted searches (i. e. , expert consultations for which data gaps were identi ed, such as in 10 languages other than English,  Angulo et al. 2021 ). Costs extracted from these sources were converted from local currencies to US$ by correcting the value with the o cial market exchange rate corresponding to the year of the value and then adjusting to 2017 US$ using in ation factors  (Diagne et al. 2020). We extracted data for invasive ant species (selecting for the family Formicidae).
Each database entry contains a cost value associated with a unique combination of cost descriptors (see "Data structure" section). We complemented the InvaCost data by adding costs found from four different targeted searches: (i) in non-English languages, speci cally focusing on the economic costs of the 19 invasive ants recognized by the IUCN ( http://www. issg. org/database ); using the same search strings as those considered in the standardized searches led in the Web of Science platform by  Diagne et al. (2020) , but with economic terms translated in different languages (Arabic, Chinese, French, German, Greek, Italian, Japanese, Portuguese, Russian, Spanish, and Ukrainian) alongside the scienti c names of the 19 ant species; (ii) in the digital database SciELO (Scienti c Electronic Library Online) ( https://www.
scielo. br ), which provides access to scienti c literature mainly originating from the South American continent, with journal articles usually published in Spanish and Portuguese; (iii) in the bibliographic database FORMIS ( http://www. ars. usda. gov/saa/cmave/ifahi/formis ), a composition of several ant literature databases, that contains citations for a large proportion of the world's ant literature. More speci cally, using EndNote X9, we searched all elds in the database version FORMIS 2018 for each invasive ant species, with the following search items "econom*", "monetary", "dollar", "$", "€", "sterling pound"; (iv) contacting key people (mainly managers and researchers) in relation with invasive ant management programs that we knew of but nancial data was either not available or was incomplete.
Data structure The Raw data contained over 60 descriptive cost variables divided into the following groups (Online Resource 1, tab "Descriptors"): (i) the bibliographic information of the documents where the costs were reported, (ii) the area impacted or where the costs were incurred (e. g. , spatial scale, location), (iii) the taxonomy of the focal species, (iv) the temporal extent over which the costs either occurred or were predicted to occur, (v) the typology of each cost reported, and (vi) the economic cost values. To describe the economic costs of invasive ants we used information mainly from the following four cost descriptors: the type of costs, the type of management, the economic sector impacted by invasive ants, and the nature of the implementation of the cost value. The type of costs (column type_of_cost_merged) assigned costs to either "damage" costs (most often corresponding to marketed costs, e.
Description of the economic costs of invasive ants To compare the number of cost entries for each invasive ant species among descriptors, we homogenized all costs recorded on an annual basis using the expandYearlyCosts function of the 'invacost' package version 0. 3-4  (Leroy et al. 2020 ) in R version 3. 6. 3 (R Core Team 2020). This function relies on the duration time of each cost entry provided as the number of years between the cost entry's starting and ending years given in the database. Hence, we obtained comparable annual costs for all cost entries. The expanded dataset resulted in 1,621 entries, from which 329 were not considered for reasons explained in the data processing section (also see, Online Resource 1, Tab "Expanded_data"). Thus, for our analyses, we considered a total of 1,292 expanded cost entries.
We also calculated the temporal trends of the economic impacts of invasive ant species using the function calculateRawAvgCosts from the invacost package version 0. 3-4 (Leroy et al. 2020) in R version 3. 6. 3 (R Core Team 2020). This function calculates average annual costs for the whole study period, providing 10-year average costs based on the annualized cost entries calculated. Other descriptions of the economic costs of invasive ants were formed by using the categorical descriptors in the InvaCost database (see section on Data structure), for instance taking into account the nature of the costs (observed vs potential costs) we describe: (i) the magnitude of costs for each invasive ant and the trend of observed costs entries per invasive ant along time; (ii) the geographic distribution of costs by country, splitting the cost in each country by each invasive ant species; (iii) the percentage of each type of cost (damage and the types of management) and we further split the type of cost with the sectors impacted; (iv) and nally, the percentage of each type of management costs (i.
Geographic coverage of the economic costs reported for invasive ants We mapped and compared the geographic locations of the costs reported in the raw database (excluding cost entries marked as "remove") with the invasive range of the ant species with reported costs. In order to obtain geographic coordinates of the costs, we used the column "Location" in the dataset, and the original documents were cross-checked to con rm location; in some cases, more than one location was reported for the same cost entry. Thus, the number of recorded geographic coordinates of the costs reported summed more than the number of cost entries. We then obtained occurrence records of the current invasive distribution of each ant species from both the GBIF (Global Biodiversity Information Facility, https://www. gbif.
50% of Acromyrmex observed costs spent in research for the optimization of the control strategies in Guadaloupe island). The observed pre-invasion management costs for multiple invasive ants ("Diverse/Unspeci ed" category) was for biosecurity and the development of educational programs with focus on invasive ants in New Caledonia (France) (Fig. 4 ). Taxonomic and geographic coverage in the economic costs of invasive ant species Although our dataset contained costs for 12 of the 19 invasive ants reported by the IUCN (and none for the other species reported as invasive by other studies), most (99. 83%) of the reported costs were only for two species, S. invicta and W. auropunctata. Yet, even for these two species, many costs are likely missing. For the other 10 species, lower costs were reported.
Notably, an analysis at the species-level found that only 19. 76% of the locations per country where S. invicta occurs (using occurrences in GBIF and AntWeb) have reported costs (Table  1 ). This outcome was predominantly driven by the USA, where there were a high number of ant occurrences but few reports. Geographic coverage of cost reporting in some other countries was signi cantly higher, such as in Australia (52. 65%, Fig. 5a ). It is worth noting that from 18 out of the 20 countries which reported costs on S. invicta did not have this species present, since costs were potential there, while other countries reporting observed costs such as New Zealand or Japan had no ant occurrences in the global databases (Table  1 ; Fig. 5a ; Online Resource 4). Similar geographic coverage per country was found for L.
As an example, research grants for scientists studying invasive alien ants are typically not recorded as economic costs and therefore largely absent from the InvaCost database. When asking colleagues worldwide about their research grants on invasive alien ants throughout the last 30 years, we came up with 45 responses providing an estimated US$ 27,000 average per research article (Online Resource 6). If one considers about 4,742 research articles during this period on this topic (with the same keyword search in WoS as described in the Methods, except for the economic components), this suggests that this research grants component alone could be in the order of US$ 127 million (Online Resource 6). This crude estimation does not account for the true cost of a research project (typically a fraction of the money received by researchers), nor the researchers salary (often not included in grants), both of which could signi cantly increase this estimated amount.
(Espadaler et al. 2007;Ugelvig et al. 2008 species stated, yet most of the costs (76. 93% of the cost entries and 99. 83% of the economic amount) are for S. invicta and W. auropunctata, and costs for the rest of highly invasive ants are lacking. Lower or nonexistent costs for other invasive ant species could be due to them being less destructive, or to signi cant underreporting. Most certainly, a lot of economic costs are neglected, especially of those invasive ant species that are not yet referenced as invasive in the global lists of invasive species, such as Tetramorium tsushimae(Steiner et al. 2006), Cardiocondyla obscurior (Heinze et al. 2006), Plagiolepis alluaudi (Wetterer 2014), Formica paralugubris (Frizzi et al. 2018) among others. Although only 19 invasive ant species are referenced in the IUCN database, Lach et al.
Automatic characterization of fluorescent labeling in intact mammalian tissues remains a challenge due to the lack of quantifying techniques capable of segregating densely packed nuclei and intricate tissue patterns. Here, we describe a powerful deep learning-based approach that couples remarkably precise nuclear segmentation with quantitation of fluorescent labeling intensity within segmented nuclei, and then apply it to the analysis of cell cycle dependent protein concentration in mouse tissues using 2D fluorescent still images. First, several existing deep learning-based methods were evaluated to accurately segment nuclei using different imaging modalities with a small training dataset. Next, we developed a deep learning-based approach to identify and measure fluorescent labels within segmented nuclei, and created an ImageJ plugin to allow for efficient manual correction of nuclear segmentation and label identification.
However, this machine learning-based approach requires large amounts of annotated data and new strategies have to be developed to process highly complex biological objects acquired with different modalities by considering small training datasets. In this paper, we propose a series of deep learningbased approaches to precisely segment nuclei and to identify fluorescently labelled cells in order to analyze the evolution of cell cycle dependent E2F protein concentration in mouse tissues. E2Fs are major regulators of the cell cycle. The members of this family of transcription factors are categorized into three subclasses in mammals: canonical activators (E2F1-3A/B), canonical repressors (E2F4-6), and atypical repressors (E2F7-8)  [10] [11] [12] [13]. Adding to the body of literature on E2F-dependent transcriptional activity in vivo, our lab previously provided quantitative evidence on the temporal expression of representative activator (E2F3A), canonical repressor (E2F4) and atypical repressor (E2F8) family members during embryonic development  [14] , all of which have been shown to be of major importance  [15] [16] [17] [18].
Using a small training dataset composed of 2D still images (with and without various forms of data augmentation), we first evaluate five different deep learning strategies to segment nuclei in microscopic images of embryonic mouse intestinal epithelium. We also design post-processing methods to improve nuclear segmentation. We then propose another deep learning-based approach for identifying nuclear markers in the epithelial cells and demonstrate the superiority of this method to the usual threshold based method  [20, 21]. Additionally, we create an ImageJ plugin  [22, 23]  named Annotater to specifically and efficiently correct nuclear segmentation and marker identification, ensuring that nuclear features are accurately quantified. Next, these image features extracted from 2D still images are used to perform a temporal analysis of E2F protein concentration over the cell cycle.
Based on three mathematical assumptions grounded in cell biology, we initialize the temporal evolution of E2F concentrations using a graph optimization method. Cell cycle markers are then used to temporally register E2F proteins' concentration with respect to cell cycle phase. The global estimation of the protein concentration of E2F3A, E2F4 and E2F8 through the cell cycle is defined as an assignment problem and solved with the Hungarian algorithm  [24, 25]. This approach is extensively evaluated with simulated data. Finally, we directly estimate the temporal evolution of E2F concentrations without using marker identification. In addition, we evaluate the impact of using different amount of images in the training datasets for nuclei segmentation on the estimation of E2F concentrations over the cell cycle.
Stardist shows a slightly better F1 score for an IoU threshold equal to 0. 5 for confocal images than Mask R-CNN (0. 864 vs 0. 858), suggesting that a few more nuclei are identified with Stardist. However, the F1 scores for higher thresholds of IoU obtained with Mask R-CNN are clearly higher than those obtained with Stardist (IoU = 0. 75, 0. 458 vs 0. 434 for confocal images and 0. 460 vs 0. 389 for widefield images). This demonstrates that Mask R-CNN better localizes the nuclei areas than Stardist. Of note, while pooling together confocal and widefield images leads to a better accuracy for all three methods, data augmentation does not improve performance for Stardist and Cellpose while it clearly ameliorates Mask R-CNN precision. This could suggest that Stardist and Cellpose are more sensitive to noise when the training dataset is small and the nuclei are densely packed.
In the following, only Mask R-CNN is compared to U-Net and Inception-V3. In contrast, the Inception-V3 approach obtains better results when both modalities are used in the training dataset (S3 Fig). While for Inception-V3 the training dataset was not increased when applying data augmentation because the computation time is already long, the high degree of overlap between input patches to which mathematical operations are applied (see Materials and methods) produces an effect similar to data augmentation, explaining in part the improved performance of the Inception-V3 approach relative to U-Net. Because Inception-V3 estimates the class one pixel at a time, both training and processing are highly compute intensive and not feasible in practice (S3d Fig). The proposed corrected watershed post-processing improves the results obtained for both U-Net and Inception-V3 (S2e, S2f, S3e and S3f Fig and  S5  and S6 Tables).
The Mask R-CNN method obtains the best nuclear segmentation for both modalities (Fig 1 ), in large part due to the improvement realized through data augmentation and in a lesser extent to the transfer learning from the coco dataset  [33] , (Fig  1a  and 1b ). While the performance of Mask R-CNN is impressive, its accuracy is not perfect and may be insufficient for many applications. Therefore, we designed the ImageJ plugin Annotater (see Materials and methods), a tool that allows users to efficiently correct the nuclear segmentation.