dataesr/scientific-paragraphs-categorization huggingface dataset:
Team Arontier_HYY The team Arontier_HYY employed a three-stage method to predict HER2 status (see Figure  6 ). In the first stage, an EfficientNet-B3 (denoted as CNN-A) was trained from scratch to classify image patches of 1024 by 1024 pixels as "tissue" or "background". The training dataset for CNN-A consisted of image patches labeled as "background" if they include regions with stains or adipose tissue, as well as if they contain severely blurred regions and patches with less than 25% of tissue (pixel intensities bellow 240), or as "tissue" otherwise. After discarding images classified by CNN-A as "background", in the second stage, a model consisting of two parallel EfficientNets (denoted as CNN-B) was used to extract a feature vector and a score representing the probability of a given tissue patch being from a HER2 positive WSI.
In one of the parallel paths, an EfficientNet-B1 was trained on image patches of 480 by 480 pixels while in the other an EfficientNet-B5 was trained on patches of size 912 by 912 pixels. Tissue patches were labeled with 0 whenever they originate from a HER2 negative WSI and with 1 otherwise. Data augmentation was then implemented to generate a more robust model and the resulting dataset was used to train CNN-B. To minimize class imbalance and inter-WSI variations, within each minibatch, all patch images originate from different WSIs and 50% were from randomly selected positive WSI while the remaining were from randomly selected negative WSI. In stage three, all feature vectors of any given WSI were sorted in decreasing order by the corresponding score of CNN-B and fed into a Long-Short Term Memory (LSTM) network with two recurrent layers and one dropout layer to assess the final WSI prediction.
Results The HEROHE Challenge was open for approximately 4 months, between October 1 st , 2019, and January 28 th , 2020. During this period, 863 participants registered and had access to the training and testing datasets. While many participants chose to work individually, 93 of them grouped into one of the 56 participant teams. From these, 21 teams or individual participants submitted their results and the final scores were evaluated to assign the final challenge's ranking (see Table  1 ). Two teams developed methods that, by construction, did not quantify the probability of a WSI belonging to one class (i. e. , did not infer a soft prediction). The model developed by the team Macaroon classified a WSI after comparing the number of patches classified as tumor by one network and as positive by the second network.
Thus, an approximation was needed to compute the AUC. The soft predictions were set to 1 whenever a WSI was classified as positive and to 0 otherwise. The model developed by team MITEL used a majority voting to assign a class to a WSI. Considering that the proposed method exported three soft predictions, the developers chose to consider as a representative soft prediction the one resulting from the overall positivity because it covers all the WSI patches, thus being considered the most exhaustive among the three. The choice of the threshold and its impact on the  1 score was assessed for the top-ranked teams by varying the threshold from 0 to 1 by steps of 0. 01 and computing the best  1 score on the test dataset, thus assessing the theoretical maximum of each model for the given test dataset.
84. Considering the distribution of cases among the four possible HER2 scores on an IHC test, the precision, recall, AUC and  1 score were also evaluated in the subset of equivocal IHC cases (score of 2+) and are presented in Table  2. Six teams achieved on this subset AUC equal or greater than 0. 84, thus higher than the AUC achieved by the models presented in recent works  [27, 29, 30]. Although the core methodologies differ between submitted models, some procedures are common among approaches. For example, 20 out of the 21 teams developed methods taking advantage of deep neural networks in one or more steps of their models and used Python as the main programming language. Reza Mohebbian was the only team that developed a "classical" Machine Learning model (without deep neural networks) coded on MATLAB.
Teams HEROH, HEROHE_Challenge, Institute of Pathology Graz, and katherandco, despite having used Python as the main programing language, used also QuPath  [51]  on some steps of their methods. Another common step was the split of each WSI into smaller patches. Only three teams used the entire WSI as input, Reza Mohebbian, that developed a non-deep learning-based model; Institute of Pathology Graz, that combined hand-crafted feature extractor developed on QuPath with a custom CNN for classification; and aetherAI, that adapted a ResNet50  [28]  to receive as input WSIs resized to a 10. 000 by 10. 000 pixels canvas (see 2. 3. Competing Solutions, and Supplementary Material for more details). Among, the 20 teams that used deep neural networks on their methods, 12 chose to rely on models pre-trained on other publicly available datasets (more details in Table  3 ), being the ImageNet  [47]  the most widespread, but other datasets were used  [34, 35, 52, 53].
Discussion As in all challenges, the definition of the metric to access the final ranking is of paramount importance. All metrics that may have been considered to evaluate the methods assess different aspects of the results so they would produce different ranks. Since our dataset was imbalanced, metrics such as accuracy (the percentage of total cases correctly classified) that is often used to evaluate classifiers is not a reliable descriptor of the model's adequacy to solve the problem at hand. For example, a model that predicts all cases as negative achieves an accuracy equal to the proportion of the negative cases, which, for highly imbalanced datasets, would result in higher classifications. Other metrics are less prone to the class imbalance problem. Among them, precision and recall are good choices nevertheless both these metrics fail in some extreme scenarios.
For example, the team MIRL, ranked tenth on the challenge (Table  1 ), achieved a perfect recall. Looking in more detail for the team MIRL's predictions (Table  4 ) one fact can be highlighted, team MIRL predicted all cases as positives. This, while guaranteeing that no false negatives would result from the predictions, and thus the recall would be 1, leads to a model that, despite having one perfect metric, is solve the problem here proposed. Cases as this one can be by comparing the model's recall with its precision. Team MIRL achieved of 0. 4, a value that corresponds to the proportion of positive cases on the test dataset (Table  1 ). Analogously, if a model only predicts as positive a very small percentage of the true positives, but none of the negatives, it will have a precision of 1, while failing to properly predict most of the relevant cases, thus having a small recall.
8 in Table  1  or in Table  2 , i. e. AUC bigger than those reported on the referred studies. The remaining methods are described in the Supplementary Material and a summary of all models is presented in Table  3. An important aspect that needs to be considered by all teams willing to participate in challenges is the decision on the threshold value to decide the final prediction. If the teams had chosen different thresholds to generate their final predictions, their results in the  1 score could have been better. For instance, with this test dataset, the model developed by the team Piaz could have achieved a  1 score of 0. 73 (instead of the actual 0. 64) if the threshold was set to 0. 39 (instead of the used 0. 5), thus being ranked at the top of the leaderboard. Of course, the value 0. 39 was obtained as the best choice for the test dataset, which, by definition, was not available to teams.
Nevertheless, the training dataset or part of it (e. g. , validation dataset) should be used to fine-tune the threshold. It is worth noting that, even if the optimum threshold was not the one resulting by the evaluation of the model performance on the training dataset, close to 0. 39 (thresholds between 0. 36 and 0. 43) would also generate a final  1 score greater than 0. 70. The comparison between the result that team MITEL reached ( 1 = 0. 67) and the one that they would have had if they used the overall connectivity as the only evaluation feature ( 1 = 0. 70 for a threshold of 0. 37) reveals that sometimes a more complex model does not outperform a simpler one. The results obtained by the teams in the subset of equivocal cases (Table  2 ) revealed that most of the top-ranked teams (8 of the top 10, including the top 7 teams) achieved better results on this subset than in the whole test dataset (Table  1 ).
For example, team Macaroon achieved a  1 score of 0. 68 on the whole test dataset, but this figure raised to 0. 79 when considering just the equivocal cases. This difference is due to the increase in precision that raises from 0. 57 to 0. 75 while the recall increases from 0. 83 to 0. 84. Similar changes occurred with other top-ranked teams (e. g. Dratur's  1 score raised from 0. 63 to 0. 73, Arontier_HYY's  1 score raised from 0. 61 to 0. 73). Equivalently, the AUC increased in most of the teams. While on the full test dataset only one team achieved an AUC greater than 0. 8 (team Piaz with  = 0. 84), on the subset of the equivocal cases 6 teams had AUC above 0. 8. The difference between the results achieved on these two groups of cases can be explained by the distribution of cases per HER2 score on both the training and test datasets with a majority being classified as equivocal on the IHC test.
Following a common trend in medical image analysis, the vast majority of the teams (20 out of 21) used at least one deep neural network on their models. Given the complexity of the task, many teams split the problem into more than one step and rely on the combination of more than one deep network to classify each WSI (Table  3 ). Since the training dataset was comprised of 359 cases, 12 teams chose to use models pre-trained on other datasets. This allowed them to train deeper networks, with up to millions of parameters, and potentially improve performance while reducing development and training time. Another common technique was splitting the entire WSI into small tiles (18 out of 21 teams), applying the HER2 classifier to each tile, and later combining the information to get the WSI classification.
This type of research and prediction can take computational pathology to a level never experienced in medicine. In the HEROHE Challenge, it was decided to consider datasets with cases without HER2 heterogeneity since it is the most frequent situation. Additionally, heterogeneous cases would require specific tumor annotations. HER2 heterogeneity corresponds to tumors with both HER2-negative and HER2-positive areas, representing a minority of situations (up to 1% of the cases) with patients requiring at least 10% of HER2-positive areas in the BC to be elected for targeted therapy  [4, [63] [64] [65]. Although there are morphological features more likely to be associated with HER2-positive BC (as discussed above), making the distinction between HER2-positive and HER2-negative theoretically possible, in cases with HER2 heterogeneity the different areas in the BC appear to be very similar, at least to the pathologist´s assessment  [64].
The proportion of cases in each IHC score was biased towards the class 2+. This decision was because these are the cases that require further assessment, namely by the evaluation of HER2 amplification with ISH, and thus considered by the challenge organizers as the most important cases. Future research to address this problem should consider these aspects. Roughly, 15% of the cases evaluated by IPATIMUP diagnostics are positives, nevertheless, the datasets here released failed to follow the real ratio between positive and negative cases. This decision was taken based on the logistics required to release a dataset according to this ratio that still had a sufficient number of WSI per class. Such a dataset would require more than 3TB of disk space and would result in a dramatic increase in the communication time required to upload and download all the data.
Conclusions The HEROHE Challenge was developed with the primary goal of promoting the development of computer-aided diagnosis tools to predict the HER2 status in invasive BC samples. Despite the complexity of the proposed task, 21 models were presented, combining different techniques, from standard image analysis to state-of-the-art DL algorithms, and promising results were achieved. Two of the top-ranked teams achieved AUC similar to those presented in other recent works while one team achieved a higher AUC. Given the biased distribution of training and test datasets, with a majority of cases classified as equivocal by the IHC test, this work also presented the AUC and  1 score of the proposed models in a subset of the test dataset with only equivocal cases (Table  2 ). Most of the teams performed better on this subsample.
Six teams achieved AUC greater than or equal to 0. 84, clearly outperforming the results presented in recently published studies. Team Arontier_HYY achieved the higher AUC (0. 88) on this dataset and the top  1 score raised from 0. 69 to 0. 79 (team Macaroon). This fact suggests that some of the presented models identified features on the H&E slides scored of 2+ on the IHC test that can be used to predict HER2 status without access to IHC or ISH assays, something that human experts are not able to do. The achieved results were not perfect and more data may lead to an improvement in the performance of the models, especially in cases scored as 0, 1+ or 3+ by the IHC test that were under-represented in the challenge datasets, but eventually in the equivocal cases also. The importance of the metric defined to assess the models' performance was shown to have a great impact on the final rank.
In this work, the importance of a proper selection of the final threshold to separate positive and negative cases was also emphasized by presenting an example revealing that with a better choice of the threshold, the same algorithm would result in a model with a significantly better performance. It was also pointed that the choice of the network, hyperparameters, and all the features of a model has to be carefully evaluated during the development, but also later, taking into consideration the evaluation of the model on the training and validation datasets. The result of the team MITEL compared to the one they could have had with a simpler model is a clear remark that is important to keep in mind. Although the challenge is now closed, the website and datasets will remain public for research purposes, thus further contributing to the development of novel solutions to automate HER2 status assessment.
-KDE The team KDE used a model that combined three deep learning modules. The first network aims to identify tumor regions, the second to extract features, and the last to classify the WSI. To segment tumor regions, a low-resolution version of the WSI was considered and an Otsu threshold and median filter were applied to identify the tissue regions. A custom convolutional neural network was trained on a manually annotated dataset to classify a patch as tumor or non-tumor, and tissue regions were classified by this network to identify tumor patches. Non-tumor patches are discarded in this stage. The remaining patches were then randomly split into five groups and a 5-fold cross-validation setup was implemented using an InceptionV3 network  [1]  as a feature extractor. For the third stage, an endto-end trainable multiple instance learning architecture, called KDE  [2] , was trained with a 10-fold cross-validation setup on the feature vectors resulting from stage 2.
-joangibert14 The team joangibert14 developed a method based on Campanella and colleagues' work  [3]. After tiling the WSI into 224x224 pixel patches and removing patches corresponding to non-tissue regions, data augmentations consisting of random horizontal and vertical flips, rotations, changes in brightness and contrast, and greyscale levels were applied to enlarge the training dataset. A ResNet101  [4]  model was then trained to classify a WSI based on the Multiple Instance Learning (MIL)  [5]  algorithm described in  [3].
-VISILAB The team VISILAB developed a model based on the SE-ResNet50 architecture  [6]. The method starts by preprocessing the input images. Each WSI is split into 299 by 299 pixel patches that are then segmented to separate tissue from background patches. Image patches are considered as tissue, and used for the next steps, if and only if the mean intensity is below 0. 96 × 255 = 244. 8. Background patches are discarded. Data augmentation, including random rotations, cropping and resizing, are then applied to the resulting dataset before training the network.
-aetherAI The team aetherAI adapted a ResNet50  [4]  model so it can support 10. 000 by 10. 000 pixels images as input. To be able to train a model with such memory requirements, a computational pipeline leveraging the NVIDIA Unified Memory technology was developed enabling GPU to directly access system memory, thus avoiding out-of-memory errors during training. In addition, batch normalization layers in the ResNet50 model were replaced by group normalization layers since the latter allowed for the implementation of the distributed computing system. Data pre-processing consisted of two steps, first WSIs were cropped to remove non-tissue regions, and then resized to a 10. 000x10. 000 pixels canvas. Data augmentation by application of random rotations and translations was also implemented to enlarge the training dataset.
Max-Max: select the higher scored  patches in the positive group and the higher scored  patches in the negative group for training. ii. Max-Min: select the higher scored  patches in the positive group and the lower scored  patches in the negative group for training. iii. Random: in the positive group, select patches with a probability greater than 0. 9 and the higher scored  patches, take the union of them, and randomly select  patches; in the negative group, randomly select  from all patches for training. 3. These strategies allowed to define an upper bound of space (Max-Max criterion) and a lower bound of space (Max-Min criterion), while Random lies between the upper and lower bounds. Max-Max and Max-Min criteria were used to select patches from the original training set to build a new supervised dataset that was later used to retrain both networks with the Random criterion to get the final model.
-HEROH Two residual networks  [9] , pre-trained on ImageNet  [8]  were used to build the model of team HEROH. The first, a ResNet18 was trained to classify tissue patches of size 128x128 pixel into one of six classes: tumor, normal, stroma, fat, necrosis, and immune cells. Patches classified as tumor with a probability above 90% were kept and the remaining discarded. To train the first network, a manually annotated dataset was created from 15 WSI, by tiling and standard data augmentations. The patches classified as tumor were then color-deconvolved to extract the hematoxylin channel and an Otsu threshold was applied to identify nuclei. A ResNet50 was then trained to predict the HER2 status of each patch. Each WSI was then classified according to the number of positive and negative patches and the corresponding probabilities.
-Institute of Pathology Graz The team Institute of Pathology Graz developed a model based on standard image analysis techniques to extract features of interest. A QuPath  [17]  script was developed consisting of a color deconvolution protocol followed by a nuclei segmentation (QuPath function WatershedCellDetection). For each nucleus, the parameters described in Table  S1  were computed and aggregated into a feature database. A custom convolutional neural network (CNN) was then created in the generated database to classify each WSI based on the described features.
-katherandco Team katherandco manually annotated tumor tissue on each WSI using QuPath  [17]. Patches with size 512x512 pixels were then created and only the regions containing invasive breast cancer were kept for the next steps. Data augmentation was applied and a ResNet50  [4]  pre-trained on the ImageNet dataset  [8]  was then re-trained on the resulting dataset. During the classification step, each WSI is classified according to the score achieved by its highest scored patch.
-QUILL The team QUILL used a model based on a DenseNet architecture  [7]. After splitting the WSI on the training dataset into 121828 superpixel patches labeled according to the WSI they were cropped from, a DenseNet network was trained and used as a feature extractor. The mean shift clustering algorithm  [18]  was then used to identify the modal vector in the feature space, and the feature vectors more distant (one positive and another negative) from the modal are considered as representative and used to predict the WSI status.
-HEROHE_Challenge Team HEROHE_Challenge built a custom convolutional neural network and pre-trained it on the Cfar10 images dataset  [19]. Then, each WSI was tiled into 200x200 pixels patches with 10 pixels overlap to extract a feature vector per patch. A K-Means algorithm  [13, 14]  was then used to group the feature vectors into 50 classes and for each class, the vector closer to the center of the class was selected as representative of its class. Finally, the representative feature vectors were concatenated and the XGBoost method  [20]  was used to distinguish between HER2 positive and negative slides.
Funding Eduardo Conde-Sousa  is supported by a post-doctoral grant of the project  PPBI-POCI-01-0145-FEDER-022122 , in the scope of  FCT National Roadmap of Research Infrastructures. Guilherme Aresta  is funded by the  FCT  grant contract  SFRH/BD/120435/2016. Teresa Araújo  is funded by the  FCT  grant contract  SFRH/BD/122365/2016.
Declaration of Competing Interest Jacob Gildenblat (JG) is the Co-Founder and CTO of DeePathology that collaborates with Roche on deep learning for digital pathology. Eldad Klaiman (EK) is affiliated to Roche Diagnostics. Despite ROCHE sponsored the challenge, the participation of JG and EK as members of the team IRISAI followed the same rules as all the other participants. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The team UC-CSSE team used an Xception network  [16]  pre-trained on the CAMELYON16 dataset  [21]  to identify tumor regions. Nuclei were then segmented with an ensemble of networks adapted from the model (https://github. com/selimsef/dsb2018_topcoders) that won the 2018 Data Science Bowl (https://www.
This paper describes the procedures and operational aspects related to the proper storage and handling of taxonomic, biogeographic and ecological data of biological specimens digitised under the AMUNATCOLL project. In the introductory phase of this process, the definition of the metadata is carried out, which is the formal handler of the structure, based upon the analysis of existing standards. The set of parameters derived from the standard is extended by data that is important according to the point of view of the specificity and functionality of the developed system. Subsequently, the database, as a key element in many IT systems, must be set up for data storage along with the suitable structure that reinforces efficiency. The process of preparing and casting a large amount of data requires automated procedures with dedicated tools attached.