dataesr/scientific-paragraphs-categorization huggingface dataset:
The third example in applying METLIN-NL is shown here for a nonendogenous drug molecule and its metabolite. The well-known cholesterol-lowering statin drug rosuvastatin  17  (trade name Crestor) and its active metabolite desmethyl rosuvastatin  18  differ in mass by 14 Da (demethylation reaction) and the MS 2 and NL data (Figure  4A, B ) of these two molecules have recently been acquired and populated within METLIN and METLIN-NL. As was observed with the oxylipins, tandem MS data was of limited utility when searching METLIN (Figure  4A ), where three fragment ions were overlapping between the two molecules. However, NL matching/detection showed near complete overlap (Figure  4B ). Further analysis of the tandem MS data as well as the molecular weight difference between the two molecules being 14 Da were consistent with loss of a methyl group.
For the rosuvastatin NL data, the overlap in the NL data clearly dominated the comparative analyses, making similarity searching much more effective using NL while the MS 2 data provided complementary information that was informative for structural determination. Overall, the NL data, which were completely derived from the MS 2 data, are more effective (than MS 2 ) at showing similarity. METLIN's molecular standards with systematically acquired experimental MS 2 data across multiple collision energies, allows for the comprehensive generation and graphical user interface (beta) visualization (Figure  4 ) of NL data. Fragment ion and NL similarity analysis 1 was originally developed to aid in the identification of novel molecules (unknowns) 1 by using fragment ion and NL data to help align an unknown molecule to compounds with similar fragmentation data within a database.
However, now with a NL database of small molecules available via METLIN-NL NL similarity analysis can be more readily applied to a host of biological and chemical challenges. Overall, The METLIN MS 2 to NL converter and METLIN-NL empirically derived data will enable new types of analyses facilitating more rapid identification of unknown compounds via both fragment ion and NL similarity searching. 2 Both biologists and chemists will be able to apply METLIN-NL to the structure elucidation of unknowns derived from animals,  19  plants,  14, 20  or microbiota;  21  and METLIN-NL can also be used as a resource for informatics  22  as well as identifying unexpected synthetic chemical or enzymatically modified drug products (e. g. , pharmaceuticals  23  ) as it is populated with both biological and chemical entities.
■ ACKNOWLEDGMENTS This research was partially funded by  National Institutes of Health  Grants  R35 GM130385  (G. S. ),  P30 MH062261  (G. S. ),  P01 DA026146  (G. S. ), and  U01 CA235493  (G. S. ) and by  Ecosystems and Networks Integrated with Genes and Molecular Assemblies (ENIGMA) , a  Scientific Focus Area Program  at  Lawrence Berkeley National Laboratory  for the  U. S. Department of Energy, Office of Science, Office of Biological and Environmental Research , under contract number  DE-AC02-05CH11231  (G. S. ).
Samples are from the dataset https://grand-challenge. org/algorithms/ quality-assessment-of-whole-slide-images-through-a/  [56]  and  [57]. domain, artifact types, and detection methods. In  [60] , Avanaki et al. proposed automatic quality estimators by adapting image quality assessment (IQA) methods that were originally developed for natural images. In particular, they considered IL-NIQE, a no-reference IQA, to detect artifacts and showed that the scores provided by this estimator enable discrimination of artifacts following the ratings given by a pathologist. In  [61] , Kothari et al. proposed a method for identifying tissue-fold artifacts in histological WSIs. Assuming that tissue folds are different from the other slides' areas w. r. t. color saturation and intensity, they considered the difference between both.
The difference is then thresholded at different values and the distribution of the remaining connected components is studied to define two thresholds that enable the extraction of tissue-folded regions. A similar approach was proposed in  [62]. In  [63]  Palokangas et al. proposed the extraction of folds using kmeans clustering in a saturation-intensity feature space. In  [4] , Janowczyk et al. proposed HistoQC  9  , an open-source quality control tool for digital pathology slides. This tool uses a combination of handcrafted features extracted from a digital slide (related to color, brightness, contrast, and edges) that can be fed to machine learning techniques for artifact extraction. The software can then be used to identify artifacts and artifact-free areas. In  [64] , Chen et al. used HistoQC for quality control of renal biopsy WSIs.
They aimed at identifying the slides that were unsuitable for computational analysis because of the presence of artifacts. They used different modules of HistoQC to identify artifacts such as tissue folds, pen markers, air bubbles, and ink stain variations. They showed that HistoQC could identify batch effects in the slides' cohorts of three pathological laboratories and concluded that a quantitative process is necessary for robust and reproducible quality control of digital slides. Kumar et al. [65] , proposed a method for the identification of artifacts in cytological cervical smears WSIs. Cells that are not epithelial are considered as artifacts: e. g. , blood cells. The cells were extracted using classical image processing methods. Features (describing intensity, shape, and texture) are extracted from the segmented cells and the latter are classified as artifact or non-artifact cells with an SVM.
They obtained an accuracy of 86. 64%. In  [66] ,  [67] , Shakhawat et al. proposed the detection of artifacts (air bubbles and tissue folds) in histological WSIs at low resolution, as the considered artifacts can be seen at low magnification and have visual properties that are very different from the surrounding tissue. They extracted luminance and saturation features, and Haralick gray-level co-occurrence matrix texture features from 100 × 100 patches. Only relevant features were retained using sequential feature selection and fed to an SVM classifier. They obtained an accuracy of 98. 98%. Smit et al. [56]  proposed a multi-class deep learning model  10   [69]  and showed that a deep learning approach to artifact segmentation  11  can produce interesting results as long as learning strategies are adapted to dataset characteristics.
In particular, artifacts in digital pathology slides are illdefined objects, which makes them particularly challenging to annotate precisely, and their work addresses this aspect of imprecise annotations  12. Babaie et al. [70] , proposed the use of a pre-trained DenseNet201 CNN as a feature extractor to characterize tissue folds, and fed an SVM classifier with these features. The DenseNet201 CNN was fine-tuned on a dataset of folded and fold-free 255 × 255 patches along with data augmentation. They obtained an accuracy of 96. 7%. Ali et al. [71]  proposed a fully automatic CNN for the classification and restoration of WSIs containing pen ink markers  13. First, a CNN detects tiles corrupted by a pen marker. Second, in these corrupted tiles, a Yolo CNN detects the bounding boxes of pen marker areas.
IV. QUALITY OF SLIDE DIGITIZATION A. QUALITY OF WSI FORMAT Many commercial scanners have appeared in the market and each has introduced its proprietary file format. Consequently, there is no established file format for storing and exchanging WSIs produced by WSI scanners (see Table  3 ). However, these proprietary file formats can share similar properties. Indeed, given the flexibility of the TIFF file format, many have adopted it to store WSI data, often in the form of a tiled multi-resolution pyramid (Figure  4 ). Nevertheless, this abundance of proprietary file formats can be a strong barrier to their use even if some formats are very similar. Indeed, as we will point it out later in this review, deep learning methods for computational pathology require large and diverse datasets from different centers potentially using different scanners.
Generative model-based methods have recently attracted considerable attention. They use adversarial learning of deep neural networks to perform a style transfer to a WSI. In addition to being more adaptive than the global color normalization and stain separation approaches, they suppress the need of selecting reference images to define the stain parameters. Indeed, generative adversarial network (GAN)  [126]  based methods consider the overall dataset of the target style as the template and approach the problem of color normalization as an image-to-image translation. GANbased color normalization approaches can be divided into supervised and unsupervised methods. Supervised methods require paired images of different staining protocols and use L1 and adversarial losses to optimize the generative networks.
They observed that the use of color normalization always causes a loss in accuracy. Tellez et al. [138]  studied the effect of color normalization  [103] ,  [113] ,  [116]  on the performance of several different CNN classification tasks on histological H&E slides. They also observed that color normalization was not necessary to achieve better performance. Hameed et al. [139]  have considered six intermediate layers of the pre-trained Xception model to extract features for the classification of histological H&E breast cancer images. None of the normalization methods they considered  [100] ,  [102] ,  [103] ,  [111]  was able to outperform the results of the original un-normalized dataset. However, with GAN-based approaches, the classification results of the unnormalized and normalized images can be very similar.
In  [140] , Ciompi et al. investigate the influence of color normalization methods  [103] ,  [113]  on tissue classification of colorectal cancer tissue samples in H&E-stained images. In contrast to the previously mentioned studies, they reported a significant gain in performance with the color normalization of  [113]. Finally, in  [141] , the influence of several color normalization algorithms  [100] ,  [103] ,  [106] ,  [108] ,  [111] ,  [113]  has been studied for the classification of three different histological cancer H&E WSIs (ovarian, breast, and pleural) using a ResNet18 network. Their finding is that color normalization does not improve performance when WSIs are from the same center. However, when normalized datasets from several centers are used for learning, an increase of classification performance can be obtained.
2) OOF datasets To be able to learn to detect OOF areas, FQA research works have considered annotated datasets. The latter can be obtained by different techniques (resulting in different dataset types). The most commonly employed techniques to label the focus quality of patches in WSIs are: • Fully manual annotation (MAN): patches are extracted from the WSI and labeled by human experts. • Automatic z-stacks (STK): patches extracted from real z-stacks with an offset value from the in-focus plane, or an absolute z-level from this location. • Semi-synthetic z-stacks (SYN) simulate the previous one by applying several degrees of synthetic blur to the in-focus patches. The labels associated to the patches are usually binary (infocus or out-of-focus) but in some works  [153] ,  [156] ,  [157]  several levels of focus have been considered.
FQA methods designed on subjectively labeled data (MAN) should provide a response closer to manual FQA according to the assessor cohort, which is usually able to distinguish a maximum of six OOF grades. Because the creation of MAN datasets is tedious, thus limiting their size, they are mainly used to evaluate the performances of FQA methods optimized with objectively labeled data (STK and SYN). The latter data, related to auto-focusing and z-stacking, are easier to create and overcome the limitations of the former (size and dependence on assessors). Note that STK and SYN are not completely objective (manually calibrated and checked), and SYN is an augmentation technique. While Gaussian blur is commonly used, Bokeh blur (2D Heaviside step function) is known to be closer to the perception of real OOF occurring in photography and microscopy.
As observed in  [156] , the human perception of blur follows an exponential relationship with the OOF level rather than a linear one. In particular, Gaussian blur tends to underestimate strong OOF which is less the case for the Bokeh blur model. Even if labeled datasets are mandatory to design a learned FQA method, still few datasets are available. FocusPath 29   [147] , and its extended version  30  , contain 864 (resp. 8640) 1024 × 1024 pathological images with 16 absolute z-levels as scores (STK), and acquired with Huron TissueScope LE1. 2 at 40×. TGCA@Focus 31   [157] , selected from the Cancer Genome Atlas  [163] , contains 14 371 manually annotated patches (MAN, 2 classes) from 52 organ types. Both of them are restricted to histology.
3) Handcrafted feature classification In  [150] , Gao et al. proposed the first work on learned WSI FQA from handcrafted features. They used 44 standard measures of image quality as features (neighborhood contrasts, derivative-based, local intensity statistics, wavelet-based), which have already been identified as good candidates for characterizing the OOF effect in the context of microscopy and auto-focusing  [164] ,  [165]. Then, an AdaBoot-based binary classifier is used to estimate if a patch is in or out of focus. They considered MAN and STK datasets  [150] , both at 20× and 40×, and trained their binary AdaBoost classifier based on 44 features for the four datasets independently. For the MAN dataset, the classification accuracy was higher at 20× with a large improvement (approximately 5%).
For the STK datasets, it was higher at 40× with a slight improvement (< 1%). Globally, STK achieved better performances than MAN (> 92. 7 and just above 91% respectively. ). The individual behavior of these handcrafted features was further analyzed by Moles Lopez et al. [152]. They observed that the discriminatory ability of these features varies from one biological structure to another for the same stain (most importantly for IHC), and from one stain to another. To cope with this, they considered several binary classifiers based on height discriminatory features and decision trees (DT), trained either on a stain-specialized dataset (H&E or IHC) or on a mixed version, with objective scores (MAN). All three learned models achieved an accuracy of at least 96%. To test the transfer ability of these three learned models, they tested them on an independent MAN dataset, and the performances dropped by up to 89%, particularly for classifiers that rely on IHC.
As classification accuracy was reduced for the mixed version in any case, mainly owing to the presence of IHC patches, they retained the stain-specialized versions with a reduced number of four features (Haralick features, mean gradient magnitude, Tenenbaum gradient, and noise used in  [149] ) and decision tree depths (approximately seven). In the context of liquid-based cervical cytology, Lahrmanb et al. proposed a binary classifier for patches containing cells automatically selected using Otsu segmentation and HSV analysis  [151]. The classification is based on a SVM and five handcrafted features expressing the mean quantity of edges and local variations, and the differences between sharpened, smoothed, and blurred versions of the patches. High accuracy and sensitivity (> 98%) were obtained for the MAN dataset.
Inspired by previous studies, Campanella et al. [153]  considered a Random Forest (RF) model with 13 features to predict a blur class (among six). Based on a feature selection on a Gaussian-SYN dataset, a RF regression model composed of 19 trees and 10 features was retained. More recently, Hosseini et al. designed several sharpness measures (HSV-MaxPol  [147]  and FQPath  [148] ) based on a symmetric FIR kernel that mimics the ability of the human visual system to boost high-frequency domain magnitudes in a balanced manner. It is defined as a superposition of multiple even-order derivative kernels, either fixed (HSV-MaxPol) or to fit the inverse PSF of the scanner optic (using the Born & Wolf model) up to a threshold frequency (FQPath). By using the FocusPath dataset (STK) to tune the parameters, good correlations with the FocusPath scores were obtained by HSV-MaxPol and FQPath compared to several state-of-the-art measures, close to the behavior of Maximum Local Variations (MLV)  [166] , for a reduced computation time.
Similar results were obtained for the binary classification  [157]  (ROC 0. 94, PR 0. 97). However, when using a different dataset (MAN) to test the transfer ability (other scanners and stain-tissue types), the performances were considerably reduced and were different for HSV-MaxPol, FQPath, and MLV (0. 56 <PR< 0. 67). This confirms the observations made in previous studies on the transfer ability of handcrafted features.
Trained by using an STK dataset and categorical cross entropy as a loss function, DeepFocus showed high classification accuracy (93. 2%) for an independent dataset (from the same scanner and stains, H&E and IHC), and confirmed that H&E-stained tiles are much easier to classify than IHC ones (< 90% for 2 slides among 3). Kohlberger et al. proposed ConvFocus  [156] , which is a similar network that distinguishes 30 OOF levels. The network is a truncated Inception V3 architecture composed of six layers to extract the features (three conv. + max pooling + average pooling + dropout), and a fully connected layer followed by a SoftMax. ConvFocus was trained using cross entropy with two Bokeh-SYN datasets, coming from two different scanners, and augmented by various transformations (orientation, brightness, contrast hue, saturation, translational jitter, Poisson noise, and JPEG compression).
It was tested using two MAN datasets (similar scanners, same stains, six grades). The predictions were highly correlated with the expert scores, mainly for one scanner (SRCC approximately 0. 8 and 0. 93 resp. ). Wang et al. proposed FocusLiteNN 33  [157] , a shallow CNN that provides a sharpness score from a single convolution layer (with k filters for each color channel) and non-linear pooling as an activation function (linear combination of the extremum feature values). FocusLiteNN was compared to the knowledge-based methods discussed in  [148] , including FQPath, HSV-MaxPol  [147] ,  [148] , and MLV  [166] , as well as five standard CNN architectures similar to those considered in previous works (DenseNet-13, EONSS, and different configurations of ResNet). All CNNs were trained on FocusPath using their initial loss function (PLCC for FocusLiteNN).
Performances on Fo-cusPath and a TGCA@Focus datasets were always higher for CNN-based models (ResNet-10 performed best), with interesting transfer ability (0. 97 <PR< 0. 99 for FocusPath and 0. 9 <PR< 0. 87 for the TGCA@Focus dataset). Differences between shallow and deeper CNN architectures were only slight, while shallow CNN improved computation time by a large margin. Albuquerque et al. [158]  performed a comparison using FocusPath to train seven other standard CNN architectures (MobileNet_V2, AlexNet, GoogleNet, ResNet18, VGG16, truncated ShuffleNet, and SqueezeNet) for a classification task (12 OOF levels). Contrary to Wang et al. [157] , data augmentation (shifting, zooming, flipping, and rotation) was applied. As most previously tested models were trained according to a non-ordinal loss function (typically cross-entropy), Albuquerque et al.
5) Discussion To conclude this tour of WSI FQA, best performances can be obtained by CNN-based methods trained on semisynthetic or stack datasets. However, most methods are dedicated to specific stains and scanners. The results on transfer ability are encouraging for subjectively assessed data containing other stain-tissue types and acquired by different scanners, even for very shallow networks  [157]. While shallow networks seem to provide a more appealing 33 https://github. com/icbcbicc/FocusLiteNN ratio between computation time and classification accuracy than handcrafted feature-based methods, their behavior for specific stain-tissue types (e. g. , IHC) and cytology images has been rarely studied. The datasets used for learning or evaluation are generally free of artifacts owing to slide preparation or manipulation.
To conclude, the field of quality control of WSIs is just in its infancy. The main issues are known and are related to artifact detection, staining quality, and focus quality. If some methods have been proposed separately for each quality issue (as exposed in the previous sections), they all have limitations in terms of accuracy and reproducibility on other datasets from other digital pathology centers. In addition, their output is at the moment limited to heat and segmentation maps that can be overlaid on the WSIs but not to evaluate the global slide quality. The two recent approaches of  [56] ,  [73]  provide good directions towards what the community should converge to: a quality control based on artificial intelligence able to recognize all the quality problems and to decide what to do with the slide from rejecting it, advising re-stain or re-scan, or considering the slide as analyzable by a CAD system.
As this is usually the case (with the popular ResNet 101, W is more than 44 million with L = 101 layers), a key issue of applying DNNs in computational pathology is therefore highly related to the availability of a huge and well-labeled learning dataset. The quantity and the quality of a learning dataset are also something that has to be taken into account when designing a computational pathology approach for CAD. Indeed, the quality of the dataset used for learning can potentially have a strong impact on the quality of the induced deep learning algorithm. We already addressed this partly when we reviewed the influence of color and focus of WSIs: once a DL algorithm has been trained, it is built only to work on data that are similar to those used to train it. Therefore, if one wants an algorithm to be as versatile as possible, it should have learned to recognize patterns in many different situations.
Therefore the evaluation of machine learning algorithms for digital pathology can be very delicate  [180]. In  [181] , Wahab et al. have studied, for the classification of different breast cells in H&E WSIs, the quality of annotations in terms of completeness, exhaustiveness, diversity, and agreement. They concluded that standardization of annotation protocols is necessary and proposed a new one. Two direct consequences of annotation quality are that: i) datasets have to be very carefully labeled to have high-quality training corpus (to cope with the lack of quality of labels), and ii) the DL algorithm can use specific learning strategies to make them more robust to variations (to cope with the lack in quantity of data). These two aspects are considered in the sequel.
A. LACK IN QUALITY OF LABELS In  [182] , Marée et al. summarized the potential data collection issues in digital pathology and proposed guidelines for constructing realistic ground-truth datasets while controlling their quality: • Examples should come from different staining and scanner equipments. • Examples should be equally represented in each class. • Biological variations of the examples should be covered in each category of examples. • Artifacts should be considered as additional categories to avoid too many false positives. • Annotations should be performed by several experts. Unfortunately, very few datasets encompass all these requirements (see in  [32]  for a list of available cancer histology datasets). Recently, Hosseini et al. [183]  proposed a new digital pathology dataset called the "Atlas of Digital Pathology".