dataesr/scientific-paragraphs-categorization huggingface dataset:
In particular, they demonstrated the quality of their image labels through pathologist validation and by training three state-of-the-art neural networks for tissue type classification. This stresses that obtaining annotations of large cohorts of representative WSIs is a very sensitive step. This is mainly because of the expertise required to generate quality labels and the limited availability of qualified experts. Therefore, pathologists require efficient annotation tools (preferably open-source) that easily enable them to label patterns in WSIs. Hopefully, such software are available (see  [184]  for a complete review): • Icy 36   [185]. • QuPath 37   [186]. • Cytomine 38   [187]. • SlideRunner 39   [188]. • Quick Annotator 40   [189]  Regardless of the chosen annotation tool, the quality of the labeled dataset must be carefully monitored, because labeling is usually performed by a single pathologist.
B. LACK IN THE QUANTITY OF DATA When the objective of a method is to detect rare items in WSIs (such as artifacts), the collection of annotated data is all the more problematic. Indeed, because rare items are by definition not often encountered, the constitution of a representative dataset is almost impossible and this is particularly true for artifact detection in quality control. Therefore, other strategies have been envisioned to address this lack of data. In computer vision, when a dataset is strongly imbalanced or its size is too limited to train a deep learning algorithm, typical strategies exist to alleviate the scarcity of annotated data  [194] : data augmentation, transfer learning, domain adaptation, and weakly-supervised learning. Data augmentation  [195]  artificially generates synthetic data from the initial dataset to enlarge it and improve the performance of the model.
Augmentation can be performed using many different transformations, such as geometric transformations or color shifts. Transfer Learning  [196]  works by training a network on a large dataset such as ImageNet and then using those weights as the initial weights in a new classification task. Transfer learning works only if the data to be processed are similar to have a valid transfer (e. g. , natural images). Domain adaptation is a type of transfer learning method. A DL algorithm learns from a source domain with a large labeled dataset and aims at achieving comparable performance for the same task on a target domain with few labeled data  [197]. Weakly supervised learning  [198]  consists in training the models with labels less expensive to collect than imagelevel annotations (e. g. , grades at the slide-level).
1) Data Augmentation Data augmentation is a frequently used solution to introduce a certain degree of invariance to a CNN and to tackle class imbalance by artificially increasing the learning dataset. In  [138] , Tellez et al. conducted a study on different kinds of data augmentation (rotations, mirroring, scaling, elastic deformation, Gaussian blur and noise, brightness and contrast, Hue-Saturation-Value and Hematoxylin-Eosin-Dab color shifts) and showed that this can lead to performance gains. In particular, stain color augmentation is crucial for achieving the best performance. In  [199] , Teramoto et al. demonstrate a consequent gain in performance with data augmentation for the classification of benign and malignant cells in cytology with a VGG CNN. In  [200] , Annuscheit et al. investigated data augmentation techniques (color-based, geometric-based, filter-based transformations, and erasing) on different datasets using multiple network architectures (VGG, Inception, DenseNet).
In  [204] , Sharma et al. compared the performance of features extracted from net-works trained on ImageNet and histopathology data. They demonstrated that specific encoders (ResNet) trained on multiple histopathology datasets result in superior features than their ImageNet trained counterparts and should be used for weight initialization in histopathology tasks. Aitazaz et al. have recently shown  [205]  that the vision transformer works better on histopathology images than CNN models pre-trained on ImageNet. In  [206] , Mormont et al. investigated various deep learning transfer learning strategies on histological and cytological datasets. They compared transfer strategies for seven different network architectures. They observed that fine-tuning always outperformed off-the-shelf features from the last layer of the network, regardless of the network.
Similar conclusions were drawn by Jang et al. in  [207]  who studied transfer learning between different types of cancers. In  [208] , Mormont et al. investigated multi-task learning as a way of pre-training networks. They gathered 22 digital pathology datasets into a single dataset and used multi-task training. The features extracted from their model are superior to those of ImageNet. This shows that domainspecific pre-training can be an interesting alternative to finetuning. Another way is to try to consider the existence of generalizable knowledge between different problems and to use weight distillation  [209]  for cross-knowledge transfer.
3) Domain adaptation When transfer learning is considered, the source and target domains are assumed to follow similar distributions. Domain adaptation deals with cases where a model trained on a source distribution is used in the context of a different (but related) target distribution. This problem has been recently addressed by employing adversarial learning: a discriminator is trained to distinguish source and target data using features extracted from a deep neural network as inputs, while the deep neural network is tuned to confuse the discriminator. This helps to map the source and target data close. In  [210] , BenTaieb et al. proposed the use of GANs to learn dataset-specific staining properties to transfer stains across datasets. Fine-tuning the obtained stain transfer network with images from a new domain can enable normalizing training images with respect to the new domain distribution.
Ren et al. [121] ,  [211]  proposed using unsupervised domain adaptation to transfer discriminative knowledge obtained from the source domain to the target domain. Adaptation is achieved through adversarial training to find an invariant feature space in the source domain along with a Siamese network architecture on the target domain to enforce regularity. Shi et al. [212]  also use stain style transfer to translate the style of a small image dataset into a large dataset style with cycleGAN. Because an inception CNN has been trained on a large dataset, it can be applied to datasets from other centers. In  [213] , a different approach has been proposed. The authors train a "universal" model to recognize diverse histological tissue types from a source domain dataset of healthy slides from various organs.
4) Weakly supervised learning Building a large dataset of cell-level labels on WSIs is tedious and time-consuming. In contrast, slide-level labels (e. g. , grading or diagnosis) are easier to obtain as they are readily available from diagnostic reports. This is the motivation of weakly supervised learning where the slidelevel diagnoses constitute weak labels. Recent reviews on that topic can be found in  [214] ,  [215]. It is then assumed that if a WSI has a cancer diagnosis, at least one of its tiles must contain cancer cells. An application of this principle, referred to as multiple instance learning (MIL), was performed by Campanella et al. [168]. They used a dataset of 44,732 WSIs using only slide-level diagnoses as labels with impressive results on test sets of prostate cancer, basal cell carcinoma, and breast cancer metastases.
In  [216] , Kanavati et al. trained a CNN based on the EfficientNet-B3 architecture, using transfer learning and weakly-supervised learning, to predict carcinoma in WSIs. They also compared fully-supervised learning and weaklysupervised learning and demonstrate that: i) fully-supervised learning performs best when cell-level labels are available, and ii) when only slide-level diagnoses are available, weakly supervised learning can be performed but requires a much larger dataset of WSIs. In  [217] , Teramoto et al. developed a weakly supervised method for the classification of benign and malignant lung cells in cytological images using attention-based deep multiple instance learning (AD MIL). Images were divided into patches images and stored in bags. Each bag was then labeled as benign or malignant, and classification was conducted using AD MIL.
WSIs with a sparse distribution of cells (PAUC +/++) often lead to the READclassification. The same effect appears in the presence of bubbles (BUBB +), blur (BLUR +/++) or poly (POLY +/++). For the other labels, the correlation between a given label and readability is less noticeable. But one can notice that for approximately 90% of the "hard to diagnose" slides, there was a joint presence of several preparation artifacts such as blood, mucus and bacteria. Similar studies have been led for histological slides  [223] ,  [224]  and reported a rate of around 20% of quality issues (mainly due to out-of-focus and stitching). Figure  10  presents the stream of slides according to their labels. Some labels and days were gathered for readability purposes when the amount of associated data in each was too small.
Regardless of the label, approximately the same proportion of WSI arrives at each time step. In addition, from the perspective of capturing examples to build a machine learning training dataset, some labels are scarcer than others. In this laboratory, it would take approximately a month to collect 100 paucicellular WSIs and a few days for atrophic WSIs. To complete this journey through the labels, we analyze the data to check the interdependence between the labels. In other words, we want to ensure that the chosen labels are good descriptors with few redundancies in the overall information they provide. The uncertainty coefficient (entropy coefficient or Theil's U) is a non-symmetric measure based on the conditional entropy between two phenomena A and B. Knowing phenomenon A, it checks if we can predict phenomena B.
In particular, most of the methods for quality control are developed on specific small-size datasets. If they can work reasonably well on the learned datasets, their generalization abilities to data coming from other centers (using different slide preparation and digitization protocols) are not enough explored. Only the recent work of  [225]  has explored how deep neural networks perform on corrupted images (with compression, focus, color, and artifact issues) and shown that this can severely decrease the prediction accuracy. This makes none of the existing quality assessment tools ready for clinical  VOLUME 4, 2016  use. However, it can be expected that novel techniques using larger datasets will converge to models close to being deployed for clinical practice. We propose to draw an outline of what a QC tool for whole slide images should do and how it could be included within a computational pathology pipeline.
AI to D -re-digitize: the scanner needs to meet the requirements for creating WSI in a compatible format and with enough information to allow the use of the AI module as intended. AI to P -re-prepare: Sometimes, color deviation from the standard is not captured by quality control made in the prior steps. According to a fine analysis of the train data and data augmentation, the AI module should be selfaware of its range of acceptability. Countermeasures can be applied with color/style transfer from the domain of the processed WSI to the domain described by the training dataset. In the same way, an AI can be self-aware of its weaknesses by incorporating a module for the detection of known perturbation (artifact, blur, glitch, or pipeline-specific disturbance such as a large presence of blood).
Their approach is constituted of four steps: (1) a pre-processing step, (2) audio data augmentation, (3) audiovisual features extraction and (4) classification of the extracted features using a one-shot learning-based framework. A Siamese Neural Network is the architecture used to implement oneshot learning. The network contains two identical feedforward neural networks, with the same parameters and weights, joined at their output and work in parallel. The network computes a distance between the output of the first network and that of the second. This distance vector represents a similarity score between both output vectors. Experiments were performed on the DAIC-WOZ dataset and a highest accuracy of 73. 68% was obtained.
Proposed approach The proposed multimodal deep learning approach is based on audiovisual cues and it consists of five steps. First, video recordings are acquired and processed, followed by an audio data augmentation as described in Section 4. 1. Second, deep audio and visual features are extracted using convolutional neural networks (Section 4. 2). Third, multimodal fusion step is performed followed by classification for depression recognition as detailed in Section 4. 3. Then, a correlation based anomaly detection framework for depression and relapse recognition is defined in Section 4. 4. Lastly, depression relapse prediction is performed based on a measure of similarity to depression as described in Section 4. 5. In the following, the different steps of the proposed approach are described.
Preprocessing and data augmentation Preprocessing. Our approach is based on video data extracted from recorded interviews between patients and a healthcare practitioner (interviewer), in which the patient answers a series of questions of the PhQ test. Only the audio and the facial data of the patient are considered in this work since the aim is to predict depression based on the patient's responses and reactions to clinical questionnaire. In our study, the timestamps of the interviewer and the subjects are provided with the audio recordings. These timestamps are used to retrieve the subjects speech. Interviewer speech portions are discarded and only the subject's speech is used in this work. The speech recording and the visual features (the action units given in the DAIC-Woz, see Section 5.
Visual features In this work, only the available 20 Action Units (AUs) in the DAIC-Woz dataset are used as low-level visual features mainly because no raw videos are made public in this corpus for ethical reasons. AUs are used as low-level visual features to encode the facial patterns of depressed subjects. The action units describe changes in facial expressions and thus they could discriminate the facial expressions of the healthy subjects from the depressed ones. In the DAIC-Woz dataset, 20 Action Units (AUs) are extracted from the upper and lower face of each subject as described in  [34]  using the OpenFace Framework. The 20 AUs are fed to a 1D-CNN. The 1D-CNN model is composed of two convolution layers with 256 filters of size 7 each, followed by a max pooling layer and a dropout layer with a probability of 0.
Correlation-based anomaly detection framework In this research work, we present an anomaly detection based approach for depression recognition and relapse prediction. two representations are learnt from anomaly-free data (depression free subjects) and anomaly data (depression subjects). The deep encoding (i. e. , deep features) of the depressed and non depressed subjects are used to learn these representations. The high dimensional feature vectors are obtained from the second dense layer of our deep neural network (presented in Section 4. 3). A test subject is diagnosed with depression or relapse as described in Algorithm 2, when its audiovisual encoding is more correlated with the model of depression (depression representation) than with the model of non-depression (control representation).
Dataset The Distress Analysis Interview Corpus Wizard-of-Oz dataset (DAIC-WOZ) is used for the experiments of the proposed methods. It was introduced in AVEC 2017 challenge  [34]  to analyze and to investigate psychological distress conditions like depression. DAIC-WOZ contains clinical interviews of an average length of 15 min of 189 participants with virtual interviewers labeled with PHQ-8 scores. Both raw speech audio data and Action Units features are available along with the timestamps of the participants answers. We randomly divide the dataset subject wise where 80% were considered for training and 20% for testing. The training data is preprocessed and augmented as described in Section 4. 1.
The 𝑝-value < 0. 0005 for the audio features while the 𝑝-value < 0. 0001 for the audiovisual features. The reported 𝑝-value of the Wilcoxon Signed Ranks test demonstrates that the distribution of the learned features from the depressed and the non-depressed subjects are significantly different. As stated before, the use of Action Units visual features on top of the log-mel spectrograms allowed to obtain a boost in performance, the proposed multimodal deep learning model managed to achieve an accuracy of 78. 97% with an improvement of over 7% compared to the 71. 18% accuracy obtained using only audio features. Furthermore, given the imbalanced nature of the DAIC-WOZ dataset where the number of the non-depressed is three times higher than the depressed. We rely on other metrics to better evaluate our approach, for instance precision and recall, we can notice that the first audio features based model is biased towards the non depression class, with high scores compared to the depression class (higher precision), this is further validated with the F1 scores of both classes (0.
507 for depression compared to 0. 796 for non-depression). A major improvement can be noticed in terms of F1 scores and both precision and recall with the proposed multimodal approach, reaching a F1 score of 0. 845 for non depression and 0. 674 for depression with an average of 0. 759, an improvement of 10% compared to the simpler model, further, the precision is much higher for the depression class (0. 618) indicating that the model is able to correctly label depression samples more efficiently. In Fig. 4(a, b ), the confusion matrices for both networks are presented, from which we can further observe the major improvements mainly in regards to the number of true positives (depression), an improvement of more than 6% out of the total accuracy, making the model less biased. Comparison experiment of resampling techniques to overcome imbalance data problem and to boost depression prediction performances: the Daic-Woz dataset is an imbalanced dataset which could have an impact on the classification model's performance.
Two techniques are considered by operating in feature space rather than sample domain: (1) a common oversampling method called the Synthetic Minority Oversampling Technique (SMOTE)  [37]  in which the minority class is oversampled by creating synthetic examples. (  2 ) The Random under-sampling (RUS) is a resampling technique whereby the majority class (non-depression) in the dataset is reduced to even the class distribution by randomly picking samples with or without replacement. Fig. 3  shows the results produced by the proposed deep neural network based approach with and without resampling techniques. The accuracy, sensitivity, and specificity rates when using SMOTE achieved (4. 70%, 2. 20%, 3. 90%), (4. 80%, 2. 40%, 3. 80%) higher than those produced by the same algorithm when accompanied by RUS and Noresampling, respectively.
Comparison with existing methods for clinical depression and relapse detection using audiovisual cues Table  5  summarizes the performance of our proposed approach in comparison to other existing approaches evaluated on the DAIC-Woz dataset in terms of accuracy and F1-scores. Salekin et al. [38]  manage to obtain the highest accuracy of 96. 7% and a F1 score of 85. 4% thanks to their feature modeling technique named NN2Vec with a multiple instance learning adaptation of a BLSTM classifier. This high performance is due to the good identification and exploitation of the inherent relationship between speakers' vocal states and symptoms/affective states and also to the good learning of the temporal dynamics of vocal states. While our audio based model obtains a slightly lower accuracy using the same leave-one-out training strategy (94.
Generalization ability to others datasets The generalization ability of our proposed deep neural network based approach is evaluated on the AVI-D corpus  [45, 46]  where 300 audio recordings are collected from 292 participants who go through two different tasks: the Northwind task (reading task) and the Freeform task (answering questions). The AVi-D corpus is based on the commonly used depression assessment test called the Beck Depression Inventory-II (BDI-II) for assessing the severity levels of depression. The audio recordings are labeled by the BDI-II scores ranging from 0 to 63. When the BDI-II score exceeds 14, the patient is labeled depressed. The deep representation is learned from the DAIC-Woz dataset. The proposed correlation-based anomaly detection framework is tested over the AVi-D data of the two tasks Freeform and Northwind separately.
Discussion The high performance of the proposed approach confirms the possibility of the deployment of our system for clinical depression monitoring and to aid clinicians in the diagnosis and prognosis of depressive disorder. It open new doors for a personalized medicine for mental and affective disorders. The proposed approach can be deployed into a web or a mobile phone application to acquire and analyze the audiovisual cues for monitoring depression and predicting relapse. However, the use of such computer-aided diagnostic system in real-world applications may poses ethical and privacy issues. The protection of the personal data of depressed patients, like others mental disorders, should be guaranteed. Besides, the proposed approach is evaluated on a dataset collected for depression recognition.
It is a proof of concept for relapse prediction as no dataset exists for depression relapse. Through this paper, we wanted to put the emphasis on an unexplored research area despite the growing and urgent need. The proposed approach presents several advantages: (1) Extensibility: our approach is extensible and others modalities could be integrated easily without training the hole framework from scratch. Each time a new modality is added, a new deep neural network is trained to learn deep features representations of it. The deep features representations of the different modalities are then concatenated and afterwards, the correlation-based anomaly detection algorithm for depression relapse prediction is performed. (2) Interpretability: a depression relapse is detected when the voice and the facial expressions of the patient become similar to the voice and the facial expressions of depressed persons.
(3) Monitoring and assistance: the proposed framework could be used to monitor depressed patients. (4) Robustness: the proposed approach outperforms the existing approaches in predicting depression. However, despite the high performances of the proposed correlationbased anomaly detection framework and the similarity measure to depression based approach, others possible encoding for depression and non-depression and others possible similarity criteria must be evaluated and compared. This will be the subject of future work. On the other hand, the proposed approach is evaluated in small and imbalanced public dataset. More data collection is needed to evaluate its applicability in real-world applications. Furthermore, a simple and lightweight deep learning-based self-attention module (SAM) as proposed and recommended in  [47]  could deal with the scarceness of datasets.