dataesr/scientific-paragraphs-categorization huggingface dataset:
Unbalanced classification HF-related events occur rarely when monitoring cardiac patients. This causes imbalanced classes where the positive class (HF event) represents a small fraction (e. g. 10%) of the whole dataset. Classification models become biased; for example, a classifier with a constant answer on a dataset with 90%-10% class proportions reaches 90% accuracy, but fails to detect the minority class. Class weighting may not be sufficient when dealing with highly imbalanced datasets. In such settings, oversampling and undersampling approaches can give better classification results. Oversampling generates synthetic samples following the distribution of the minority class to balance the dataset, such as Synthetic Minority Over-sampling Technique (SMOTE)  [21] , or Adaptive Synthetic sampling (ADASYN)  (Haibo [22].
Undersampling removes samples from the majority class. This can be done randomly or with prior information such as TOMEK links  [23]. One should note that oversampling could lead to overfitting when dealing with noisy data, while undersampling can be inefficient when the minority class is very small. Finally, undersampling the majority class combined inside an ensemble method is a promising approach (Xu-Ying  [24]. This creates a bagged classifier from multiple weak classifiers. Each classifier is trained over iteratively balanced datasets generated with undersampling. The majority class subsamples may overlap or not. This yields a classifier trained on the whole dataset without any synthetic data, and can achieve better accuracy.
It is hard to compare these methods since they use different datasets, with different number of patients and different follow-up duration. Validation schemes are different and are often within patient, i. e. same patient at different periods in train and test subsets. This makes the generalization of ML models less reliable. Recent works rely on deep neural networks that are able to ingest raw data. Recurrent Neural Networks are specifically designed for timeseries. They use the sequential relationship between timepoints to process the output Rumelhart et al. [25]. For instance, Long-Short Term Memory (LSTM)  [26]  networks are successfully used for speech recognition  [27]  and language processing  [28]. In Ref. [29] , the authors show that LSTM can predict patient diagnosis from electronic health records.
Feature engineering This step defines the inputs of the ML model. For each patient, medical records (time-series) are segmented into sub-periods of N consecutive days. In this study, we set N = 55 to ensure the benchmark of all the medical guideline rules (e. g. rule R2 in as mentioned in section 2. 1). Each sub-period followed by a medical event is then tagged positively (target = 1). Sub-periods corresponding to the stable state are tagged negatively (target = 0). We apply a lag from 7 to 14 days between each patient consecutive sub-periods (Fig. 6 ). This avoids any overlap between stable and degradation sub-periods or any contamination between medical events. Following the data representation steps, a reference dataset is constructed from patients daily tele-transmitted weight and symptoms records.
The first day of each interval is the beginning of the telemonitoring in the corresponding sub-period whilst the last day corresponds to the day of a medical event if occurred. The dataset is then well formatted for training ML algorithms with target representation for classification setting: • Y = 1 stands for HF decompensations from medical events P0 and P1. • Y = 0 stands for the healthy conditions corresponding to stable state of patients (P6) or no proven worsening (P2, P3, P4, P5). Another data representation option would be to include uncertain situations (P2, P3, P4) as positive targets. Our experiments have shown that this increases the number of false alarms considerably. This implies a considerable workload of nurses without a significant improvement in the prediction of HF-related events.
Validation schemes We evaluate the model through two validation schemes: • Held-out test set validation; where the reference dataset is split into a train subset (75% of the whole dataset) on which the model is built. The test subset (25% of the whole dataset) is used as a ground-truth on which the overall performance is measured. • K-fold cross-validation; here, the initial train subset (75% of the whole dataset) is split with a stratification, in order to have homogeneous subsets, resulting in subsets having the same proportions of the target. As shown in Fig. 7 , we group the samples (features) by patient during the split. This means that patients of the test subset are different from those in the train subset. Such an approach ensures a better generalisation of the fitted classifier.
HeartPredict performance HeartPredict algorithm, implemented with balanced random forests, is compared to alarming rules derived from medical guidelines in the literature. Fig. 8  shows the ROC curve of HeartPredict computed using the probabilities' outputs on validation data. The sensitivity and specificity of guidelines rules are also reported corresponding to a fixed decision point. We also show sensitivity and specificity scores from the literature, although these results come from different datasets. HeartPredict achieves better performance compared to algorithms from the state-of-the art on heart failure detection. It is slightly better than the HeartPhone  [9]  algorithm. However, this comparison is provided for information only, since we do not have access to the dataset used in the related studies.
More importantly, HeartPredict surpasses any guideline rule found in the literature, on the same dataset. Indeed, Table  4  shows that for a similar sensitivity (SEN) level (53%), the number of false alarms is decreased by 51% (guidelines: 9. 4 to Heart-Predict: 4. 6 false alarms per patient per year). For similar specificity (SPE) (84%), HF prediction is improved by 19 points of sensitivity (Guidelines: 53% vs. HeartPredict: 72%). That means a gain of 19% of HF prediction. Further analysis in cross-validation setting confirms this observation. Fig. 9  shows sensitivity and specificity variations over 10 cross-validation folds. We observe that HeartPredict outperforms significantly rules from guidelines performance. HF prediction is improved by 13 points of sensitivity in average (Guidelines: 56% vs.
Optimal threshold selection HeartPredict outputs probability scores related to cardiac decompensation. Alarms are generated according to a defined threshold. The ROC curve in Fig. 8  shows that there is an interval in which HeartPredict Fig. 7. Cross-validation scheme -In order to perform a generalisable model selection, we split the dataset into train/test folds by keeping the proportions of the target (stratification), and by isolating patients between train and test. Here, each patient (rows) has multiple time-series (columns). performance is better than rules from guidelines in both sensitivity and specificity. If the decision threshold is selected at the upper bound (0. 69), we have similar sensitivities of between rules and HeartPredict (53%) but HeartPredict specificity is better (94% vs 84%).
g. Deep Learning) needs a downstream post-hoc model to interpret their predictions. HeartPredict is developed with a Random Forests, an ensemble decision trees algorithm considered as "black box" or a "gray box", which predictions could be interpreted using a model-agnostic module. We use two methods for model-agnostic interpretability: LIME  [32]  local surrogate works by taking a point and its black box prediction. A new dataset is generated with local variations around this point of interest. Then, an interpretable algorithm (e. g. decision trees, Fig. 12. HeartPredict prediction distribution on the validation set. The distribution of HF predictions is concentrated towards one, while healthy state predictions are concentrated towards zero. The distribution of surveillance events is skewed towards high prediction scores, suggesting the conservative tendency of HeartPredict.
logistic regression, …) is trained using the generated dataset and their corresponding "black box" predictions used as targets. The resulting interpretable model can then be considered as a good local approximation of the "black box" algorithm at this point of interest. Anchors  [33]  model explanation works by considering some perturbation distribution at the point of interest. Then, it finds some features that are sufficient to anchor the prediction using a rule i. e. a set of predicates. Afterwards, the algorithm verifies if any counterfactual instance is not found by changing the other features. Finally, the resulting predictions on the features represent a rule interpreting locally the "black box" prediction at this point. Some examples of simple rules interpreting HeartPredict predictions using Anchors approach are illustrated in Fig.
Conclusion and future work We introduced HeartPredict, a new machine learning algorithm for early detection of heart failure episodes and related complications. HeartPredict relies on a consistent data set (n = 385), resulting from a large clinical study. The algorithm uses home care data in real patient monitoring setting. Measurements are recorded during 18 months follow-up whilst the medical events (labels) are collected from hospitals and other care centers. We rely on a thorough medical expertise to set the HF prediction methodology. This allowed us to define a consistent stratification of medical events, the prediction horizon window, and the time series observation window. Several tests have been carried out to identify the most relevant sensitive to HF episodes, using patients' weight and symptoms time-series, in addition to sociodemographics.
Our experiments show also that HeartPredict prediction scores are consistent with patient health condition. Indeed, the distributions of prediction scores allow to establish HF risk stratification levels. Such a feature can help care professionals to optimise resources by prioritizing patients with higher risk of HF episode. The main limitation regarding HeartPredict, as any other ML algorithm, is the lack of theoretical guarantees of statistical performance. The performance is defined on empirical data sampled from a theoretical distribution. Another limitation is the difficulty to have a quantitative description of the medical events, that could ensure a more accurate analysis of HF episodes. Also, HeartPredict relies on a study with patients that have more than one month of follow-up. This can lead to a participation bias as only more compliant patients are included in the analysis.
Such bias should be monitored in order to estimate its impact. Future works call for a better integration of model explainability in order to facilitate adoption from the practitioners. This includes developing adapted tools for time series explanations such as shapelet  [34]. In perspective, a prospective pilot with care professionals at Air Liquide Home Care Services is being designed to assess the algorithm. The pilot relies in an evaluation phase in "silent mode", by monitoring the behaviour of the algorithm without using its outputs for medical decision-making. This will help to safely confirm HeartPredict performance. It will also help to collect operational feedback and more data to enhance the model. After running the pilot, it is important to identify and to implement key monitoring indicators in order to detect ML algorithm drift or data distribution deviations.
Purpose: The purpose of this study was to measure the cancer detection rate of computer-aided detection (CAD) software in preoperative automated breast ultrasonography (ABUS) of breast cancer patients and to determine the characteristics associated with false-negative outcomes. Methods: A total of 129 index lesions (median size, 1. 7 cm; interquartile range, 1. 2 to 2. 4 cm) from 129 consecutive patients (mean age±standard deviation, 53. 4±11. 8 years) who underwent preoperative ABUS from December 2017 to February 2018 were assessed. An index lesion was defined as a breast cancer confirmed by ultrasonography (US)-guided core needle biopsy. The detection rate of the index lesions, positive predictive value (PPV), and false-positive rate (FPR) of the CAD software were measured. Subgroup analysis was performed to identify clinical and US findings associated with false-negative outcomes.
Results: The detection rate of the CAD software was 0. 84 (109 of 129; 95% confidence interval, 0. 77 to 0. 90). The PPV and FPR were 0. 41 (221 of 544; 95% CI, 0. 36 to 0. 45) and 0. 45 (174 of 387; 95% CI, 0. 40 to 0. 50), respectively. False-negative outcomes were more frequent in asymptomatic patients (P<0. 001) and were associated with the following US findings: smaller size (P=0. 001), depth in the posterior third (P=0. 002), angular or indistinct margin (P<0. 001), and absence of architectural distortion (P<0. 001). Conclusion: The CAD software showed a promising detection rate of breast cancer. However, radiologists should judge whether CAD software-marked lesions are true-or false-positive lesions, considering its low PPV and high FPR. Moreover, it would be helpful for radiologists to consider the characteristics associated with false-negative outcomes when reading ABUS with CAD.
Several studies have reported that computer-assisted detection (CAD) software may enhance the diagnostic performance of ABUS  [19] [20] [21] [22]. Improvement in the performance of CAD software has been observed with advances in machine learning algorithms  [19]. However, to the best of our knowledge, no previous research has analyzed the shortcomings of ABUS CAD software in identifying breast cancer. Knowledge of the imaging features of breast cancers missed by the CAD software (false-negative cases) is crucial for optimizing the accuracy of diagnoses based on ABUS. The purpose of this study was to measure the cancer detection rate of CAD software in preoperative ABUS of breast cancer patients and to determine the characteristics associated with false-negative outcomes.
ABUS CAD Software All volumetric data of ABUS were loaded onto a workstation dedicated for a deep learning-based commercialized CAD software (QVCAD version 2. 1. 2, QView Medical, Los Altos, CA, USA). The CAD software was applied to all ABUS examinations. The output of the CAD software could be presented in two forms: (1) markers intended to highlight potentially malignant lesions and (2) minimum intensity projection images of the coronal section in areas where CAD software detected abnormalities. We only used CAD markers to evaluate the diagnostic performance of the CAD software (Fig. 2 ). The CAD markers did not display an exact value of the probability of malignancy potential. The number of CAD markers displayed per ABUS volume could be adjusted by changing the values of the false-positive rate (FPR) in the configuration setting of the CAD software.
When searching for an index lesion on ABUS, the radiologists were not allowed to refer to the results of the CAD software. Instead, they were permitted to refer to clinical information and findings from available imaging such as HHUS performed during US-guided core needle biopsy, mammography, and preoperative MRI. After identifying the index lesion, the size, nipple-to-lesion distance, depth, shape, margin, and echo pattern of the lesion, as well as the background tissue echotexture and architectural distortion were recorded. If an index lesion was visualized in multiple ABUS volumes, the largest size of the index lesion and the shortest nipple-to-lesion distance were recorded. To determine the depth of the index lesion, the depth of the breast was divided into three parts: the anterior third, the middle third, and the posterior third of the fibroglandular tissue.
In a lesion with more than 1 layer, the depth was determined by the layer on which the center of the lesion was located. The background tissue echotexture was measured using the method suggested by Kim et al. [24]. Otherwise, the findings on ABUS were assessed based on the Breast Imaging Reporting and Data System (BI-RADS)  [25]. After the image interpretation, the radiologists checked whether the CAD marker correctly pointed to the index lesion on the ABUS volume data in the CAD software-dedicated workstation. For an index lesion visualized in multiple volumes of ABUS, they considered that the CAD marker indicated the index lesion correctly if the CAD marker was placed at the index lesion on at least one ABUS volume.
Data Analysis and Statistical Analysis The detection rate of index lesions of the CAD software on ABUS was measured. The detection rate was defined as the number of index lesions correctly indicated by the CAD marker divided by the total number of breast cancer patients in this study. Additionally, the positive predictive value (PPV) and FPR were measured to evaluate the diagnostic performance of the CAD software. PPV was defined as the number of markers properly located in the index lesion divided by the number of total markers in both non-cancer and cancer volumes. To check whether the FPR measured in our patient group was similar to the preset value of 0. 2 in the CAD software, the definition of the FPR was identical to that in the manual of the CAD software  [23]  (i. e. , the number of false-positive CAD markers in non-cancer volumes divided by the number of non-cancer volumes).
In the present study, we measured the FPR using the ABUS volume data obtained from the contralateral side of breast cancer that was categorized as BI-RADS category 1 or 2 on preoperative ABUS. During the calculation of FPR, contralateral breasts that were classified as BI-RADS category 3 or above were excluded from the analysis. The results of preoperative MRI and US-guided core needle biopsy performed on the contralateral side of the breast were also checked to ensure the absence of breast cancer in the contralateral breast. Subgroup analysis was performed to identify characteristics associated with false-negative outcomes of the CAD software. The Student t test and Fisher exact test were used for continuous and categorical variables, respectively. The tested variables were patients' age, menopausal status, family history of breast cancer, symptoms, histopathology, molecular subtype, and ABUS findings (size, nippleto-lesion distance, depth, shape, margin, and echo pattern of the lesion, background tissue echotexture, and architectural distortion).
The ABUS findings in the true-positive and false-negative groups are listed in Table  2. A smaller size of the index lesion was observed in the false-negative group (mean±SD, 2. 1±1. 1 cm vs. 1. 3±0. 7 cm; P=0. 001). The depth of the lesion was significantly different between the two groups (P=0. 002), as depth in the posterior third was more frequent in the false-negative group (5. 5% [6 of 109] vs. 25. 0% [5 of 20] ). The margin of the index lesion was associated with a lower cancer detection rate using the CAD software (P<0. 001), as angular margins (6. 4% [7 of 109] vs. 35. 0% [7 of 20] ) and indistinct margins (28. 4% [31 of 109] vs. 45. 0% [9 of 20] ) were more frequent in the false-negative group. Absence of architectural distortion was associated with a higher falsenegative rate (24.
Discussion In this study, we measured the cancer detection rate of CAD software in preoperative ABUS of breast cancer patients and analyzed the characteristics associated with false-negative outcomes. The detection rate of index lesions of the CAD software was 0. 84, and its PPV and FPR were 0. 41 and 0. 45, respectively. False-negative outcomes were more frequent in asymptomatic patients and were associated with the following US findings: smaller lesion size on ABUS, depth in the posterior third, an angular or indistinct margin, and absence of architectural distortion. The low PPV and high FPR observed in our study can be explained by the following reasons. First, a heterogeneous background tissue echotexture on ABUS was noted in a considerable portion of our included patients, which may lower the detection rate and increase Ultrasonography 40(1), January 2021 e-ultrasonography.
org the FPR of CAD. Second, Asian women tend to have small and dense breasts. In a previous study performed in women with dense breasts, the PPV of the CAD software-based ABUS reading was 0. 50 (95% CI, 0. 45 to 0. 55)  [26]. Moreover, there is a possibility of artifacts owing to insufficient compression. Third, while we set the FPR of our CAD based on data assuming a screening setting of a Western general population  [23] , our study results reflect the hypothetical screening setting of Asian breast cancer patients. The smaller size of the index lesions led to a decrease in the cancer detection rate of the CAD software. Generally, it is known that small cancers can be overlooked on ABUS used in the screening setting  [27]. A recent study that utilized the same CAD software as ours showed that the cancer detection rate of the ABUS CAD software was associated with tumor size, and the detection rate was around 90% for invasive ductal carcinoma larger than 1 cm  [28].
Furthermore, Kim et al. [29]  reported that a small size of the tumor (<8 mm) was associated with false-negative outcomes of the CAD software, although their CAD software was different from that used in the present study. This trend is maintained even when ABUS is interpreted by radiologists. A previous study reported that the detection rate of malignancy in the ABUS read by radiologists increased in proportion to the size of the lesion  [30]. When the size of the target lesion was larger than 1. 2 cm, radiologists could reliably find the lesion  [13]. Therefore, the results of the present study could reflect the weakness of ABUS itself in visualizing small masses. Absence of architectural distortion was associated with a decreased cancer detection rate of the CAD software. Some articles demonstrated that 3D ABUS helped radiologists detect a spiculated margin of breast cancer or architectural distortion by showing the coronal section of the breasts  [10, 13].
Furthermore, it was suggested that the detection of architectural distortion may contribute to the  a) Two patients were confirmed to have ductal carcinoma in situ on core needle biopsy but they were lost to follow-up before they underwent surgery. Thus, the absence of an invasive component in their breast cancers was not guaranteed due to the lack of a surgical specimen. For that reason, they were classified as having inconclusive histopathology. b) Immunohistochemistry results were missing from four patients. timely detection of breast cancer on ABUS  [27]. Thus, the absence of architectural distortion might lower the diagnostic performance of the CAD software in detecting malignancy on ABUS. Depth in the posterior third was associated with higher falsenegative rate of the CAD software. Lesions located in the deep posterior tissue are inherently difficult to visualize using transducers with a high insonating frequency due to attenuation of the US beam  [31].
Moreover, detection of the masses in the deeper portion of the breast can be hindered by the nipple shadow or posterior shadowing of another lesion in a superficial location. Angular and indistinct margins of the lesion were associated with a decreased cancer detection rate of the CAD software. Subtle noncircumscribed margins, artifacts, and architectural distortions may be difficult to capture on static images of US  [31]. Moreover, an inability to freely adjust the degree of compression or scan angle could be a potential cause of false-negative outcomes on ABUS. Therefore, all available volume scan images must be sufficiently evaluated not only in the axial plane, but also in the coronal and sagittal planes, to overcome the false-negatives of the CAD software. The CAD software used in the present study was based on a deep learning algorithm.