dataesr/scientific-paragraphs-categorization huggingface dataset:
Data This study was part of a larger cross-sectional survey on informal care; details of the research have been described elsewhere  [16]. In brief, data were collected between November 2018 and January 2019 in Hungary, Poland and Slovenia using a web-based questionnaire. Recruitment was carried out and data collected via commercial online panels in each country by a market research and public opinion polling company (Big Data Scientist Ltd. ) via their partnership with these panels. Quotas were applied to ensure the representativeness of the sample by age (up to age 65 years and a sizeable sample above), sex, educational level, and residency. The target sample size was 1000 respondents per country to ensure representativeness of the sample and to involve a reasonable number of participants with some experience with informal care, either as a recipient or a provider.
The DCE design applied the priors from the original survey in the Netherlands  [13, 14]. This provided an efficient experimental design, constructed in Ngene software (ChoiceMetrics, Australia), optimized for D-efficiency. The underlying utility function consisted of two dummy variables per CarerQol-7D domains, interaction terms for all combinations and a constant term for the first alternative in the choice set. The design contained 40 choice sets, which were blocked into four groups (i. e. each respondent was presented with ten choice sets in the survey). See more about the creation of the design in Hoefman et al. [13, 14].
Analyses were performed in Stata 15. 1 (StataCorp, Texas, USA). To illustrate the differences and similarities between domain importance in the different countries, the relative weights of the domains in the total score were plotted (Fig. 1 ) for each country involved in our study and for the reference countries. Moreover, CarerQol-7D utility scores were calculated for six selected caregiving situations and were presented in a radar plot (Fig. 2 ), created in Microsoft Excel. These six caregiving situations were selected to (1) be easy to explain and imagine, (2) cover states from the entire scale from the worst to the best situation, and (3) include some of the most frequent caregiving situations observed in real life (as observed in the caregivers in our dataset). Additionally, CarerQol-7D scores were calculated for current informal caregivers in our samples using Hungarian, Polish, Slovenian and the original Dutch tariffs.
The training dataset includes 24,530 dermoscopic images of seven skin disease categories, which is by far the largest dataset of skin cancer. The performance metrics of six proposed methods are evaluated on a test dataset of 2,453 images. Our proposed EfficientNetB4-CLF model achieves the highest accuracy of 89. 97% and also the highest mean recall of 86. 13% with the smallest recalls' standard deviations of 7. 60%. Compared to the original methods, our proposed solution not only surpasses 4. 65% (86. 13% vs 81. 48%) of mean recalls but also reduces 4. 24% of the recalls' standard deviations (from ±11. 84% to ±7. 60%). This result indicates that our hybrid method is highly effective in training the Deep CNN network on the skin-disease imbalanced dataset. It addresses the problem of slow learning of the minority classes in the networks by combining the data level method of balanced mini-batch logic followed by the real-time image augmentation with the algorithm level method of the newly designed loss function.
, 2019 [9]  introduce skin lesion classification from dermoscopy images using hand-crafted features. While Celebi et al. uses SVM classifier; Barata et al. uses SVM, k-Nearest Neighbor, AdaBoost and Bag of Features as classifiers; Pham et al. compares the performances of six classifiers (SVM, Logistic Regression, Random Forest, AdaBoost, Balanced Bagging, and Balanced Random Forest) in combination with seven hand-crafted features methods and four data preprocessing steps on the two datasets of skin cancer. Recently, deep convolutional neural networks (CNN) have achieved excellent results in image recognition and exceeded human accuracy in some problems with large datasets  [10] -  [12]. Many recent studies have used Deep CNN for the classification of skin lesions  [13] -  [19]  but there are still open challenges due to the data limitation and imbalance problems  [13] ,  [15] ,  [19] -  [22].
There have been many studies using artificial intelligence to classify popular diseases  [23] ,  [24]  and skin-disease  [7] ,  [8] ,  [22] ,  [25] ,  [26] ,  [9] ,  [13] -  [19]. Our analysis of the state of the art reveals important remaining problems: 1) the limited and imbalanced dataset and 2) the imbalance of classification performance among disease classes (especially between Melanoma and Nevus). Specifically, the study  [7]  of  Celebi et al. , 2007  uses only 564 dermoscopy images, and  Barata et al. , 2014 [8]  uses a dataset of only 176 skin lesion images. Besides, many studies such as  [9] ,  [13] ,  [14] ,  [17] ,  [19] ,  [25] -  [28]  are binary melanoma classification while multi-class classification is much more difficult. Recently, 2018 and 2019 ISIC challenges  [4] ,  [29]  have released the largest dermoscopy image datasets and received results from 200 and 64 research teams respectively.
Therefore, deep learning approaches for resolving imbalanced datasets have been applied, such as 1) Data level methods, 2) Algorithm level methods, and 3) Hybrid methods. A. DATA LEVEL METHODS 1) OVERSAMPLING this method randomly augments samples from minority classes until their proportions are equal to the majority class. The augmentation is repeated until no class has smaller samples than the largest one. This is when the balance achieved. The impact of imbalanced training datasets on CNN performance in image classification is explored by Hensman and Masko  [32]. They train neural networks on the imbalanced dataset created from the CIFAR-10 dataset and suggest that oversampling is an effective approach to deal with the impact of imbalances in the training data. 2) UNDER-SAMPLING this method randomly removes samples from majority classes until they weigh equally to the minority class.
After that, the selected images are augmented by the Real-time Image Augmentation component and then used in neural networks training. In this system, we design fully connected layers with two hidden layers to increase the learning ability of the network for the skin-disease dataset. In these layers, we also use batch normalization and dropout layers to improve the efficiency of our solution. This network is trained with the new custom loss function of the seven classes detailed below. In addition, during network training, we do not fix the learning rate but update it with CyclicLR  [40]. In this hybrid method, we combine an algorithm level method of designing new loss function and a data level method of balanced mini-batch logic integrated with a real-time image augmentation for dealing with a class imbalance of skin lesion dataset.
B. CNN AND CUSTOM FULL CONNECTED LAYERS 1) CNN Popular CNNs are used as feature extractors. We investigate outstanding CNN architectures, such as InceptionV3  [41] , ResNet50  [42] , DenseNet169  [43] , and Efficient-NetB4  [44]  and selected two best performance architectures (DenseNet169 and EfficientNetB4) to evaluate and analyze the efficiency of our multiple skin-disease classification system. DenseNet169 is also the architecture used by Pham et al. [13]  to achieve excellent performance in melanoma binary classification. In this study, the CNNs are transferred from the network trained by ImageNet, and the output of last layers (named relu with DenseNet169 and top_conv with EfficientNetB4) is passed as input features of fully connected layers. Because the skin-disease dataset has distinctive features from ImageNet, we retrain all CNN layers and our new custom fully connected layers.
Specifically, when the loss decreases, the performance must increase. In classification, performance is usually measured with ACC, but for classification with an imbalanced dataset and medical applications such as the multiple skin-cancer classification problem, performance is evaluated by the mean recall (mRecall). This classification is indeed an imbalance problem between illness or not (for example, Table  4  NV accounts for 52. 22% while AK is only 3. 47%). Thus, the performance of a diagnosis is not the percentage of correctly diagnosed people from the total number of people examined but instead is the combination of the two ratios: a) the percentage of people correctly diagnosed as being ill and b) the percentage of people correctly diagnosed as not being ill. For a multi-class problem, the combination of these ratios is mean recall.
Corresponding to the true label y i,c , the predicted value y * i,c is a real number with a value in the range [0: 1]. We demonstrate the loss function on the subset of the c th class which has P samples in the formula (2) based on the y i,c and y * i,c. Finally, we propose the formula (3) as the l CLF of the whole dataset which has C classes. In this formula, l i is the loss of the i th class which is calculated by the formula (2) mentioned above, and coefficient a i is the adjusting value of the i th class. To understand better why we use this l CLF function, we demonstrate three examples of a 100-images dataset includes three classes of M, N, and A (26, 70, and 4 images respectively). The value of l CLF is calculated with a i = 1 for all three classes M, N, and A. The Accuracy (ACC), mean Recall (mRecall), l MSE , and l CLF of three examples are shown in Table  1.
From Table  1 , it can be seen that from Example 1 to Example 3, mRecall of l CLF decreases accordingly, while that of l MSE increases together with ACC. This indicates that l CLF is the better function to optimize mean recall in multiclass classification. To be more specific, Example 3 perfectly reflects what we aim at since it has the lowest loss value (0. 0326) and the highest mRecall (84. 3%). In this study, we propose l CLF for seven classes of the skindisease dataset for handling class imbalance. This loss is calculated by the formula (3). The a i coefficients are adjustable parameters that must be tuned in order to obtain a model with optimal performance. In this study with limited GPU resources, we used the Trial and error method to find parameters for high performance in order to demonstrate that our proposed loss function be able to increase the learning ability of the network for the imbalanced skin diseases dataset.
D. BALANCED MINI-BATCH LOGIC Deep learning requires a large training dataset to achieve high performance, but the optimizer's calculation over the entire dataset is complex and resource consuming. Therefore, to speed up the training process, Deep CNNs split the training dataset into smaller subsets or batches. The splitting is done by applying a specific logic called batch logic. Samples of each batch are randomly selected resulting in a change in the distribution of the classes in each batch, especially when compared to their distribution in the training set. Normal balanced mini-batch logic fixes the NIC and NIC is usually the same in all mini-batches of all epochs. But there are differences in our Mini-Batch Logic, which are: 1) The NIC varies between mini-batches and between epochs. 2) In each mini-batch, the NIC is not fixed and is selected in the range of predefined MIN, MAX in Table  3.
A. MATERIALS In this study, we use the HAM10000 dataset of 10,000 images belonging to 7 categories  [29] ,  [48] : Melanoma (MEL), Melanocytic nevus (NV), Basal cell carcinoma (BCC), Actinic keratosis (AK), Benign keratosis (BKL), Dermatofibroma (DF), and Vascular lesion (VASC). Additionally, we add more images of these seven categories from ISIC 2019  [4] ,  [48] -  [50]  dataset, and remove duplicate content images. Finally, there are 24,530 images  [4] ,  [29] ,  [48] -  [50]  for this study. These images are center cropped and resized to the size of 256 × 192px. Then, they are randomly divided into train, validation, and test sets (namely Train-80, Validation-10 and Test-10 sets respectively) with the corresponding ratios of 80%, 10%, and 10% of the total images. The splitting must ensure randomness while keeping the same proportion of classes as in the original dataset.
The summary of all datasets is shown in Table  4  below. Although this dataset comes with a large number of images (24,530), its distribution among classes is strongly skewed. NV accounts for more than 52% of the data and is the most popular skin lesion image. Meanwhile, the second most common MEL only accounts for 18% of total images, which is a little over one-third of NV. VASC, DF and AK are the least with only 1. 03%, 0. 97%, and 3. 47% of images respectively.
Accuracy = The number of correct samples The number of all samples (4) Recall = TP TP + FN (5) Precision = TP TP + FP (6) stdev = N i=1 (x i -x) 2 N , with x = N i=1 x i N ( 7 ) In these formulas, recall and precision are metrics applied at each output class. For example, with Class A, TP (true positive) represents the number of cases correctly identified as Class A; FP (false positive) represents the number of cases incorrectly identified as Class A; and FN (false negative) represents the number of cases incorrectly identified as Not Class A. Results are analyzed focusing on these criteria: 1) the trend of ACC and mRecall during training and 2) the performance over the Test-10 dataset and compare with TOP-1 and TOP-7 in terms of mRecall and standard deviation.
C. THE TREND OF ACC AND MRECALL DURING TRAINING In the research of Anand et al. [31]  in the 1990s, it is indicated that when training with imbalanced data, the minority classes learn much slower than the majority ones. In this study, we analyze mRecall and ACC values during training instead of the loss values. As described in formula (  1 ) and (  2 ) in the previous section, ACC is the rate of correct samples out of all samples, while mRecall is the mean of recalls over classes in our dataset. In the most ideal and balanced case, the recall of each class is the same, mRecall will be equal to ACC. For detail, we can conclude that: • When mRecall equals to ACC, the training achieves the best optimization, or the minority classes are learned faster than the majority classes by the networks.
• When mRecall is much smaller than ACC, the recall performance on each class has a much difference and imbalance, or the minority classes are still learned slower than the majority classes by the networks. To The left graph demonstrates the models' performance in the training dataset. In general, during training, both BON's and CLF's ACC and mRecall are almost the same and there is no deviation. This is because the two ACC's and mRecall's lines of DenseNet169 and EfficientNetB4 are almost identical in all epochs from 1 to 100 in both left and right figures. In contrast, with ORI, these two architectures have ACC greater than mRecall at the initial epochs, right from the epoch of 20th (with DenseNet169) and 40th (with Effi-cientNetB4), these two measures are similar. This demonstrates that BON (customized balanced batch logic) and CLF (customized balanced batch logic combined with custom loss function) are very effective in training the network for optimal mRecall, better than ORI with both architectures.
The right graph displays the performance of the validation dataset. We can see that in both architectures combined with three scenarios, ACC tends to be higher than mRecall. Besides, the difference between ACC and mRecall of Effi-cientNetB4 is smaller than that of DenseNet169 except for ORI of EfficientNetB4. This proves that EfficientNetB4's combined with BON and CLF methods achieve more balance on the validation dataset than that of DenseNet169. Of both DenseNet169 and EfficieneNetB4 architectures, ACC and mRecall of BON and CLF are almost equal, while ORI's is different in the initial epochs of training. This proves that BON and CLF methods help the networks learn minority classes faster on the imbalanced skin dataset. About mRecall on the validation dataset, with DenseNet169 architecture, Figure  2  shows that from epoch 1 to 55, CLF's mRecall is the best, and ORI's is the worst.
While from epoch 56 to 100, this value of BON is better than that of CLF, and ORI's is still the worst. As for EfficientNetB4, among all epochs, BON's mRecall is the best and ORI's is still the worst. This suggests that the two proposed solutions, BON and CLF, improve the mRecall efficiency of the deep neural network significantly. Finally, to compare DenseNet169 with EfficientNetB4 architectures, both provide good results when trained with BON and CLF, meanwhile, DenseNet169 outperforms Effi-cientNetB64 in the ORI scenario. However, on the validation dataset, EfficientNetB4 is not only more efficient in terms of ACC and mRecall values, but also has a better deviation of ACC and mRecall (the smaller the better).
D. EVALUATION PERFORMANCE OVER THE TEST-10 DATASET In this study, we evaluate the proposed multiple skindisease classification system by using the Test-10 dataset of  Overall, we observe a considerable increase of mean recall from both architectures in models that applied batch logic and loss function changes. With the ORI scenario, the mean recalls achieve 81. 22% (DenseNet169) and 81. 48% (Effi-cientNetB4), whereas with CLF, this metric reaches 84. 45% (DenseNet169) and 86. 13% (EfficientNetB4). Among models of two architectures, AK and DF are categories that come with the lowest recalls, especially in the ORI scenario. However, these values are raised in both BON and CLF. Specifically, AK and DF's recall values grow from 67. 1% and 62. 5% in ORI models to 65. 9% and 79. 2% in BON, then reach 75.
In addition, our solution has halved the number of MEL that is misclassified to BKL, another benign type. To summarize, our proposed system contributes a medically meaningful solution to multiple skindisease classification. Another important summary of the six models we should look at is presented in Table  5. It compares the performance of models in terms of accuracy, mean recall and mean precision. In general, mRecall and mPrecision are increased from ORI, BON to CLF methods, except for the DenseNet169 architecture mPrecs. It is noticed that mRecalls produced by CLF are the highest in both DenseNet169 (84. 45%) and Effi-cientNetB4 (86. 13%) architectures. The table also includes standard deviation values (stdev), which represent how spread out the values are in the data set. The higher the standard deviation, the wider its values spread out from the mean.
13% vs 83. 36% of TOP-1 and 78. 93% of TOP-7) and with smaller stdev (7. 60% vs 9. 76% of TOP-1 and 9. 35% of TOP-7). In addition, to evaluate the effectiveness of the solutions, we compare the recall of the two majority classes (MEL and NV have 70. 05% images in all Train-10, Validation-10, and Test-10 datasets) in two criteria: 1) the average recall of MEL and NV (namely AVG-MEL-NV), and 2) deviation recall between MEL and NV (namely DEV-MEL-NV). The higher the AVG-MEL-NV, the better the solution, whereas the smaller DEV-MEL-NV the better the solution. Our best solution of EfficieneNetB4-CLF has an AVG-MEL-NV of 88. 4%, which is the highest and is greater than TOP-1 and TOP-7 of 10. 95% and 8. 2% respectively. The CLF solution is also more balanced. Its DEV-MEL-NV is only 13. 8%, smaller 16.
V. CONCLUSION In this study, we have proposed a new approach for multiple skin-disease classification by proposing a hybrid method, which combines designing new loss function with a data level method of balanced mini-batch logic followed by a real-time image augmentation. The major results of this research are listed below. 1) Our proposed hybrid method, which combines the algorithm level method of new designed loss function and the data level method of balanced mini-batch logic integrated with the real-time image augmentation, is effective in handling class effectiveness of networks optimization on the imbalanced dataset because it helps the networks learn the minority classes faster. Our solution is superior to improve the balance of recalls among classes and improve the overall performance significantly.
The proposed Deep CNN system is suitable for multiple skin-disease classification and with the combination of EfficientNetB4-CLF achieved highest ACC at 89. 97% and mRecall at 86. 13% on Test-10 dataset of 2,453 dermoscopic images. 2) The combination of balanced mini-batch logic and real-time image augmentation is effective in training the networks with imbalanced skin dataset, which  increases the performance on both of these networks: DenseNet169 and EfficieneNetB4. 3) Compared to ORI and BON, CLF is efficient in increasing mRecall when applied in both CNN architectures. In addition, by reducing stdev, CLF has proved to improve the learning effectiveness of the minority classes on an imbalanced dataset. 4) Our best EfficientNetB4-CLF solution not only increased mRecall by 4. 65% (86. 13% vs 81.
48%) but also reduced 4. 24% on the standard deviation of recalls (from 11. 84% to ±7. 60%) compared to the original method. This study shows that our hybrid method is a very important direction for high-performance Deep CNN architectures to classify imbalanced medical images. We plan to investigate more intensively on the loss function of this solution so that it could apply not only to multiple skin-disease classification but also to other medical image analysis and common imbalanced datasets.
[13] ,Pham et al. propose a new custom loss function based on the errors of each class. Their solution achieves the strongest result on a 100 images test dataset and outperforms every dermatologists working in German hospitals and universities. In this study, we design a new loss function based on the mean squared error of each class. A similar idea was proposed by author Wang et al. [34]  in 2018, evaluated with the CIFAR-100 dataset which contains 60,000 images belonging to 100 classes. Unlike other works, so as to adjust the balance of multi-class recall, we add a coefficient a i to our loss function to adjust the accuracy according to i th class. These below formulas explain this for:(1) mean squared error (namely l MSE ); (2) loss on the subset of the c th class of P samples (namely l c ); and (3) our custom loss function on the full dataset of C classes (namely l CLF ):.
define whether the proposed solution resolves the problem of training Deep CNN on imbalanced data, we analyze the trend of ACC and mRecall during the model training process. We built a custom metric in Keras to calculate mRecall during training and validating at the end of each epoch. The maximum mRecall on the Validation-10 dataset model is saved as the best final model. The trend of ACC and mRecall of the six methods are demonstrated in Figure 2, DenseNet169's, and EfficientNetB4's methods are shown on the left and right, respectively.
453 skin lesion images (448 melanoma, 1,281 melanocytic nevus, 331 basal cell carcinoma, 85 actinic keratosis, 259 benign keratosis, 24 dermatofibroma, and 25 vascular images). This Test-10 dataset is bigger than the ISIC 2018 test dataset of 1,512 medical images. The performances of six combined proposed methods are shown in Figure 3.
2 , 2 453 skin lesion images (448 melanoma, 1,281 melanocytic nevus, 331 basal cell carcinoma, 85 actinic keratosis, 259 benign keratosis, 24 dermatofibroma, and 25 vascular images). This Test-10 dataset is bigger than the ISIC 2018 test dataset of 1,512 medical images. The performances of six combined proposed methods are shown in Figure 3.