New fully automatic approach for tissue identification in histopathological examinations using transfer learning

IET Image ProcessingVolume 16, Issue 11 p. 2875-2889 ORIGINAL RESEARCHOpen Access New fully automatic approach for tissue identification in histopathological examinations using transfer learning Yongzhao Xu, Yongzhao Xu Department of Physics and Optoelectronic Engineering, Dongguan University of Technology, Dongguan, ChinaSearch for more papers by this authorMatheus A. dos Santos, Matheus A. dos Santos orcid.org/0000-0003-3854-7178 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorLuís Fabrício F. Souza, Luís Fabrício F. Souza orcid.org/0000-0002-3156-1359 Federal University of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorAdriell G. Marques, Adriell G. Marques orcid.org/0000-0001-6274-1961 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorLijuan Zhang, Corresponding Author Lijuan Zhang [email protected] orcid.org/0000-0002-8924-2449 DGUT-CNAM Institute, Dongguan University of Technology, Dongguan, China Correspondence Lijuan Zhang, DGUT-CNAM Institute, Dongguan University of Technology, China. Email: [email protected]Search for more papers by this authorJosé Jerovane da Costa Nascimento, José Jerovane da Costa Nascimento orcid.org/0000-0002-7361-4316 Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorVictor Hugo C. de Albuquerque, Victor Hugo C. de Albuquerque orcid.org/0000-0003-3886-4309 Federal University of Ceará, Ceará, BrazilSearch for more papers by this authorPedro P. Rebouças Filho, Pedro P. Rebouças Filho orcid.org/0000-0002-1878-5489 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this author Yongzhao Xu, Yongzhao Xu Department of Physics and Optoelectronic Engineering, Dongguan University of Technology, Dongguan, ChinaSearch for more papers by this authorMatheus A. dos Santos, Matheus A. dos Santos orcid.org/0000-0003-3854-7178 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorLuís Fabrício F. Souza, Luís Fabrício F. Souza orcid.org/0000-0002-3156-1359 Federal University of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorAdriell G. Marques, Adriell G. Marques orcid.org/0000-0001-6274-1961 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorLijuan Zhang, Corresponding Author Lijuan Zhang [email protected] orcid.org/0000-0002-8924-2449 DGUT-CNAM Institute, Dongguan University of Technology, Dongguan, China Correspondence Lijuan Zhang, DGUT-CNAM Institute, Dongguan University of Technology, China. Email: [email protected]Search for more papers by this authorJosé Jerovane da Costa Nascimento, José Jerovane da Costa Nascimento orcid.org/0000-0002-7361-4316 Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this authorVictor Hugo C. de Albuquerque, Victor Hugo C. de Albuquerque orcid.org/0000-0003-3886-4309 Federal University of Ceará, Ceará, BrazilSearch for more papers by this authorPedro P. Rebouças Filho, Pedro P. Rebouças Filho orcid.org/0000-0002-1878-5489 Federal Institute of Education, Science and Technology of Ceará, Ceará, Brazil Laboratory for Processing Images, Signals and Computer Science - LAPISCO, Ceará, BrazilSearch for more papers by this author First published: 09 March 2022 https://doi.org/10.1049/ipr2.12449AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The use of computational techniques in the processing of histopathological images allows the study of the structural organization of tissues and their changes through diseases. This study aims to develop a tool for classifying histopathological images from breast lesions in the benign and malignant classes through magnification scales by an innovative way of using transfer learning techniques combined with machine learning methods and deep learning. The BreakHis dataset was used in the experiments, consisting of histopathological images of breast cancer with different tumor enlargement scales classified as Malignant or Benign. In this study, various combinations of Extractor-Classifiers were performed, thus seeking to compare the best model. Among the results achieved, the best Extractor-Classifier set formed was CNN DenseNet201, acting as an extractor, with the SVM RBF classifier, obtaining accuracy of 95.39% and precision of 95.43% for the 200X magnification factor. Different models were generated, compared to each other, and validated based on methods in the literature to validate the experiments, thus showing the effectiveness of the proposed model. The proposed method obtained satisfactory results, reaching results in the state-of-the-art for the multi-classification of subclasses from the different scale factors found in the BreakHis dataset and obtaining better results in the classification time. 1 INTRODUCTION Breast cancer is the most common cause of cancer death among women aged 40 to 45 years and being the main factor of mortality in females [1]. According to the World Health Organization (WHO), in 2020, about 2.3 million women were diagnosed with breast cancer. By the end of the same year, about 7.8 million women lived after receiving the diagnosis. Breast cancer is considered the leading cause of female mortality worldwide, accounting for 15% of total female mortality [2]. Finally, in 2020, there were about 685,000 deaths worldwide; these data affirm breast cancer as the most prevalent in the world [3]. In the United States of America, it is estimated that about 255,000 new cases of invasive cancer occurred in 2017, with the overwhelming majority of cases detected in women, with about 252,000 patients. Many of these cases could have been treatable if the disease had been diagnosed early. [4]. From a financial point of view, American health insurers estimate an annual expense of US$29,044 per patient with direct health charges related to breast cancer and its sequelae, such as hospitalizations to control systemic infections, hospitalization costs, transport, medicines etc. A patient with this type of pathology spends about the US$ 2,300 a year directly, and about the US$ 3,325 to US$ 5,545 with indirect costs due to the medical expenses of the disease [5]. Even with several existing techniques in medicine for the detection of breast cancer, the Digital Mammography [6] and the MRI of the breast [7] are examples of this vast inventory. An open surgical biopsy (OSB) is the most recommended exam when there is a need for an accurate diagnosis for surgical preparation, especially when it is not possible to determine the cause of the tumor with a needle biopsy [8] or other techniques. This examination is performed through a cut in the breast, in which the protuberance is removed in part or its entirety so that the tumor can be seen under a microscope. The OSB obtains practically all the suspicious injuries in its procedure with a high rate of reliability. Then, this exam is a technique considered adequate for defining the entire pathology in cases where the mass is small and complex to locate by touch or if an area looks suspicious on a mammogram but cannot be felt [9]. Histopathological images are, by definition, microscopic images captured from tissue damaged by the disease. Thus it is possible to diagnose with more excellent safety and precision not only the type of pathology that affects the tissue but its degree of evolution and severity[10]. In the case of cancer, histopathological images help to classify it as to its class, whether the tumor is benign or malignant [11]. The microscope images provided by the OSB exam are histopathological images [12]; after obtaining this type of image, as already mentioned, it is necessary to classify the tumor as malignant or benign. Usually, this the diagnostic process is manual, performed by the specialist physician, In which, based on his experience, he will detect specific characteristics by the exam image that help to classify the tumor as Malignant or Benign [13]. However, manual diagnosis is subject to a series of limiting factors, such as the long delay for the final result and errors caused by the operator, whether due to lack of experience of a professional at the beginning of their career or due to high workloads, even difficulties in diagnosing an exam in abnormal conditions of exam acquisition. [14]. So, taking into account the limiting factors of manual diagnosis, especially about the waiting time for results, many Computer Aided Diagnosis (CAD) methods have been proposed over the years to make this type of classification and several other medical applications faster and more accurate. These methods are mainly based on digital image processing [15], machine learning [16], and deep learning [17], which are characterized by using artificial intelligence, or assisting, in human activities in an intuitive way and need [18]. For the histopathological classification of cancer, the CAD system must classify the pathology between benign and malignant based on the attributes extracted from the exam image to train the CAD system, the help of the specialist physician is necessary: The training of the CAD system is carried out through the use of datasets previously classified by the expert physician, in which the images are already properly labeled, leaving the system to understand which characteristics delimit the classes [19]. With the growing need for datasets for training these CAD tools, numerous institutions and research groups have been concentrating their efforts not only on the development of such CAD tools but also on the construction of large datasets made publicly available through events and challenges[20]. In this context, The Breast Cancer Histopathological Image Classification (BreakHis) emerged, consisting of about 9,109 microscopic images of breast tumor tissue using different magnifying factors, 2,480 of which are benign and 5.429 malignant samples, thus, due to the large number of images previously classified as to the type of tumor by specialist physicians, training CAD systems based on deep learning with BreakHis becomes a more manageable task [21]. As examples of CAD systems, works such as Sharma et al. [22] proposed two classification methods based on machine learning and deep learning algorithms for the set of the multiclass present in the BreakHis [21] dataset. The first proposal is based on handcrafted resources extracted using Hu Moments, color histogram, and Haralick textures for later classification of these attributes to be classified by conventional classifiers. The second proposal is based on VGG16, VGG19, and ResNet50 networks to extract and classify features. The results were satisfactory, especially for convolutional neural networks, reaching an accuracy of 93.97%. However, it is noteworthy that the authors had difficulties in multi-classification due to variations in the images, with the appearance and resolution of these classes. Aiming better results in multi-classification, the study by Murtaza et al. [23] sought to develop a tree-based breast cancer multi-classification model via Deep Learning for the extraction of discriminative characteristics; for this, the BreakHis dataset was used to train this algorithm. In their results, the authors obtained a minimum precision of 87.50% for the breast cancer subtypes of the dataset, thus proving the method's effectiveness. However, for some subclasses, the algorithm had difficulty classifying, switching the labels to carcinoma classifications. Taking into account the importance of CAD systems for quickly and efficiently classification of pathologies, and based on different techniques in the literature with the same purpose for the same dataset, the proposed method seeks to classify the classes of histopathologies of breast cancer, especially the classification between benign or malignant classes through the use of machine and deep Learning algorithms. As contributions, this study presents significantly better results than the methods found in the current literature, besides having a quick classification time. We show excellent results for classification according to the BreakHis dataset magnification factor and for generalized classification of the dataset, that is, with all scales at once. So, Aware of the high female mortality of breast cancer, the high financial cost for treatment, medication, and hospitalization, Aware of the scope of operation of a CAD system for the histopathological classification of breast cancer images. Motivated by the facilitation of training a Deep Learning network with the BreakHis dataset and the challenges in the multi-classification of different subclasses from the scales used in the dataset. This study aims to develop a tool for classifying histopathological images from breast Lesions in the benign and malignant classes through magnification scales by an innovative way the use of transfer learning techniques combined with machine learning methods and deep learning. Given this context, this study addresses the following main contributions: The proposal of an innovative method for the classification of histopathological images from breast lesions in the benign and malignant classes, both through magnification scales. The study brings in an innovative way the use of Transfer learning techniques combined with machine learning methods and the use of deep learning. The study proposes the generation of different computational models of deep extractors combined with machine learning classifiers to obtain the best combination for the classification of histopathological tissue images. The method aims to aid efficient medical diagnosis, have an innovative point an effective pre-prognosis, and enable a possible treatment in an agile and efficient way. 2 RELATED WORKS The use of deep learning has gained strength in recent years, when applied as an approach to problems involving machine learning, such as object recognition and classification [24]. A convolutional neural network (CNN) is composed of convolutional layers (used to generate attributes), layers of pooling (used to join information from a region) and layers called fully-connected, used in classification. In some cases, to improve the performance of these systems, pré-processing techniques are used on the image [25, 26]. Usually, any application involving Deep Learning initially starts with its training for the proposed problem, that is, the adequacy of the tool for its scope. This training process is carried out through the use of Datasets consisting of a large amount of previously labeled samples with the classes you want to train the network to classify [27]. For medical applications, the datasets used to consist of large amounts of medical exams previously organized by medical specialists [28]. The transfer learning technique uses convolutional neural networks and pooling layers, while replacing the fully-connected layers with traditional classifiers (such as KNN, SVM etc.) [29]. In this way, the convolutional neural network functions as a feature extractor[30]. Histopathological images (H&E) have been used to detect different pathologies, such as breast cancer, colorectal cancer, and lung cancer. The various uses of this type of medical image demonstrate the vast scope for improvements in histological exams and the importance of contrasting these approaches [31]. In the studies by Celik et al. [32], there was an investigation regarding the automatic detection of invasive ductal carcinoma (IDC), which is the most common subtype of breast cancer, using the technique deep transfer, especially with the aid of the ResNet-50 and DenseNet-161 extractors for IDC detection. The method developed by the authors was applied to the BreaKHis image data set for classification between benign and malignant tumors of the breast, obtaining mean accuracy of 88% for the classification of the whole set, without separation by scale. However, it is worth mentioning that the dataset has been balanced for better performance, which may compromise the method's generalizability. The work of Zhi et al. [33] investigates the use of transfer learning with Convolutional Neural Networks to automatically diagnose breast cancer in spots of histopathological images provided by BreakHis. The authors Combined transfer learning with CNN VGGNet in a more superficial custom architecture. In addition, as far as classification is concerned, it was separated into scales the dataset, and the two main classes were considered malignant and benign. The method obtained satisfactory results, with metrics superior to the same model of CNN trained without transfer Learning, with an accuracy of 94% for the classification on the scale of 200%. However, the comparison between the proposed model and other methods found in the literature for the BreakHis dataset was based only on accuracy, which may compromise the reliability of such a comparison, such as the absence of observation of other points of the method, such as its false detection positives and false negatives. Also, taking into account the advantages of this approach, and the same dataset, Mehra et al. [34] had Transfer Learning compared to networks fully trained in the histopathological imaging modality. It analyzed three pre-trained networks: VGG16, VGG19, and ResNet50. Was observed the behavior of the networks to enlarge the image scale, the classification took into account the two main types of cancer. The method obtained 92% accuracy and 95% ROC, proving the effectiveness of using Transfer Learning over networks. However, the authors balanced the dataset for better performance without specifying this procedure in more detail, which makes the method's performance less reliable. In addition, classification was not performed by scale. Song et al. [35] presented an approach based on transfer of learning for the classification of histopathological images. In addition, we used the Fisher Image Coding Vector (FV) resource of local characteristics, extracted using the model of Convolutional Neural Network (CNN) pre-trained on ImageNet. Again, it uses BreaKHis the imaging data set to classify benign and malignant breast tumors. The authors used the only accuracy as an evaluation metric, which precludes a broader comparison with state-of-the-art works, and only performed the classification by scale. Deniz et al. [36] guided the theme focused on breast cancer in which the transfer of learning and methods of extracting characteristics are used to adapt a pre-trained CNN model to the problem in question. The AlexNet and Vgg16 models are used for resource extraction. The attributes obtained are classified by support vector machines (SVM). The data set used the BreakHis, being divided between the image scales for the malignant and benign classes. The method reached 95% accuracy with the SVM RBF classifier in the 200X scale. However, the classification was not carried out without distinction by the scale factor, which could prove the ability to generalize the method. Aware of the challenge of classification regarding subclasses arising from the scaling factor of the BreakHis dataset, Boumaraf et al. [37] carried out an approach based on the transfer of learning for the automated classification of breast cancer from histopathological images constituted in the BreakHis dataset. The CNN ResNet-18, pre-trained by ImageNet, was used for the category, with its results improved by the fine adjustment. The authors obtained 92% accuracy in the multi-scale classification, being limited by the fine-tuning techniques, which could not enhance the classification for some subclasses arising from the different scale factors. Liew et al. [38] proposed a technique titled Deep Learning and eXtreme Gradient Boosting (DLXGB) on breast cancer histopathological images using the BreaKHis dataset; the authors applied preprocessing on the dataset, which increased the performance of the network in the binary and multi-scale classification, thus obtaining an accuracy of 97% in both. However, the authors did not carry out a study about the classification time of the entire process, which compromises the applicability of the technique for applications as a CAD tool. So, considering the importance of a CAD system for the classification of histopathologies quickly and efficiently, based on different techniques in the literature with the same purpose. In addition to the BreakHis dataset, the proposed method seeks to classify the other classes of histopathologies of breast cancer, especially the classification between benign or malignant, through Machine and Deep Learning algorithms. As contributions, this study presents significantly better results than the methods found in the current literature:The proposed method obtained satisfactory results, reaching results in the state-of-the-art for the multi-classification of subclasses from the different scale factors found in the BreakHis dataset and obtaining better results in the classification time. 3 MATERIALS AND METHODS In this section, the methods used in the experiments are presented in different combinations of ways through the transfer learning technique for the generation of other computational models to classify tissues in histopathological exams. In this section, the methods used in the experiments are presented, in the different combinations of algorithms, using the transfer learning technique to generate diverse computational models to classify tissues in histopathological exams and the evaluation metrics used in the experiments. Attribute extraction through convolutional neural networks The Transfer Learning method consists of using a convolutional neural network (CNN), pre-trained with an extensive database, of extracting shape and texture characteristics from another dataset, to remove the attributes of the images used in the classification step, using traditional classifiers [39]. In other words, to be used exclusively as a feature extractor, CNN lacks its classification layers, called fully connected layers [40, 41]. Thus, the network does not need to be trained, unlike what happens with other methods such as fine-tuning or learning from scratch, which seeks to train the network with the dataset in question [42]. In this work, we used CNN's topologies with weights initially trained with the large ImageNet [43] image bank, which consists of millions of images of everyday objects in 1000 different classes. The fully connected layers were removed for all of them, with the output of each network being a vector resulting from the last convolutional or pooling layer. The CNN's architectures used in this paper were two: the VGG [44] architecture and the DenseNet [45] architecture. The first was implemented in two different configurations (VGG16 and VGG19), while the second was implemented in three (DenseNet121, DenseNet169, and DenseNet201). The VGG16 and VGG19 configurations differ by their number of weights, with 16 layers and the second with 19 layers. This architecture is differentiated by using small convolutional filters, which increases its depth power [44]. In parallel, the three different DenseNet configurations are differentiated by the different number of layers that compose them, being formed by convolutional, transition layers and by the so-called DenseBlocks. Its main characteristic is its dense connections between the layers, feeding a system of solid propagation of attributes to the subsequent layers and its reuse, requiring few parameters [45]. Classification using machine learning techniques After the feature extraction stage, the attribute vectors extracted by the topologies presented in the previous section have different sizes, depending on the network, and are provided to the seven classifiers used in this work. They are Naive Bayes [46], MLP [47], Nearest Neighbors [48], Random Forest [49], and three different versions of SVM (Linear, Polynomial, and RBF) [50]. The first classifier used, Naive Bayes, is an algorithm that makes a statistical analysis of the vector of attributes, based on the Bayes Decision Rule, on conditional analysis, and the probability density function. The method calculates a probability value for a sample to belong to each of the classes in question at the end labels, it with the most likely class [46]. The MLP classifier (Multi-layer Perceptron) is an algorithm composed of several layers of the artificial neuron, called perceptron [51]. Between the input, which receives the attributes, and the output of the MLP, there are several layers of perceptrons with different weights that propagate the initial information throughout its length, learning from the values provided to it to predict the sample class in its output [47]. Nearest Neighbors, or KNN, is a supervised machine learning method that classifies a sample through its spatial distribution with the others already labeled [48]. It is based on its k parameter, which must be odd and is the number of neighbors closest to the current sample. Therefore, the new sample receives the most frequent label among the k neighbors. Based on the decision trees and the form of classification of the human brain, we have the Random Forest [49] algorithm. This method is considered unsupervised and has a random startup, using estimators to handle the input information. The Support Vector Machine (SVM), in turn, are classification methods that use statistical analysis and optimal separation hyperplanes that depend directly on the kernel used to analyze the spatial layout of the samples [50]. The kernels used for SVM's were Linear, Polynomial, and RBF (Radial Basis Function). Evaluation metrics The metrics used to assess the classification of each combination were: accuracy (Acc), precision, recall or sensitivity, F1-Score (F1), and Matthews Correlation Coefficient (MCC) [31]. All equations use true positives (VP), false negatives (FN), true negatives (VN), and false positives (FP), all present in the confusion matrix to calculate the evaluation metrics that can be seen in Equations (1)–(5). A confusion matrix is a tool that compares the actual class of each classified sample and the class predicted by the method. A c c = V P + V N V P + V N + F P + F N , \begin{equation} \hspace*{-7.5pc}Acc=\frac{ VP+VN }{ VP+VN+FP+FN }, \end{equation} (1) P r e c i s i o n = V P V P + F P , \begin{equation} \hspace*{-13pc}Precision=\frac{ VP }{ VP+FP }, \end{equation} (2) R e c a l l = V P V P + F N , \begin{equation} \hspace*{-12pc}Recall=\frac{ VP }{ VP+FN } , \end{equation} (3) F 1 = 2 V P 2 V P + F P + F N = 2 × P r e c i s i o n × R e c a l l P r e c i s i o n + R e c a l l , \begin{equation} \hspace*{-2pc}F1=\frac{ 2VP }{ 2VP+FP+FN } = 2 \times \frac{Precision \times Recall}{Precision + Recall}, \end{equation} (4) M C C = V P × V N − F P × F N ( V P + F P ) ( V P + F N ) ( V N + F P ) ( V N + F N ) \begin{eqnarray} MCC=\frac{VP\times VN - FP \times FN}{\sqrt {(VP+FP)(VP+FN)(VN+FP)(VN+FN)}}\nonumber\\ \end{eqnarray} (5) Accuracy (ACC) is a metric that measures the number of samples correctly classified by the total number of samples, and since it deals directly with the method's success rate, it is often placed as the main one. F1-Score, on the other hand, uses intermediate metrics, Precision and Recall, to measure a balance between them through their harmonic mean. Thus, a high F1-Score index is considered a uniformity factor in the classification of tissues, as it shows a low number of false positives and false negatives [31]. Finally, the Matthews Coefficient (MCC) is a correlation measure widely used in binary classification problems. Based on Pearson's correlation indices, this coefficient can be treated as a balanced measure even when there is an unbalanced dataset. There is a metric result for each class in question, simple arithmetic averages of the class results for representation in tables were made to facilitate the analysis of the results. Values will be represented as a percentage. Statistical test The Kolmogorov–Smirnov test is non-parametric, commonly used to assess the statistical similarity between two attribute vectors. Through the statistical test, it is possible to affirm with a more significant property if an average with its respective standard deviation is different from the other, since not only its point values are compared, but the vector of values that generated it [52]. It is necessary to establish a reli