Cancer is a fatal disease caused by a combination of genetic diseases and a variety of biochemical abnormalities. Lung and colon cancer have emerged as two of the leading causes of death and disability in humans. The histopathological detection of such malignancies is usually the most important component in determining the best course of action. Early detection of the ailment on either front considerably decreases the likelihood of mortality. Machine learning and deep learning techniques can be utilized to speed up such cancer detection, allowing researchers to study a large number of patients in a much shorter amount of time and at a lower cost. In this research work, we introduced a hybrid ensemble feature extraction model to efficiently identify lung and colon cancer. It integrates deep feature extraction and ensemble learning with high-performance filtering for cancer image datasets. The model is evaluated on histopathological (LC25000) lung and colon datasets. According to the study findings, our hybrid model can detect lung, colon, and (lung and colon) cancer with accuracy rates of 99.05%, 100%, and 99.30%, respectively. The study’s findings show that our proposed strategy outperforms existing models significantly. Thus, these models could be applicable in clinics to support the doctor in the diagnosis of cancers. Keywords: Feature Extraction, Transfer Learning, Machine Learning, Ensemble Learning, Lung Cancer, Colon Cancer 1. Introduction Cancer is a broad classification that encompasses a variety of illnesses that can damage organs in the human body. It is also a rapid reaction of aberrant cells that expand beyond their acceptable boundaries, allowing them to infect neighboring areas and travel to neighboring tissues which are known as metastasis. Metastasis is the prime factors of cancer-related death (Organization, 2022). The disease’s prevalence has risen significantly over time, most likely as a result of increased sensitivity to particular cancers as age advances. Colorectal cancer, which affects the colon and rectum, is one of the major induces of death. Lung cancer is probably the most commonly recognized malignancy (12.2%), but colorectal cancer, which affects the colon and rectum, is also the leading trigger of death ∗Mohammad Ali Moni ∗∗Md Manowarul Islam Email addresses: mdalamintalukdercsejnu@gmail.com (Md. Alamin Talukder), manowar@cse.jnu.ac.bd (Md. Manowarul Islam), ashraf@cse.jnu.ac.bd (Md Ashraf Uddin), arnisha@cse.jnu.ac.bd (Arnisha Akhter), fida.hasan@rmit.edu.au (Khondokar Fida Hasan), m.moni@uq.edu.au (Mohammad Ali Moni) 1 Accepted in Expert Systems with Applications on 28 May 2022 (10.7%) (Lyon, 2020). It is one of the major drivers of fatality globally, although its impact is not evenly distributed (Vineis and Wild, 2014). When it comes to identifying cancer symptoms, there are quite a variety of approaches. With the advancements, the amount of data stored in archives is growing by the day (Godkhindi and Gowda, 2017). It is extremely challenging to analyze a huge quantity of data and extrapolate using traditional approaches (Yildirim and Cinar, 2022). The increased availability of healthcare data provides scientists with a fresh opportunity to improve existing approaches for further comprehensive clinical analysis (Sarwinda et al., 2020). In the medical profession, machine learning and deep learning are commonly utilised technologies for analyzing biomedical data (Park et al., 2018) (Das et al., 2020) (Baldi, 2018) (Lenz et al., 2019). Machine learning (ML) is a limb of AI technologies that arose from the field of pattern identification and cogni- tive acquisition concepts. It produces mechanisms that effectively adapt through a vast group of information and generate forecasts based on historical evidence (Park et al., 2018). In several sectors where developing explicit tech- niques with acceptable efficiency seemed hard and nearly impossible, machine learning had indeed been successfully employed with remarkable outcomes (Hussain et al., 2021) (Assegie, 2021) (Xue et al., 2021). Deep learning (DL), on the contrary, is a sophisticated based pattern classification technology that has shown exceptional results in fea- ture extraction, object detection and voice recognition, and other domains that require multilevel data processing (Pyrkov et al., 2018). It can extract meaningful significant patterns from images and has been demonstrated to obtain state efficiency, occasionally outperforming humans (Ciregan et al., 2012). Transfer learning (TL) entails repurposing previously acquired knowledge to tackle a specific situation (Kandaswamy et al., 2016). It can be used in two ways: as a baseline algorithm, which is used to train the image dataset and determine performance (Hijab et al., 2019) (Chougrad et al., 2020) (Khan et al., 2019); as a feature extractor, which is used to extract features from image datasets and then use machine or deep learning algorithms to determine performance (Deniz et al., 2018) (De Matos et al., 2019) (Abbasi et al., 2020) (Phankokkruad, 2021). Ensemble learning relies upon the combination of many learning models that are purposefully developed as well as integrated to solve issues like categorization (Polikar, 2012). It is a kind of machine learning that aims to upgrade predicted efficiency by combining several models. Moreover, it’s a significant subject for increasing base classification models (Onan, 2015). A number of papers have been conducted in the literature, each of which is predicated on a distinct strategy for detecting cancer (Hatuwal and Thapa, 2020) (Tasnim et al., 2021) (Liang et al., 2020) (Chen et al., 2021). A wide variety of medical images have already been identified and signified with the help of machine learning procedures. DL strategies have made it possible for machines to assess exalted dimensional information such as pictures, multimodal pathology scans, and video files (Adeoye et al., 2022) (Alsinglawi et al., 2022) (Tsai et al., 2022). Numerous supervised machine learning techniques have been produced, as they are extremely adept at handling biological images (Bukhari et al., 2020) (Wu and Zhao, 2017) (Papp et al., 2021) (Urbanos et al., 2021). To classify lung and colon cancer efficiently, we need to train our model with a large dataset where all cancer possibilities are included. To do so, our research emphasizes designing a resourceful and well-organized framework where we can take large histopathological image (HPI) datasets (LC25000) and a number of pre-processing methods are applied to datasets. k-fold cross-validation, and feature extraction, all are applied to it. To build a robust model, we have used an ensemble method at the end of our implementations getting better performance in cancer classifications. Accepted in Expert Systems with Applications on 28 May 2022 The performance of our proposed hybrid model will be evaluated using high-performance filtering (HPF) with combination of ensemble learning. We have evaluated a variety of performance indicators in our investigation including accuracy, recall, precision , f1-score, MAE, MSE, RMSE, confusion matrix, AUC score and the ROC Curve. The classifier algorithms in the proposed framework can detect cancer (lung and colon) having an accuracy rate of above 99%. The following are the major accomplishments of this study. •First of all, to detect lung and colon cancer, we proposed an efficient hybrid technique that combines deep feature extraction, high-performance filtering, and an ensemble learning strategy. •Secondly, A deep feature extraction technique is proposed that is uniquely applied using different transfer learning models to extract features. •Finally, we analyze our proposed model on HPI datasets using a set of metrics and showed how to diagnose lung and colon cancers, thereby contributing to the new knowledge of AI-based applications. The remainder of the paper is arranged as follows: Section 2 leads a literature review to identify the state-of- art of lung and colon cancer diagnoses using machine learning. Section 3 gives the methodology and the dataset description of the study. The experimental setup and performance evaluation are presented in Section 4 and Section 5, respectively before concluding the paper in section 6. 2. Literature Review Categorizing histopathological image datasets of many cancer kinds, such as breast, lung, colon, colorectal, and skin cancer, has received much attention. Various approaches are utilized using ML, DL, and TL to detect lung and colon cancer. 2.1. Lung and colon cancer using computer-aided diagnosis (CAD) approach A CAD method was introduced by (Nishio et al., 2021) to categorize histopathology images of lung tissues automatically. They evaluated the effectiveness of eight ML algorithms on two datasets using traditional texture analysis (TA) and homology-based image processing (HI) to visual feature extraction. The CAD system with HI outperformed the TA system in both datasets. As a result, they determined that HI was much more beneficial for CAD systems than TA and that a precise CAD system for lung tissues might be developed. Similarly, (Mangal et al., 2020) created a CAD system by examining digital pathology images for recognizing lung and colon cancer utilizing a convolutional neural network (CNN). In comparison to classical ML models as well as deep CNN models utilizing TL trained on a similar collection though utilizing state-of-the-art feature descriptors, their empirical findings on the LC25000 indicated better accuracy of 97.89% for lung and 96.61% for colon gained by CNN. Moreover, (Shandilya and Nayak, 2022) devised a CAD technique for categorizing histological pictures of lung tissues. They utilized a publicly accessible dataset of histological photos of lung tissue for the design and validation of CAD. Multi-scale processing was performed to extract image features. Furthermore, a comparative analysis was conducted using seven pre-trained hyper-tuned CNN models for lung cancer prediction where, ResNet101 provided the highest accuracy Accepted in Expert Systems with Applications on 28 May 2022 of all, at 98.67 percent. This study will assist researchers in developing more effective CNN-based lung cancer detection models. 2.2. Lung and colon cancer using CNN approach A CNN approach was proposed by (Hatuwal and Thapa, 2020) for detecting lung cancer using histopathology pictures. Their model revealed a considerable improvement in classification accuracy and got training and validation accuracy of 96.11% and 97.20%, respectively. Similarly, (Tasnim et al., 2021) introduced three CNN models to analyze colon cell imaging data. The models were trained and tested at various epochs to estimate the learning rate. The average pooling and max-pooling layers are revealed to be 95.48% and 97.49% accurate, respectively. The MobileNetV2 exceeds the other two models with the highest accuracy of 99.67% and the lowest loss rate of 1.24%. It would be developed in collaboration with medical professionals at hospitals or clinics that specialize in colon cancer treatment. Qasim et al. (2020) produced a CNN model for predicting colon cancer distinguished by speed and accuracy with a small number of parameters. Their model was based on two routes, each of which was in charge of building 256 feature maps to enhance the number of features at various levels and so raise the accuracy and sensitivity. The visual geometry group (VGG16) model was created and trained within the same dataset to assess the effectiveness of the proposed scheme. The acquired accuracy is 99.6% for the proposed model and 96.2% for VGG16. The performance indicated that it was useful in identifying colon cancer. On the other hand, (Sikder et al., 2021) proposed a unique method for segmenting, recognizing, classifying, and detecting various types of malignant cells in both MRI and RGB scans. They used a CNN model and a SegNet approach with a morphological operation that was superior to the typical SegNet model to minimize training time and boost segmentation results. The proposed technique produced an average accuracy rate of 93 percent in detecting cancer cells from multiple cancer datasets. They were able to overcome the limitations of employing separate cancer identification techniques for MRI and Histopathology data. 2.3. Lung and colon cancer using DL and ML approach An ML approach to diagnosing lung and colon cancer based on DL was proposed by (Masud et al., 2021) that analyzed pathological photos of lung and colon cancers to identify five different types of tissues. They extracted the features using 2D Fourier and 2D Wavelet (2D FW) feature extraction process and combine the features and trained them with a CNN model. The obtained findings revealed that the presented architecture has the highest accuracy of 96.33 cents in identifying cancer tissues. The use of this approach will aid healthcare practitioners in the development of a computerized and efficient method for detecting different forms of lung and colon cancers. Similarly, (Sarwinda et al., 2020) evaluated the effectiveness of deep feature extraction for colorectal cancer diagnosis using two TL models as well as traditional ML algorithms. The findings revealed that employing DenseNet121 to extract the features outperforms ResNet50 for all ML algorithms and achieved 98.53% accuracy utilizing the KNN algorithm. The developed approach proved adequate for identifying colon tissue. To enhance picture classification efficiency, (Bansal et al., 2021) presented an approach integrating deep features generated with VGG19 with other customized feature extraction techniques, such as SIFT, SURF, ORB, and the Shi-Tomasi corner detector algorithm. Additionally, the collected features from these methods were categorized using a variety of ML algorithms. The Accepted in Expert Systems with Applications on 28 May 2022 empirical findings showed that Random Forest (RF) with the cooperative features beat other classifiers with 93.73 percent accuracy. According to the findings, a single feature extractor is insufficient to provide consistent outcomes. As a result, for picture classification, a mixed method employing DL features as well as classical handmade features is preferable. To˘ ga¸ car (2021) introduced a DL model where the image classes were built from the ground up using the DarkNet19 model. The feeble features were selected from the feature set retrieved from the DarkNet19 model using the Equilibrium and Manta Ray Foraging optimization approach and separated from the rest of the set’s features, yielding an optimal feature set. The Support Vector Machine (SVM) approach was accustomed to aggregating and categorizing the suitable characteristics generated by the two optimization algorithms utilized. The overall classifier performance percentage was 99.69%. It was discovered that combining the complementary approach with certain optimizing strategies increased the dataset’s categorization ability. To determine lung and colon cancer using histopathological images with improved augmentation strategies, (Garg and Garg, 2020) proposed a method by modifying 8 TL models. According to the findings, the 8 models achieved notable accuracy ranging from 96 percent to 100 percent accuracy. To visualize the focus images of CNN models, SmoothGrad and GradCAM were then employed that had previously been learned to differentiate between cancerous and benign images. 2.4. Lung and colon cancer using Hybrid approach In (Phankokkruad, 2021), for diagnosing lung cancer three TL models were utilized. The findings exhibited that the proposed models attained accuracy levels of 62 percent, 90 percent, and 89 percent for VGG16, ResNet50V2, and DenseNet201, respectively. Lastly, a combination of the three proposed CNN models called the ensemble model exceeded the other models with a validation accuracy of 91%. The employment of the ensembles of the TL model, which enhances efficiency, is beneficial in overcoming the challenge of diagnosing lung cancer. Another, (Chen et al., 2021) proposed a hybrid classifying framework that combines the Inception v3, hog, and daisy extracting features modules to distinguish lung cancer and normal tissue from lung pathology images. The experimental findings showed that the proposed model achieved an accuracy of 99.60 percent, which was a gratifying outcome greater than other comparable trials. It demonstrated that hybrid deep learning models may be used to evaluate a reliable cancer diagnosis approach. On the other hand, (Liang et al., 2020) developed a shearlet transform-based multi-scale feature fusion (MFF-CNN) model to recognize histopathology images of colon cancer. It achieved a 96 percent detection performance for colorectal after feature learning and feature fusion. Its improvement offers innovative opportunities for cancer screening instantaneously, with objectivity as well as accuracy. Yildirim and Cinar (2022) created a new method for identifying colon cancer images called MA ColonNET. The algorithm reached a 99.75 cent accuracy rate for detecting and classifying colon cancer which will help to avoid human errors that are common in older methods. They demonstrated that the presented methodology can identify colon cancer faster than originally thought. The therapy strategy will be more effective as a result of this as well. Furthermore, (Adu et al., 2021) offered the DHS-CapsNet, a revolutionary dual horizontal squash capsule network for classifying lung and colon tumors on histopathological pictures. Encoder feature fusion (EFF) and a unique horizontal squash (HSquash) algorithm were used to build DHS-CapsNet. The collected feature from the 2-lane convolutional layers was aggregated by the EFF, which provided valuable insights for greater accuracy. The claiming prediction error for the DHS-CapsNet was 0.77 percent, as opposed to 14.45 percent for the regular CapsNet. It enhanced CapsNet Accepted in Expert Systems with Applications on 28 May 2022 which can be used as a CAD approach to aid clinicians in diagnosing lung and colon cancer. The related paper summary is shown in Table 1. 3. Methodology In this section, we have covered image pre-processing, deep feature extraction, high-performance filtering, and ensemble learning method as well as our proposed model. We also give a basic rundown of the ML techniques that are used to detect lung and colon cancer. We developed our proposed model using feature extraction and machine learning algorithms on LC25000 lung and colon histopathological image datasets to assure a prognosis of lung and colon cancer. Figure 1 represents the schematic block diagram of our proposed paradigm. The stages of our proposed model are as follows: •Step-1: Initially, we take the histopathological image datasets (LC25000) for lung and colon and combine both (lung and colon) cancer to conduct our experiment. There are three types of lung cancers, and two types of colon cancers are available in our datasets. •Step-2: In the pre-processing step, the pre-processing is accomplished by resizing the image into 128x128, converting the image into bgr2rgb, and then the image to NumPy array. After that, we perform feature scaling where we use the generalization method in the image; we divide easy image array values by 255, which is the maximum value (max. intensity value of an image), and perform labeling by assigning a label (0,1,2) for each image. •Step-3: In the deep feature extraction step, we extract features from the image datasets using five different transfer learning such as VGG16, VGG19, DenseNet169, DenseNet201 and collect the features for next the step. •Step-4: In this step, successfully extracted features are fit into the six well-known ML algorithms such as Random Forest (RF), Support Vector Machine (SVM), Logistic Regression (LR), Multi-Layer Perceptron (MLP), Extreme Gradient Boosting (XGB), and Light Gradient Boosting (LGB) to evaluate the performance. •Step-5: In this High-Performance Filtering (HPF) step, we filter our ML algorithms based on the accuracy. By using this filtering, we select the top three algorithms for the next step. •Step-6: In this Ensemble Learning step, after getting the top three algorithms from HPF, we apply majority (hard) and weighted average (soft) voting classifiers on these algorithms for all TL and select the best ensemble voting classifier for each TL model. •Step-7: Finally, the performance is assessed based on the accuracy for each TL and selecting the best TL model. After that, the performance metrics such as accuracy, recall, precision, f1-score, MAE, MSE, RMSE, confusion matrix, AUC score, and ROC Curve are used to evaluate the selected model of our experiment and also showed the comparison analysis with other existing models. Accepted in Expert Systems with Applications on 28 May 2022 SL. No Author Dataset Approach Accuracy(In %) 1 Nishio et al. (2021) Histopathology CAD 99.23 2 Mangal et al. (2020) Histopathology CAD97 (Lung) 96 (Colon) 3 Shandilya and Nayak (2022) Histopathology CAD 98.67 4 Hatuwal and Thapa (2020) Histopathology CNN 97.2 5 Tasnim et al. (2021) Histopathology CNN 99.67 6 Qasim et al. (2020) Histopathology CNN 99.6 7 (Sikder et al., 2021)Brain MRI Image BreCaHAD SN-AM HistopathologyCNN98.59 (IoU) 91.16 (IoU) 93.02 (IoU) 90.24 (IoU) 8 Masud et al. (2021) Histopathology 2D FW+ CNN 96.33 9 Sarwinda et al. (2020) Histopathology DenseNet121 + KNN 98.53 10 Bansal et al. (2021) Caltech101 VGG19 + RF 93.73 11 To˘ ga¸ car (2021) Histopathology DarkNet19 + SVM 99.69 12 Garg and Garg (2020) HistopathologyVGG16 ResNet50 InceptionV3 InceptionResNetV2 MobileNet Xception NASNetMobile DenseNet16998 96 100 100 100 100 97 100 13 Phankokkruad (2021) Histopathology Ensemble Model 91 14 Chen et al. (2021) Histopathology Ensemble Model 99.6 15 Liang et al. (2020) Histopathology MFF-CNN 96 16 Yildirim and Cinar (2022) Histopathology MA ColonNET 99.75 17 Adu et al. (2021) Histopathology DHS-CapsNet 99.23 Table 1: Related work summary Accepted in Expert Systems with Applications on 28 May 2022 Pre-ProcessingCancer Dataset VGG16 MobileNetDenseNet169Lung Cancer Colon Cancer Transfer Learning Models Feeding into Machine Learning Models SVM MLPXGB LGBDeep Feature Extraction Feature Extraction Ensemble Learning DenseNet201 VGG19 RF LR HPF (High Performance Filter) Evaluate the Performance Figure 1: The proposed hybrid model for lung and colon cancer detection Accepted in Expert Systems with Applications on 28 May 2022 Lung Cancer No. of Images Benign lung tissue 1400 Lung adenocarcinomas 1400 Lung squamous cell carcinoma 1400 Total 4200 Table 2: Distribution of Lung Cancer dataset Colon Cancer No. of Images Benign colon tissue 1400 Colon adenocarcinomas 1400 Total 2800 Table 3: Distribution of Colon Cancer dataset 3.1. Dataset Collection In our proposed work, we have employed the lung and colon cancer dataset called histopathological images (LC25000) dataset. The dataset consists of two cancerous datasets called lung and colon cancer datasets. It contains 25000 images where 10000 for colon cancer and 15000 for lung cancer images. The lung cancer dataset contains 3 labels of cells such as adenocarcinomas, squamous cell carcinoma and benign tissue. The colon cancer dataset contains 2 labels of cells, such as adenocarcinomas and benign tissue. The LC25000 dataset was created using a snippet of HIPAA-compliant as well as confirmed