## 3 Materials and Methods
### Methods of Improvement
#### 3.1.2 Stacking Ensemble Learning  
Stacking is a technique that leverages multiple machine learning models, or estimators, to generate predictions. Unlike mere averaging, stacking feeds these predictions into a new model which subsequently forms its own predictions based on the earlier results. Within this framework, models have specific designations: those used in the primary ensemble step are termed 'zero-level models' or 'weak learners', while the subsequent model that consolidates these predictions is the 'first-level model'. Typically, stacking follows a two-tier hierarchy, though more layers can be introduced. In the context of deep learning, transfer learning is an adaptation of stacking. As touched upon in the Background section, transfer learning is frequently cited in the litera  
Figure 1: Example of the bagging algorithm where the blue and red dots represent the normal and abnormal images, with the CNN structures in the first experiment Alexnet, then VGG-16, and lastly with Inception. With the results, there is an averaging of predictions that results in the final voting. ture as a means to utilize pretrained weights and architectures, which diminishes computational demands and paves the way for the implementation of intricate architectures. However, given that these transfer learning architectures are originally trained on natural images rather than medical ones, their efficacy in medical image classification remains a topic of debate. In this study, the initial phase of the stacking process employs transfer learning using the VGG-16 architecture for feature extraction. This approach taps into the capabilities of a deep learning algorithm while synergizing it with other models. Subsequently, as an alternative to VGG-16, transfer learning is executed with the Inception architecture to assess its comparative performance. Notably, akin to the scenario with the bagging ensemble, Alexnet was excluded from the transfer learning process. This is due to its relatively simpler architecture, which generally produces superior outcomes when trained from scratch. Using pretrained weights to derive features from images generates a numerical vector. This vector, representative of distinct image attributes, offers a format readily interpretable by conventional machine learning algorithms. Despite stacking potentially increasing processing time, initial feature extraction renders the data more manageable. As illustrated in Figure 2, following feature extraction, prevalent machine learning algorithms such as decision tree, random forest, \(k\)-neighbors, and support vector classifier are employed to interpret the transformed data. These algorithms are accessible within the _sklearn_ Python library [38]. We use deep learning at the beginning of the stacking and combine various machine learning models used for classification. A more in-depth explanation of each of the algorithms can be found in [36; 39; 40]. Nevertheless, in the context of the stacking ensemble, the models can be summarized as follows:  
* **Decision tree:** As a base learner, a decision tree can be quick to train and has the advantage of simplicity. However, it might be prone to overfitting on its own. * **Random forest:** This classifier is more robust than a single decision tree. It can reduce overfitting by averaging the results of individual trees. It is commonly used as a base learner in stacking due to its efficiency and high accuracy. Figure 2: Example of the stacking ensemble where the blue and red dots represent the normal and abnormal images. The first part consists of a transfer learning method as a feature extractor that transforms the data set into a manageable format for machine learning algorithms. The second one contains the weak learners used for intermediate predictions, and the third one corresponds to the final estimator given by logistic regression. * **K-nearest neighbor:** It can capture complex patterns in the data without requiring explicit model training.