# Examination of Landslide Susceptibility Modeling Using Ensemble Learning and Factor Engineering
## 3 Methodology
### Factor Engineering  
Factor engineering is a critical preprocessing step in data science and machine learning, involving the creation, selection, and transformation of factors from raw data to build more efficient and accurate predictive models. The primary goal of factor engineering is to maximize the utilization of information in the dataset, enhance model performance, simplify the model structure, and expedite training processes. In this study, a comprehensive factor engineering strategy was adopted, leveraging advanced ensemble learning models to evaluate and rank the importance of all the potential influencing factors.  
This study departs from traditional factor collinearity detection methods, focusing instead on examining the performance of influencing factors across various ensemble learning models. Specifically, it explores the strategies employed by random forest, XGBoost, LightGBM, and CatBoost in evaluating and ranking factor importance. Random forest assesses factor importance by analyzing the purity gain contributed by each factor during decision tree node splits. This process typically uses Gini impurity or entropy as metrics, calculating the reduction in model error associated with each factor. The number of splits involving each factor across all the trees and the average purity gain they provide are accumulated to form a comprehensive factor importance score. XGBoost uses \"gain\" as its primary evaluation metric, quantifying the improvement in model performance contributed by each factor when making splits. This is specifically represented by the average reduction in the loss function following a split. Similarly, LightGBM also relies on gain to determine factor importance, but its calculation method differs slightly due to the incorporation of Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) techniques, which focus on evaluating the total gain in performance improvement from each factor during splits. CatBoost employs a method similar to the aforementioned models, assessing factor importance based on the performance improvement each factor brings during splits. Additionally, it considers the frequency of factor usage across different trees to evaluate the overall contribution of each factor, thereby determining its importance ranking.  
Based on the evaluation mechanisms of different models, the specific implementation steps of factor engineering in this study are as follows: (1) The dataset was independently trained using models such as random forest, XGBoost, LightGBM, CatBoost, bagging, and stacking. Each model provided factor importance scores, indicating the contribution of each factor to the prediction task. (2) By integrating the importance scores from each model, a consolidated factor importance list was created using either the average scores or more complex statistical methods. (3) Based on the importance list, the least significant factors were gradually eliminated, with the model retrained after each removal to assess its impact on prediction accuracy. (4) Model performance was compared before and after factor removal to analyze which eliminations significantly affected the outcomes. This step helps identify and exclude factors with minimal impact, simplifying the model and potentially improving efficiency. (5) A subset of factors that minimally impacted the prediction accuracy, while retaining those critical to model performance, was selected. This approach optimized the factor set, enhanced the model's generalization capability, and improved prediction accuracy. By employing this systematic factor engineering method, machine learning models were constructed that were both efficient and accurate, demonstrating the critical role of factor engineering in improving predictive performance.