# Exploiting Data Analytics and Deep Learning Systems to Support Pavement Maintenance Decisions
## 3 Methodology
### Feature Selection Workflow
#### 3.2.1 Developing Feature Selector  
Once the data were preprocessed, the next step was preparing a feature selector tool adapted to the particular dataset but whose configuration could be modified based on the type of data typically available. Feature selection involves using only appropriate features that explain the dependent variable and the process helps produce good learners and models [47]. In this case, the dependent variable is the year when road intervention took place for a given road in the network. A feature selector is a tool that can be used for producing strong datasets for machine learning purposes. For the study, a feature selector tool was developed considering a selector developed in python utilizing the LightGBM library [48]. For the feature selector, the tool's backbone model was redesigned to identify feature importance using a gradient boosting machine from the CatBoost (Category Boosting) [49] library instead. This construction was done based on the strength of CatBoost to handle categorical features and the high presence of these features within this study and typical of this type of data. For the feature selector's backbone, several ensemble deep learning gradient boosting algorithms were considered. Gradient boosting algorithms are considered powerful tools that work by building ensemble tree predictors carrying out gradient descent within a functional space [49]. The algorithms sequentially create base models. The accuracy of these models versus others are considered higher given the production of multiple models sequentially and the emphasis on training cases that are more difficult to evaluate, making the mistake more evident. In the process, the examples that were considered harder to estimate in the earlier base models are forced to appear more frequently in the training data. The subsequent base models are all aimed at correcting the prior mistakes in earlier models. The boosting process utilizes weak base models that are easier to predict and combines them to get one highly accurate model. The process fits subsequent models that allow for minimization of particular loss function objectives and errors averaged across the training data [50]. The pseudocode for [PERSON]'s gradient boosting algorithm is subsequently shown in Algorithm 1 [51]. Gradient boosting methods offer a high level of customizable flexibility to any specific data-driven job [52] making it appealing for this study. There are several available gradient boosting models that are applied for prediction and regression problems. These include XGBoost [53], AdaBoost [54] and LightGBM [55] with the recent introduction of the CatBoost [49] libraries which provide a new approach, especially for categorical features. As a result of this, the CatBoost library was chosen given its ability to handle categorical features and previous studies identifying a higher performance of this algorithm versus the others previously mentioned [56; 57]. This is particularly important to the model given the number of categorical features present in the dataset. CatBoost is different from the other gradient boosting algorithms in the following ways:  
* The library handles categorical features during training as opposed to during the preprocessing time. It also utilizes the entire dataset for training. For each training example, the library carries out a random permutation of the dataset and calculates an average label value for the particular example with the same category value placed before the one provided by the permutation [58]. * The library allows unbiased boosting with categorical features and feature combinations where all the categorical features can be combined as a new feature. [49]
* The use of a fast scorer allows utilizing oblivious trees as base predictors [58]. The CatBoost library [49; 59] introduced two vital advances: ordered boosting, which is a permutation driven alternative to the classic choice and a state-of-the-art algorithm to process categorical features.