# Enhanced Feature Selection via Hierarchical Concept Modeling
## 1 Introduction  
At present, big data are often confronted, and dataset sizes have increased dramatically in machine learning classification applications. Feature selection is typically applied in these classification tasks. The aim of feature selection is to improve the predictive performance of classifiers by removing redundant or irrelevant features. A single feature subset to generate a binary classifier tends to have features that are useful for distinguishing these specific classes but useless for distinguishing others [1; 2; 3]. Feature selection is a common task in applications of pattern recognition, data mining, and machine learning since it can help improve prediction quality, reduce computation time, and allow building more human-understandable models. Thus, it is used in many applications before training a classifier [3]. However, while there are a lot of state-of-the-art approaches for feature selection in standard feature space [4], only a few approaches for feature selection in hierarchical feature space have been proposed in the literature [1; 5]. Hierarchical modeling is one approach to feature selection for enhanced predictive performance of classifiers. It is not only advantageous for identifying the hierarchical model itself but is also helpful for selecting a feature subset for each node [1; 5; 6; 7; 8]. Classes in a hierarchical structure have both parent-children relationships and sibling relationships [2; 9; 10]. Moreover, classes with a parent-children relationship are similar to each other and may share common features for classification, while distinguishing between classes with a sibling relationship may require different features [1; 8; 11]. The authors developed a method for joint feature selection and hierarchical classifier design using genetic algorithms. Thus, this work focuses on using hierarchical structure in feature selection to improve the predictive performance in classification. In this study, we designed a feature selection algorithm based on the hierarchical information structure. The hierarchical models were built by using two strategies: bottom-up and top-down. The bottom-up strategy starts by placing each object in its own cluster and then merges these atomic clusters into larger and larger clusters until all the objects are in a single cluster or until some other termination conditions are satisfied. Later, the top-down strategy performs the reverse by starting with all objects in one cluster. It subdivides the cluster into smaller and smaller pieces until each object forms a cluster on its own or until some other termination conditions are met, such as a desired number of clusters or that the distance between the two closest clusters is above a certain threshold [12; 13; 14]. Thus, this work applied both bottom-up and top-down approaches, which were performed with formal concept analysis (FCA) and a decision tree, respectively. Both FCA and the decision tree approach were considered for feature selection in order to reduce the attribute count while still retaining the efficiency of data classification. In the context of feature selection, the hierarchical approach provides distinct advantages over non-hierarchical or flat methods, particularly when applied to complex datasets where relationships among features are not uniform. Traditional feature selection techniques, while effective for certain tasks, often operate in a flat structure and may overlook nuanced, multi-level relationships between features. Hierarchical models, such as those using FCA and DT, address these limitations by organizing features in a structured, multi-level manner that reflects inherent data patterns more accurately. Thus, FCA is advantageous for building hierarchical feature selection models because it organizes features based on generalization and specialization relationships. This lattice-based hierarchy helps identify clusters of related features and isolate distinctive attributes within classes.