# Reducing Model Complexity in Neural Networks by Using Pyramid Training Approaches
## 3 Methodology
### Size Reduction  
Feature size reduction, also known as feature selection or dimensionality reduction, plays a vital role in machine learning by enhancing model efficiency and generalization capabilities. A primary advantage of this technique is its ability to mitigate the curse of dimensionality. High-dimensional feature spaces often lead to increased computational complexity and a higher risk of overfitting.  
Figure 1: Visual representation of feature maps extracted from a convolutional layer of a neural network trained on MNIST dataset [14]. These maps highlight the potential for similarity-based feature reduction, as several maps exhibit similar activation patterns.  
In our approach, a similarity matrix facilitates the identification of redundant features, guiding the selection of feature pairs that exceed a predefined similarity threshold. These pairs are deemed combinable. The merging of similar features is accomplished by replacing the original features with their mean or another suitable aggregation method, effectively reducing the number of features iteratively until no additional pairs meet the threshold. This results in a dataset with a minimized set of features, maintaining essential information while eliminating redundancies.  
To manage the dimensionality reduction dynamically, we introduce a'reduction scale' parameter. This parameter adjusts according to the evolving feature set, starting low when feature numbers are small and incrementing as the feature count increases. This adaptive mechanism allows the reduction process to become more aggressive as the complexity of the dataset grows, thereby ensuring efficient handling of varying feature spaces and achieving more optimized reductions as necessary.  
The mean method is commonly used in this reduction process:  
\[\mu=\frac{1}{n}\sum_{i=1}^{n}F_{i} \tag{3}\]  
where \(n\) is the number of similar features to be combined, and \(F_{i}\) represents the \(i\)th feature. While this method is straightforward and maintains the central tendencies of the features, it may not account for variations in feature importance and is susceptible to outliers. An alternative is the weighted mean method, which provides a more nuanced combination:  
\[\mu=\frac{\sum_{i=1}^{n}w_{i}\cdot F_{i}}{\sum_{i=1}^{n}w_{i}} \tag{4}\]  
where \(w_{i}\) represents the weight assigned to the \(i\)th feature, allowing for differential importance among features. This method enhances adaptability and allows for customization according to the specific characteristics of the dataset.  
Both methods serve to consolidate multiple features into a single, composite feature. The choice between these methods should be dictated by the dataset's characteristics and the specific goals of the analysis. A systematic approach involving experimentation and validation is essential to determine the most effective method for a given scenario.  
Let us assume we have features of shape \((B,N,H,W)\), where \(B\) represents the batch size, \(N\) represents the feature size, \(H\) represents the height, and \(W\) represents the width. The general formulation of feature size reduction for both mean and weighted mean methods can be seen below.  
\[\text{Mean Reduction}(X)_{i,j}=\frac{1}{N}\sum_{f=1}^{N}X_{i,j,f} \tag{5}\]  
\[\text{Weighted Mean Reduction}(X,W)_{i,j}=\frac{\sum_{f=1}^{N}W_{f}\cdot X_{i, j,f}}{\sum_{f=1}^{N}W_{f}} \tag{6}\]  
where \(W\) represents the weight vector for each feature, and the equation calculates the weighted mean reduction at each spatial location (\(i\),\(j\)), considering the importance assigned by the weights. In both equations, \(X_{i,j,f}\) denotes the feature map value at position (\(i\),\(j\)) for the \(f\)-th feature, and \(n\) is the total number of features in the tensor. The weighted mean reduction incorporates the weights, \(W_{f}\), associated with each feature for a more nuanced reduction process.