# Reducing Model Complexity in Neural Networks by Using Pyramid Training Approaches
## 3 Methodology
### Similarity Comparison  
Within the convolutional layers of a CNN, feature maps are generated that encapsulate the essential characteristics of the input data. These maps often exhibit redundancy, especially in deeper layers of the network, which can be computationally wasteful and detract from model efficiency. To address this, we employ a similarity comparison strategy using cosine similarity metrics to identify and consolidate redundant features [14]. Cosine similarity measures the cosine of the angle between two vectors (feature maps in this context), providing a value between \(-1\) and \(1\) that indicates how similar the feature maps are in terms of their information content. The formula is given by:  
\[\text{cosine similarity}(A,B)=\frac{A\cdot B}{\|A\|\cdot\|B\|} \tag{1}\]  
where \(A\cdot B\) denotes the dot product of vectors \(A\) and \(B\), and \(\|A\|\) and \(\|B\|\) represent their respective magnitudes. This measure allows us to quantify the degree of similarity between pairs of feature maps. Feature maps that exhibit a high degree of similarity (cosine similarity close to 1) are considered redundant. These are then combined using averaging techniques, reducing the overall number of feature maps and, consequently, the model's complexity and computational load. To illustrate this concept, an example feature map representation from the MNIST dataset is shown in Figure 1. This visualization demonstrates how similar feature maps, which emerge naturally during the training of convolutional layers, can be identified and merged. By highlighting these redundancies through the feature maps of the well-known digit images, we underline the potential for significant reductions in network complexity through our similarity comparison technique. Let us assume we have a set of 16 features, represented as vectors \(F_{1},F_{2},\dots,F_{16}\). To evaluate the similarity between these feature vectors, we calculate the cosine similarity for each pair, which measures the cosine of the angle between two vectors in a multi-dimensional space. The cosine similarity is particularly useful as it normalizes the feature scale, focusing solely on the information content. As an alternative to cosine similarity, Euclidean distance can also be used to measure the similarity between features. Euclidean distance calculates the straight-line distance be tween two points in a multidimensional space, making it a geometric measure of similarity. It is defined by the following equation:  
\[\text{Euclidean Distance}(\mathbf{A},\mathbf{B})=\sqrt{\sum_{i=1}^{n}(A_{i}-B_{ i})^{2}} \tag{2}\]  
where \(\mathbf{A}\) and \(\mathbf{B}\) represent the feature vectors, \(\mathbf{n}\) is the number of dimensions, and \(A_{i}\) and \(B_{i}\) are the \(i\)th components of vectors \(\mathbf{A}\) and \(\mathbf{B}\), respectively. Unlike cosine similarity, which normalizes the vectors and focuses solely on their orientation, Euclidean distance takes into account both the magnitude and direction of the vectors. This characteristic means that Euclidean distance is sensitive to the scale of the vector components, which can be both an advantage and a limitation depending on the application. While Euclidean distance provides a direct measure of the physical 'distance' between feature vectors, it may not be as effective in high-dimensional spaces where different features may vary widely in scale or where the direction of the vectors is more informative than their magnitude. In contrast, cosine similarity, by normalizing vector magnitude, offers resilience to variations in scale and is often preferred in applications such as text analytics and other forms of semantic similarity where the direction of the feature vector (i.e., the angle between vectors) is more critical than their length. Therefore, the choice between Euclidean distance and cosine similarity should be guided by the specific characteristics of the dataset and the requirements of the application.