# Reducing Model Complexity in Neural Networks by Using Pyramid Training Approaches
## 3 Methodology
### Method Description  
In the pyramid training framework, we introduce a novel approach for dynamically optimizing neural network models during the training process. This methodology involves training smaller network blocks independently and then integrating them to form a more complex and capable model. This section details the steps involved in our methodology and highlights the innovative aspects of our approach. We start by training two foundational blocks, Block 1 and Block 2, which are designed to capture and process diverse features from the input data. Initially, Block 1 is trained to extract primary features and patterns essential for the initial stages of learning. Once Block 1 is sufficiently trained, its fully connected (FC) layers are discarded to preserve spatial information when integrating with Block 2. This step ensures that essential spatial relationships are maintained, enhancing the overall model performance. To address potential dimensional mismatches and redundancies in feature maps, we employ a similarity comparison strategy. This involves constructing a similarity matrix using cosine similarity metrics to identify highly correlated feature maps. Redundant features, identified by high similarity scores, are combined using averaging techniques. This process effectively reduces the feature map count, ensuring compatibility between blocks and enhancing the network's capacity to handle complex patterns. The similarity comparison and feature size reduction steps are critical innovations in our approach. They allow us to dynamically adjust the model complexity during training, reducing redundancies and maintaining essential information. This strategy not only \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Layer** & **Type** & **Input Channels** & **Output Channels** & **Kernel Size** & **Stride** & **Padding** \\ \hline Conv1 & Conv2d & 3 & 16 & \(3\times 3\) & 1 & 1 \\ Pool1 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ Conv2 & Conv2d & 16 & 32 & \(3\times 3\) & 1 & 1 \\ Pool2 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ FC1 & Linear & - & 128 & - & - & - \\ FC2 & Linear & - & 10 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Block 1 model details. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Layer** & **Type** & **Input Channels** & **Output Channels** & **Kernel Size** & **Stride** & **Padding** \\ \hline Conv3 & Conv2d & 16 & 32 & \(3\times 3\) & 1 & 1 \\ Pool1 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ Conv4 & Conv2d & 32 & 128 & \(3\times 3\) & 1 & 1 \\ Pool2 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ FC3 & Linear & - & 128 & - & - & - \\ FC4 & Linear & - & 10 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Block 2 model details. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Layer** & **Type** & **Input Channels** & **Output Channels** & **Kernel Size** & **Stride** & **Padding** \\ \hline Conv1 & Conv2d & 3 & 16 & \(3\times 3\) & 1 & 1 \\ Pool1 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ Conv2 & Conv2d & 16 & 32 & \(3\times 3\) & 1 & 1 \\ Pool2 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ Conv4 & Conv2d & 32 & 128 & \(3\times 3\) & 1 & 1 \\ Pool2 & MaxPool2d & - & - & \(2\times 2\) & 2 & 0 \\ FC3 & Linear & - & 128 & - & - & - \\ FC4 & Linear & - & 10 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Block 1 combined with Block 2 model details. optimizes computational resources but also enhances the adaptability and efficiency of the model. The combined architecture, consisting of Block 1 and Block 2, undergoes further training to refine and enhance the feature representations. This iterative process continues, progressively adding and integrating more blocks (Block 3, Block 4, etc.) until the network achieves the desired level of accuracy. Each cycle of this process enriches the network's feature understanding and optimizes its overall performance. A key feature of our pyramid training methodology is the ability to freeze initial blocks once they are sufficiently trained.