# Reducing Model Complexity in Neural Networks by Using Pyramid Training Approaches
## 1 Introduction  
In the rapidly evolving field of machine learning, convolutional neural networks (CNNs) are extensively used for visual information processing, playing pivotal roles in areas such as image classification [1, 2], object detection [3, 4], facial recognition [5], medical imaging [6, 7], and autonomous driving [8]. As these tasks grow in complexity, so too do the models designed to tackle them, often resulting in increased model size and computational demands. Consequently, state-of-the-art models require significant hardware resources, which can be prohibitively expensive for individual researchers or small organizations. Notably, deep architectures such as very deep convolutional networks have demonstrated substantial improvements in large-scale image recognition tasks by significantly increasing the network depth [9], and deep residual networks [10] have also demonstrated substantial improvements in large-scale image recognition tasks by significantly increasing the network depth.  
Addressing the challenge of CNN model size optimization has spurred a variety of strategies. Techniques such as quantization [11], pruning [12], knowledge distillation [13], weight sharing [14], factorization [15], low-rank approximation [16], and dynamic network surgery [17] have been developed to manage model complexity by reducing size either before or after training. However, these methods often require trade-offs between model performance and efficiency.  
This study introduces a novel \"pyramid training\" methodology, an innovative approach designed to dynamically adjust and optimize CNNs during the training process itself. Pyramid training involves building smaller networks incrementally--starting with a simple model (A) and progressively integrating it with other networks (B, C, etc.) to form larger, more complex structures. This method not only conserves resources but also adaptsto increasing accuracy demands without the exponential growth in computational load typically associated with larger models.  
A distinctive feature of our methodology is the implementation of a feature size reduction strategy during network combination. By employing a similarity comparison between features using a cosine similarity matrix, our approach identifies and merges similar feature maps. This process not only helps in condensing the network's complexity efficiently but also ensures that essential information is preserved, optimizing both the model size and its computational efficiency.  
The contributions of this study are summarized as follows:  
* Introduction of the pyramid training paradigm for efficient optimization of CNNs.
* Detailed methodology for iterative network construction and integration to progressively achieve desired accuracy.
* Implementation of a feature size reduction strategy using a similarity matrix and averaging technique to enhance the network's architectural efficiency.  
The remainder of this paper is structured as follows. Section 2 reviews related work in model size optimization techniques. Section 3 presents the proposed methodology in detail. Section 4 describes the experimental setup and datasets used for evaluating our approach. Section 5 discusses the experimental results, highlighting the effectiveness of our methods. Finally, Section 6 concludes the paper with a summary of findings and suggestions for future research.