# Reducing Model Complexity in Neural Networks by Using Pyramid Training Approaches
## 2 Related Work  
Quantization has garnered significant attention in machine learning due to its potential to reduce model size without substantially compromising accuracy, a crucial factor in on-device applications where balance between accuracy and latency is vital [11; 18]. It is also effective in large-scale multimedia retrieval, managing high-dimensional data efficiently [19]. This method has been extensively used to compress machine learning models for deployment on field-programmable gate arrays (FPGAs) [20]. Moreover, quantization plays a crucial role in optimizing machine learning models for deployment on microcontrollers, with studies comparing quantization and pruning strategies for efficient model deployment [21]. Recent advancements like SmoothQuant have introduced accurate and efficient post-training quantization for large language models, achieving significant efficiency improvements while maintaining high accuracy [22]. However, there is a loss of accuracy during the mapping process, which may degrade a model's performance, especially in highly sensitive applications. Pruning techniques enhance neural network efficiency by removing redundant model components, thereby reducing computational overhead without affecting accuracy. Innovative approaches, such as the formulation by [PERSON] et al., have made significant contributions to efficient inference [23; 24]. Further, studies by [PERSON] et al. emphasize pruning's utility in removing unnecessary network structures post-training, improving performance [25]. Another study proposes the use of model structural pruning techniques to reduce the parameter count of deep learning models, as discussed in [26]. Despite these benefits, pruning can sometimes lead to suboptimal architectures and may require iterative retraining, which can be time-consuming and computationally expensive. Knowledge distillation involves training a smaller, more efficient \"student\" model to emulate a larger \"teacher\" model, thereby accelerating and compressing machine learning models without losing predictive power, proving especially useful in domains like computer vision and natural language processing [13; 27]. A multiscale semantic graph mapping (SGM) loss function is proposed to enable more comprehensive knowledge transfer between teacher and student networks at multiple feature scales, as discussed in [28]. However, the effectiveness of knowledge distillation heavily depends on the quality and architecture of the teacher model. If the teacher model is not well-optimized, the student model may inherit its deficiencies. Weight sharing reduces the number of parameters in a neural network, enhancing generalization and reducing overfitting. It has shown particular promise in neural architecture search (NAS), where careful management of weight sharing can significantly impact model performance [14; 29]. Nevertheless, weight sharing can sometimes limit the expressiveness of a model, potentially leading to underfitting if not managed properly. Additionally, it can complicate the training process and requires careful hyperparameter tuning. Low-rank approximation techniques streamline the training of deep neural networks by leveraging the inherent structures within the data. [PERSON] et al. 's method of low-rank matrix factorization exemplifies how these techniques facilitate efficient training processes by focusing on the most expressive features of the data [16]. Additionally, the adaptive quantization method introduced by [PERSON] integrates low-rank approximations to reduce computational burdens, which is crucial in resource-constrained settings [11]. Moreover, low-rank approximations have been explored in diverse fields such as optical coherence tomography image processing, where low tensor train and low multilinear rank approximations are utilized for compression and de-speckling of images [30].