# Efficient Pruning of Detection Transformer in Remote Sensing Using Ant Colony Evolutionary Pruning
## 3 Methodology  
### Algorithm Overview  
Neural networks are described as the amalgamation of weight parameters and architecture parameters [49]. Weight parameters signify the specific values of the neural network parameters, whereas architecture parameters delineate the neural network structure. Typically, the neural network learns weight parameters automatically through dataset training, whereas architecture parameters necessitate manual design by seasoned experts through extensive experimentation. The objective of NAS is to supplant manual design with an automated search mechanism, enabling us to acquire the optimal neural network architecture. This study employs this concept to autonomously explore the redundancy of architecture parameters in the initial neural network, which facilitates the pruning process. The exploration of architecture parameters can be formulated as the subsequent optimization problem:  
\[\text{a}^{*}=\underset{\text{a}\in\text{A}}{\text{argmax}}(\text{N}(\text{a}, \text{W})), \tag{1}\]  
where \(\text{f}(\cdot)\) denotes the testing accuracy of the optimized neural network structure, \(\text{N}(\text{a},\text{W})\) embodies distinct architectures, W represents the initial neural network weights, and A = {a1,a2, \(\cdots\)} comprises a range of diverse model architectures obtained through binary coding. Evolutionary-based traditional NAS approaches often rely on a single population for comprehensive exploration. Certain individuals may discover the optimal architecture within specific modules yet exhibit subpar overall performance and hence are sieved out. To safeguard the contributions of these individuals, this paper introduces an innovative ant colony evolutionary algorithm. This algorithm leverages the strengths and complementarity among different populations to effectively augment diversity and flexibility in search trajectories, transcending from local maxima to global maxima, thereby enhancing the overall architectural efficacy. The overall algorithm is shown in Figure 1.  
To search the architectural space more efficiently, we first decompose the neural network structure so that it can be divided into multiple smaller submodules for searching. Next, during the population initialization phase, as shown in the Initialization section of Figure 1, we generate initial populations of the same size for each submodule. These populations adopt different pruning strategies for different neural network structures. Subsequently, the initial populations undergo pre-search according to their respective pruning strategies, as depicted in the Pre-Search section of Figure 1. In this process, the ant colonies prune channels or attention heads in the neural network according to their individual search strategies, thereby fully exploiting the importance of local structures. Then, by integrating the local search results of each module, an initial global reference is constructed which will guide the subsequent global optimization process, as illustrated in the Globe Search and Bayesian Pruning section of Figure 1. Within this framework, each heterogeneous population engages in intense competition and evolution, ultimately retaining only the most advantageous populations. This ensures that the final solution achieves excellent performance on a global scale. The proposed pruning approach encompasses three fundamental phases: (1) population initialization: the establishment of initial search clusters for each module, delineating a strong foundational standpoint concerning diversity and directionality; (2) pre-search: leveraging the initial populations to delve into the design space of each module and fabricating diverse local optimal solutions; (3) global search and Bayesian pruning: merging the optimization findings of each module and applying Bayesian optimization methodologies for pruning constraints to ensure the competitive prowess of the target architecture in the global arena.