# Block-Wisely Supervised Network Pruning with Knowledge Distillation and Markov Chain Monte Carlo
## 3 Method
### Block-Wise Network Pruning  
As different parts of DNN deliver distinct patterns of an image [10], we divide the baseline network into blocks and perform channel pruning on individual blocks to improve efficiency. Each block contains an arbitrary number of layers, including convolutional and fully connected (FC) layers. Block shrinkage is independently applied to each block to yield a reduced substructure, then all the resulting substructures are connected to constitute the final subnet. Inspired by the recent success of neural architecture search in efficiently training supernets [10], we utilize block-wise representation of the baseline network to supervise subnet search, such that each block can be pruned independently in an analogous manner. As shown in Figure 1, the (\(i\)-1)-th block \(B_{i-1}\) of the teacher network outputs feature map \(\mathcal{F}_{i-1}\), the \(i\)-th block \(S_{i}\) of the student network takes \(\mathcal{F}_{i-1}\) as input and outputs feature map \(\hat{\mathcal{F}}_{i}\) to imitate the block-wise output \(\mathcal{F}_{i}\) of the teacher network. To effectively measure the knowledge transferred from the \(i\)-th block, we define an accuracy metric based on the MSE loss between \(\hat{\mathcal{F}}_{i}\) and \(\mathcal{F}_{i}\):  
\[\mathcal{R}_{a}(Z;\mathbf{X})=\exp\left(-\frac{1}{2n}\sum_{k}\
orm{\mathcal{F }_{i}(\mathbf{x}_{k})-\hat{\mathcal{F}}_{i}(\mathbf{x}_{k})}^{2}\right) \tag{1}\]  
where \(Z\) is a substructure of the \(i\)-th block, \(\mathbf{x}_{k}\in\mathbf{X}\) denotes input image, \(\mathbf{X}\) represents validation dataset, and \(n\) is the number of samples in \(\mathbf{X}\). Note that our block-wise search does not rely on labels of input data.  
To further discriminate between different substructures that show similar accuracy but different computational efficiency, the FLOPs' reduction rate is employed to define the efficiency metric:  
\[\mathcal{R}_{e}(Z)=1-\frac{\#\text{FLOPs}(Z)}{\#\text{FLOPs}(B_{i})} \tag{2}\]  
where \(\#\text{FLOPs}(Z)\) and \(\#\text{FLOPs}(B_{i})\) denote the FLOPs of the reduced and baseline blocks, respectively. FLOPs of FC layer are calculated as \(M_{h}\times M_{w}\times C_{in}\times C_{out}\); here, \(M_{h}\) and \(M_{w}\) are the height and width of the feature map, and \(C_{in}\) and \(C_{out}\) are the numbers of input and output channels. In addition, the FLOPs of the convolutional layer are measured as \(M_{h}\times M_{w}\times K_{h}\times K_{w}\times C_{in}\times C_{out}\); here, \(K_{h}\) and \(K_{w}\) are the height and width of the convolutional kernel.  
By taking a tradeoff between \(\mathcal{R}_{a}(Z;\mathbf{X})\) and \(\mathcal{R}_{e}(Z)\), we derive a metric to score each substructure by considering both model capability and efficiency:  
\[\mathcal{R}(Z;\mathbf{X})=\alpha\mathcal{R}_{e}(Z)+(1-\alpha)\mathcal{R}_{a}( Z;\mathbf{X}) \tag{3}\]  
where \(\alpha\in[0,1]\) is introduced to control the weights of two terms. Higher values of \(\alpha\) will give a preference to substructures that have more significant FLOP reductions. Based on the above definition, for each block of student network we need to find the best-performing substructure that shows the highest value of \(\mathcal{R}\).