# Neuoplasticity-Based Pruning Method for Deep Convolutional Neural Networks
## 3 Neuroplasticity-Based Pruning Method
### Retraining  
After pruning sub-structures from the network and reconnecting the remaining ones with the translator, the network needs to be retrained to compensate for the accuracy loss.  
Most state-of-the-art pruning methods apply an iterative pruning-retraining process, assuming the network connectivity may be harmed if relevant (important) connections are removed using a one-shot strategy, producing a drastic accuracy drop.  
On the one hand, pruning-retraining iteratively reduces the parameters gradually and monitors the accuracy during the pruning process. On the other hand, the iterative process may require several pruning-retraining loops to achieve similar results as in the original models, which implies using more computational resources and time.  
This work proposes adopting a one-shot pruning-retraining strategy because our method's key objective is not to reduce parameters gradually.  
The one-shot process assumes that a group of \(P\) consecutive layers (pruned layers) can be removed and compressed into a new block \(T\) (translator) on the network. Thus, during retraining, the translator block learns a transformation for the feature maps that is equivalent to all the transformations produced by \(P\), the pruned layers. This can be observed in Figure 5.  
The knowledge from the removed sub-structures should be compressed using the learnable parameters introduced in the convolutional layer of the translator block. Thus, the layers in the network must be frozen while retraining (i.e., their parameters will not be  
Figure 4: Example of VGG16 pruned architecture. Blocks 3 and 4 are pruned, thus, block 2 and 5 are reconnected with the translator block.  
Figure 5: Example of replacing a group of layers \(P\) (blocks 3 and 4) for the translator block.  
adapted during the retraining phrase), except for the convolutional layer in the translator. This can be observed in Figure 6.  
Leaving the translator active during retraining allows recreating the knowledge in a specific network region. Most of the previously learned tasks will be remapped into a different (new in this case) region of the neural network. Similar to the biological counterpart, this process can be seen as the biological neural network recovery process after injury.  
In our method, multiple origin and destination layers can be selected as candidates, which produce a combination of \(\mathcal{C}\) possible sub-models where, for each sub-model, a group of layers \(P_{i}\) (with \(i=1,2,...,C\)) will be pruned, and the model will be retrained once.  
In contrast to other pruning methods, our pruning-retraining strategy generates different sub-models that may recover (or even improve) the base model's performance, using a single retraining step with a fixed number of epochs. Furthermore, each sub-model is different and non-dependent from the other generated models. Hence, the one-shot pruning-retraining process for each sub-model can be computed individually and even in a parallel model.  
Using a one-shot pruning-retraining should be faster than other pruning methods, which apply the process iteratively until desired results are achieved or a maximum number of iterations are reached.