# Exploiting the Relationship between Pruning Ratio and Compression Effect for Neural Network Model Based on TensorFlow

Authors: Bo Liu; Qilin Wu; Yiwen Zhang; Qian Cao
Journal: Security and Communication Networks
 (2020)

---

## Abstract

Pruning is a method of compressing the size of a neural network model, which affects the accuracy and computing time when the model makes a prediction. In this paper, the hypothesis that the pruning proportion is positively correlated with the compression scale of the model but not with the prediction accuracy and calculation time is put forward. For testing the hypothesis, a group of experiments are designed, and MNIST is used as the data set to train a neural network model based on TensorFlow. Based on this model, pruning experiments are carried out to investigate the relationship between pruning proportion and compression effect. For comparison, six different pruning proportions are set, and the experimental results confirm the above hypothesis.


---

## Body

## 1. Introduction

Model compression is a common method to transplant artificial intelligence from the cloud to the embedded terminal. Network pruning is a particularly effective compression solution for models [1, 2]. In [1, 3], Han et al. proposed a method of compression based on pruning but did not investigate the relationship between pruning proportion and compression effect. At the same time, He et al. [2] studied channel pruning for accelerating very deep neural networks, yet the pruning rate on the prediction effect is not stated. In fact, some studies of pruning methods have been carried out in recent years. However, to the best of our knowledge, there are very few studies on the relationship between the pruning proportion and the size, accuracy, and computing time which is used to make predictions. It is also the motivation of our research.In a trained neural network model, pruning sets all parameters with values less than a specific threshold to zero. After pruning, retraining and sparsification are normally conducted, where sparsification can delete connections with the zero values to compress the size of the model [4, 5]. As an example, the two figures show the comparison before and after pruning, where Figure 1 shows the original structural diagram, and Figure 2 shows the structural diagram after pruning.Figure 1
Original structural diagram.Figure 2
Structural diagram after pruning.Here, based on TensorFlow, we will use MNIST as the data set to train a neural network model. TensorFlow is an open-source machine learning framework. Specifically, it is software, and users need to build mathematical models by programming in Python and other languages. These models are used in the application of artificial intelligence. MNIST data set is a handwritten data set with 60,000 handwritten digital images in the training library and 10,000 in the test library. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.In this paper, we make the hypothesis that the pruning proportion is positively correlated with the compression scale of the model but not with the prediction accuracy and calculation time. So, our research object is the preliminary relationship between pruning proportion and compression effect in the neural network model. Specifically, this paper studies the relationship from three aspects: first, the relationship between pruning proportion and model size; second, the relationship between pruning proportion and model prediction accuracy; lastly, the relationship between pruning proportion and computing time for model predictions. For the above objective, a great number of experiments are carried out to investigate the relationship between pruning proportion and compression effect, and the above hypothesis is confirmed, which is our main contribution in this paper.The rest of this paper is organized as follows. In Section2, the neural network model is proposed first. To test the hypothesis, an original model and an experimental plan are introduced in Section 3. Section 4 gives the experimental procedures, and Section 5 gives the experimental results and analysis. Finally, Section 6 concludes this paper.


## 2. Neural Network Model

A neural network is constituted by one input layer, one or several hidden layers, and one output layer, and every layer is constituted by a certain number of neurons. These neurons are interconnected, just like the nerve cells of humans. Figure3 shows the structure of the neural network.Figure 3
Structure of the neural network model.We assume thatXit=xi,1t,xi,2t,…,xi,Dt is the ith individual (solution) in the population. The mutation operator aims to generate mutant solutions. For each solution Xi, a mutant solution Vi is created by the corresponding mutation scheme. There are some classical mutation schemes listed as follows:(1)
DE/rand/1:(1)vi,jt=xi1,jt+F⋅xi2,jt−xi3,jt.(2)
DE/rand/2:(2)vi,jt=xi1,jt+F⋅xi2,jt−xi3,jt+F⋅xi4,jt−xi5,jt.(3)
DE/best/1:(3)vi,jt=xbest,jt+F⋅xi1,jt−xi2,jt.(4)
DE/best/2:(4)vi,jt=xbest,jt+F⋅xi1,jt−xi2,jt+F⋅xi3,jt−xi4,jt,where i1,i2,i3,i4,andi5 are five randomly selected individual indices between 1 and N, and i1≠i2≠i3≠i4≠i5⋅F∈0,1 is usually used. Xbest is the global best individual (solution).The crossover operator focuses on recombining two different individuals and creates a new one. In DE, a trial solutionUi is created based on the following crossover operation:(5)swapXi1,Xi2,iffXi2<fXi1,where CR is called the crossover rate, the random value randj is in the range [0, 1], and jr is a randomly selected dimension index. As seen, Ui inherits from Vi and Xi based on the value of CR. For a large CR, most dimensions of Ui are taken from Vi. For a small CR, most dimensions of Ui are taken from Xi. For the latter case, Ui is similar to its parent Xi.


## 3. Design of the Experiment

### 3.1. Structure of the Original Model

The basic neural network structure consists of the following layers in sequence: convolutional layer, pooling layer, convolutional layer, pooling layer, and two fully connected layers [6, 7], which is shown in Figure 4. In the experiment plan, pruning is performed by default on the weight parameters w of the two fully connected layers. Alternative pruning is performed on all network parameters, and the specific operations are executed by changing the command line parameters [8, 9].Figure 4
Basic structure of the neural network of the experiment.


### 3.2. Experiment Plan

The experiment is based on the TensorFlow framework and used MNIST as the dataset. An original model is trained in the beginning, and then six pruning practices with different pruning proportions are employed [10, 11]. For each pruning, retraining and sparsification are subsequently performed. When all three operations are completed on the original model, the task of pruning compression is also finished [12, 13]. Then, the data are collected and analysed for comparison (size, accuracy, and computing time for making predictions).


## 3.1. Structure of the Original Model

The basic neural network structure consists of the following layers in sequence: convolutional layer, pooling layer, convolutional layer, pooling layer, and two fully connected layers [6, 7], which is shown in Figure 4. In the experiment plan, pruning is performed by default on the weight parameters w of the two fully connected layers. Alternative pruning is performed on all network parameters, and the specific operations are executed by changing the command line parameters [8, 9].Figure 4
Basic structure of the neural network of the experiment.


## 3.2. Experiment Plan

The experiment is based on the TensorFlow framework and used MNIST as the dataset. An original model is trained in the beginning, and then six pruning practices with different pruning proportions are employed [10, 11]. For each pruning, retraining and sparsification are subsequently performed. When all three operations are completed on the original model, the task of pruning compression is also finished [12, 13]. Then, the data are collected and analysed for comparison (size, accuracy, and computing time for making predictions).


## 4. Experimental Procedures

### 4.1. Run Command of the Pruning Experiment

Model pruning is executed by the following command: python train.py −1 −2 −3 --train_data_dir /tmp/mnist_data --train_dir /tmp/mnist_train --variables_dir /tmp/mnist_variables --max_steps 10000 --batch_size 32 --sparse_ratio 0.9 --pruning_variable_names w_fc1, w_fc2. Table1 specifies the parameters in this command [14–16].Table 1
Descriptions of parameters of the pruning command.
Parameter
Description
Default value
−1
Most basic model training
−2
Pruning is performed on the most basic model
−3
Pruning is performed, and network parameters of sparsification are stored
--train_data_dir
Folder path containing training data
/tmp/mnist_data
--train_dir
Log data during training and folder path containing model data
/tmp/mnist_train
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations of the basic model
10,000
--batch_size
Sample data size of each batch during model training
32
--sparse_ratio
Percentage of compression parameter of pruning/volume of parameters set to 0
0.9
--pruning_variable_names
Parameters on which pruning could be performed. Optional parameters include w_conv1, w_conv2, w_fc1, and w_fc2
w_fc1, w_fc2


### 4.2. Pruning Effect View Command

The effects of the −1 or −2 parameters can be viewed through eval_predict_with_dense_network.py. The specific command is python eval_predict_with_dense_network.py --test_data_dir /tmp/mnist_data --checkpoint_dir /tmp/mnist_train/step_2_2 --batch_size 32 --max_steps 10. Table2 specifies the parameters in this command [17–19].Table 2
Descriptions of parameters of the pruning effect view command (view effects of −1 or −2 training).
Parameter
Description
Default value
--test_data_dir
Folder path containing test data
/tmp/mnist_data
--checkpoint_dir
Folder path of the trained model
/tmp/mnist_train/step_2_2
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations during model testing
10
--batch_size
Size of sample data of each batch used for calculation during model testing
32The effect of −3 sparsification can be viewed through eval_predict_with_sparse_network.py. The specific command is python eval_predict_with_sparse_network. py --test_data_dir /tmp/mnist_data --checkpoint_dir /tmp/mnist_train/step_3 --batch_size 32 --max_steps 10. Table3 specifies the parameters in the command.Table 3
Descriptions of parameters of the pruning effect view command (view effects of −3 sparsification).
Parameter
Description
Default value
--test_data_dir
Folder path containing test data
/tmp/mnist_data
--checkpoint_dir
Folder path of the trained model
/tmp/mnist_train/step_3
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations during model testing
10
--batch_size
Size of sample data of each batch used for calculation during model testing
32


### 4.3. Hardware and Software Configuration of the Experiment

Pruning experiments are based on the following hardware and software parameters and versions [20, 21]:Operating system: Windows 10GPU: NVIDIA GeForce GTX 1080 Ti 11.0 GCPU: Intel(R) Core(TM) i3-4160 CPU @ 3.60 GHzMemory: 16.0 GB DDR3Disk: Lenovo SSD SL700 240GSoftware: TensorFlow-GPU 1.5.0, Python 3.6


### 4.4. Construction of Experimental Environment

The experiment is based on the MNIST dataset and the TensorFlow framework. The experimental environment was constructed by the following three steps [22–24]:Step 1: constructing the Python environment. Directly following Anaconda and then directly adding and running Anaconda3-4.3.1-Windows-x86_64.exe.Step 2: constructing the plug-ins of NVIDIA GPU. Directly running the cuda_9.0.103_win10.exe for installation. Unzipping cudnn-9.0-windows10-x64-v7.zip and copying its contents to the folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0.Step 3: constructing the TensorFlow environment. Executing the installation command on the CMD commands: pip install tensorflow-gpu = = 1.5.0.


## 4.1. Run Command of the Pruning Experiment

Model pruning is executed by the following command: python train.py −1 −2 −3 --train_data_dir /tmp/mnist_data --train_dir /tmp/mnist_train --variables_dir /tmp/mnist_variables --max_steps 10000 --batch_size 32 --sparse_ratio 0.9 --pruning_variable_names w_fc1, w_fc2. Table1 specifies the parameters in this command [14–16].Table 1
Descriptions of parameters of the pruning command.
Parameter
Description
Default value
−1
Most basic model training
−2
Pruning is performed on the most basic model
−3
Pruning is performed, and network parameters of sparsification are stored
--train_data_dir
Folder path containing training data
/tmp/mnist_data
--train_dir
Log data during training and folder path containing model data
/tmp/mnist_train
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations of the basic model
10,000
--batch_size
Sample data size of each batch during model training
32
--sparse_ratio
Percentage of compression parameter of pruning/volume of parameters set to 0
0.9
--pruning_variable_names
Parameters on which pruning could be performed. Optional parameters include w_conv1, w_conv2, w_fc1, and w_fc2
w_fc1, w_fc2


## 4.2. Pruning Effect View Command

The effects of the −1 or −2 parameters can be viewed through eval_predict_with_dense_network.py. The specific command is python eval_predict_with_dense_network.py --test_data_dir /tmp/mnist_data --checkpoint_dir /tmp/mnist_train/step_2_2 --batch_size 32 --max_steps 10. Table2 specifies the parameters in this command [17–19].Table 2
Descriptions of parameters of the pruning effect view command (view effects of −1 or −2 training).
Parameter
Description
Default value
--test_data_dir
Folder path containing test data
/tmp/mnist_data
--checkpoint_dir
Folder path of the trained model
/tmp/mnist_train/step_2_2
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations during model testing
10
--batch_size
Size of sample data of each batch used for calculation during model testing
32The effect of −3 sparsification can be viewed through eval_predict_with_sparse_network.py. The specific command is python eval_predict_with_sparse_network. py --test_data_dir /tmp/mnist_data --checkpoint_dir /tmp/mnist_train/step_3 --batch_size 32 --max_steps 10. Table3 specifies the parameters in the command.Table 3
Descriptions of parameters of the pruning effect view command (view effects of −3 sparsification).
Parameter
Description
Default value
--test_data_dir
Folder path containing test data
/tmp/mnist_data
--checkpoint_dir
Folder path of the trained model
/tmp/mnist_train/step_3
--variables_dir
Folder containing data of the last trained model
/tmp/mnist_variables
--max_steps
Number of iterations during model testing
10
--batch_size
Size of sample data of each batch used for calculation during model testing
32


## 4.3. Hardware and Software Configuration of the Experiment

Pruning experiments are based on the following hardware and software parameters and versions [20, 21]:Operating system: Windows 10GPU: NVIDIA GeForce GTX 1080 Ti 11.0 GCPU: Intel(R) Core(TM) i3-4160 CPU @ 3.60 GHzMemory: 16.0 GB DDR3Disk: Lenovo SSD SL700 240GSoftware: TensorFlow-GPU 1.5.0, Python 3.6


## 4.4. Construction of Experimental Environment

The experiment is based on the MNIST dataset and the TensorFlow framework. The experimental environment was constructed by the following three steps [22–24]:Step 1: constructing the Python environment. Directly following Anaconda and then directly adding and running Anaconda3-4.3.1-Windows-x86_64.exe.Step 2: constructing the plug-ins of NVIDIA GPU. Directly running the cuda_9.0.103_win10.exe for installation. Unzipping cudnn-9.0-windows10-x64-v7.zip and copying its contents to the folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0.Step 3: constructing the TensorFlow environment. Executing the installation command on the CMD commands: pip install tensorflow-gpu = = 1.5.0.


## 5. Experimental Results and Analysis

In this section, six different pruning proportions are employed in this experiment. The six groups of tables show the specific data of pruning proportion, model size, accuracy, and computing time for predictions.


### 5.1. 10% Pruning Proportion

First, the pruning proportion is set to 10%; Table4 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.012034996 and 0.013038448. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 2,890,137 and 9,215, respectively, making exactly 10% of the parameter values of the two fully connected layers equal to 0. However, the model size after the pruning, retraining, and sparsification is 66.5 M, which is larger than the size (37.5 M) of the original model. Hence, no compression effect is achieved. In addition, compared with the original model, the accuracy does not change, and the computing time for predictions slightly increases [25, 26].Table 4
Descriptions of parameters of the pruning effect in the first scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.012034996
2,890,137
w_fc2
10,240
0.013038448
9,215


### 5.2. 30% Pruning Proportion

Second, the pruning proportion is set to 30%; Table5 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.036936015 and 0.039559085. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 2,247,884 and 7,167, respectively, making exactly 30% of the parameter values of the two fully connected layers equal to 0. However, the model size after the pruning, retraining, and sparsification is 51.8 M, which is larger than the size (37.5 M) of the original model. Hence, no compression effect was achieved. Again, compared with the original model, the accuracy does not change, and the computing time for predictions slightly increases.Table 5
Descriptions of parameters of the pruning effect in the second scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.036936015
2,247,884
w_fc2
10,240
0.039559085
7,167


### 5.3. 50% Pruning Proportion

Third, the pruning proportion is set to 50%; Table6 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.06429165 and 0.068891354. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 1,605,631 and 5,119, respectively, making exactly 50% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 37.1 M, which is slightly smaller than the size (37.5 M) of the original model. Here, compression takes effect. Besides, both accuracy and computing time for predictions slightly decrease as compared with those of the original model.Table 6
Descriptions of parameters of the pruning effect in the third scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.06429165
1,605,631
w_fc2
10,240
0.068891354
5,119


### 5.4. 70% Pruning Proportion

Fourth, the pruning proportion is set to 70%; Table7 shows the parameters of pruning effect in the fourth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.09749276 and 0.10360378. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 963,379 and 3,071, respectively, making exactly 70% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 22.3 M, which is smaller than the size (37.5 M) of the original model. The compression effect is obvious. Moreover, both accuracy and computing time for predictions slightly decrease as compared with those of the original model.Table 7
Descriptions of parameters of the pruning effect in the fourth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.09749276
963,379
w_fc2
10,240
0.10360378
3,071


### 5.5. 80% Pruning Proportion

Fifth, the pruning proportion is set to 80%; Table8 shows the parameters of pruning effect in the fifth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.11903707 and 0.12662686. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 642,252 and 2,047, respectively, making exactly 80% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 14.9 M, which is smaller than the size (37.5 M) of the original model, and compression is 60%. Additionally, as compared with the original model, the accuracy slightly decreases and the computing time for predictions slightly increases.Table 8
Descriptions of parameters of the pruning effect in the fifth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.11903707
642,252
w_fc2
10,240
0.12662686
2,047


### 5.6. 90% Pruning Proportion

Lastly, the pruning proportion is set to 90%; Table9 shows the parameters of pruning effect in the sixth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.14814831 and 0.15710811. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 321,126 and 1,023, respectively, making exactly 90% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 7.6 M, which is compressed by 80%. Furthermore, both accuracy and computing time for predictions slightly decreased as compared with those of the original model.Table 9
Descriptions of parameters of the pruning effect in the sixth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.14814831
321,126
w_fc2
10,240
0.15710811
1,023


### 5.7. Comparison Results

Figure5 shows the comparison results for persistence model size of the four networks, with the pruning ratio increases, and the model size represented by the red columns decreases gradually. Apparently, the pruning proportion is positively correlated with the model size.Figure 5
Comparison results for persistence model size of the four networks.Figure6 shows the comparison results for testing accuracy of the four networks, with the pruning ratio increases, and the testing accuracy represented by the red columns has no obvious changes. This means that there is no positive relationship between pruning proportion and accuracy.Figure 6
Comparison results for testing accuracy of the four networks.Figure7 shows the comparison results for computing time of the four networks. With the pruning ratio increases, the computing time for prediction represented by the red columns changes irregularly. Also, there is no positive relationship between pruning ratio and computing time for predictions.Figure 7
Comparison results for computing time of the four networks.


## 5.1. 10% Pruning Proportion

First, the pruning proportion is set to 10%; Table4 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.012034996 and 0.013038448. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 2,890,137 and 9,215, respectively, making exactly 10% of the parameter values of the two fully connected layers equal to 0. However, the model size after the pruning, retraining, and sparsification is 66.5 M, which is larger than the size (37.5 M) of the original model. Hence, no compression effect is achieved. In addition, compared with the original model, the accuracy does not change, and the computing time for predictions slightly increases [25, 26].Table 4
Descriptions of parameters of the pruning effect in the first scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.012034996
2,890,137
w_fc2
10,240
0.013038448
9,215


## 5.2. 30% Pruning Proportion

Second, the pruning proportion is set to 30%; Table5 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.036936015 and 0.039559085. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 2,247,884 and 7,167, respectively, making exactly 30% of the parameter values of the two fully connected layers equal to 0. However, the model size after the pruning, retraining, and sparsification is 51.8 M, which is larger than the size (37.5 M) of the original model. Hence, no compression effect was achieved. Again, compared with the original model, the accuracy does not change, and the computing time for predictions slightly increases.Table 5
Descriptions of parameters of the pruning effect in the second scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.036936015
2,247,884
w_fc2
10,240
0.039559085
7,167


## 5.3. 50% Pruning Proportion

Third, the pruning proportion is set to 50%; Table6 shows the parameters of pruning effect in the first scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.06429165 and 0.068891354. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 1,605,631 and 5,119, respectively, making exactly 50% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 37.1 M, which is slightly smaller than the size (37.5 M) of the original model. Here, compression takes effect. Besides, both accuracy and computing time for predictions slightly decrease as compared with those of the original model.Table 6
Descriptions of parameters of the pruning effect in the third scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.06429165
1,605,631
w_fc2
10,240
0.068891354
5,119


## 5.4. 70% Pruning Proportion

Fourth, the pruning proportion is set to 70%; Table7 shows the parameters of pruning effect in the fourth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.09749276 and 0.10360378. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 963,379 and 3,071, respectively, making exactly 70% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 22.3 M, which is smaller than the size (37.5 M) of the original model. The compression effect is obvious. Moreover, both accuracy and computing time for predictions slightly decrease as compared with those of the original model.Table 7
Descriptions of parameters of the pruning effect in the fourth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.09749276
963,379
w_fc2
10,240
0.10360378
3,071


## 5.5. 80% Pruning Proportion

Fifth, the pruning proportion is set to 80%; Table8 shows the parameters of pruning effect in the fifth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.11903707 and 0.12662686. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 642,252 and 2,047, respectively, making exactly 80% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 14.9 M, which is smaller than the size (37.5 M) of the original model, and compression is 60%. Additionally, as compared with the original model, the accuracy slightly decreases and the computing time for predictions slightly increases.Table 8
Descriptions of parameters of the pruning effect in the fifth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.11903707
642,252
w_fc2
10,240
0.12662686
2,047


## 5.6. 90% Pruning Proportion

Lastly, the pruning proportion is set to 90%; Table9 shows the parameters of pruning effect in the sixth scene. In this group of experiments, the parameter threshold values of the two fully connected layers are set to 0.14814831 and 0.15710811. In this way, the valid parameter numbers are reduced from 3,211,264 and 10,240 to 321,126 and 1,023, respectively, making exactly 90% of the parameter values of the two fully connected layers equal to 0. The model size after pruning, retraining, and sparsification is 7.6 M, which is compressed by 80%. Furthermore, both accuracy and computing time for predictions slightly decreased as compared with those of the original model.Table 9
Descriptions of parameters of the pruning effect in the sixth scene.
Original number of parameters
Threshold value
Valid parameter number
w_conv1
800
0.0
800
w_conv2
51,200
0.0
51,200
w_fc1
3,211,264
0.14814831
321,126
w_fc2
10,240
0.15710811
1,023


## 5.7. Comparison Results

Figure5 shows the comparison results for persistence model size of the four networks, with the pruning ratio increases, and the model size represented by the red columns decreases gradually. Apparently, the pruning proportion is positively correlated with the model size.Figure 5
Comparison results for persistence model size of the four networks.Figure6 shows the comparison results for testing accuracy of the four networks, with the pruning ratio increases, and the testing accuracy represented by the red columns has no obvious changes. This means that there is no positive relationship between pruning proportion and accuracy.Figure 6
Comparison results for testing accuracy of the four networks.Figure7 shows the comparison results for computing time of the four networks. With the pruning ratio increases, the computing time for prediction represented by the red columns changes irregularly. Also, there is no positive relationship between pruning ratio and computing time for predictions.Figure 7
Comparison results for computing time of the four networks.


## 6. Conclusions

By comparing the experimental data of six different pruning proportions, it is found that pruning does not necessarily compress the size of the model. Compression takes effect only when the pruning proportion reaches 50% or more. Furthermore, we found a positive relationship between the pruning proportion and the model size. However, there was no positive relationship between pruning proportion and accuracy and between pruning proportion and computing time for predictions.Since there is no specific experimental verification for other models, the conclusion does not apply to other models. Additionally, the experimental is based on the pruning method, pruning is only one of the compression methods of various models; thus, the conclusion of this study is not applicable to other compression methods [27–29].



---

*Source: 5218612-2020-04-30.xml*