Question: What are the novel contributions from this work?
Evidence: The efficiency of neural networks is essential in large-scale deployment scenarios such as mobile applications, internet of things, and edge computing. For a given performance requirement, an efficient neural network should use the simplest network architecture with a minimal number of parameters and connections. In this paper, we introduce a framework to analyze and obtain efficient neural networks. In summary, our main contributions are three-fold. Our first contribution is the subnetwork hypothesis to address overfitting issues and help explain the effectiveness of several key techniques in training efficient networks: 1) softmax normalization in output layers may be one major cause of overparameterization; 2) using log likelihood ratio representation in output layers can reduce overfitting; 3) weight decaying and structural regularization can also effectively reduce overfitting. The second contribution is a simple and effective snapshot-based procedure to prune a well-trained network that minimizes overfitting -pruning unimportant weights and connections first, and simply adjust remaining non-weight parameters using the backpropagation algorithm. Besides, the snapshot-based pruning method can also be used to evaluate the efficiency of trained networks. Finally, we hypothesize that there exist lower bounds of the total number of bits for representing parameters and connections regarding performance metrics for a given optimization problem. Rather than focusing on improving the sole accuracy metric with more complex network architectures, it is also important to explore the trade-offs between accuracy and the total number of representation bits, when comparing different network architectures and implementations.
Our first contribution is the subnetwork hypothesis to address overfitting issues and help explain the effectiveness of several key techniques in training efficient networks: 1) softmax normalization in output layers may be one major cause of overparameterization; 2) using log likelihood ratio representation in output layers can reduce overfitting; 3) weight decaying and structural regularization can also effectively reduce overfitting. The second contribution is a simple and effective snapshot-based procedure to prune a well-trained network that minimizes overfitting – pruning unimportant weights and connections first, and simply adjust remaining non-weight parameters using the backpropagation algorithm. Besides, the snapshot-based pruning method can also be used to evaluate the efficiency of trained networks. Finally, we hypothesize that there exist lower bounds of the total number of bits for representing parameters and connections regarding performance metrics for a given optimization problem. Rather than focusing on improving the sole accuracy metric with more complex network architectures, it is also important to explore the trade-offs between accuracy and the total number of representation bits, when comparing different network architectures and implementations.
Answer: The main contributions of this work: * We propose the subnetwork hypothesis that could explain the issues with Softmax normalization and ReLU layers. Although not rigorously proved, this view could be adopted for further theoretical study of neural network. * We propose the LLR representation and a new loss function (bipolar soft-plus), which is first adopted in this paper. LLR representation could help with the explainability of neural network. In our follow-up work, LLR representation has been extend to all hidden layers. * We show that a well-trained network can be pruned and retrained by adjusting only the non-weight parameters, and hence the novel snapshot-base pruning procedure. This demonstrates that the distribution of weights are very important and correlated with the efficiency of neural networks. * The fundamental framework in the paper for analyzing network efficiency is novel, which can be used to analyze existing optimization techniques. By adopting a holistic view on accuracy and model efficiency, many conventional techniques for improving accuracy may need to be revisited.