Question: Perhaps providing the evidence of Loshchilov & Hutter (2018)'s AdamW algorithm increasing overparameterization would strengthen the anecdote at the end of S2.3?
Evidence: We first conduct experiments on the MNIST dataset using the LeNet-300-100 and LeNet-5 architectures. In Figure (left), the distribution of weights when training with LLR representation is compared with the case of softmax normalization. Using LLR representation yields a better distribution of weights -the probability of small weights around zero is higher and these weights can be pruned with less impact on the performance. Furthermore, weight decaying can push the weights aggressively towards zero as shown in Figure (right). While softmax normalization primarily affects output layers, the ReLU function may cause overfitting in all layers. Therefore, the effect of weight decaying is more prominent and effective as shown in Figure. This observation is consistent with the analysis and hypotheses from Section 2. Figure shows the trade-off curves for test errors vs. total number of effective parameters for all experiment results on the MNIST dataset. Each curve represents 20 trained networks with the same training settings, each point represents the average total number of weights and average top-1 errors of the pruned networks for each of the 10 different pruning settings. We can see that using LLR representation instead of softmax normalization can reduce the total number of parameters for the same accuracy requirement. Using weight decaying also significantly improve the efficiency of the trained networks. Using both methods yields the most efficient neural networks with better performance than the ones using the iterative pruning approach from, as shown in  We found that the AdamW optimizer with weight decaying may increase training accuracy by increasing overfitting and yield less efficient networks, as demonstrated in Figure. Compared with previous results, the optimal pruned model sizes are dramatically increased and using weight decaying does not improve the efficiency of trained networks.  Therefore, the trade-off curves can be used to judge the efficiency of trained networks. Curves closer to the bottom-left region on the figure represent more efficient networks. Both ResNet and DenseNet architectures show similar trends in terms of efficiency. The trade-off curves for the same architecture with different initial model sizes seem to be bounded by a single theoretical curve. In the energy efficient region with small number of parameters, the error rate goes down rapidly with a small increase in the number of parameters; while in the high accuracy region, small increases in accuracy require exponentially increasing numbers of parameters. In this regard, the ResNet-32 and the DenseNet-60 architectures may offer better alternative trade-offs in efficiency. For all experiments using either softmax normalization or LLR representation, the same weight-decaying settings are used. In terms of efficiency trade-offs, we see a similar trend as before: linear increase in accuracy tends to require exponential increase in network capacity. We notice that the difference between softmax normalization and LLR representation is less prominent for the DenseNet architecture. One possible reason is that the BSP loss function is not yet fully optimized for the DenseNet architecture. Another possible reason is that the effects of weight decaying are more prominent than softmax normalization for the DenseNet architecture with larger initial model sizes.
We found that the AdamW optimizer with weight decaying may increase training accuracy by increasing overfitting and yield less efficient networks, as demonstrated in Figure 3. Compared with previous results, the optimal pruned model sizes are dramatically increased and using weight decaying does not improve the efficiency of trained networks.****For DenseNet-60 with less than 90K parameters, the performance is comparable to ResNet-56 with 200K parameters. Therefore, overfitting issues with the DenseNet architecture are less prominent than the ResNet architecture. Figure 7 summarize the efficiency trade-offs for both ResNet and DenseNet architectures. Compared with ResNet architecture, the initial DenseNet model sizes are larger, the effects of weight decaying are more prominent than the softmax normalization, and the difference between softmax normalization and LLR representation for DenseNet is smaller.
Answer: Additional results and analysis for using AdamW algorithm are added to Section 3.2 to show that the AdamW algorithm may product less efficient networks. Additional analysis of the overfitting issue is also provided in the Appendix.