Title: CAPACITY OF DEEP NEURAL NETWORKS UNDER PA-RAMETER QUANTIZATION

Abstract: Most deep neural networks (DNNs) require complex models to achieve high performance. Parameter quantization is widely used for reducing the implementation complexities. Previous studies on quantization were mostly based on extensive simulation using training data. We choose a different approach and attempt to measure the per-parameter capacity of DNN models and interpret the results to obtain insights on optimum quantization of parameters. This research uses artificially generated data and generic forms of fully connected DNNs, convolutional neural networks, and recurrent neural networks. We conduct memorization and classification tests to study the effects of the number and precision of the parameters on the performance. The model and the per-parameter capacities are assessed by measuring the mutual information between the input and the classified output. We also extend the memorization capacity measurement results to image classification and language modeling tasks. To get insight for parameter quantization when performing real tasks, the training and test performances are compared.

1. INTRODUCTION:

Deep neural networks (DNNs) have achieved impressive performance on various machine learning tasks. Several DNN architectures are known, and the most famous ones are fully connected DNNs (FCDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).

It is known that neural networks do not need full floating-point precision for inference (Dundar & Rose, 1995; Hwang & Sung, 2014; Lin et al., 2016) . A 32-bit floating-point parameter can be reduced to 8-bit, 4-bit, 2-bit, or 1-bit, but this can incur performance degradation. Therefore, precision should be optimized, which is primarily conducted by extensive computer simulations using training data. This not only takes much time for optimization but also can incorrectly predict the performance in real environments when the characteristics of input data are different from the training data.

In this study, we attempt to measure the capacity of DNNs, including FCDNN, CNN, and RNN, using a memorization and classification task that applies random binary input data. The per-parameter capacities of various models are estimated by measuring the mutual information between the input data and the classification output. Then, the fixed-point performances of the models are measured to determine the relation between the quantization sensitivity and the per-parameter capacity. The memorization capacity analysis results are extended to real models for performing image classification and language modeling, by which the parameter quantization sensitivity is compared between memorization and generalization tasks.

The contributions of this paper are as follows.

• We experimentally measure the memorization capacity of DNNs and estimate the perparameter capacity. The capacity per parameter is between 2.3 bits to 3.7 bits, according to the network structure, which is FCDNN, CNN, or RNN. The value is fairly independent of the model size. • We show that the performance of the quantized networks is closely related to the capacity per parameter, and FCDNNs show the most resilient quantization performance while RNNs suffer most from parameter quantization. The network size hardly effects the quantization performance when DNN models are trained to use full capacity.

• We explain that severe quantization, such as binary or ternary weights, can be employed without much performance degradation when the networks are in the over-parameter region.

• We suggest the sufficient number of bits for representing weights of neural networks, which are approximately 6 bits, 8 bits, and 10 bits for FCDNNs, CNNs, and RNNs, respectively. This estimate of the number of bits for implementing neural networks is very important considering that many accelerators are designed without any specific training data or applications.

• The study with real-models shows that neural networks are more resilient to quantization when performing generalization tasks than conducting memorization. Thus, the optimum bits obtained with the memorization tasks are conservative and safe estimate when solving real problems.

The paper is organized as follows. In Section 2, previous works on neural network capacity and fixedpoint optimization are briefly presented. Section 3 explains the capacity measurement methods for DNN models. Section 4 presents parameter capacity measurement results for FCDNNs, CNNs, and RNNs. The quantization performances measured on DNNs are presented in Section 5. Concluding remarks follow in Section 6.

2. RELATED WORKS AND BACKGROUNDS:



2.1. NEURAL NETWORK CAPACITY:

The capacity of neural networks has been studied since the early days of DNN research. Although the capacity can be defined in many ways, it is related to the learnability of networks. The capacity of networks is shown as the number of uncorrelated random samples that can be memorized (Cover, 1965) .

A single-layer perceptron with n parameters can memorize at least 2n random samples (Gardner & Derrida, 1988) . In other words, the network can always construct a hyperplane with n parameters that divides 2n samples. Additionally, the capacity of a three-layer perceptron is proportional to the number of parameters (Akaho & Amari, 1990) . Recently, RNNs were trained with random data to measure the capacity per parameter (Collins et al., 2017) . Our study is strongly motivated by this research, and extends it to the quantization performance interpretation of generic DNN models, including FCDNN, CNN, and RNN. Recent studies have showed that neural networks have a generalization ability even if the expressive capacity of the model is sufficiently large (Zhang et al., 2017; Arpit et al., 2017) . In this paper, we also discuss the effect of network quantization when performing generalization tasks.

2.2. FIXED-POINT DEEP NEURAL NETWORKS:

Early works on neural network quantization usually employed 16-bit parameters obtained by directly quantizing the floating-point numbers (Dundar & Rose, 1995) . Recently, a retraining technique was developed to improve the performance of quantized networks (Hwang & Sung, 2014; Lin et al., 2016) .

Retraining-based quantization was applied to CNN and RNN models, showing superior performance compared to directly quantized ones (Anwar et al., 2015; Shin et al., 2016) . Many studies attempting extreme quantization have been published, such as 2-bit ternary (Hwang & Sung, 2014; Li et al., 2016; Zhu et al., 2017) , 1-bit binary weight quantization, and XNOR networks (Courbariaux et al., 2015; Rastegari et al., 2016) . Some aggressive model compression techniques also employed vector quantization or table look-up (Han et al., 2015a; Boo & Sung, 2017) . However, not all CNNs show the same quantization performance. For example, AlexNet (Krizhevsky et al., 2012) shows almost the same performance with only 1-bit quantized parameters. However, the same quantization technique incurs a very severe performance loss when applied to ResNet (Rastegari et al., 2016) . A previous study shows that large sized networks are more resilient to severe quantization than smaller ones (Sung et al., 2015) . Theoretical works and many practical implementation optimization techniques have been studied (Han et al., 2015b; Iandola et al., 2016; Kim & Smaragdis, 2016; Sakr et al., 2017; Louizos et al., 2017) . Recent work increases the number of network parameters to preserve the performance under low-precision quantization (Mishra et al., 2017) . Our works are not targeted to a specific data or model, but introduce the general understanding of parameter quantization.

3. NETWORK CAPACITY MEASUREMENTS OF DNNS:



3.1. CAPACITY MEASUREMENTS ON A MEMORIZATION TASK:

We assess the network capacity of DNN models using a random data memorization and classification task (Collins et al., 2017) . In this task, N random binary vectors, X, are generated and each is randomly and uniformly assigned to the output label Y . The size of the binary vector depends on the DNN model. For FCDNN, the input X is a one dimensional vector whose size is determined by the hidden layer dimension. In CNN, the input needs to be a 2-D or 3-D tensor. Input samples of CNNs are generated by concatenating and reshaping random binary vectors. During the training process, the DNN is trained to correctly predict the label, which is 0 or 1, of the random input X. As the number of input data size, N , increases, the classification accuracy drops because of the limited memorization capacity. Note that the accuracy for the memorization task refers to the training performance after convergence because there is no proper test dataset for random training samples.

The capacity is measured using the mutual information, defined as a measure of the amount of information that one random variable contains about another random variable (Cover & Thomas, 2012) . The mutual information of a trained network with N input samples is calculated as follows:

EQUATION

)

where p is the mean classification accuracy for all samples under trained parameter θ. The network capacity is defined as

EQUATION

The accuracy, p, may vary depending on the training method of the model. We find N and p that maximize the mutual information of the networks by iteratively training the models. This optimization employs both grid search-and Bayesian optimization-based hyper-parameter tuning (Brochu et al., 2010) . The optimization procedure consists of three stages. First, we try to find the largest input data size whose accuracy is slightly lower than 1. Second, we perform a grid search to determine the boundary values of the hyper-parameters. The searched hyper-parameters can include initialization, optimizer, initial learning rate, learning rate decay factor, batch size, and optimizer variables. Finally, we conduct hyper-parameter tuning within the search space using Scikit-learn library (Pedregosa et al., 2011) . We add the number of training samples N as a hyper-parameter and use the mutual information of Eq. ( 1) as the metric for the optimization.

3.2. NETWORK QUANTIZATION AND PARAMETER CAPACITY:

Quantization of model parameters perturbs the trained network, therefore, fixed-point training or retraining with full-precision backpropagation is usually needed (Hwang & Sung, 2014; Li et al., 2016; Courbariaux et al., 2015; Zhou et al., 2017) . However, the performance of the quantized networks does not always meet that of the floating-point models, even after retraining. This suggests that model capacity is reduced by quantization, especially when the number of bits used is very small.

In this research, we observe the memorization capacity degradation caused by quantization in generic FCDNN, CNN, and RNN models. The uniform quantization is used for the sake of convenient arithmetic, and the same step size is assigned to each layer in the FCDNN, each kernel in the CNN, or each weight matrix in the LSTM layer. The bias values are not quantized, because they have a large dynamic range. It is important to note that the weights connected to the output are not quantized, because their optimum bit-widths depend on the number of labels in the output. Quantization is performed from floating-point to 8-bit, 6-bit, 5-bit, 4-bit, 3-bit, and 2-bit precision, in sequence.

Retraining is performed after every quantization, but requires only a small number of epochs, because only fine-tuning is needed (Hwang & Sung, 2014) . 

2

EQUATION

Here, L(θ) is the loss according to the network parameters. The distribution of weights may vary depending on the model size and learning method. We apply the normalized filter noise to the θ noise for fair comparison on different models (Li et al., 2017) .

We employ two real networks, one is for image classification with CIFAR-10 dataset and the other is language modeling with Penn Tree Bank (PTB) dataset. One large and one small model are trained for these networks. We quantize those networks with the precision of 8, 6, 4, and 2 bits and analyze the variation of the surface according to the precision. θ noise is added to the quantized parameters. In order to reduce the error due to randomness of noise, all loss values are measured with 10 different trials and the average values are plotted.

4. EXPERIMENTAL RESULTS ON CAPACITY OF FLOATING-POINT DNNS:

The capacities of FCDNNs, CNNs, and RNNs are measured via the memorization task explained in Section 3.1. The models used for the test employ floating-point parameters.

4.1. CAPACITY OF FCDNNS:

The training data for FCDNNs is a 1-D vector of size n in . N input data are used as for the training data. The output, Y , is the randomly assigned label, either 0 or 1, for each input. Thus, inputs, X and Y , are represented as X ∈ {0, 1} N ×nin and Y ∈ {0, 1} N , respectively. The input data dimension, n in , should be larger than log 2 N so that no overlapped data is contained among N input data. In the experiments for FCDNNs, the input vector dimension, n in , is chosen to be equal to the number of units in the hidden layer.

We conduct experiments for FCDNNs with hidden layer dimensions of 32, 64, 128, and 256, and with hidden layer depths of 1, 2, 3, and 4. The initialization method chosen is the 'He' initialization (He et al., 2004) and gradients are updated following the rule in SGD, with momentum, which shows the best performance in our grid search. The initial learning rate for hyper parameter tuning is chosen between 0.001 and 0.05 on the log scale. The decay factor and momentum are set to have even distance values in the linear scale between 0.1 and 0.5 and between 0.6 and 0.99, respectively. For each model, experiments are conducted to measure the accuracy of memorization while increasing the size of the input data, N . Note that only the training error is measured in this memorization task, because there is no unseen data. Experimental results are based upon the best accuracy obtained when attempted with different hyper parameters. The capacity of the model is estimated according to Eq. ( 1), where p is the training accuracy. The experimentally obtained memorization capacities of the FCDNN models are presented in Fig. 1 , where depths of 1, 2, 3, and 4, and widths of 32, 64, 128, and 256 are used. When the number of hidden layers is the same, the amount of data that can be almost perfectly memorized quadruples when the dimension of the hidden layer is doubled. This means that the memorization capacity is linearly proportional to the number of parameters. Similarly, the FCDNN models with 2, 3, or 4 hidden layer depths can memorize 2, 3, or 4 times the input data as compared to the single layer DNN, respectively. Fig. 2 (a) shows the memorization accuracy and the mutual information obtained using Eq. ( 1) on the FCDNN. The model is composed of three layers and the hidden layer of size 64. Here, we find that the amount of mutual information steadily increases as the input data size grows. However, it begins to drop as the input size grows farther, and the memorization accuracy drops. By analyzing the accuracy trend of the model, it is possible to distinguish the input data size into three regions: the over-parameterized, the maximum performance, and the under-parameterized sections, as shown in Fig. 2(a) . For example, if the model is trained to memorize only 10,000 data, it can be regarded as over-parameterized. The number of data that can be memorized by maximally utilizing all the parameters is between 30,000 and 40,000. In over-parameterized regions, performance can be maintained, even if the capacity of the networks is reduced.

The per-parameter capacity of FCDNNs is shown in Fig. 2(b ). Regardless of the width or depth, one parameter has a capacity of 1.7 to 2.5 bits, and FCDNNs have an average of 2.3-bit capacity per parameter. This result is consistent with theoretical study (Gardner & Derrida, 1988; Akaho & Amari, 1990) . The total capacity of the model may be interpreted as an optimal storage that can store a maximum of random binary samples (Gardner & Derrida, 1988; Bollé et al., 1991) .

4.2. CAPACITY OF CNNS:

The capacity of CNNs is also measured via a similar memorization task. CNNs can have a variety of structures according to the number of channels, the size of the kernels, and the number of layers. The kernel size of CNNs in this test are either (3 × 3) or (5 × 5), which are the same for all layers, the number of convolution layers from 3 to 9. The dimensions of the inputs are n height = n width = 32 and n channel = 1 for all experiments. Three max-pooling operations are applied to reduce the number of parameters in the fully connected layer. CNN structures used in our experiments are shown in Supplementary materials.

The CNN models contain not only convolution layers but also fully connected layers. Thus, the per-parameter capacity for convolution layers is calculated after subtracting the capacity for fully connected layers from the measured total capacity. We assume the per-parameter capacity of the fully connected layer as 2.3 bits to calculate the capacity for convolution layers. As shown in Fig. 2(b) , the convolution layers have the per-parameter capacity of between 2.86 and 3.09 except the smallest model, which is higher than that of FCDNNs. The average capacity per parameter of the tested models is 3.0 bits. Results show that the per-parameter capacity of CNNs is higher than that of FCDNNs, even when CNNs memorize uncorrelated data. Note that one parameter of FCDNNs is used only once for each inference. However, the parameter of CNNs is used multiple times. This parameter-sharing nature of CNNs seems to increase the amount of information that one parameter can store.

4.3. CAPACITY OF RNNS:

It has been shown that the various structures of RNNs all have similar capacity per parameter of 3 to 6 bits (Collins et al., 2017) . We train RNNs with a dataset with no sequence correlation to show the capacity of the parameters. The random input dataset is composed of inputs, X ∈ {0, 1} N ×nseq×nin and labels Y ∈ {0, 1} N , which are uniformly set to 0 or 1. The training loss is calculated using the cross-entropy of the label at the output of the last step.

We train RNNs with a single LSTM layer of 32-D. The input dimension, n in , is also 32-D and the amount of unrolling sequence, n seq , is five-step. It has been reported that unrolling of five-step almost saturates the performance in this setup (Collins et al., 2017) . We apply 5 input random vectors, X 0 , X 1 , X 2 , X 3 , and X 4 , each with 32-D, and assign one label to this 160-D vector at the last time step. The error propagates from the last step only, and the outputs at intermediate time-steps are ignored.

The number of parameters in the network is 8,386. In this case, the maximum mutual information is obtained when the number of samples is 32K, and the memorization accuracy is 99.52 %. Therefore, the per-parameter capacity of the model is 3.7 bits. The RNN shows higher per-parameter capacity than FCDNNs and CNNs. We have shown that FCDNNs, CNNs, and RNNs have different per-parameter capacities. According to the parameter-data ratio, a trained DNN can be an over-parameterized, max-capacity, or underparameterized model. Thus, we can assume that the DNN performance under quantization would depend on not only the network structure, such as FCDNN, CNN, or RNN, but also the parameter-data ratio. The experiments are divided into two cases. The first is to measure performance degradation via quantization precision when each model is in the maximum capacity region. The second analyzes performance when the models are in the over-parameterized region.

When the FCDNN, CNN, and RNN are trained to have the maximum memorization capacity, the performances with parameter quantization are shown in Fig. 3 (a). The FCDNN, CNN, and RNN models are shown. The fixed-point performances of two FCDNNs, two CNNs, and two RNNs are illustrated. With 6-bit parameter quantization, the FCDNN shows no accuracy drop. However, those for CNNs and RNNs are 5 % and 18 %, respectively. Because the RNN contains the largest amount of information at each parameter, the loss caused by parameter quantization seems to be the most severe. We also find that there is no decline in performance until the parameter precision is lowered to 6-bit for FCDNNs, 8-bit for CNNs, and 10-bit for RNNs, even when all models use full capacity.

Next, we show the fixed-point performance of DNNs when they are trained to be in the overparameterized region. Note that the per-parameter capacity is lowered in the over-parameterized region. We conducted simulation with half size of the maximum number of data that can be memorized. For example, an FCDNN used for the measurement has 3 hidden layers with a hiddenlayer dimension of 128; the capacity of the corresponding model is about 2 17 bits. The network is

Under review as a conference paper at ICLR 2019 2-bit 3-bit 4-bit 5-bit 6-bit 8-bit float 0.85 0.9 0.95 1 Precision Accuracy FC( 8) FC( 16) FC( 32) FC( 64) FC( 128 

5.2. QUANTIZATION EXPERIMENTS ON REAL TASKS:

We have assessed the required precision of networks for performing memorization tasks. The memorization test only uses the training data that are artificially generated. However, most neural networks should conduct more than memorization because the test data are not seen during the training. In this section, we analyze the effects of network quantization for performing real tasks. We train two different sized CNN models with CIFAR-10 data. The structures of the two models are as follows:

Small : 2 × 16C -M P 2 -2 × 32C -M P 2 -2 × 64C -M P 2 -2 × 128F C -10out (4) Large : 2 × 64C -M P 2 -2 × 128C -M P 2 -2 × 256C -M P 2 -2 × 512F C -10out.

The size of kernels of both models is (3×3), 16C represents a convolution layer with 16 channels and 128F C means a fully connected layer with 128-dimension. The number of parameters is 0.22M for the small model and 3.5M for the large one. Both models were trained with the same hyper-parameter setting.

To analyze the impact of network quantization on the test performance, we plot the loss and accuracy surfaces of floating-point and quantized CNN models in Fig. 5 . For simplicity, the results of floatingpoint and 2-bit fixed-point CNN are given. Please refer to the Supplementary materials for other results. When applying the training data that may have been memorized during the training phase, the large model shows indifferent performance surface regardless of the parameter precision. But, for the small model, the 2-bit model shows quite degraded performance when compared to the floating-point network. However, the test accuracy of small 2-bit model is not much lowered. We can notice that the loss surface of the 2-bit model shown in Fig. 5 is much wider than that of the floating-point model. This result is consistent with recent studies on generalization (Keskar et al., 2017; Li et al., 2017) . This observation suggests that the quantized networks are more resilient when performing generalization tasks. Thus, the required precision of the network obtained with the memorization task can be considered a conservative estimate. T iny model has the same structure as the small model, but reducing the number of channels by half and the size of the fully connected layers by a quarter. The RNNs are trained for language modeling with Penn Treebank corpus (Marcus et al., 1993) , and the models consists of two LSTM layers with the same dimension. Here, both for CNN and RNN models, we can confirm that large networks are more robust to quantization. Also, the networks need more parameter precision when conducting memorization tasks using the training set, rather than solving real problems using the test set.

We have measured fixed-point DNN performance on real tasks and results are shown in Fig. 7 . FCDNN models are trained with MNIST dataset, ResNet (He et al., 2016) models are trained using ILSVRC-2012 dataset (Russakovsky et al., 2015) and RNN based word-level language models (WLMs) are designed using PTB dataset. Experimented FCDNNs and RNNs are composed of two FC layers and two LSTM layers with the same dimension, respectively. 4-bit quantized FCDNNs shows almost same performance compared to the floating-point networks even when the number of neurons are only 8. Performances are preserved up to 6 bits on ResNets and RNN WLMs.Their resiliency to quantization increases as networks become larger.

6. CONCLUDING REMARKS:

Quantization of parameters is a straightforward way of reducing the complexity of DNN implementations, especially when VLSI or special purpose neural processing engines are used. Our study employed simulations on varying sizes of generic forms of DNN models. Memorization tests using random binary input data were conducted to determine the total capacity by measuring the mutual information. Our simulation results show that the per-parameter capacity is not sensitive to the model size, but is dependent on the structure of the network models, such as FCDNN, CNN, and RNN. The maximum per-parameter capacities of FCDNNs, CNNs, and RNNs are approximately 2.3 bits, 3.0 bits, and 3.7 bits per parameter. Thus, RNNs have the tendency of demanding more bits when compared to FCDNNs. We quantized DNNs under various capacity-utilization regions and showed that the capacity of parameters are preserved up to 6 bits, 8 bits, and 10 bits on FCDNNs, CNN, and RNNs, respectively. The performance of the quantized networks was also tested with image classification and language modeling tasks. The results show that networks need more parameter precision when conducting memorization tasks, rather than inferencing with unseen data. Thus, the precision obtained through the memorization test can be considered a conservative estimate in implementing neural networks for solving real problems. This research not only gives valuable insights on the capacity of parameters but also provides practical strategies for training and optimizing DNN models.

C LOSS SURFACES OF FIXED-POINT DNNS:

The loss surfaces of fixed-point CNNs are shown in Fig. 8 

:



Figures and Tables:

If the training accuracy is 1, the model memorizes all random samples and the I(Y ; Ŷθ |X) becomes the number of samples N . If the training accuracy is 0.5, I(Y ; Ŷθ |X) goes to 0. Please consult Supplementary materials for details of Eq. (1).

Figure 1: Memorization performances according to the (a) width and (b) depth of the FCDNN.

Figure 2: (a) Mutual information according to the number of inputs N . (b) The relationship between the number of parameters and the capacity of networks in FCDNNs and CNNs.

Figure 3: Quantization effect when networks use the (a) full capacity and (b) half capacity.

Figure 5: Loss surface and accuracy of floating-point and fixed-point CNNs. α denotes the scale of the parameter noise.

Figure 7: (a) Accuracy of fixed-point FCDNN on MNIST dataset. (b) Top-1 test accuracy of fixed-point ResNets trained on ImageNet dataset. (c) Perplexity of fixed-point WLMs on PTB testset.

Fig. 6 shows the training and test data based performance of fixed-point CNN and RNN on real data.T iny model has the same structure as the small model, but reducing the number of channels by half and the size of the fully connected layers by a quarter. The RNNs are trained for language modeling with Penn Treebank corpus(Marcus et al., 1993), and the models consists of two LSTM layers with the same dimension. Here, both for CNN and RNN models, we can confirm that large networks are more robust to quantization. Also, the networks need more parameter precision when conducting memorization tasks using the training set, rather than solving real problems using the test set.

Figure 8: Loss surface of fixed-point large model trained with CIFAR-10 dataset. The top and bottom plots are the results for the training and test dataset, respectively.

Figure 9: Loss surface of fixed-point small model trained with CIFAR-10 dataset.

Figure 10: Loss surface of fixed-point RNN LM with two 32-dimensional LSTM layers trained using PTB dataset.

Peer review (iclr2019):
 Missing references about VC dimension /shattering, Rademacher complexity, etc. I would like to see more discussion and motivation of using MI to measure capacity (of NNs, or in other contexts). I am not an expert in this area, so I do not know what the appropriate references would be, but only one reference about MI is cited. The paragraph under eqn (1) is helpful, but the relationship between classification accuracy and MI still seems unclear to me. I also find it is not clear exactly what the MI is measured between - this should be made clearer in the main text, and also in appendix B. Xi and f(theta, Xi) are defined, but nothing else is. It is fairly easy for someone familiar with the notation to guess what the variables refer to, but this should be explicitly stated to avoid confusion and ambiguity. What exactly is Y? The output of the softmax? Aside from the lack of clarity about notation/what symbols represent, I have two main issues with appendix B. First is the claim "Note that X and Y are independent as well as Yi and Yj when i neq j"; I think it should be made clearer that X and Y are independent only in this setting because the experiments are with random data, and it does not seem obvious to me that Yi and Yj are independent. Second, using average accuracy as the probability that the class is the true class seems strange to me for random data. Maybe it*s fine and I just haven*t wrapped my head around it, or I misunderstood something, but I would like to understand why it makes sense to do this. Is it standard not to quantize bias values? why does having a large dynamic range mean they shouldn*t be quantized? I find section 3 not very clear. I understand why the loss under perturbation is interesting, but this should be discussed along with what are the results of your experiments. The procedure for doing this should also be cited, unless you came up with it on your own in which case the justification for it should be further discussed. I assume you don*t do this, but the way the section is worded makes it sound like you train networks with a noisy loss. It would be interesting to investigate the claims about parameter-sharing in CNNs (and RNNs) increasing their capacity as measured by MI; e.g. plot MI vs. #times filter is used for different convnet configurations, and #timesteps for RNNs. Quality: 6/10 The writing is mostly clear to follow, although paragraphs don*t always flow well into each other and some sections assume a lot of knowledge. Clarity: 7/10 Results are clearly presented, although more intuition and context would be helpful, and some sections are not well explained or contextualized Originality: 2/10 The methods are not novel, to my knowledge, and there is little exploration or insight given for the extensive experimental results Significance: 7/10 measuring and reporting results is valuable, and the reported results are interesting Pros: - interesting results, good plots - overall well structured and explained Cons: - plots could use more explanation and interpretation in the captions, and more investigation and insight from the experiments - some sections are not clearly worded and I think the objective/interpretation of the experiments would not be clear to someone even a little unfamiliar with the approaches cited Specific comments/nits (in order reading through paper): 1. First paragraph of intro is kind of fluff/unnecessary. I would suggest beginning with the 3rd paragraph and work in the second. 2. It is usually advised in scientific writing not to use phrases like "it is known". "It has been shown by [refs]" would be better. 3. "in the over-parameter region" -> overparameterized 4. "real-models" -> real models (although this seems like a bit of an odd thing to say to me. What are fake models?? maybe "empirical study" would be better) 5. "Related work and backgrounds" remove s 6. "capacity ... is related to the learnability of networks" I don*t think this makes it more clear; it just brings up the question about what you mean by learnabilty. Maybe "how much a network can learn to capture about a dataset" or something like that. 7. "is shown as" odd phrasing, maybe "is defined as? 8. "classification task (Collins et al)" -> "classiification task, as in Collins et al" 9. mention what scikit learn uses for hyperparameter tuning / why you use this 10. Figure 1 would benefit from a bit more explanation in the caption (e.g. memorization performance on/of what; explain what the acronyms mean and what the results tell us) 11. Some of the hyperparameter specifications here could be put in an appendix if you need space. Some of this information is also not specific to FCDNNs and should go in the preamble before the subsections for each architecture 12. Figure 2: Why does accuracy go to 1.4? Like Figure 1, caption would benefit from more information and some interpretation. Why do you think the MI levels off/goes up on the right hand side of A? 13. References about capacity here at the end of 4.1 should go in the background and related work section 14. "Result is consistent with theoretical study" In what ways? How consistent is it? What does the theory say? 15. Mention if you use truncated bptt in the RNN experiments 16. Mention what it means for "all models use full capacity" 17. VLSI mentioned without definition 18. "RNNs have the tendency of demanding more bits when compared to FCDNNs" I find this sentence unclear 19. "conducting memorization tasks, rather than inferencing with unseen data" -> saying "generalization tasks" would be more accurate; maybe this is being too picky but they can inference with unseen data just fine wihtout requiring more parameter precision; it*s only if we care about the generalization performance that they require more parameter precision.