Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a meta learning approach to select the best trade-of between memory usage and bit precision for DNN training. This approach is evaluated on 87 datasets and 99 bit-precision configurations. The result shows that in general 6 bit floats (1,4,1) for activations and weights and 14 bit floats (1,6,7) for optimizer parameters is the best choice in term of memory and accuracy trade-off.  
main_review: Strengths:
1- The proposed approach is evaluated in large and variant benchmarks which means the proposed approach can be generalized to various datasets and models
2- The paper is well written, and the algorithms and methods are clearly explained.

Weaknesses:
1- The quality of the proposed approach based on convergence metric and hypervolume difference is not obvious.  
2- The overhead of meta learning approach in term of time needs to be reported ( how many hours it takes to select precision configuration ) 
3- I would recommended to compare the meta learning approach with other multi objective approaches and report the overhead of each method.
4- The related works such as [1,2] on selecting trade-off between precision and others metric is not discussed in the paper. 
[1] Hashemi, Soheil, et al. "Understanding the impact of precision quantization on the accuracy and energy of neural networks." Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017. IEEE, 2017.
[2]  Langroudi, Hamed F., et al. "Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge." arXiv preprint arXiv:1908.02386 (2019).
summary_of_the_review: As a reviewer mentioned, the idea behand the paper is interesting. However, there are couple of comments needs to be addressed. I recommend this paper marginally accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two-step methodology for evaluating the error-memory trade-off when employing low-precision training/inference in neural networks. Although low-precision training/inference in deep learning is a widely studied topic, this paper seems to be the first one to study this error-memory tradeoff without exhaustive search. 

The proposed idea is inspired from the multi-objective optimization field. The authors propose to identify the Pareto frontier that would allow a user to identify the optimal precision (lowest error) given a certain memory constraint. The proposed system has two phases: training and testing. 

At training time, given a set of datasets, a network architecture and a set of low-precision configurations, the method samples dataset-configuration pairs to train and computes the misclassification error for each selected pair. Matrix factorization is used to compute low-dimensional embeddings for each configuration. 

At test time, a subset of configurations is chosen for evaluation (using Design Experiments with Matrix Factorization) and the computed errors are used as input together with the configuration embeddings to a linear regression model in order to estimate the errors on the non-evaluated configurations. For both training and testing, the memory matrix is fully computed based on the network architecture and the low-precision format.

The method is evaluated in terms of convergence and Hypervolume difference of the true and estimated Pareto frontier and it outperforms in most of the cases other methods such a Bayesian Optimization or Random Selection with Matrix Factorization.

main_review: The authors propose an interesting, one-of-a-kind two-step approach for error-memory trade-off in low-precision training/inference of neural networks. I appreciate the design of the method, the utility of the method (I find it very useful for the community working in the field) as well as the extensive set of measurements. The paper however could be improved especially in terms of clarity, especially in the experimental section. Please find below a few questions below:

- Figure 1(b): it is not clear what the numbers in the parenthesis are. Memory requirements?
- Is the meta-training performed per neural network architecture? This is my understanding, and if this is so, please clearly specify this in the description of the method (e.g., in Figure 2).
- Was regularization used in the linear regression model? If so, how sensitive is the estimated Pareto frontier to the regularization parameter?
- Configuration formats A and B (Section 2): the weights should also be mentioned, as you are using one of the two formats for weights as well, not only for the optimizer and activation.
- How is the memory computed? Maximum memory measured at train/inference time? Or theoretical computation knowing the neural network architecture? My understanding is the latter, as you compute the full memory matrices both at train and test times. It should be clearly mentioned in the paper.
- Figure 6b deserves more discussion. How it was generated, what it means. This is important as it supports one of the main properties of the method: the rapid decaying of the singular values of the error matrix. On a related note, how to choose the dateset-configuration data-points to guarantee this property?
- Please define the propensity score.
- In section 3.2, please define all the notations (e.g., yi).
- In section 4.2 how was rank 3 chosen?
- Figure 11: it is not clear what the x axis represents. For setting I, there is no memory cap, thus what does relative memory usage mean?
- In Section 4, the authors mention different neural network architectures: ResNet-18, ResNet-34 and VGG. For which architecture are the results shown in Section 4? If for all of them, how are the performance metrics aggregated across architectures? Averaged across architectures? This is unfortunately not clear.
summary_of_the_review: Although there are some items that should be better explained / clarified, overall I enjoyed reading this paper and find the addressed topic of high interest for the community working in the field of low-precision training/inference of neural networks. Although I am not familiar with the prior art, the paper seems to be the first to propose a method for analyzing the error-memory trade-off in a deep learning low-precision training/inference scenario without exhaustive search. 

I like the two-step meta-training/meta-test approach, however I am not yet fully convinced about the main property of the meta-training algorithm, namely the rapid decaying of the singular values of the error matrix. In terms of evaluation, I appreciate the authors analyzing the performance of the meta-training approach using multiple metrics and providing recommendations for the optimal sampling conditions. 

Also for the meta-test phase, the authors analyze the performance of the ED-MF approach to other methods such as bayesian optimization, random selection and QR with matrix factorization. ED-MF out-performs these methods in most of the cases. Some results are not clear unfortunately - see my questions in the main review above. Hopefully the results will become more clear during the paper discussion / rebuttal phase.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to find Pareto frontiers for the precision given memory budget. The overall objective is to allow ML engineers to select optimal precision that lets them train the desired model with less energy. To do this, the paper formulates the problem into multi-objective optimization problem and use meta-learning to minimize the number of low-precision training data points for this optimization. Then, the paper uses the term "meta-test" that refers to transferring the information from known tasks to estimate Pareto frontiers of low-precision training on new unseen tasks.
main_review: The paper formulates the determination of low-precision configuration as a multi-objective optimization. The overall objective is to allow ML engineers to select optimal precision that lets them train the desired model with less energy.

The key idea seems to be using meta-learning and testing to determine these Pareto frontiers. The overall methodology is sound and is evaluated thoroughly. However, one limitation is that it doesn't provide evaluation on ImageNet which usually is a good reference point for the readers.

I am not an expert in low-precision training. However, I like the overall idea where they formulated the configuration determination into a multi-objective optimization setting.

Questions:
* Are there results for energy reduction?
summary_of_the_review: I believe the paper proposes interesting approach in selecting low-precision training configurations. Formulating into multi-objective optimization and using meta-training/testing to find Pareto frontier of configurations for the overall accuracy seems like the key contribution of the paper. I am not an expert on low-precision training, but to the best of my knowledge, the overall approach seems reasonable and the evaluation seems okay. However, I believe it would provide better reference for the readers if the authors provide evaluation on ImageNet.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To achieve the goal of efficiently selecting the best low-precision configuration within the memory budget, this paper proposed Pareto Estimation to Pick the Perfect Precision (PEPPP) by using matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. Although this paper presents a solution for low-precision training that has recently gained a lot of attention, there are some concerns as a result of my review of this paper. (Please refer to my main review)
main_review: 1. Incomplete experimental results
1-1) There is a limitation that most of the experiments were performed on CIFAR-10. Also, experimental results on more networks (e.g., MobileNet, EfficientNet, more ResNets) should be included.
1-2) Although the proposed method considers both training and inference, performance comparison with existing SOTA studies for each phase should be performed.
1-3) It is necessary to show whether the proposed method is effective not only for classification but also for more practical detection and segmentation networks.

2. It is difficult to understand the actual use-case of the proposed method. Most of the experimental results also appear to support the feasibility of meta-training. Therefore, it is difficult to be sure how practical the proposed method will be. It would be good to think about this comment in relation to the comments about additional experiments (i.e., Comment 1).

3. It is difficult to agree with the argument of this paper because the motivation of this paper is not sufficiently presented. The motivation of this study needs to be clearly presented at the beginning to secure differentiation from many recently published low-precision training studies.

4. It is necessary not to just consider memory capacity as a resource, but to consider practical indicators such as power consumption or memory R/W portion as the target of trading.

5. Although the proposed method suggests optimization considering the memory budget, it is necessary to consider the computational power together. It is not practical to consider only the memory budget because various designs using memory reuse, etc. are possible depending on the computation method/capability.

6. It would be nice if comparative analysis and comparison results with custom hardware designs are presented.
summary_of_the_review: The motivation and contribution need to be made clearer. In addition, the experiment part that supports the contribution needs to be thoroughly supplemented. Please address my concerns in "Main Review" through the rebuttal process.
review_title: null

Meta-review: The paper considers a learning problem to determine the best low-precision configuration within the memory budget.  It is an interesting problem that could be of interest to the community.  Overall, the reviewers were fairly positive on the paper and believe the paper give interesting insights into how to use limited memory for learning.
Decision: Accept (Poster)