How Low Can We Go: Trading Memory for Error in Low-Precision Training
Authors: Chengrun Yang
Ziyang Wu
Jerry Chee
Christopher De Sa
Madeleine Udell
Venue: ICLR 2022 Poster (2022)
Keywords: low-precision training
meta-learning
Pareto frontier
error-memory tradeoff
active learning
matrix factorization

Abstract: Low-precision arithmetic trains deep learning models using less energy, less memory and less time. However, we pay a price for the savings: lower precision may yield larger round-off error and hence larger prediction error. As applications proliferate, users must choose which precision to use to train a new model, and chip manufacturers must decide which precisions to manufacture. We view these precision choices as a hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. In this paper, we introduce Pareto Estimation to Pick the Perfect Precision (PEPPP). We use matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. For any given memory budget, the precision that minimizes error is a point on this frontier. Practitioners can use the frontier to trade memory for error and choose the best precision for their goals.

Published as a conference paper at ICLR 2022
HoW Low CAN WE Go: TRADING MEMORY FOR ERROR IN LOW-PRECISION TRAINING
Chengrun Yang', Ziyang Wu",Jerry Chee, Christopher De Sa, Madeleine Udell Cornell University {cy438, zw287, jc.3444, cmd353, udell 1 @cornell cemmmelll.e edu
ABSTRACT
Low-precision arithmetic trains deep learning models using less energy, less mem- ory and less time. However, we pay price for the savings: lower precision may yield larger round-off error and hence larger prediction error. As applications proliferate, users must choose which precision use train chip manufacturers must decide which precisions to manufacture. We view these precision choices hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. In this paper, we introduce Pareto Estimation Pick the Perfect Precision (PEPPP). We use matrix factorization to find non-dominated configurations (the Pareto frontier) with lim- ited number of network evaluations. For any given memory budget, the precision that minimizes eπror is point on this frontier. Practitioners can use the frontier to trade memory for error and choose the best precision for their goals.
model, and
INTRODUCTION
Training modern-day neural networks is becoming increasingly expensive as task and model sizes increase. The energy consumption of the corresponding computation has increased alarmingly, and raises doubts about the sustainability of modern training practices (Schwartz et al., (019). Low-precision training memory (Sohoni et al., 2019a), thus minimizing the cost and energy to train larger models and cesource-limited users.
reduce consumption of both computation (De Sa et al.] 2018) and
making deep learning more accessible
Low-precision training replaces 32-bit or 64-bit floating point numbers with fixed or floating point numbers that allocate fewer bits the activations, optimiz- ers, and weights. A rich variety of meth-
0.8
test error
0.6
2636 (180)
0.4
14% (10a)
0.2
ods appear in recent literature, includ- ing bit-centering (De Sa et al.) 2018). loss scaling (Micikevicius et al. 2017) and mnixed-precision training (Zhou et al.
150
200
250
300
20% (156)
22%
memory (MB)
(a) error vs memory on
(b) # activation bits in non-
CIFAR-10 Krizhevsky
dominated configurations
2016). There is however a fundamental tradeoff: lowering the number of bits of precision increases the quantization er- ror, which may disrupt convergence and increase downstream error (Zhang et al. 2016, Courbariaux et al.) 2014 Gupta et al. 2015). The low-precision configura- tion is thus hyperparameter to chosen according to the resource constraints of the practitioner.
2009
Figure 1: Test error vs memory for ResNet- 18 across 99 low-precision floating point configurations. Figure (a) shows the tradeoff on CIFAR-10. (Non-dominated points are blue circles.) Figure (b) shows that the best precision to use varies depending the memory budget, 87 image datasets. See Section 4 for experimental details.
How should we choose this hyperparameter? Many believe the highest allowable precision (given memory budget) generally produces the lowest error model. However, there are typically many ways
Equal contribution.

---

Published as a conference paper at ICLR 2022
to use the same amount of memory. As shown in Figure I(a), some of these configurations produce much lower error than others! Figure [[bb] shows that no one configuration dominates all others. We might consult previous literature to choose a precision; but this approach fails for new applications.
shows
Our goal is to efficiently pick the best low-precision configuration under a memory budget. Efficiency is especially important for snoourc-constrainee practitioners, such individual users or early-stage startups. To promote efficiency, we use a meta-learning (Lemke et al.) 2015; Vanschoren 2018, Hospedales et al. 2020) approach: we train a small number of cheap very-low-precisionn models on the dataset to choose the perfect precision. The gains from choosing the right low low-precision format can offset the cost of this extra training - but each precision must be chosen carefully to realize the benefits of low- oowpprecision training.
We use ideas from multi-objective optimization to characterize the tradeoff between memory and error and identify the Pareto frontier: the set of non-dominated solutions. Users will want to understand the tradeoff between error and memory so they can determine the resources needed to adequately train a model for a given machine learning task. This tradeoff may also influence the design of application-specifie low-precision hardware, with profound implications for the future (Hooker) 2020). For example, among all 99 low low-precision configurations we tried, we identified some configurations that are Pareto optimal across many different tasks (listed in Appendix D). These results could help hardware manufacturers decide which precision settings worth manufacturing.
Computing the Pareto frontier by training models for all low-precision configurations is expensive and unnecessary. Cloud computing platforms like Google Cloud and Amazon EC2 charge more for machines with the same CPU and double memory: 25% more on Google Cloud and 100% more on Amazon EC2 as of mid-September 2021. We use techniques from meta-learning to leverage the information from other low-precision training runs on related datasets. This approach allows us to estimate the Pareto frontier without evaluating all of the www-reciisonn configurations.
Our system, Pareto Estimation to Pick the Perfect Precision (PEPPP), has two goals. The first goal to find the Pareto frontiers of a collection of related tasks, and is called meta-training in the meta-learning literature. Meta-training requires a set of measurements, each collected by training and testing neural network with a given precision on a dataset. This information can be gathered offline with a relatively large resource budget, or by crowdsourcing amongst the academic or open-source community. Still, it is absurdly expensive to exhaustively evaluate all measurements: that is, every possible low-precision configuration on every task. Instead, we study how to choose a subset of the possible mcasurements to achieve the best estimate. The second goal we call meta-test: using the information learned on previous tasks. how can we transfer that information to new task to efficiently estimate its Pareto frontier? This goal corresponds to a resource- conottrccon individual or startup who wants to determine the best low-precision configuration for a new dataset.
Both meta- meta-training and meta-test rely on matrix completion and active learning techniques to avoid exhaustive search: we make subset of all possible measurements and predict the rest. We then estimate the Pareto frontier to help the user make an informed choice of the best configuration. We consider two sampling schemes: uniform and non-uniform sampling. Uniform sampling is straightforward. Non-uniform sampling estimates the Pareto frontier more efficiently by making fewer or even no high-memory measurements.
To the best of our knowledge, PEPPP is the first to study the error-memory tradeoff in low-precision training and inference without exhaustive search. Some previous works show the benefit of low- precision arithmetic in significant energy reduction at the cost of a small accuracy decrease (Hashemi et al. (017). ) and propose hardware-softwaar codesign frameworks to select the desired model (Lan- groudi et al. 2019). Many other papers focus on low-precision inference only and use it for model compression (Wu et al. 2018, Cai & Vasconcelos) 2020). Another line of work seeks state-of-the-art (SOTA) performance with low-precision training. using carefully designed learning schedules (Sun) (ct al. 22019) 2020). Our work imagines low-precision as a way to reduce training or deployment costs of customized models trained on proprietary datasets (not SOTA models on ML benchmarks). Even so, we show in Section 4 that our work picks promising models for every memory budget, which enables near-SOTA results on CIFAR-10.
2

---

Published as a conference paper at ICLR 2022
meta-training
matrix
configuration embeddings
configuration
Coafiig
Coufie
meta-training datasets
configuration errors
compule
datasets
0.71 0.32
matrix Factorizarion
0.80
0,80.0.0
Cunfig 3
Confii 1
10.45
weights
optimizer
meta-test
predicted configuration error
Confii
configuration error
Config 2
select configurations evaluate
linear regressioc
0.74
0:30
0.62: 74.0...74.0.32
configuration memory
new (metn-lest) dataset
test IeST
compute memories
601 805 1105 1032
finul selected configuration
exciated deniastd
estimated Denisited
valid (beyend meeaay limi)
memory
memory
limit
Figure 2: The PEPPP workflow. We begin with a collection of (meta-) training datasets and low precision configurations. In the meta-tret meta-training phase, sample and compute the misclassification error. We use matrix factorization to compute low dimensional embedding of every configuration. In the meta-test phase, our goal is to pick the perfect precision (within our memory budget) for the meta-test dataset. We compute the memory required for each configuration, and we select subset of fast, informative configurations to evaluate. By regressing the errors of these configurations on the configuration embeddings, we find an embedding for the meta-test dataset, which we use predict the error of every other configuration (including more expensive ones) and select the best subject to our memory budget.
pairs to train,
Figure 2 shows flowchart of PEPPP. The rest of this paper is organized as follows. Section introduces notation and terminology. Section DeationnJdecribbeee describes the main ideas we use to actively sample configurations and approximate Pareto frontiers. Section 4 shows experimental results.
NOTATION AND TERMINOLOGY
low-precision formats. We use floating point numbers for low-precision training in this paper. As an example, Figure shows an 8-bit floating point number with exponent bits and 4 mantissa bits.
4
3
specific low-preed oowprrcciiion representation with certain number of bits for each part is called a low-precision format. We may different low-precision formats for the weights, activations, and optimizer. In the case of using two formats train and represent a neural network (as discussed in Section 3.1. these two formats are called Format A and B. A specific combination of these two formats a hyperparameter setting of the neural network, and we call it low-precision configuration.
sign 3-bit exponent
mantissa
Figure 3: An 8- bit floating point number rep- resenting (-1)sign 2expoment- -.... by,b,,, by- -..bbbbb,
Math basics. define [n] = (1, for positive integer n. With Boolean variable X, indicator function 1 (X') equals if X' is true, and 0 otherwise. With a scalar variable we use x4 to denote max 0}. C and C denote subset and strict subset, respectively.
I Linear algebra. We denote vector and matrix variables respectively by lowercase letters (a) and capital letters (A), All vectors are column vectors. The Euclidean norm of vector a I2 is
||a|| == VΣ a. To denote a part of a matrix A E Rnxd we use a colon to denote the varying dimension: Ar. and A. (or a,) denote the ith row and jth column of A, respectively, and Ag denotes the (i. entry. With an ordered index set = (si...., $x) where $ < ... < 8 E [d. s E Ra Given vectors 2r, y IR x X means 3; y
denote A.s [A.s:
3

---

Published as a conference paper at ICLR 2022
for each i [n]. Given matrix A R"xd and set of observed indices as Q2 S [n] x [d]. the partially observed matrix (2(4) E Raxd has entries (Pa(A)),) = Ayy if (i,5) E Ω and otherwise.
Pareto frontier. Multi-objective optimization simultancously minimizes costs {c, A feasible
point e(1) = (cf" ..... cnl) is Pareto optimal if for any other feasible point c! (cl (2) ..... (2))
cl2) 3 c(l) implies cl2) = c(L) BBoyd& & Vandenberghe 2004). The set of Pareto optimal points is the Pareto frontier.
Tasks, datasets, measurements and evaluations. A lask carries out process (classification, regression, image segmentation, etc.) on a dataset. Given a deep learning model and a dataset, the training and testing of the model on the dataset is called measurement. In our low-precision context, we evaluate a configuration on a dataset to make a mcasurement.
Meta-learning. Meta-learning transfers knowledge from past (meta-training) tasks to better under- stand the new (meta-test) task. To evaluate the error of veta-training, we use meta-leave-one rross-validation (metv- and the others as meta-training, and average the results over all splits.
collection of tasks, each time we treat one task meta-test
Error matrix and memory matrix. Given a neural network, the errors of different low-precision configurations meta-training datasets form an error matrix, whose (i. entry Eig is the test error of the jth configura- tion on the ith dataset. To compute the error matrix, we split the i-th dataset into training and test subsets (or use given split), train the neural network at the j-th configuration on the training subset, and evaluate the test error on the test subset. The memory required for each measurement forms memory matrix M, which has the same shape as the correspond- ing error matrix. Example error and memory matrices are shown in Figure 4
low-precision configurations
low-precision configurations
b
d
datasets
0.85
0.71
0.32
0.28
datosets
269
272
295
272
245
1032
(a) error matrix E
(b) memory matrix M (MB)
Figure 4: Example error and memory matrices for some datasets and low- low-precision configurations. Dataset 1: CIFAR-10, 2: CIFAR- 100 (fruit and veg- etables), 3: ImageNet-stick (Deng et al. 2009). Con- liguration Format A (exponent bits, mantissa bits), Format B (exponent bits, mantissa bits). a: (3, 1), (6. 7); b: (3, 4), (7. 7); c: (4, 3). (8, 7); d: (5, 3), (6. 7).
3 METHODOLOGY
PEPPP operates in two phases: meta-training and meta-test. First in meta-training, we learn configu- ration embeddings by training a neural network of a specific architecture at different cow-precision configurations on different datasets. As listed in Appendix [A] we study image classification tasks and a range of low-precision formats that vary the number of bits for the exponent and mantissa for the activations, optimizer, and weights. Our hope is to avoid enumerating every possible configuration since exhaustive search is prohibitive in practice. To this end, we make a few measurements and then predict the other test errors by active learning techniques. We also compute the memory matrix to understand the memory consumption each measurement.
Then in meta-test, we assume that we have completed meta-training, and hence know the configuration embeddings. The meta-test goal is to predict the tradcoff between memory usage and test error on (meta-test) dataset. This step corresponds inference in traditional machine learning; it must be quick and cheap to satisfy the needs of essource-constrinnes practitioners. To select the configuration that has smallest test error and takes less than the memory limit, we measure a few selected (informative and cheap) configurations on the meta-test dataset, and use the information of their test errors to predict the rest. The problem of choosing measurements known as the "cold-start" problem in the literature on recommender systems. Then we use the model built on the meta- meta-training datasets estimate values for the other measurements on the meta-test dataset.
3.1 META-TRAINING
On the n meta-training datasets, first theoretically compute the memory needed to evaluate each of the d low- low-precision configurations to obtain the full memory matrix M E xx The total memory consists of memory needed for model weights, activations, and the optimizer (gradients, gradient
4

---

Published as a conference paper at ICLR 2022
2019b). Among these three types, activations typically dominate, as shown in Figure[3] Thus using lower precision for activations drastically reduces memory usage. Addi- tionally, empirical studies (Zhou et al. 2016) have shown that adopting higher precision formats for the optimizer can substantially improve the accuracy of the trained model. Since the optimizer usually re- quires relatively small memory, it is often the best to use higher precision format for this component. Thus for each low-precision configuration, it is typical to use a lower precision for network weights and activations, and higher precision for the optimizer, resulting in combinatorially many choices in total. An example of the memory usage in this low- wwwprecisioon scenario is also shown in Figure
accumulators, etc.) (Sohoni et al.
In practice ML tasks are often correlated: for example, meta-training datasets might be subsets of a large dataset like CIFAR-100 or Im-
mommor [MB)
500
filll precision
low precision
ageNet. The ranking of different configurations tends to be simi- lar on similar tasks. For example, we computed the test errors of 99 low-precision configurations on 87 datasets (both listed in Ap- pendix A to form a performance vector in IR for each dataset. We use the Kendall tau correlation to characterize the alignment be- tween the ranking of errors incurred by different configurations on two datasets: the Kendall tau correlation 1 if the order is the same and -1 the order is reversed. As shown in Figure 6(a), similar datasets have larger correlations: for example, datasets with indices 38-87 correspond to ImageNet subproblems such as distinguishing types of fruit or boat. Notice also that some dataset pairs have configuration performance rankings that are negatively correlated. The corresponding error matrix E concatenates the performance vector for each dataset. not low rank, but its singular values decay rapidly, as shown in Figure 6(b) Hence expect low rank approximation of this matrix from a few measurements to work well: is not necessary to measure every configuration.
600-
400
200
133
37
12 6
activations optiinizer
Weights
Figure 5: Memory us- age under two training paradigms. Both train a ResNet-18 on CIFAR-I0 with batch size 32.
In the uniform sampling scheme, we sample mea- surements uniformly at random to obtain partially observed error matrix (E). We then estimate the full error matrix E using a low rank matrix comple- tion method to form estimate É. In this paper, we use SOFTIMPUTE (Mazumder al. 2010 Hastie et al. 2013) (Algorithm [J Appendix B Using the estimated error matrix E and computed memory matrix M, we compute the Pareto frontier for each dataset understand the (estimated) mrooryemory tradcoff. Section 4 Figure shows an example.
distssct index
10
Inoge Not CIPRIe 100
100
datizeed
40
10"
index
(a) datasct correlation (b) singular value decay
Figure 6: Kendall tau correlation of test er- ror of all configurations between all pairs of datasets, and singular value decay of corre- sponding error matrix. Strong correlations al- low PEPPP to succeed with a few measure- ments. Details in Appendix
In the non-uniform sampling scheme, sample measurements with non-uniform probabilities, and weighted variant of SOFTIMPUTE, weighting by the inverse sampling probability, to complete the error matrix (Ma & Chen 2019). Then we estimate the Pareto frontier in the same way as above. Non-uniform sampling is useful to reduce the memory needed for measurements. To this end, we construct probability matrix P by entry-wise transforma- tion P. = o (1/M,,), in which the monotonically increasing : R + [0, 1] maps inverse memory usage to a sampling probability. this way, we make more low-memory measurements, and thus reduce the total memory usage.
3.2 META-TEST
Having estimated the embedding of cach configuration, our goal is to quickly compute the error- memory tradeoff on the meta-test dataset, avoiding the exhaustive search of all possible measurements.
We first compute the memory usage of each low-precision configuration on the meta-test dataset. With this information and guided by meta-training, evaluate only a few (cheap but informative) configurations and predict the rest. Users then finally select the non-dominated configuration with highest allowable memory. An example of the process is shown in Figure
5

---

Published as a conference paper at ICLR 2022
PEPPP uses Experiment Design with Matrix Factorization (ED-MF) described below to choose informative configurations. With more time, ED-MF evaluates more configurations, improving our estimates. We may also set hard cutoff memory: we do not evaluate low www-precision configurations exceeding the memory limit. This setting has been studied in different contexts, and is called active learning sequential decision making.
ED-MF picks measurements to minimize the variance of the resulting estimate. Specifically, we factorize the true (or estimated) error matrix E E RTxd (or E) into its best rank-k approximation, and get dataset embeddings
E Ra*k and configuration embeddings Y Rid as R X y (Fusi et al. 2018)) Yang et al.] (019). On a meta-test dataset, we denote the error and memory vectors of the configurations as gnew IRd and new IRd respectively. We model the error on the new dataset as enew Y new + IR where xeew R is the embedding of the new dataset and E Rdd accounts for the errors from both measurement and low rank decomposition. We estimate the embedding new from few measurements (entries) of enow by least squares. Note that this requires at least k measurements on the meta-test dataset to make meaningful estimations, in which & the rank for matrix factorization.
If E (N000 the variance of the es- timator is (Ejes UoU, in which Vj is the jth column of Y. D-optimal experi-
memory limit informative estinmted moe-deeniuaee estimated docuinated
test. ertor
ment design selects measurements on the
valid
new dataset by minimizing (a scalarization of) the variance,
950
200 memory (MB)
250
300
(beyour menocyy liuit)
final selected configuration
minimize subject to
log det EjE
|S| <1 S SC [d].
Figure 7: Meta-test CIFAR-10. After meta-training on all other datasets in Appendix A Table 2, we use ED-MF to choose six informative measurements (or- ange squares) with a 275MB memory limit for each measurement CIFAR-10 10. Then estimate test er- rors of other configurations by ED-MF, and restrict our attention to configurations that we estimate to be non-dominated (red x x's). Note some of these are in fact dominated, since we plot true (not estimated) test error! Finally we select the estimated non- -onmimnnetedd configu- ration with highest allowable memory (blue square).
(1) to find S, the set of indices of config- urations to evaluate. The positive inte- ger 7 bounds the number of measurements, Given memory cap Mmax. we replace the second constraint above by S C (,, in which T' (j E [d] myew < mimax) } is the set of feasible configurations. Since Problem D is combinatorially hard, we may either relax it to convex optimization problem by allowing decision variables to have non-integer values between and 1, or use a greedy method to incrementally choose measurements (initialized by measurements chosen by the column-pivoted QR decomposition (Gu & Eisenstat, 1996; Golub & Van Loan 2012)). ).. We compare these two approaches in Appendix C
In Section 4.2 we compare ED-MF with competing techniques and show works the best to estimate the Pareto frontier and select the final configuration.
4 EXPERIMENTS AND DISCUSSIONS
The code for PEPPP and experiments is in the GitHub repository at htttps::/ com/ punnggrunnann//ppppp We use QPyTorch (Zhang al. 2019) to simulate low-precision formats on standard hardware. The 87 datasets and 99 oowprecision configurations in experiments are listed in Appendix A The datasets consist of natural and medical images from various domains. Apart from CIFAR- 10. the datasets include 20 CIFAR-100 partitions from mutually exclusive subsets, based on the superclass labels. They also include 50 subsets of ImageNet (Deng et al.) 2009), which contains over 20,000 classes grouped into multiple major hierarchical categories like fungus and amphibian; each of the 50 datasets come from different hierarchies. Finally, we use 16 datasets from the visual domain benchmark (Wallace& & Hariharan 2020 increase the diversity of domains.
The 99 low www-precisoon configurations use mixed sixee-preccsion as described in Section 3.1 Format A
(for the activations and weights) uses 5 to bits; Format B (for the optimizer) ranges from 14 to

---

Published as a conference paper at ICLR 2022
20 bits. For each, these bits may be split arbitrarily between the exponent and mantissa. Across all configurations, we use ResNet-18, ResNet-34 and VGG (Simonyan & Zisserman (0144) variants (1 13, 16, 19) with learning rate 0.001, momentum 0.9, weight decay 0.0005 and batch size 32. Each training uses only 10 epochs as an early stopping strategy (Yao et al. 2007 to prevent overfitting. All the measurements take 35 GPU days on NVIDIA GeForce RTX 3090.
Developing method to select the above hyperparameters at the same time as the low- low-precision format an important topic for future research, but not our focus here. In Section 4.3 we demonstrate how PEPPP can naturally extend to efficiently select both optimization and low-precision hyperparameters.
number of metrics may be used to evaluate the quality of the obtained Pareto frontier in multi- objective optimization problem (Wu & Azarm 2001 Li et al.. 2014 Audet et al.) 2020 the distance between approximated and true frontiers (convergence), the uniformity of the distances between neighboring Pareto optimal points (uniformity), how well the frontier is covered by the approximated points (spread), etc. our setting, the memory accurate and the test error is estimated, so we do not have full control of the uniformity and spread of the 2-dimensional frontiers between test error and memory, As illustrated Figure 8 the following two metries:
we evaluate the quality of our estimated Pareto frontiers by
Convergence (Deb et al., 2002) between the sets of true and estimated Pareto optimal points P. P is [P] Ever dist(u; u, P), where the dis-
upper boundary
test erter
test enrur
tance between point and set P is dist(e, P) min - w| w E p). This is a surrogate for the distance between Pareto frontiers.
non-dominated
memorry
menoory
X estimtted non-dominated
(a) convergence
(b) HyperDiff
Hypervolume difference (HyperDiff) (Zitzler & Thiele 1998 Coello & Sierra 1999 Wu &
Figure 8: Illustraioon of Pareto frontier metrics. (a) Convergence is the average distance from each estimated Pareto optimal point to its closest true point: average(d], d2, d3). (b) HyperDiff is the absolute difference in area of feasible regions given by the true and estimated Pareto optimal points: the shaded area between Pareto frontiers.
Azarm 2001) is the absolute difference between
the volumes of solution spaces dominated by the true and estimated Pareto frontiers. This metric improves with better convergence, uniformity, and spread. Its computation requires an upper boundary for each resource.
When computing these metrics, we normalize the memories by proportionally scaling them to between and 1. To evaluate the matrix completion performance, we use the relative error defined as ||o /|l// in which 0 is the predicted vector and v is the true vector.
4.1 META-TRAINING
We study the effectiveness of uniform and non-uniform sampling schemes. For simplicity, we regard all 87 available datasets as meta- veea-training in this section. In SOFTIMPUTE, we use A = 0.1 (chosen by volssvvaliationn from a logarithmic scale); and we choose a rank 5 approximation, which accounts for 78% of the variance in the error matrix (as shown in Figure 6(b)
In the uniform sampling scheme, we investigate how many samples we need for accurate estimates: we sample the error matrix at a number of different ratios, ranging from 5% to 50%, and complete the uniform sampling, the sampling ratio also
error matrix from each set of sampled measurements. the percentage of memory we need to make the measurements in parallel, compared to exhaustive search. Hence we show the relationship between Pareto frontier estimation performance and sampling ratio in Figure We can see the estimates more accurate at larger sampling ratios, but sampling 20% the entries already suffices for good performance.
As more intuitive example, Figure 10 shows the error-memory tradeoff on CIFAR-10. Compared to the estimated Pareto optimal points at sampling ratio 5%, the ones at 20% are closer to the true Pareto frontier (better convergence). lie more evenly (better uniformity) and cover the true Pareto frontier better (better spread), as shown by the respective convergence and HyperDiff values.
The non-uniform sampling scheme has similar trend and is shown in Appendix E.2
7

---

Published as a conference paper at ICLR 2022
0.4
CHEMR 30
CHRRR:O
CHEAR X
0.4
0.2
0.
2
test arror
0.8
Lest GES
0.8
0.6
0.6
0.0
21
0.0
0.4
Sampping tatio
Gampling FITE
samplin ratio
0.2
0.2
(a) relative error
(b) convergence
(c) HyferDiff
950
200
250
500
150
200
250
300
memory (MB)
memory (MB)
Figure 9: Pareto frontier estimation in PEPPP meta- training. with uniform sampling of configurations. The violins show the distribution of the performance on individual datasets, and the error bars (blue) show the range. The red error hars show the standard de-
(a) sampling ratio 5%
(b) sampling ratio 20%
deauasted
oxtuntod condomiistttd
Figure 10: Error vs memory on CIFAR-10 with true and estimated Pareto frontiers from uni- form sampling in PEPPP meta-training. A 20% uniform sample of entries yields better esti- mate of the Pareto frontier (convergence 0.03 and HyperDiff 0.02) compared 5% sample (convergence 0.09 and HyperDiff 0.16).
viation of the error on CIFAR-10 across 100 random
samples of the error matrix. Figure [a) shows the matrix completion error for each dataset; Figure (b)
and [c]]w the performance of the Pareto frontier es-
timates. Modest sampling ratios (around 0.1) already
yield good performance.
4.2 META-LEAVE-ONE-OUT CROSS-VALIDATION (META-LOOCV)
Now suppose that we have already collected measurements on the meta-training datasets form meta-training error matrix E (or its low rank approximation E). On the meta-test dataset, PEPPP estimates the Pareto frontier by the active learning technique ED-ME. We compare ED-MF with few other active learning techniques: Random selection with matrix factorization (RANDOM-M) MF)) QR decomposition with column pivoting and matrix factorization (QR and two Bayesian optimization techniques (BO-MF and BQ-FULL), to understand whether the strong assumptions in ED-MF (low rank and Gaussian errors) are hindrance help. An introduction to these techniques can be found in Appendix E.I
We rank 3 for matrix factorization: Y.j= E R3 for each j [d]. In BO, we tune hyperpa- rameters on logarithmic scale and choose the RBF kernel with length scale 20, white noise with variance 1, and & = 0.01. Ta- ble D shows the meta-LOOCV settings for each acquisition technique. We compare the techniques at range of number of configura- tions to measure in each meta-LOOCY split, resembling what practitioners do hyperparameter tuning: evaluate an informative subset and infer the rest. Setting is the most basic, Setting IV and VI are the most practical. We only show results of Setting I and IV in the main paper, and defer the rest to Appendix
Table 1: Meta-LOOCV experiment settings
meta-training error matrix
memory cap on meta-test?
no
yes
full uniformly sampled
III IV
non-uniformly sampled
V
VI
In Setting I, we do meta-LOOCV with the full meta-training error matrix in each split and do not cap the memory for meta-test. This means we evaluate every configuration on the meta-training datasets. We can see from Figure TT and that:
ED-MF stably outperforms under both metrics, especially with fewer measurements.
QR-MF overall matches the performance of ED-MF.
BO-MF, BO-FULII and RANDOM-M MF underperform ED-MF and QR-MF at lower memory usage, but often match their performance with higher memory.
Practitioners may have only collected part of the meta-training performance, and desire or are limited by memory cap when they do meta-test on the new dataset. In Setting IV, we cap the conegllconii memory usage for meta-test at 816MB, the median memory of all possible mea- surements across configurations and datasets. Additionally, we uniformly sample 20% configurations from the meta-trai tettatraiinng error matrix in each split. In Figure [[[]) and [I(d] we can see similar trends as Setting I, except that QR-MF is slightly worse.
8

---

Published as a conference paper at ICLR 2022
con convergence (2nea 3
Eygee
HyperDif (anea 1:
(anea 1
44
3%
10%
10%
207
40%
20%
40%
relative metnory usage
relative menocy usage
relative metnory usage
relstive metnory usage
(a) Setting convergence
(b) Setting HyperDift (c) Setting IV convergence (d) Setting IV HyperDiff
ED-MF (PEPPP)
QR-MF
BO-MF
BO-FULL
RANDOS-
Pareto frontier estimates in meta-LOOCV Setting and IV (with 20% meta-training sampling ratio and an 816MB meta-test memory cap). Each error bar is the standard error across datasets. The axis measures the memory usage relative to exhaustively searching the permissible configurations. ED-MF consistently picks the configurations that give the best PP estimates.
Figure 11:
Ultimately, users would want to select a configuration that both achieves a small error and takes lower memory than the limit. As shown in Figure 2 PEPPP offers users the predicted Pareto frontier and chooses the non-dominated configuration with the highest allowable memory and hence the lowest error. We compare the acquisition techniques with the "random high-memory" baseline that randomly chooses a configuration that takes the highest allowable memory: an approach that follows the "higher memory, lower error" intuition. Figure 12 shows example.
Overall, among all approaches and across all memory us- age, ED-MF outperforms, especially at smaller number of measurements, Compared to BO techniques. ED-MF also enjoys the benefit of having less hyperparameters to tune. Although techniques like QR-MF and RANDOM- MF are easier to implement, the additional cost ED-MF is much smaller than the cost of making measurements: neural network training and testing.
shorebird bousing
curtilagioons fish
fondstul
auper
world monlery
14
error dillased
(reletive to ED-MP)
4.3 TUNING OPTIMIZATION HYPERPARAMETERS
BO-MP BO-FULL
RANDOM-MF
candom memeoyy
Previously, we have shown that PEPPP can estimate the error-memory tradeoff and select promising configura- tion with other hyperparameters fixed. In practice, users may also want to tune hyperparameters like learning rate to achieve the lowest error. In Appendix [E.4] we tune the number of epochs and learning rate in addition to pre- cision, and show that the methodology can used broader settings.
Figure 12: Relative performance with re- spect to ED-MF in meta-test Setting IV when making 3 measurements (memory usage 10%) on 10 ImageNet partitions. ED-MF outperforms in most cases.
4.4 META-LEARNING ACROSS ARCHITECTURES
The meta-lcarning in previous sections were conducted on ResNet-18, In Appendix E.5 we show that on 10 ImageNet partitions, PEPPP also capable of estimating the error-memory tradeoff of ResNet-34, VGG-11, VGG-13, VGG-16 and VGG-19 competitively. Moreover, the meta-learning across architectures works better than considering each architecture separately.
5 CONCLUSION
This paper proposes PEPPP, a meta-learning system to select low-precision configurations that leverages training information from related tasks to efficiently pick the perfect precision given a memory budget. Built on low rank matrix completion with active learning. PEPPP estimates the Pareto frontier between memory usage and model performance to find the best low-precision configuration at each memory level. By reducing the cost of hyperparameter tuning in low low-prccision training, PEPPP allows practitioners efficiently train accurate models.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Madeleine Udell and Chengrun Yang acknowledge support from NSF Award LIS-194313,, the ONR Young Investigator Program, and the Alfred P. Sloan Foundation. Christopher De Sa acknowledges gift from SambaNova Systems, Inc., and Jerry Chee acknowledges support from the NSF Research Traineeship Program (Award # 1922551). The authors thank Tianyi Zhang for helpful discussions, and thank several anonymous reviewers for useful comments.
REFERENCES
Charles Audet, Jean Bigeon, Dominique Cartier, Sébastien Le Digabel, and Ludovic Salomon. Performance indicators in multiobjectiie optimization. European journal of operational research, 2020.
Stephen R Becker, Emmanuel J Candès, and Michael C. Grant. Templates for convex cone problems with applications to sparse signal recovery. Mathematical programming computation, 3(3):165, 2011.
C.M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, 2006. ISBN 9780387310732.
Stephen Boyd and Lieven Vandenberghe. Corvex optimization. Cambridge university press, 2004.
Leon Bungert, Tim Roith, Daniel Tenbrinck, and Martin Burger. A bregman learning framework for sparse neural networks. arXiv preprint arXiv:2105.043// 2021.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed-precision neural net- works. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2349-2355 2020.
Carlos A Coello Coello and Margarita Reyes Sierra. Multiobjective evolutionary algorithms: classifi- cations, analyses, and new innovations. In Evolutionary Computation. Citeseer, 1999.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv preprint arXiv: 141777044 2014.
Christopher De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Christopher R Aberger, Kunle Olukotun, and Christopher Ré. High-accuracy low-precision training. arXiv preprint arXiv: 1803.038, 2018.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjec- tive genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2):182-197, 2002.
Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.0000 2018.
Nicolo Fusi, Rishit Sheth, and Melih Elibol. Probabilistic matrix factorization for automated machine learning. Advances in neural information processing systems, pp. 3348-3357 2018.
Gene H Golub and Charles F Van Loan. Matrix computations, volume 3. JHU press, 2012.
Ming Gu and Stanley C Eisenstat. Efficient algorithms for computing strong rank-revealing qr factorization. SIAM Journal on Scientific Computing, 17(4):848-86) 869, 1996.
Suyog Gupta, Ankur Agrawal. Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. International conference on machine learning, pp. 1737-1746. PMLR, 2015.
David A Harville. Matrix algebra from statistician's perspective, 1998.
10

---

Published as a conference paper at ICLR 2022
Soheil Hashemi, Nicholas Anthony, Hokchhay Tann, R Iris Bahar, and Sherief Reda. Understanding the impact of precision quantization on the accuracy and energy of neural networks. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017, pp. 1474-1479. IEEE, 2017.
Trevor Hastie, Rahul Mazumder, Jason D Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares. The Journal of Machine Learning Research, 16(1):3367-3422. 2015.
Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489 2020.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, arXiv preprint arXiv:2004.0544 05439, 2020.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Hamed F Langroudi, Zachariah Carmichael, David Pastuch, and Dhireesha Kudithipudi. Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge. arXiv preprint arXiv: 02386, 2019.
Christiane Lemke, Marein Budka, and Bogdan Gabrys. Metalcarning: survey of trends and technologies. Artificial intelligence review, 44(1):117-130 2015.
Miqing Li, Shengxiang Yang, and Xiaohui Liu. Diversity comparison of Pareto front approximations in mony-ojeective optimization. IEEE Transactions Cybernetics, 44(12):2568-22 -2584, 2014.
Wei Ma and George H Chen. Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. In Advances Neural Information Processing Systems, volume 32, pp. 14900-14909, 2019.
Vivek Madan, Mohit Singh, Uthaipon Tantipongpipat, and Weijun Xie. Combinatorial algorithms for optimal design. In Conference on Learning Theory, pp. 2210-2258. PMLR, 2019.
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. The Journal of Machine Learning Research, 1 11:2287-2322. 2010.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 1oXXv:17100037400 2017.
Jonas Močkus. On Bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. Green ai. arXiv preprint
arXiv: 2019.
Jack Sherman and Winifred Morrison. Adjustment of an inverse matrix corresponding to a change in one element of given matrix. The Annals of Mathematical Statistics, 21(1):124-12) 1950.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszezynski, Jian Zhang, and Christopher Ré. Low-memory neural network training: A technical report. arXiv preprint 1904..1904.1 10631, 2019a.
Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszezynski, Jian Zhang, and Christopher Ré. Low-memory neural network training: A technical report, 2019b.
Xiao Sun, Jungwook Choi, Chia- Yu Chen. Naigang Wang, Swagath Venkataramani, Vijayalak- shmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. Advances in Neural Informa- fion Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2022
Xiao Sun, Naigang Wang, Chia-Yu Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Joaquin Vanschoren. Meta-learning: A survey. arXiv preprint arXiv:1810.03558, 2018.
Bram Wallace and Bharath Hariharan. Extending and analyzing self-supervised learning across domains. In European Conference on Computer Vision, pp. 717-734. Springer, 2020.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:18812.00090, 2018.
Jin Wu and Shapour Azarm. Metrics for quality assessment of a multiobjectivv design optimization solution set. J. Mech. Des., 123(1): 2001.
Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. Oboe: Collaborative filtering for automl model selection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1183-118,, 2019.
Chengrun Yang. Jicong Fan, Ziyang Wu, and Madeleine Udell. Automl pipeline selection: Efficiently navigating the combinatorial space. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1446-1456, 2020.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-31) 315, 2007.
Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, and Ce Zhang. The zipml framework for training models with end-to-end low precision: The cans, the cannots, and little bit of deep 2016.
learning. arXiv preprint
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. QPyTorch: A low-precision arithmetic simulation framework, 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 16o::666.000 (066606660. 2016.
Eckart Zitzler and Lothar Thiele. Multiobjective optimization using evolutionary algorithms comparative case study. In International conference parallel problem solving from nature, pp. 292-301. Springer, 1998.
12

---

Published as a conference paper at ICLR 2022
All the tables in this appendix shown in the last page.
A DATASETS AND LOW-PRECISION FORMATS
The 87 datasets chosen in our experiment are listed in Table Note that the datasets contain images of two different resolutions: 32 denotes resolution 3 x 32 x 32, and 64 denotes 3 64 x 64. We list our choices Format A and in Table and respectively. In each evaluation, the low-precision configuration is composed of one Format A (for the activations and weights) and one Format B (for the optimizer), as detailed in Section 3.1 Therefore, the total number of low-precision configurations used in our experiment is 99.
B THE SOFTIMPUTE ALGORITHM
There several versions of SOFTIMPUTE; (Hastie al. 2013) gives a nice overview. We the version in (Mazumder et al. 2010).
At high level, SOFTIMPUTE iteratively applies soft-thresholding operator Sx the partially observed error matrix with series decreasing À values. Each Sx replaces the singular values {or with ((o, }.+ The regularization parameter can be set in advance or ad hoc, by convergence dynamics.
The pseudocode for the general SOFTIMPUTE algorithm shown Algorithm 1 in which Pa (F) matrix with the same shape as F,, and has the (i, j)th entry being Fag if (i,j) E Ω and otherwise. In our implementation, A At; for each step i, and t; is the step size from TFOCS backtracking (Becker et al. 2011).
Algorithm 1 SOFTIMPUTE
Input: partially observed matrix (a(E) E R number of iterations I, a series of decreasing values {Xi)
Output: an estimate É
for 1 to do
E + Pa(E) + Pac(E)
U,E,V V + svd((E)
UVVV
C ALGORITHMS FOR EXPERIMENT DESIGN
As mentioned in Section 3.2 there are mainly two algorithms to solve Problem I the D-optimal experiment design problem: convexification and greedy.
The convexification approach relaxes the combinatorial optimization problem to the convex optimiza- tion problem
log det (E 2,8,,,)
minimize
subject to
Σ 1
E
[0, 1j. E [d]
that can be solved by a convex solver (like SLSQP). Then we sort the entries the optimal solution o² IRd and set the largest entries and the rest to O.
The greedy approach (Madan et al. 2019 Yang et al. 2020) function by first choosing initill set of configurations by maximizes the submodular objective QR decomposition, and then greedily adding new configuration to the solution set in each step until |S]] = The greedy stepwise selection algorithm is shown Algorithm 2 in which the new configuration is chosen by
13

---

Published as a conference paper at ICLR 2022
the Matrix Determinant Lemma (Harville 1998) and X is updated by the close form from the Sherman-Me Morrman-Morrioon Formula (Sherman & Morrison, 1950). The column-pivoted QR decomposition selects top k pivot columns of Y R*x" to get the index set So, ensuring that Xo Ejeso ViV is non-singular.
Algorithm 2 Greedy algorithm for D-optimal experiment design
Input: design vectors {2, { in which yj E IR*; maximum number of selected configurations I; initial set of configurations So _ [d]. s.t. Xo Ejeso Myyj is non-singular
Output: the selected set of designs C [d]
S f- So
2 while |S| - 1 do
3
- argmax jeldm suj 'y
4
S uu
X Xx vo! 5
The convexification approach empirically works because most entries of the optimal solution v² are
close either 0 or The histogram in Figure 13 shows example when we use rank k 5 to factorize the entire error matrix and set / 20.
relative (mean + standard error)
In terms of solution quality, the relative
0.1
error plot in Figure [3] shows that the greedy approach consistently outperforms
60
convextsoation
greddy
0.08
in terms of the relative matrix completion error for each dataset ED-MF solutions.
20
0.05
0.2
0.4
0.6
0..8
1.0
Since the greedy approach is also more than 10x faster than convexification (im- plemented by scy),, we use the greedy approach throughout all experiments for the rest of this paper.
15
cutry values of
(a) histogram o" entries
(b) relative error
Figure 13: Convexification vs greedy for ED.
D INFORMATION TO HARDWARE DESIGN
Among all 87 datasets (each has its own error-memory tradcoff), the promising configurations that we identify among all 99 configurations are as below (number of sign bits number of exponent bits -number of mantissa bits).
Format A: 1-4-1, Format B: 1-6-7 (appears 60 out of 87 Pareto frontiers)
Format A: 1-4-1, Format B: 1-7-7 (appears on 33 out 87 Pareto frontiers)
Format A: Format B: 1-7-7 (appears 30 out of 87 Pareto frontiers)
Format A: 1-4-2, Format B: 1-6-7 (appears on 24 out of 87 Pareto frontiers)
Format A: 1-5-2, Format B: 1-6-7 (appears 21 out of 87 Pareto frontiers)
E MORE DETAILS ON EXPERIMENTS
We first show the plot of explained variances of top principal components in Figure 14 how much variance in data do the first several principal components account for (Bishop) (006). This quantity is computed by the ratio of sum of squares of the first & singular values to that of all singular values. In Figure 14 vary from 30, corresponding the decay of singular values shown in Figure 6(b) We can see the first singular value already accounts for 99.0% of the total variance, and the first two singular values account for more than 99.5%. This means we can pick a small rank for PCA in meta-training and still keep the most information in our meta-training data.
'The Matrix Determinant Lemma states that for any invertible matrix A E 1 xxk and a, b R². det(A + (e + The Sherman- Morrison Sherman-Morrison Formula states that for any invertible matrix A E R and a, b E R (A al ct A!
ab
a> Thus argmax s det(X, + yı argmax je X, V/-
ab )- A
145
14

---

Published as a conference paper at ICLR 2022
100.0%
explained variance
99.5%
99.0%
10
20
30
# first several singular values
Figure 14: Explained variance of the first several singular values in Figure 6(b)
50
# configuree cnniinraations ttons
* distaees
40
# config confiuuration urations
2000
2000
30
20
1000
1000
10
0.0
0.2
0.6
1.0
600
1000
configurations below median memory
30 40
50
60
70 80 90 100
bast error
memory (MB)
Figure 16: Histogram of datasets by the number of configurations that take memo- ries less than the overall median of 816MB.
Figure 15:
Histograms of error and memory. The
dashed lines are the respective medians.
We the ratio of incorrectly classified images error and memory values in our error and memory matrices from evaluating 99 configurations on 87 datasets. The vertical dashed lines show the respective medians: 0.78 for test error and 816MB for memory. We can see both the test етrоr and memory values span wide range. The errors come from training wide range of low-precision configurations with optimization hyperparameters not fine- tuned, and are thus larger than SOTA results. In Section 4.3 of the main paper and Section S.4/eer,, we show that training for larger number of epochs and with some other optimization hyperparameters yield the same arrormemory tradeoff as Figure ((a) in the main paper.
metric. Figure 13 shows histograms of
In meta- Leta- settings with a meta-test memory cap at the median memory 816MB, Figure shows histogram of number of feasible configurations on each of the 87 datasets. There are "cheap" (resolution 32) datasets on which each of the 99 configurations takes less than the cap, and "expensive" (resolution 64) datasets which the feasible configurations are far less than 99.
E.1 INTRODUCTION TO RANDOM-MF, QR-MF AND BO
Random selection with matrix factorization (RANDOM-MF), Same as ED-MF, RANDOM- MF predicts the unevaluated configurations by linear regression, except that it selects the configurations to evaluate by random sampling.
QR decomposition with column pivoting and matrix factorization (QR-MF). QR-MF first selects the configurations to evaluate by QR decomposition with column pivoting: EP = QR, in which the permutation matrix P gives the most informative configurations. Then predicts unevaluated configurations in the same way as ED-MF and RANDOM-MF.
Bayesian optimization (BO). Bayesian optimization is sequential decision making framework that learns and optimizes black-box function by incrementally building surrogate models and choosing new measurements (Frazier 2018). It works best for black-box functions that expensive to evaluate and lack special structures or derivatives. We compare ED-MF with two BO techniques. The first technique, BO-MF, applies BO to the function fi R E that maps low-dimensional configuration embeddings (Y.g 1d 1==1 to the test enrors of the configurations
q the meta-test dataset {enew Jj=1 (Fusi al. 2018). The embeddings come from the same low-rank factorization of the error matrix E as in ED-MF. The second, BO-FULL, applies that directly maps columns of the error matrix {E. 1- To learn either of these black-box functions, we start by evaluating a subset of configurations S S [d] and then incrementally select new configurations that maximize the expected improvement (Močkus) 1975 Jones al. 1998]
BO to the function fe : R"
to fehew d
15

---

Published as a conference paper at ICLR 2022
E.2 ADDITIONAL META- TRAINING RESULTS: NON-UNIFORM SAMPLING
In non-uniform sampling, we sample each entry of the er- ror matrix E with probabilities
0.6
CIFAR J0
convengenee
CIEAR 10
HypeeDDit
11.6
CITAR
Po (1/Way)) in which o maps {1/Wa} into an interval
relaiiv
0.2
0.4
0.2
0.4
[0, Pmax] C [0.1] 1) according to the cumulative distribution func- tion {1/Wa;}- By varying Pmax. we change how we ag- gressively sample the configu- rations, how different the sam- pling probabilities are between large and small memory configu- rations, and also the percentage of memory needed. In Figure we vary Pmax from 0.1 to and see that the quality of the estimated Pareto frontier improves with more memory.
0.0
20%
10%
0.0
2076
10%
0.0
20%
40%
relative merrory Gsage
yelative menoryy
Gsage
relative memory usago
(a) relative error
(b) convergence
(c) HyperDiff
Figure 17: Pareto frontier estimation performance in PEPPP meta- training with non-uniform sampling of configurations. The violins and scatters have the same meaning as Figure | The x axis measures the memory usage relative to exhaustive search.
E.3 ADDITIONAL META-LOOCY RESULTS
In the main paper, we have shown the performance of Pareto frontier estimates and configuration selection for Setting and IV. For the rest of the settings in Table we conduct meta-LOOCV in the same way and show the results of Pareto frontier estimates in Figure [8]
Setting II. We have the 816MB memory cap for meta-test, and have the full meta-training error matrix in each split.
conaggge (uaan x))
Cozeaggnn (coon #
200
0.03
HypeeDill (mnenn
relaiim monoory sssiee
ssoige
pdative
relative
Setting II convergence
(b) Setting II HyperDiff (c) Setting III convergence (d) Setting IIII HyperDiff
conoegennc (inoo (inon m))
0000 0000 grane (quonn th m])
Eypes bal (mean m))
Hopecbiff
I0%
1056
2155
relaiie
volative
telation
(e) Setting V convergence
(1) Setting V HyperDifff (g) Setting VI convergence (h) Setting VI HyperDiff
ED-MP (PEPPP)
QR-MP
BO-MF
BQ-FULI
RANDOM- MF
Pareto frontier estimates in meta-LOOCV Setting II (full meta-training error matrix, 816MB memory cap), Setting III (uniformly sample 20% meta-training measurements, no meta-test memory cap), Setting V (non-uniformly sample 20% meta-training measurements, meta-test memory cap), and Setting VI (non-uniformly sample 20% meta-training measurements, an 81 816MB meta-test memory cap). Each error bar is the standard error across datasets. ED-MF is among the best in every setting and under both metrics.
Figure 18:
Setting III. We have no memory cap for meta-test, and uniformly sample 20% configurations for meta-training in each split.
Setting V. We have no memory cap for meta-test, and non-uniformly sample 20% configurations for meta-training in each split. The sampling method is the same in metainraining (Figure I7].
Setting VI. We have the 816MB memory cap for meta-test, and non-unifommly sample 20% meta- training configurations in the same way Setting V above.
16

---

Published as a conference paper at ICLR 2022
100 epochs
epochs
Figure 19: Errors of 99 configurations trained for different numbers of epochs.
0.8
dr
tost ertor
Lel emo
tt fi
0.6
0.6
0.4
0.4
An tyk.
0.2
0.2
0.50
250
300
4.50
200
250
300
memury (MB)
memory (MB)
(a) fixed learning rate
(b) tuned learning rate
dominated
non-dominated
Figure 20: 1IFAR-10 error-memory tradeoff. Figure (a) has learning rate 0.001 for all low-precision
configurations. Figure (b) shows the tradeoff with tuned learning rates: at each low-precision configuration, the lowest test error achieved by learning rates {0.01, 0.001, 0.0001 is selected.
E.4 TUNING OPTIMIZATION HYPERPARAMETERS
E.4.1 NUMBER OF EPOCHS
The low-precision networks still underfit after 10 epochs of training. This situation typical: underfitting due to budget constraints is unfortunately common in deep learning. Luckily, meta- learning the best precision will succeed so long as the validation errors are correctly ordered, even if they all overestimate the error of the corresponding fully trained model. Indeed, our validation errors correlate well with the errors achieved after further training: on CIFAR- 10, the Kendall tau correlation between ResNet- 18 emrors 10 epochs and 100 epochs is 0.73, shown in Figure 19 The lowest error at 100 epochs is 9.7%, only approximately 2% higher than SOTA (Bungert et al. 2021).
E.4.2 LEARNING RATE
Previously, we have shown that PEPPP estimate the error-memory tradeoff and select promising configuration with other hyperparameters fixed. In practice. users may also want to tune hy hyperparam- eters like learning rate achieve the lowest error. Here, we tune learning rate in addition precision, and show that the methodology can be used in broader settings of hyperparameter tuning.
Figure shows that the error-memory tradeoff still exists with a fine-tuned learning rate for each configuration. With the learning rate tuned across (0.01, 0.001, 0.0001), }. the test errors are in general smaller. but higg-memory configurations still achieve lower errors in general. Thus the need to efficiently select a low-precision configuration persists.
Our approach can naturally extend efficiently selecting optimization and low-precision hyperpa- rameters. We perform the meta-training and meta-LOOCV experiments on subset of CIFAR-100 partitions with multiple learning rates (0.01, 0.001, 0.0001]. The error and memory matrices we use here have 45 rows and 99 columns, respectively. Learning rate and oowprrccsioon configuration are collapsed into the same dimension: each row corresponds to a combination one CIFAR- 100 subset and one of the learning rates {0.01, 0.001. 0.0001]. shown in Table 5 We say that these error and memory matrices are LR-runed. Figure 20aure2TTshae shows the LR-tuned error matrix also has a fast singular value decay. The other hyperparameters are the same in LR-fixed experiments, except that
17

---

Published as a conference paper at ICLR 2022
10
10
10a
0
10
index
Figure 21: Singular value decay of the LR-tuned error matrix.
relsfive OFFEr
0.3
0.3
0.4
Hyper Diff
0.2
0.2
0.1
$
0.2
0.1
0.0
3
&
0.0
1
0.0
4
10%
30%
10%
30%
10%
30%
relative memory usage
relative memory ussge
relative memory usage
(a) relative error
(b) convergence
(c) HyperDiff
CIFAR-100 100 (aquatic mammals) with learning rate 0.01
The Pareto frontier estimation performance meta-training, with uniform sampling of configurations on the LR-tuned error and memory matrices. Similar to Figure the violins show the distribution of the performance on individual datasets, and the error bars (blue) show the range. The red error bars show the standard deviation of the error CIFAR-100 aquatic mammals and learning rate 0.01, across 100 random samples of the error matrix. Figure (a) shows the matrix completion error for each dataset; Figure (b) 3]dand (c) show the performance of the Pareto frontier estimates in convergence and HyperDiff.
Figure 22:
batch size 128 and train for 100 epochs. The meta- veta-training and meta-LOOCV results are consistent with those in Sections 4.1 and 4.2 respectively:
In meta-training, we first uniformly sample the error matrix and study the performance of matrix completion and Pareto frontier estimation. Figure shows the matrix completion error and Pareto frontier estimation metrics. Then we do non-uniformly sampling and get Figure 233 the L.Rtuuedd version of Figure 17 in the main paper.
In meta-test, we evaluate settings in Table in the main paper. We get the performance of Pareto frontier estimates in Figure 24 the LR-tuned version of Figure I1 in the main paper. ED-MF steadily outperforms.
0.6
relative error
convergaro:
Hyper Diff
0.4
0.4
0.2
0.2
0.0
20%
40%
0.0
20%
40%
0.0
20%
40%
relative memory usage
relative memory usage
relative memory usage
(a) relative error
(b) convergence
(c) HyperDiff
CIFAR-100 (aquatic mammals) with learning rate 0.01
Figure 23: The Pareto frontier estimation performance in meta-training, with non-uniform sampling of configurations on the LR-tuned error and memory matrices. The violins and scatters have the same meaning as Figure in the main paper.
18

---

Published as a conference paper at ICLR 2022
con convergence (2nea
Hyped (anea 1
HyperDif
(anea 1:
10%
10%
10%
20/5
30%
10%
2076
30%
relative metoory usage
relative menooyy usage
relative metnory usage
relstive memoory usage
(a) Setting convergence
(b) Setting HyperDiff (c) Setting IV convergence (d) Setting IV HyperDiff
ED-MP (PEPPP)
QR-MP
BO-MF
BO-FULI
RANDOM-MP
Figure 24: Pareto frontier estimates meta-LOOCV settings on the LR-tuned error and memory
matrices. Each error bar is the standard error across datasets.
convergizz
HyporDil (toean seld
Lostvereeze [1000
Hyper (tuoean =t
390
10%
10%%
3%
10%
momorry usage
relutive memmory usage
memorry usagye
relutive memoory usagy
(a) ResNet-34
(b) ResNet-34
(c) VGG variants
(d) VGG variants
ED-MF (PEPPF)
QR-MF
BO-MF
BO-FULL
RANDOM--EP MFF
Figure 25: Pareto frontier estimates in meta-LOOCV Setting when learning across architectures: from ResNet- 18 to either ResNet-34, -34, or to VGG variants. Each error bar is the standard error across datasets. The axis measures the memory usage relative to exhaustively searching the permissible configurations. ED-MF consistently picks the configurations that give the best PF estimates.
E.5 LEARNING ACROSS ARCIITECTURES
We show that on 10 ImageNet partitions, PEPPP with ED-MF is able to estimate the error-memory tradeoff of the low-pre low-precision configurations of ResNet-34, VGG-11, VGG-13, VGG-16 and VGG- 19. The 10 ImageNet partitions have WordNet IDs [n02470899, 481487 n02022684, n03546340, n07566340, n0001961 1611,, n01772222, n03915437, n02489589, n021 (02127808) and randomly selected from the 50 ImageNet subsets on which we collected the eπror matrix. On these ImageNet partitions, we use the performance ResNet-18 as meta treintta data, and either the performance of ResNet- 18 or VGG variants as meta-test data. In Figure 25 we can see that ED-MF is steadily among the best in Pareto frontier estimation, and there is no statistical difference between the estimation performance on ResNet-34 and VGG variants.
Next, we compare the performance of the following two cases:
i Meta-learning across datasets with performance from the same architecture: For example, to learn the error-memory tradeoff of ResNet-18 on n02470899, we only use the tradeoff' of ResNet-18 other ImageNet partitions the meta-training data.
Meta-learning across datasets with performance from both the same and other architectures: For example, the error-memory tradeoff of ResNet-18 on n02470899, we not only use the tradeoff's of ResNet- 18 9 other ImageNet partitions, but also those of ResNet-34, VGG-11, VGG-13, VGG-16 and VGG-19 on the partitions as the meta-training data.
Figure 26arewsoss shows that Case ii outperforms Case better estimating the error-memory tradeoff's on different architectures and datasets.
19

---

Published as a conference paper at ICLR 2022
0.05
conver genne
mmaan + Se
ED-MF (Case: 3)
ED-MF (Case ii)
0.02
5%
10%
relative memory usage
Figure 26: Benefit of meta-learning across architectures. Each error bar is the standard error across architecture-dataset combinations (e.g., ResNet-18 + n02470899 is a combination). The x axis measures the memory usage relative to exhaustively searching the permissible configurations.
20

---

Published as a conference paper at ICLR 2022
Table 2: Datasets
Table 3: Format A (for activations and weights)
index dataset nome
resolation
4 points
index
exponent bits
mantissa Bits
total bit width
1 CIFARIO
32
(00000
2
CIFARIOO laguatic mammaks)
32
3000
CIFAR (f-hh)
3000
CIFARIDI (flowes)
3000
5
CIFARIO0 (food comsinners)
32
3000
1
6
6
CIFARIDE 100 (Trit and vegetahlle))
32
3000
2
CIFARIO0 (houschlld electrical devices)
3000
8
CIFARIO0 (houschold furniture)
3000
4
9
CIFARIO0 (issec)s
32
3000
1
10
CIFARI00 (lagge cornivores)
32
3000
10
CIFAR (Jarge man- made outdoor things)
3000
11
CIFAR (lange natural outdoor scsnes)
3000 32
13
CIFAR100 (Jange oomnivares and herbivores)
3000
14 CIFARIOD 100 (medium--izel mammaals
3000 32
CIFAR
invertdwetes)
3000
16
CIFARIO0 (people)
3000 32
17
I0F (reptile))
3000
Table 4: Format B (for optimizer)
18 CIFAR 100 (small mannnaa))
32
3000
CIFARIOO (trees)
3000
CIFAR (vahieles
3000
index
exponent bits
mantissa Bits
sotil bit width
21 CIEAR1O0 (vebácles 2)
32
3000
22
mirenatt
6667
2
9
16
24
did
3760
3
11
18
25
isie
64
22802
26
marael
64
1890
9
17
scenes
6
11
19
28
uc110]
12024
7
16
29
daimlerpedels
64
20400
gisbb
26640
kather
4000
32
umniglot
25968
35
svhn
64 73257
34
'gp- "ge-flowers
20-40
bach
36
protein atlss
12113
Table 5: Datasets and learning rates in Section 4.3
mine 37
64
51750
ImageNet (bug)
6319
ImageNet (ectriever)
6668
index. dataset name
learning rale
40
ImageNet (domestic cat)
64
6750
CIFARI00 (oquatic mammalss)
0.01
42
41
ImageNett (urile)
ImageNet (stick)
64
6750
6750
2
CIFAR100 (Bowwrs)
CIFAR 100 (fisk)
0.01
0.01
ImageNet (tinch)
6750
CIFAR
containers)
45
44
ImageNet (walchslog)
ImageNet (Gootwear)
ImageNet (salamander)
64
6750
6587
6404
CIPAR100 (houschuld electrical devices)
CIFARIOU (fruit and vegetables)
CIFAR (00 (houschuld furnitune)
0.01
0.01
0.01
ImageNet Canthropod upe)
6750
CIEAR100 (insects)
48
ImageNet (elasmebranch)
64
6750
CIPAR100 (large carnivores)
0.01
49
ImageNet (housing)
ImageNet (shureburd)
64
6605
6750
10
CIFAR10 (large natural ousdoer scenes)
CIFARIO0 (large mum made outdoor things)
0.01
ImageNet (foodstuff)
6750
CIPAR100 (large omnivorre berbévures)
52
ImageNet (substance)
6643
13
CIFAR100 (medium--sizel mammals)
0.01
53
ImageNet (New World monkey)
ImageNet (spider)
ImageNet (percussion instrument)
64
S100
9082
S100
14
15
CIPAR100 (non insect invertebrates)
CHERR100 (roptlies)
CIFARIO0 (00 (people)
0.01
0.01
57
56
ImageNet (fabri)
ImageNet (box)
ImageNet (big cat)
64
64
8100
7862
7829
17
18
CIARR10 (fowers)
CIFAR100 (aquatic mammals)
CIFAR 100 (fish)
0
0.001
0.001
60
ImageNet (kitclsen applianee)
ImageNet (mollusk)
64
7773
8100
21
CIFAR100 (Friit and vegetables)
CIEAR100 (food containess)
0.001
0.001
61
ImageNet (bunerfly)
ImageNet (hand tool)
64
8100
S054
23
22
CIFAR100 (houschold electrical devices)
CIFAR100 (houschold furniture)
0
0.001
64
65
ImageNet (stringed instrument)
ImageNet (rodent)
ImageNer (hoat)
ImageNet (lilltry))
64
64
7522
8006
8100
8006
26
25
CIFAR100 (large man made oundoor things)
CIFAR1O (large carnivores)
CIFARIO0 (insects)
CIFARIOR (large natural outdoor scenes)
0
0.001
0.001
0.001
68
ImageNer (comsuuter)
ImageNet (shop)
64
76%6
9400
29
CIPAR100 (mediuu sized mammals))
CHEAR100 (large omnivvrres berbevares)
0.001
0.001
69
ImageNet (Old World monkey)
ImageNet (musteline manmal)
64
9450
9450
30
CIFAR100 (non-insect invertebeates)
CIFRR1O (poople)
0
0.001
73
72
ImageNet (bottle
ImageNet (fungus)
ImageNet (track)
64
64
9309
9205
9450
34
32
33
CIFAR10U (fuit)
CIFAR1O0 (aquatic mammals)
CIFARIOR 100 (reptiles)
0.0001
0.000
0.001
74
ImageNet (spaniel)
9119
CIFAR 100 (flowers
0.000
76
ImageNatt (sports equipment)
ImageNet (gume bind)
64
D450
9490
37
CIFAR100 (fruit and vegetables)
CIFARIO0 (food containess)
0.0001
77
ImagoNer
ImageNet (seat)
64
9450
9126
38
CIFAR 100 (houschold elecnrical devices)
CIFARIO 100 (houschold furniture)
(1000
0.000
81
80
ImageNet (wespon)
ImageNet (beede)
ImageNer (lny dog)
ImageNet (decaped crastacena)
64
64
9450
9832
10800
10800
41
42
CIFAR ton (large natural ourdivee
CIFAR 100 (insects)
CIFARION (large carnivores)
CIFARIOO 100 (large man- made oundoce things)
(10001
0.0001
0.0001
ImageNat (Gattner)
10675
CIFARION (O0) (large omnivores herbivores)
85
84
ImageNet (dish)
ImageNet (limepicoz)
64
64
10556
10164
45
CIFARIOU (medicas- sized mammals)
0.0001
TmageNet (moclunical device)
ImageNer (oohibrid snaee)
12150
21


Reviews:
[{"review_id": "_qREJ1WRyO", "reviewer": "Reviewer_ezUH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an interesting, one-of-a-kind two-step approach for error-memory trade-off in low-precision training/inference of neural networks. I appreciate the design of the method, the utility of the method (I find it very useful for the community working in the field) as well as the extensive set of measurements. The paper however could be improved especially in terms of clarity, especially in the experimental section. Please find below a few questions below:\n\n- Figure 1(b): it is not clear what the numbers in the parenthesis are. Memory requirements?\n- Is the meta-training performed per neural network architecture? This is my understanding, and if this is so, please clearly specify this in the description of the method (e.g., in Figure 2).\n- Was regularization used in the linear regression model? If so, how sensitive is the estimated Pareto frontier to the regularization parameter?\n- Configuration formats A and B (Section 2): the weights should also be mentioned, as you are using one of the two formats for weights as well, not only for the optimizer and activation.\n- How is the memory computed? Maximum memory measured at train/inference time? Or theoretical computation knowing the neural network architecture? My understanding is the latter, as you compute the full memory matrices both at train and test times. It should be clearly mentioned in the paper.\n- Figure 6b deserves more discussion. How it was generated, what it means. This is important as it supports one of the main properties of the method: the rapid decaying of the singular values of the error matrix. On a related note, how to choose the dateset-configuration data-points to guarantee this property?\n- Please define the propensity score.\n- In section 3.2, please define all the notations (e.g., yi).\n- In section 4.2 how was rank 3 chosen?\n- Figure 11: it is not clear what the x axis represents. For setting I, there is no memory cap, thus what does relative memory usage mean?\n- In Section 4, the authors mention different neural network architectures: ResNet-18, ResNet-34 and VGG. For which architecture are the results shown in Section 4? If for all of them, how are the performance metrics aggregated across architectures? Averaged across architectures? This is unfortunately not clear.", "summary_of_the_paper": "The paper proposes a two-step methodology for evaluating the error-memory trade-off when employing low-precision training/inference in neural networks. Although low-precision training/inference in deep learning is a widely studied topic, this paper seems to be the first one to study this error-memory tradeoff without exhaustive search. \n\nThe proposed idea is inspired from the multi-objective optimization field. The authors propose to identify the Pareto frontier that would allow a user to identify the optimal precision (lowest error) given a certain memory constraint. The proposed system has two phases: training and testing. \n\nAt training time, given a set of datasets, a network architecture and a set of low-precision configurations, the method samples dataset-configuration pairs to train and computes the misclassification error for each selected pair. Matrix factorization is used to compute low-dimensional embeddings for each configuration. \n\nAt test time, a subset of configurations is chosen for evaluation (using Design Experiments with Matrix Factorization) and the computed errors are used as input together with the configuration embeddings to a linear regression model in order to estimate the errors on the non-evaluated configurations. For both training and testing, the memory matrix is fully computed based on the network architecture and the low-precision format.\n\nThe method is evaluated in terms of convergence and Hypervolume difference of the true and estimated Pareto frontier and it outperforms in most of the cases other methods such a Bayesian Optimization or Random Selection with Matrix Factorization.\n", "summary_of_the_review": "Although there are some items that should be better explained / clarified, overall I enjoyed reading this paper and find the addressed topic of high interest for the community working in the field of low-precision training/inference of neural networks. Although I am not familiar with the prior art, the paper seems to be the first to propose a method for analyzing the error-memory trade-off in a deep learning low-precision training/inference scenario without exhaustive search. \n\nI like the two-step meta-training/meta-test approach, however I am not yet fully convinced about the main property of the meta-training algorithm, namely the rapid decaying of the singular values of the error matrix. In terms of evaluation, I appreciate the authors analyzing the performance of the meta-training approach using multiple metrics and providing recommendations for the optimal sampling conditions. \n\nAlso for the meta-test phase, the authors analyze the performance of the ED-MF approach to other methods such as bayesian optimization, random selection and QR with matrix factorization. ED-MF out-performs these methods in most of the cases. Some results are not clear unfortunately - see my questions in the main review above. Hopefully the results will become more clear during the paper discussion / rebuttal phase.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WYixWaT3Ga", "reviewer": "Reviewer_ANon", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Incomplete experimental results\n1-1) There is a limitation that most of the experiments were performed on CIFAR-10. Also, experimental results on more networks (e.g., MobileNet, EfficientNet, more ResNets) should be included.\n1-2) Although the proposed method considers both training and inference, performance comparison with existing SOTA studies for each phase should be performed.\n1-3) It is necessary to show whether the proposed method is effective not only for classification but also for more practical detection and segmentation networks.\n\n2. It is difficult to understand the actual use-case of the proposed method. Most of the experimental results also appear to support the feasibility of meta-training. Therefore, it is difficult to be sure how practical the proposed method will be. It would be good to think about this comment in relation to the comments about additional experiments (i.e., Comment 1).\n\n3. It is difficult to agree with the argument of this paper because the motivation of this paper is not sufficiently presented. The motivation of this study needs to be clearly presented at the beginning to secure differentiation from many recently published low-precision training studies.\n\n4. It is necessary not to just consider memory capacity as a resource, but to consider practical indicators such as power consumption or memory R/W portion as the target of trading.\n\n5. Although the proposed method suggests optimization considering the memory budget, it is necessary to consider the computational power together. It is not practical to consider only the memory budget because various designs using memory reuse, etc. are possible depending on the computation method/capability.\n\n6. It would be nice if comparative analysis and comparison results with custom hardware designs are presented.", "summary_of_the_paper": "To achieve the goal of efficiently selecting the best low-precision configuration within the memory budget, this paper proposed Pareto Estimation to Pick the Perfect Precision (PEPPP) by using matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. Although this paper presents a solution for low-precision training that has recently gained a lot of attention, there are some concerns as a result of my review of this paper. (Please refer to my main review)", "summary_of_the_review": "The motivation and contribution need to be made clearer. In addition, the experiment part that supports the contribution needs to be thoroughly supplemented. Please address my concerns in \"Main Review\" through the rebuttal process.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UXMnoWrdW5N", "reviewer": "Reviewer_HPBj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper formulates the determination of low-precision configuration as a multi-objective optimization. The overall objective is to allow ML engineers to select optimal precision that lets them train the desired model with less energy.\n\nThe key idea seems to be using meta-learning and testing to determine these Pareto frontiers. The overall methodology is sound and is evaluated thoroughly. However, one limitation is that it doesn't provide evaluation on ImageNet which usually is a good reference point for the readers.\n\nI am not an expert in low-precision training. However, I like the overall idea where they formulated the configuration determination into a multi-objective optimization setting.\n\nQuestions:\n* Are there results for energy reduction?", "summary_of_the_paper": "The paper aims to find Pareto frontiers for the precision given memory budget. The overall objective is to allow ML engineers to select optimal precision that lets them train the desired model with less energy. To do this, the paper formulates the problem into multi-objective optimization problem and use meta-learning to minimize the number of low-precision training data points for this optimization. Then, the paper uses the term \"meta-test\" that refers to transferring the information from known tasks to estimate Pareto frontiers of low-precision training on new unseen tasks.", "summary_of_the_review": "I believe the paper proposes interesting approach in selecting low-precision training configurations. Formulating into multi-objective optimization and using meta-training/testing to find Pareto frontier of configurations for the overall accuracy seems like the key contribution of the paper. I am not an expert on low-precision training, but to the best of my knowledge, the overall approach seems reasonable and the evaluation seems okay. However, I believe it would provide better reference for the readers if the authors provide evaluation on ImageNet.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7moA7-q_eb", "reviewer": "Reviewer_M7yh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1- The proposed approach is evaluated in large and variant benchmarks which means the proposed approach can be generalized to various datasets and models\n2- The paper is well written, and the algorithms and methods are clearly explained.\n\nWeaknesses:\n1- The quality of the proposed approach based on convergence metric and hypervolume difference is not obvious.  \n2- The overhead of meta learning approach in term of time needs to be reported ( how many hours it takes to select precision configuration ) \n3- I would recommended to compare the meta learning approach with other multi objective approaches and report the overhead of each method.\n4- The related works such as [1,2] on selecting trade-off between precision and others metric is not discussed in the paper. \n[1] Hashemi, Soheil, et al. \"Understanding the impact of precision quantization on the accuracy and energy of neural networks.\" Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017. IEEE, 2017.\n[2]  Langroudi, Hamed F., et al. \"Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge.\" arXiv preprint arXiv:1908.02386 (2019).", "summary_of_the_paper": "The paper proposes a meta learning approach to select the best trade-of between memory usage and bit precision for DNN training. This approach is evaluated on 87 datasets and 99 bit-precision configurations. The result shows that in general 6 bit floats (1,4,1) for activations and weights and 14 bit floats (1,6,7) for optimizer parameters is the best choice in term of memory and accuracy trade-off.  ", "summary_of_the_review": "As a reviewer mentioned, the idea behand the paper is interesting. However, there are couple of comments needs to be addressed. I recommend this paper marginally accepted.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers a learning problem to determine the best low-precision configuration within the memory budget.  It is an interesting problem that could be of interest to the community.  Overall, the reviewers were fairly positive on the paper and believe the paper give interesting insights into how to use limited memory for learning.
Decision: Accept (Poster)