How Low Can We Go: Trading Memory for Error in Low-Precision Training

1 INTRODUCTION . Training modern-day neural networks is becoming increasingly expensive as task and model sizes increase . The energy consumption of the corresponding computation has increased alarmingly , and raises doubts about the sustainability of modern training practices ( Schwartz et al. , 2019 ) . Low-precision training can reduce consumption of both computation ( De Sa et al. , 2018 ) and memory ( Sohoni et al. , 2019a ) , thus minimizing the cost and energy to train larger models and making deep learning more accessible to resource-limited users . Low-precision training replaces 32-bit or 64-bit floating point numbers with fixed or floating point numbers that allocate fewer bits for the activations , optimizers , and weights . A rich variety of methods appear in recent literature , including bit-centering ( De Sa et al. , 2018 ) , loss scaling ( Micikevicius et al. , 2017 ) and mixed-precision training ( Zhou et al. , 2016 ) . There is however a fundamental tradeoff : lowering the number of bits of precision increases the quantization error , which may disrupt convergence and increase downstream error ( Zhang et al. , 2016 ; Courbariaux et al. , 2014 ; Gupta et al. , 2015 ) . The low-precision configuration is thus a hyperparameter to be chosen according to the resource constraints of the practitioner . How should we choose this hyperparameter ? Many believe the highest allowable precision ( given a memory budget ) generally produces the lowest error model . However , there are typically many ways to use the same amount of memory . As shown in Figure 1 ( a ) , some of these configurations produce much lower error than others ! Figure 1 ( b ) shows that no one configuration dominates all others . We might consult previous literature to choose a precision ; but this approach fails for new applications . Our goal is to efficiently pick the best low-precision configuration under a memory budget . Efficiency is especially important for resource-constrained practitioners , such as individual users or early-stage startups . To promote efficiency , we use a meta-learning ( Lemke et al. , 2015 ; Vanschoren , 2018 ; Hospedales et al. , 2020 ) approach : we train a small number of cheap very-low-precision models on the dataset to choose the perfect precision . The gains from choosing the right low-precision format can offset the cost of this extra training — but each precision must be chosen carefully to realize the benefits of low-precision training . We use ideas from multi-objective optimization to characterize the tradeoff between memory and error and identify the Pareto frontier : the set of non-dominated solutions . Users will want to understand the tradeoff between error and memory so they can determine the resources needed to adequately train a model for a given machine learning task . This tradeoff may also influence the design of application-specific low-precision hardware , with profound implications for the future ( Hooker , 2020 ) . For example , among all 99 low-precision configurations we tried , we identified some configurations that are Pareto optimal across many different tasks ( listed in Appendix D ) . These results could help hardware manufacturers decide which precision settings are worth manufacturing . Computing the Pareto frontier by training models for all low-precision configurations is expensive and unnecessary . Cloud computing platforms like Google Cloud and Amazon EC2 charge more for machines with the same CPU and double memory : 25 % more on Google Cloud1 and 100 % more on Amazon EC22 as of mid-September 2021 . We use techniques from meta-learning to leverage the information from other low-precision training runs on related datasets . This approach allows us to estimate the Pareto frontier without evaluating all of the low-precision configurations . Our system , Pareto Estimation to Pick the Perfect Precision ( PEPPP ) , has two goals . The first goal is to find the Pareto frontiers of a collection of related tasks , and is called meta-training in the meta-learning literature . Meta-training requires a set of measurements , each collected by training and testing a neural network with a given precision on a dataset . This information can be gathered offline with a relatively large resource budget , or by crowdsourcing amongst the academic or open-source community . Still , it is absurdly expensive to exhaustively evaluate all measurements : that is , every possible low-precision configuration on every task . Instead , we study how to choose a subset of the possible measurements to achieve the best estimate . The second goal we call meta-test : using the information learned on previous tasks , how can we transfer that information to a new task to efficiently estimate its Pareto frontier ? This goal corresponds to a resource-constrained individual or startup who wants to determine the best low-precision configuration for a new dataset . Both meta-training and meta-test rely on matrix completion and active learning techniques to avoid exhaustive search : we make a subset of all possible measurements and predict the rest . We then estimate the Pareto frontier to help the user make an informed choice of the best configuration . We consider two sampling schemes : uniform and non-uniform sampling . Uniform sampling is straightforward . Non-uniform sampling estimates the Pareto frontier more efficiently by making fewer or even no high-memory measurements . To the best of our knowledge , PEPPP is the first to study the error-memory tradeoff in low-precision training and inference without exhaustive search . Some previous works show the benefit of lowprecision arithmetic in significant energy reduction at the cost of a small accuracy decrease ( Hashemi et al. , 2017 ) , and propose hardware-software codesign frameworks to select the desired model ( Langroudi et al. , 2019 ) . Many other papers focus on low-precision inference only and use it for model compression ( Wu et al. , 2018 ; Cai & Vasconcelos , 2020 ) . Another line of work seeks state-of-the-art ( SOTA ) performance with low-precision training , using carefully designed learning schedules ( Sun et al. , 2019 ; 2020 ) . Our work imagines low-precision as a way to reduce training or deployment costs of customized models trained on proprietary datasets ( not SOTA models on ML benchmarks ) . Even so , we show in Section 4 that our work picks promising models for every memory budget , which enables near-SOTA results on CIFAR-10 . Figure 2 shows a flowchart of PEPPP . The rest of this paper is organized as follows . Section 2 introduces notation and terminology . Section 3 describes the main ideas we use to actively sample configurations and approximate Pareto frontiers . Section 4 shows experimental results . 1https : //cloud.google.com/ai-platform/training/pricing 2https : //aws.amazon.com/ec2/pricing/on-demand 2 NOTATION AND TERMINOLOGY . Low-precision formats . We use floating point numbers for low-precision training in this paper . As an example , Figure 3 shows an 8-bit floating point number with 3 exponent bits and 4 mantissa bits . A specific low-precision representation with a certain number of bits for each part is called a low-precision format . We may use different low-precision formats for the weights , activations , and optimizer . In the case of using two formats to train and represent a neural network ( as discussed in Section 3.1 ) , these two formats are called Format A and B . A specific combination of these two formats is a hyperparameter setting of the neural network , and we call it a low-precision configuration . Math basics . We define [ n ] = { 1 , . . . , n } for a positive integer n. With a Boolean variable X , the indicator function 1 ( X ) equals 1 if X is true , and 0 otherwise . With a scalar variable x , we use x+ to denote max { x , 0 } . ⊆ and ⊂ denote subset and strict subset , respectively . Linear algebra . We denote vector and matrix variables respectively by lowercase letters ( a ) and capital letters ( A ) . All vectors are column vectors . The Euclidean norm of a vector a ∈ Rn is ‖a‖ : = √∑n i=1 a 2 i . To denote a part of a matrix A ∈ Rn×d , we use a colon to denote the varying dimension : Ai , : and A : ,j ( or aj ) denote the ith row and jth column of A , respectively , and Aij denotes the ( i , j ) -th entry . With an ordered index set S = { s1 , . . . , sk } where s1 < . . . < sk ∈ [ d ] , we denote A : S : = [ A : ,s1 · · · A : ,sk ] ∈ Rn×k . Given two vectors x , y ∈ Rn , x y means xi ≤ yi for each i ∈ [ n ] . Given a matrixA ∈ Rn×d and a set of observed indices as Ω ⊆ [ n ] × [ d ] , the partially observed matrix PΩ ( A ) ∈ Rn×d has entries ( PΩ ( A ) ) ij = Aij if ( i , j ) ∈ Ω , and 0 otherwise . Pareto frontier . Multi-objective optimization simultaneously minimizes n costs { ci } i∈ [ n ] . A feasible point c ( 1 ) = ( c ( 1 ) 1 , . . . , c ( 1 ) n ) is Pareto optimal if for any other feasible point c ( 2 ) = ( c ( 2 ) 1 , . . . , c ( 2 ) n ) , c ( 2 ) c ( 1 ) implies c ( 2 ) = c ( 1 ) ( Boyd & Vandenberghe , 2004 ) . The set of Pareto optimal points is the Pareto frontier . Tasks , datasets , measurements and evaluations . A task carries out a process ( classification , regression , image segmentation , etc . ) on a dataset . Given a deep learning model and a dataset , the training and testing of the model on the dataset is called a measurement . In our low-precision context , we evaluate a configuration on a dataset to make a measurement . Meta-learning . Meta-learning transfers knowledge from past ( meta-training ) tasks to better understand the new ( meta-test ) task . To evaluate the error of meta-training , we use meta-leave-one-out cross-validation ( meta-LOOCV ) : on a collection of tasks , each time we treat one task as meta-test and the others as meta-training , and average the results over all splits . Error matrix and memory matrix . Given a neural network , the errors of different low-precision configurations on meta-training datasets form an error matrix , whose ( i , j ) -th entry Eij is the test error of the jth configuration on the ith dataset . To compute the error matrix , we split the i-th dataset into training and test subsets ( or use a given split ) , train the neural network at the j-th configuration on the training subset , and evaluate the test error on the test subset . The memory required for each measurement forms a memory matrix M , which has the same shape as the corresponding error matrix . Example error and memory matrices are shown in Figure 4 . 3 METHODOLOGY . PEPPP operates in two phases : meta-training and meta-test . First in meta-training , we learn configuration embeddings by training a neural network of a specific architecture at different low-precision configurations on different datasets . As listed in Appendix A , we study image classification tasks and a range of low-precision formats that vary in the number of bits for the exponent and mantissa for the activations , optimizer , and weights . Our hope is to avoid enumerating every possible configuration since exhaustive search is prohibitive in practice . To this end , we make a few measurements and then predict the other test errors by active learning techniques . We also compute the memory matrix to understand the memory consumption of each measurement . Then in meta-test , we assume that we have completed meta-training , and hence know the configuration embeddings . The meta-test goal is to predict the tradeoff between memory usage and test error on a new ( meta-test ) dataset . This step corresponds to inference in traditional machine learning ; it must be quick and cheap to satisfy the needs of resource-constrained practitioners . To select the configuration that has smallest test error and takes less than the memory limit , we measure a few selected ( informative and cheap ) configurations on the meta-test dataset , and use the information of their test errors to predict the rest . The problem of choosing measurements is known as the “ cold-start ” problem in the literature on recommender systems . Then we use the model built on the meta-training datasets to estimate values for the other measurements on the meta-test dataset .

Summary: The paper proposes a meta learning approach to select the best trade-of between memory usage and bit precision for DNN training. This approach is evaluated on 87 datasets and 99 bit-precision configurations. The result shows that in general 6 bit floats (1,4,1) for activations and weights and 14 bit floats (1,6,7) for optimizer parameters is the best choice in term of memory and accuracy trade-off.