How Low Can We Go: Trading Memory for Error in Low-Precision Training

1 INTRODUCTION . Training modern-day neural networks is becoming increasingly expensive as task and model sizes increase . The energy consumption of the corresponding computation has increased alarmingly , and raises doubts about the sustainability of modern training practices ( Schwartz et al. , 2019 ) . Low-precision training can reduce consumption of both computation ( De Sa et al. , 2018 ) and memory ( Sohoni et al. , 2019a ) , thus minimizing the cost and energy to train larger models and making deep learning more accessible to resource-limited users . Low-precision training replaces 32-bit or 64-bit floating point numbers with fixed or floating point numbers that allocate fewer bits for the activations , optimizers , and weights . A rich variety of methods appear in recent literature , including bit-centering ( De Sa et al. , 2018 ) , loss scaling ( Micikevicius et al. , 2017 ) and mixed-precision training ( Zhou et al. , 2016 ) . There is however a fundamental tradeoff : lowering the number of bits of precision increases the quantization error , which may disrupt convergence and increase downstream error ( Zhang et al. , 2016 ; Courbariaux et al. , 2014 ; Gupta et al. , 2015 ) . The low-precision configuration is thus a hyperparameter to be chosen according to the resource constraints of the practitioner . How should we choose this hyperparameter ? Many believe the highest allowable precision ( given a memory budget ) generally produces the lowest error model . However , there are typically many ways to use the same amount of memory . As shown in Figure 1 ( a ) , some of these configurations produce much lower error than others ! Figure 1 ( b ) shows that no one configuration dominates all others . We might consult previous literature to choose a precision ; but this approach fails for new applications . Our goal is to efficiently pick the best low-precision configuration under a memory budget . Efficiency is especially important for resource-constrained practitioners , such as individual users or early-stage startups . To promote efficiency , we use a meta-learning ( Lemke et al. , 2015 ; Vanschoren , 2018 ; Hospedales et al. , 2020 ) approach : we train a small number of cheap very-low-precision models on the dataset to choose the perfect precision . The gains from choosing the right low-precision format can offset the cost of this extra training — but each precision must be chosen carefully to realize the benefits of low-precision training . We use ideas from multi-objective optimization to characterize the tradeoff between memory and error and identify the Pareto frontier : the set of non-dominated solutions . Users will want to understand the tradeoff between error and memory so they can determine the resources needed to adequately train a model for a given machine learning task . This tradeoff may also influence the design of application-specific low-precision hardware , with profound implications for the future ( Hooker , 2020 ) . For example , among all 99 low-precision configurations we tried , we identified some configurations that are Pareto optimal across many different tasks ( listed in Appendix D ) . These results could help hardware manufacturers decide which precision settings are worth manufacturing . Computing the Pareto frontier by training models for all low-precision configurations is expensive and unnecessary . Cloud computing platforms like Google Cloud and Amazon EC2 charge more for machines with the same CPU and double memory : 25 % more on Google Cloud1 and 100 % more on Amazon EC22 as of mid-September 2021 . We use techniques from meta-learning to leverage the information from other low-precision training runs on related datasets . This approach allows us to estimate the Pareto frontier without evaluating all of the low-precision configurations . Our system , Pareto Estimation to Pick the Perfect Precision ( PEPPP ) , has two goals . The first goal is to find the Pareto frontiers of a collection of related tasks , and is called meta-training in the meta-learning literature . Meta-training requires a set of measurements , each collected by training and testing a neural network with a given precision on a dataset . This information can be gathered offline with a relatively large resource budget , or by crowdsourcing amongst the academic or open-source community . Still , it is absurdly expensive to exhaustively evaluate all measurements : that is , every possible low-precision configuration on every task . Instead , we study how to choose a subset of the possible measurements to achieve the best estimate . The second goal we call meta-test : using the information learned on previous tasks , how can we transfer that information to a new task to efficiently estimate its Pareto frontier ? This goal corresponds to a resource-constrained individual or startup who wants to determine the best low-precision configuration for a new dataset . Both meta-training and meta-test rely on matrix completion and active learning techniques to avoid exhaustive search : we make a subset of all possible measurements and predict the rest . We then estimate the Pareto frontier to help the user make an informed choice of the best configuration . We consider two sampling schemes : uniform and non-uniform sampling . Uniform sampling is straightforward . Non-uniform sampling estimates the Pareto frontier more efficiently by making fewer or even no high-memory measurements . To the best of our knowledge , PEPPP is the first to study the error-memory tradeoff in low-precision training and inference without exhaustive search . Some previous works show the benefit of lowprecision arithmetic in significant energy reduction at the cost of a small accuracy decrease ( Hashemi et al. , 2017 ) , and propose hardware-software codesign frameworks to select the desired model ( Langroudi et al. , 2019 ) . Many other papers focus on low-precision inference only and use it for model compression ( Wu et al. , 2018 ; Cai & Vasconcelos , 2020 ) . Another line of work seeks state-of-the-art ( SOTA ) performance with low-precision training , using carefully designed learning schedules ( Sun et al. , 2019 ; 2020 ) . Our work imagines low-precision as a way to reduce training or deployment costs of customized models trained on proprietary datasets ( not SOTA models on ML benchmarks ) . Even so , we show in Section 4 that our work picks promising models for every memory budget , which enables near-SOTA results on CIFAR-10 . Figure 2 shows a flowchart of PEPPP . The rest of this paper is organized as follows . Section 2 introduces notation and terminology . Section 3 describes the main ideas we use to actively sample configurations and approximate Pareto frontiers . Section 4 shows experimental results . 1https : //cloud.google.com/ai-platform/training/pricing 2https : //aws.amazon.com/ec2/pricing/on-demand 2 NOTATION AND TERMINOLOGY . Low-precision formats . We use floating point numbers for low-precision training in this paper . As an example , Figure 3 shows an 8-bit floating point number with 3 exponent bits and 4 mantissa bits . A specific low-precision representation with a certain number of bits for each part is called a low-precision format . We may use different low-precision formats for the weights , activations , and optimizer . In the case of using two formats to train and represent a neural network ( as discussed in Section 3.1 ) , these two formats are called Format A and B . A specific combination of these two formats is a hyperparameter setting of the neural network , and we call it a low-precision configuration . Math basics . We define [ n ] = { 1 , . . . , n } for a positive integer n. With a Boolean variable X , the indicator function 1 ( X ) equals 1 if X is true , and 0 otherwise . With a scalar variable x , we use x+ to denote max { x , 0 } . ⊆ and ⊂ denote subset and strict subset , respectively . Linear algebra . We denote vector and matrix variables respectively by lowercase letters ( a ) and capital letters ( A ) . All vectors are column vectors . The Euclidean norm of a vector a ∈ Rn is ‖a‖ : = √∑n i=1 a 2 i . To denote a part of a matrix A ∈ Rn×d , we use a colon to denote the varying dimension : Ai , : and A : ,j ( or aj ) denote the ith row and jth column of A , respectively , and Aij denotes the ( i , j ) -th entry . With an ordered index set S = { s1 , . . . , sk } where s1 < . . . < sk ∈ [ d ] , we denote A : S : = [ A : ,s1 · · · A : ,sk ] ∈ Rn×k . Given two vectors x , y ∈ Rn , x y means xi ≤ yi for each i ∈ [ n ] . Given a matrixA ∈ Rn×d and a set of observed indices as Ω ⊆ [ n ] × [ d ] , the partially observed matrix PΩ ( A ) ∈ Rn×d has entries ( PΩ ( A ) ) ij = Aij if ( i , j ) ∈ Ω , and 0 otherwise . Pareto frontier . Multi-objective optimization simultaneously minimizes n costs { ci } i∈ [ n ] . A feasible point c ( 1 ) = ( c ( 1 ) 1 , . . . , c ( 1 ) n ) is Pareto optimal if for any other feasible point c ( 2 ) = ( c ( 2 ) 1 , . . . , c ( 2 ) n ) , c ( 2 ) c ( 1 ) implies c ( 2 ) = c ( 1 ) ( Boyd & Vandenberghe , 2004 ) . The set of Pareto optimal points is the Pareto frontier . Tasks , datasets , measurements and evaluations . A task carries out a process ( classification , regression , image segmentation , etc . ) on a dataset . Given a deep learning model and a dataset , the training and testing of the model on the dataset is called a measurement . In our low-precision context , we evaluate a configuration on a dataset to make a measurement . Meta-learning . Meta-learning transfers knowledge from past ( meta-training ) tasks to better understand the new ( meta-test ) task . To evaluate the error of meta-training , we use meta-leave-one-out cross-validation ( meta-LOOCV ) : on a collection of tasks , each time we treat one task as meta-test and the others as meta-training , and average the results over all splits . Error matrix and memory matrix . Given a neural network , the errors of different low-precision configurations on meta-training datasets form an error matrix , whose ( i , j ) -th entry Eij is the test error of the jth configuration on the ith dataset . To compute the error matrix , we split the i-th dataset into training and test subsets ( or use a given split ) , train the neural network at the j-th configuration on the training subset , and evaluate the test error on the test subset . The memory required for each measurement forms a memory matrix M , which has the same shape as the corresponding error matrix . Example error and memory matrices are shown in Figure 4 . 3 METHODOLOGY . PEPPP operates in two phases : meta-training and meta-test . First in meta-training , we learn configuration embeddings by training a neural network of a specific architecture at different low-precision configurations on different datasets . As listed in Appendix A , we study image classification tasks and a range of low-precision formats that vary in the number of bits for the exponent and mantissa for the activations , optimizer , and weights . Our hope is to avoid enumerating every possible configuration since exhaustive search is prohibitive in practice . To this end , we make a few measurements and then predict the other test errors by active learning techniques . We also compute the memory matrix to understand the memory consumption of each measurement . Then in meta-test , we assume that we have completed meta-training , and hence know the configuration embeddings . The meta-test goal is to predict the tradeoff between memory usage and test error on a new ( meta-test ) dataset . This step corresponds to inference in traditional machine learning ; it must be quick and cheap to satisfy the needs of resource-constrained practitioners . To select the configuration that has smallest test error and takes less than the memory limit , we measure a few selected ( informative and cheap ) configurations on the meta-test dataset , and use the information of their test errors to predict the rest . The problem of choosing measurements is known as the “ cold-start ” problem in the literature on recommender systems . Then we use the model built on the meta-training datasets to estimate values for the other measurements on the meta-test dataset .

Summary: The paper proposes a two-step methodology for evaluating the error-memory trade-off when employing low-precision training/inference in neural networks. Although low-precision training/inference in deep learning is a widely studied topic, this paper seems to be the first one to study this error-memory tradeoff without exhaustive search.   The proposed idea is inspired from the multi-objective optimization field. The authors propose to identify the Pareto frontier that would allow a user to identify the optimal precision (lowest error) given a certain memory constraint. The proposed system has two phases: training and testing.   At training time, given a set of datasets, a network architecture and a set of low-precision configurations, the method samples dataset-configuration pairs to train and computes the misclassification error for each selected pair. Matrix factorization is used to compute low-dimensional embeddings for each configuration.   At test time, a subset of configurations is chosen for evaluation (using Design Experiments with Matrix Factorization) and the computed errors are used as input together with the configuration embeddings to a linear regression model in order to estimate the errors on the non-evaluated configurations. For both training and testing, the memory matrix is fully computed based on the network architecture and the low-precision format.  The method is evaluated in terms of convergence and Hypervolume difference of the true and estimated Pareto frontier and it outperforms in most of the cases other methods such a Bayesian Optimization or Random Selection with Matrix Factorization.