EFloat: Entropy-coded Floating Point Format for Deep Learning

In a large class of deep learning models , specifically vector embedding models1 in NLP , we observe that floating point exponent values tend to cluster around2 few unique values , presenting entropy encoding opportunities . The proposed3 EFloat floating point number format encodes frequent exponent values and signs4 with Huffman codes to minimize the average exponent field width while keeping5 the original exponent range unchanged . Saved bits then become available to the6 significand increasing the EFloat numeric precision on average by 4.3 bits compared7 to other low-precision floating point formats of equal bit budget . The EFloat format8 makes 8-bit and smaller floats practical by preserving the full exponent range9 of a 32-bit floating point representation . We currently use the EFloat format for10 compressing and saving memory used in large NLP deep learning models while11 I/O and memory bandwidth savings in GPUs and AI accelerators are also possible.12 Using RMS-error as a precision metric , we demonstrate that EFloat provides more13 accurate floating point representation than other formats with the same bit budget.14 EF12 with 12-bit budget has less end-to-end application error than the 16-bit15 BFloat16 . EF16 RMS-error is 17 to 35 times less than BF16 RMS-error for a range16 of datasets . Using the NDCG metric for evaluating ranked results of similarity and17 dissimilarity queries in NLP , we demonstrate that EFloat matches the result quality18 of other floating point representations with larger bit budgets.19 1 Introduction20 As natural language processing ( NLP ) models expand their capabilities , complexity , and training costs,21 the model sizes have been increasing dramatically . For example , state-of-the-art transformer-based22 NLP models such as BERT ( Vaswani et al . ( 2017 ) ) , Megatron-LM ( Shoeybi et al . ( 2020 ) ) , Open AI23 GPT-3 ( Brown et al . ( 2020 ) ) , or Google Switch-C Transformers ( Fedus et al . ( 2021 ) ) , contain from24 hundreds of millions , to even trillion parameters ( Hoefler ( 2020 ) ; Fedus et al . ( 2021 ) ) . Although25 NLP model compression is a very active area of research ( Section 6 ) , its current focus is on model26 inference scenarios , in which reduced precision and integer quantization are commonly used , given27 that the original model need not be restored.28 The primary goal for this work is to explore compression strategies for large vector embedding models29 such that one can recover or minimize the loss in the original model , or use the same compressed30 model in the inference phase . The database embedding ( db2Vec ) , a vector embedding technique31 designed to develop semantic models from multi-modal relational database tables ( Bordawekar and32 Shmueli ( 2016 , 2017 ) ) , forms the impetus behind this exploration . Db2Vec differs from its NLP33 counterparts , such as Word2Vec ( Mikolov et al . ( 2013 ) ) and GloVe ( Pennington et al . ( 2014b ) ) , in34 that its source data follows the relational data model ( Date ( 1982 ) ) ( the source data is not a natural35 language document but a relational database table ) . Considering the relational database tables can36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Number of bits 1 2 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 ( a ) IEEE SINGLE ( FP32 ) S E E E E E E E E M M M M M M M M M M M M M M M M M M M M M M M ( b ) BFLOAT16 S E E E E E E E E M M M M M M M ( c ) IEEE HALF ( FP16 ) S E E E E E M M M M M M M M M M ( e ) EFLOAT16 ( 8-bit logical exponent ) S ( f ) EFLOAT16 Frequent Exponents S E E E M M M M M M M M M M M M ( g ) EFLOAT16 Infrequent Exp . S E E E E E E M M M M M M M ( h ) EFLOAT16 Average ( 3 to 4-bit typical ) S E E E M M M M M M M M M M M M ( i ) EFLOAT11 Average S E E E M M M M M M M N bits ( 15-N ) bits . compute cycles with capacity and I/O bandwidth savings . A factor of 3 reduction in memory68 footprint is achieved converting FP32 values to EF11 . Hardware conversion is possible for69 FPn to EFn and vice versa with simple Static RAM based lookup tables.70 • For a given dataset , many different FP to EF conversion tables are possible . Tables may be71 optimized for maximum significand width ( highest exponent compression ) at the expense of72 worse precision for few floats with infrequent exponents ( less significand bits for outliers ) 73 and vice versa.74 • Since vector embedding models are used in a wide array of NLP transformer architectures,75 the EFloat format can be used for a much wider ( and more space consuming ) class of NLP76 models.77 In Section 2 , we first present the analysis of various vector embedding models . The EFloat format78 is presented in Section 3 . Section 4 describes key steps in conversion between EFloat and other FP79 formats . Section 5 presents an error analysis of various EFloat widths ( EFn ) against BF16 and FP16.80 In Section 6 , a review of related work on model compression and floating-point formats for deep81 learning is presented . Finally , Section 7 presents conclusions and outlines future directions.82 2 Analyzing vector embedding models83 Vector embedding models are extensively used in natural language processing ( NLP ) to capture84 and exploit semantic relationships of word entities ( e.g. , words , sentences , phrases , paragraphs , or85 documents ) . A trained vector embedding model consists of a set of vectors , each vector encoding86 a distributed representation of inferred semantics of a word entity , i.e. , a single vector captures87 different attributes of the inferred semantics ( Hinton et al . ( 1986 ) ) , created , in part , by contributions88 by other word entities . Every vector embedding model implements some variant of the log-bilinear89 language ( LBL ) model that predicts the probability of the next word wi given the previous words90 ( context ) ( Hinton ( 2013 ) ; Almeida and Xexeo ( 2019 ) ; Bender and Koller ( 2020 ) ) . The LBL model91 first predicts a real-valued vector representation of a word by linearly combining the real-valued vector92 representations of its context words . Then the distributed representation of the word is computed93 based on the similarity between the predicted representation and the representations of all words in94 the vocabulary . This step is accomplished using the normalized exponential or Softmax function over95 the associated feature vectors . The output of the Softmax function is the probability distribution over96 V different possible outcomes , where V is the vocabulary size.97 Figure 2 presents histograms of exponent values in multiple pre-trained vector embedding models,98 where the X-axis represents exponent values ( from the 8-bit exponent portion of a 32-bit IEEE 75499 floating point value ) . For each X-axis value , the Y-axis represents normalized number of occurrences100 of that exponent value , i.e. , a histogram . Vector embedding and related NLP models presented in Fig . 2101 include word embedding ( word2Vec ) ( Mikolov et al . ( 2013 ) ; Zhang et al . ( 2019 ) ) , sentence ( sent2Vec ) 102 and document embedding ( doc2Vec ) ( Le and Mikolov ( 2014 ) ; Chen et al . ( 2019 ) ) , GloVe ( Pennington103 et al . ( 2014a ) ) , subword embedding ( FastText ) ( Bojanowski et al . ( 2017a , b ) ) , database embedding104 ( db2Vec ) , graph embedding ( PyTorch BigGraph Lerer et al . ( 2019 ) ) , and Google ’ s transformer-based105 universal sentence encoder ( Cer et al . ( 2018 ) ; Google ( 2021 ) ) using the Brown corpus ( Brown-106 corpus ( 2021 ) ) . All these models implement different variations of the LBL model . The word2Vec107 based models , e.g. , word2Vec , sent2Vec , doc2Vec , db2Vec , and FastText , use a neural network with108 different versions of Softmax as the activation function . GloVe , on the other hand , is a count-based109 optimization approach that uses a word co-occurrence matrix and weighted least-square as the110 optimization function . The FastText subword model ( Joulin et al . ( 2016 ) ; Bojanowski et al . ( 2017a ) ) 111 assigns a vector for every character n-gram , using an extended skip-gram model ( Mikolov et al.112 ( 2013 ) ) and then , words are represented as the sum of these representations . The universal sentence113 encoder generates embedding vectors for sentences using a standard Transformer architecture that114 takes word embedding vectors as input and uses a Softmax function to compute attention ( Vaswani115 et al . ( 2017 ) ) . Irrespective of the model type , we observe that exponent values cluster around a116 certain range of values , and display a distinct peak . The only exception is the doc2Vec model that117 exhibits two peaks as the doc2Vec first builds fine-grained embeddings for words and then uses them118 to build embeddings for coarser-grained entities such as paragraphs via concatenating and averaging119 individual word vectors which results in a smaller second peak as observed in Fig . 2.120 The Softmax family of activations functions used in vector embedding models is responsible for the121 clustering behavior of exponents ( Figure 2 ) . To understand the reasons , let us delve deeper into the122 training of an embedding model . For illustration purposes , we use database embedding ( db2Vec ) 123 of the Telcom Churn data ( IBM ( 2020 ) ) as an example . db2Vec is an adaptation of the word2Vec124 approach , and has been designed to build an embedding model from structured database tables that125 adhere to the relational data model . Like word2Vec , db2Vec also uses Skipgram with Negative126 Sampling ( SGNS ) as the training approach . The SGNS approach uses a binary classifier based on127 the logistic ( Sigmoid ) function instead of using the Softmax-based predictor . The overall training128 process involves multiple back-propagation iterations to update model weights using the gradient of129 the Sigmoid function . Weights get updated iteratively during the back-propagation process by the130 error computed for that iteration . Practically , the error is computed using the gradient of the activation131 function . During model training , we observe that the weights rapidly converge ( Fig.4 ) to their final132 values . Their exponents are substantially clustered at the slope of the Sigmoid curve , the 2−8 to 20133 output range of Sigmoid , as evidenced by Figures 3 and 4 . Training eliminates smaller exponents134 from the model because the activation function output is practically zero for any input value when135 weights are small , and large exponents are non-existent of normalization of weights.136 3 The EFloat format : EFn137 The key idea behind the EFloat format is the variable-width encoding of exponents using the well-138 known Huffman algorithm . Frequency of unique exponent values in the dataset determine the139 coded-exponent widths which may vary between as small as 1-bit and some software configurable140 maximum , e.g. , 8-bit ( Figure 1 ( e , f , g ) ) . Thanks to the entropy coding of the Huffman algorithm,141 frequent exponent values are coded with fewer bits and infrequent exponents are coded with more142 bits as observed in Fig.5.143 Bits saved from the exponent become part of the significand , therefore increasing the floating point144 precision compared to other float formats with the same bit budget . An N -bit coded-exponent in145 an EF16 float results in a ( 15-N ) -bit significand as shown in Fig.1 ( e ) . Since Efloats with frequent146 exponents have wide significands , the entire dataset has a greater precision on average . Efloats with147 infrequent exponents have narrow significands . But , their contribution is relatively small in the148 common calculations used in model training and inferencing , such as dot-products , vector-sums , and149 cosine-similarity ( EFloat precision is quantified and compared to prior formats in Section 5 . ) .150 EFloat on average have greater precision and range than any other fixed-field FP format with the151 same bit budget . For example , EF16 with a 3-bit coded-exponent has 12-bits of significand compared152 to the 7-bit significand found in a BF16 ( Figures 1 ( h , b ) ) . EFloat exponent ’ s logical width is always153 8-bit , the same as for FP32 and BF16 , irrespective of EFloat width . Even for extremely narrow floats154 such as EF8 , the logical exponent width can be 8-bit since encoding compresses the exponent field.155 The EFloat format compresses special values of IEEE 754 , such as signed zeros and infinities156 losslessly . NaN are semantically compressed losslessly : converting a NaN to and from FP32 to EFn157 and vice-versa still results in a NaN . Denormal floats may round to zero since least significant bits of158 significands are truncated during encoding.159 4 EFloat encoding and decoding160 The Huffman algorithm : is a popular lossless compression algorithm used in many compression161 tools and compressed data formats ( Salomon ( 2004 ) ) . Data symbols are encoded with variable-length162 binary codes whose length are determined by the symbol probabilities in the data stream . The163 algorithm builds a binary tree with each leaf assigned a symbol . Higher probability symbols are164 closer to the tree root than others . The path from the tree root to the leaf is the binary coding of the165 symbol . To demonstrate with a trivial example , the letters A , B , C occuring with probabilities of166 0.5 , 0.25 , and 0.25 may be encoded with the bit patterns 0 , 10 , and 11 , respectively . The algorithm167 yields 1.5-bit/symbol compression efficiency , better than 8-bits/symbol using an ASCII representation168 or 2-bits/symbol using a simplistic mapping of the 3 letters to 2-bit integers . Huffman coding is169 optimal when symbol probabilities are negative powers of 2 . However , it is an effective compression170 method even for non-optimal data distributions . Fig.5 shows the Huffman coded exponent widths as171 a function of exponent frequencies of a word2vec trained model.172 Huffman codes have the prefix property which states that no code is a prefix of a longer code ( due173 its tree structure . ) As a result , the Huffman code not only encodes the original symbol but the174 code-length as well . We use this property to locate the bit position of the movable boundary between175 the exponent and the significand fields when decoding EFloats ( Fig.1 ( e ) ) .176 Length-Limiting : The basic Huffman algorithm , depending on probabilities , may produce extremely177 wide codes consuming the entire width of EFloats and more . We use the Length-Limiting variant of178 the Huffman algorithm to set a maximum coded-exponent width ( Abali et al . ( 2020 ) ) . In Fig.5 , the179 maximum code-width is set to 8-bits resulting in infrequent exponents encoded with that maximum.180 The software-defined limit presents an opportunity to tune the EFloat precision to a particular NLP181 application requirements . Figure 6 shows the effect of limiting maximum width of coded-exponents to182 5 , 8 , and 10-bits . As the limit is increased , the least frequent exponents are coded with the maximum-183 width codes , therefore their respective significands lose precision . On the other hand , with increasing184 limits the most frequent exponents are coded with fewer bits reducing both the minimum and the185 average coded-exponent widths , therefore their respective significands gain precision . Therefore,186 EFloat not only compresses the regular floats but for a given EFn budget of n-bits the application can187 optimize the end-to-end precision by adjusting the maximum coded-exponent width.188 That the minimum and average code widths are inversely proportional to the maximum code-width189 may appear counterintuitive ( Figure 6 ) . Let the maximum coded-exponent width be K-bits ( K = 5,190 K = 8 , etc. ) . K-bits are sufficient to represent N unique exponents in the dataset provided191 dlog2 ( N ) e ≤ K holds . When K is chosen much larger than the minimum dlog2 ( N ) e , some codes192 can have fewer than K-bits . Consider a short L-bit code such that L ≤ K ( e.g. , assume K = 8,193 L = 2 and the short-code = 00 . ) Due to the prefix property the L-bit code consumes 2K−L bit194 patterns out of the 2K maximum possible ( e.g. , all 8-bit patterns whose 2-bit prefix is 00 , 64 patterns195 in total , are consumed by the code 00 . ) The remaining N − 1 exponents can still be encoded if196 N − 1 ≤ 2K − 2K−L holds . We observed in practice that increasing the maximum code-width by 2197 to 3 bits over the minimum dlog2 ( N ) e gives a good compression ratio.198 EFloat Encoding and the Code-Table : During the conversion from FP32 to an EFn ( e.g. , EF16 ) ,199 exponents in the original dataset are histogrammed first , e.g. , Fig.2 . The histogram representing200 probabilities of the exponents is the input to the Length-Limiting Huffman algorithm . The output is a201 256-entry ( 28 ) code-table indexed by the original 8-bit exponent . Each table entry contains a pair , the202 variable-width coded-exponent and its width . Note that the code-table is quite small , tens of bytes in203 practice , since few unique exponents are present in most NLP datasets as Fig.2 shows.204 When the sign bits have a skewed distribution , e.g. , if they are substantially positive , then the sign205 bit and the 8-bit exponent may be treated as a single 9-bit integer when histogramming . Thanks to206 Huffman coding , a skewed sign bit distribution may provide up to one additional bit of precision to207 the significand.208 Using the code-table , the entire dataset is converted from FP32 to the chosen EFn width ( e.g. , EF16 ) 209 replacing original exponents with coded-exponents . Least significant bits of the FP32 significand are210 truncated to match the EFn width . For example , in Fig.5 , the algorithm encodes the most frequent211 exponent with 2-bits . Accounting for the sign bit , this yields a 13-bit significand in EF16 by truncating212 the bottom 10-bit of the 23-bit significand of FP32 . We use the round-to-nearest method to provide213 on average 0.5 bits of additional precision : when the leading bit of the truncated part is 1 the upper214 part of the significand is incremented +1 provided it doesn ’ t overflow in to the exponent field.215 For large datasets , a statistically representative subset may also be used to reduce histogram collection216 time . When the histogram is known in advance , a pre-built code-table may be used . Pre-built217 code-tables eliminate the overhead of executing the Huffman algorithm . During training exponents218 rapidly converge to their final values as observed in Fig.4 . The exponent distribution is practically219 identical for all iterations 11 to 2481 , Therefore , a single pre-built code-table optimized for final220 iterations may serve for all iterations start to finish . The same pre-built table , although suboptimal221 for early iterations , may be used because significand precision is not as important at that point in222 time ; model weight updates are dominated by exponent updates . Once exponents settled to their final223 values the significand precision becomes important since model weights updates progressively get224 smaller.225 EFloat Decoding : For EFloat to FP32 conversion ( i.e. , decoding ) we use a inverse mapping of the226 code-table described earlier . A decoder-table indexed by the coded-exponent may be used to decode227 the original exponent value and the significand ’ s leading bit position in constant time . Each table entry228 contains the original exponent and width of the coded-exponent . To index the decoder-table with229 variable-width codes many entries are filled with duplicates . For example , the 2-bit coded-exponent230 00 is duplicated 64 times in the table at locations 00000000 through 00111111 with each location231 containing the pair ( original exponent and code-width= 2 ) . Duplicating entries is equivalent to having232 logical don ’ t care bits in the index which is a useful in hardware based decoder implementations.233 The second element of each table entry contains the EFloat significand width . Since the significand234 was truncated earlier during the FP32 to EFloat conversion , the missing least significant bits must be235 padded with zeros to match the original FP32 width.236 5 Evaluating the EFloat representation237 In this section , we evaluate the efficacy of the EFloat format using two sets of experiments . The238 first set measures the loss of precision in representing FP32 data in BF16 , FP16 , and EFloat formats239 with bit budgets from 16 to 8 bits . The second set of experiments compares the quality of ranked240 results for similarity and dissimilarity queries using the Normalized Discounted Cumulative Gain241 ( NDCG ) score for BF16 , FP16 , and various EFloat formulations . Table 1 presents the list of models242 used in these experiments , along with their characteristics : model types , model size ( stored using243 FP32 ) , number of unique exponents , range of exponent bits generated by the Huffman algorithm , the244 average count of exponent bits , and minimum and maximum average count of significand bits . For245 EF16 , the average significand length is 4.3 bits more than BF16 ( with 7-bit significand ) and 1.2 bits246 more than FP16 ( with 10-bit significand ) .247 The first set of experiments compares the loss of precision due to the least significant significand248 bits being truncated during conversion from FP32 to various lower-precision formats . Given a249 low-precision format ( e.g. , EF16 or BF16 ) , the values are converted back to FP32 , and the arithmetic250 difference , fo − f c , of the original FP32 value , fo , and the regenerated FP32 value , f c , is computed.251 This difference represents the precision loss due to conversion . Root Mean Square Error ( RMSE ) 252 metric is then used to summarize the loss of precision across a dataset of N floats as:253 RMSE = √√√√ 1 N N∑ k ( fok − f ck ) 2 We then compare the errors of BF16/FP16 and EFn by dividing RMSEBF16/FP16 by RMSEEFn254 in Table 2 . Ratios greater than 1.0 indicate that the EFloat error is less than BF16 or FP16 errors.255 For EF16 , across all models , we observe an average RMSE error ratio of 24.1 for BF16 , and 1.5 for256 FP16 . Note that for these experimental results , the datasets were encoded with a minimum of 3-bit257 and a maximum of 6-bit coded-exponents resulting in an average width in the range of 3.4 to 4.2-bits258 ( Table 1 ) . Accordingly , for EF16 , the minimum significand width is 10-bit which is 3-bit wider than259 BF16 , and of the same length as FP16 . Therefore , EF16 has significantly higher precision against260 BF16 than FP16 . Also , Table 2 shows that EF12 has the same to slightly better RMSE than BF16261 since the RMSE ratios are in the 1.0 to 2.2 range . Thus , EF12 uses 25 % less bandwidth and memory262 capacity than BF16 for similar floating-point precision.263 Note that the RMSE method amplifies larger errors due to the squaring of differences . EFloat coded264 floating point values with short significands ( i.e. , those with infrequent exponents ) are disproportion-265 ately represented in the RMSE summation . However , the true measure of error for vector embedding266 models will be the evaluation of ranked results for similarity queries for different floating point267 formats . Unlike the binning in traditional classification inference tasks , ranked results from similarity268 queries are far more sensitive to numerical precision . We use the Normalized Discounted Cumulative269 Gain ( NDCG ) metric ( Järvelin and Kekäläinen ( 2002 ) ; Wang et al . ( 2013 ) ) , to evaluate the quality270 of ranked results for different floating point formats . NDCG is widely used in information retrieval271 and web search to evaluate the relevance of retrieved documents . NDCG is a normalization of the272 Discounted Cumulative Gain ( DCG ) measure . DCG is calculated as a weighted sum of the degree of273 relevancy of the ranked items , where the weight is a decreasing function of the position of an item.274 NDCG is computed by normalizing DCG by IDCG , which is the DCG measure for a perceived ideal275 ranking result . Thus , the NDCG measure always lies within [ 0.0,1.0 ] .276 For a given vector embedding model , we choose q = 20 randomly selected distinct query points.277 For each query point , we compute similar and dissimilar points by computing cosine similarities278 over the corresponding vectors . For similarity queries , the result contains a list of points sorted in279 decreasing order of their similarity scores ( most similar pair of items will have score closer to 1.0 ) ,280 and for dissimilarity queries , the result list is sorted in increasing order of their similarity scores ( most281 dissimilar pair of items will have score closer to -1.0 ) . For each query point , we run similarity and282 dissimilarity queries for different floating point formats , and use the top k = 10 results for each test283 to compute the NDCG score , ( NDCG @ 10 ) . In our evaluation , we use the ranked results for FP32284 as the baseline for calculating the IDCG . For each model , we report the average NDCG @ 10 score285 computed over 20 query points using BF16 , FP16 , and various EFn from EF16 to EF8.286 Figure 7 presents NDCG10 results for similarity queries , and Figure 8 presents NDCG10 results for287 dissimilarity queries . For both similarity and dissimilarity queries , EF16 matches or exceeds the288 quality of BF16 or FP16 ( in particular , among the three formats , BF16 provides the worst qaulity289 results ) . Furthermore , EF14 and EF12 provide similar quality results as EF16 in many instances . The290 two lower-precision EFn , EF10 and EF8 , consistently generate the least quality results.291 In summary , results from the two sets of experiments ( Table 2 , and Figures 7 and 8 ) , conclusively292 demonstrate that : ( 1 ) Given a bit budget , EFloat has higher accuracy than other formats , ( 2 ) In many293 scenarios , EFn with reduced bit budget ( e.g. , EF14 or EF12 ) provides results of quality comparable to294 higher precision formats , e.g. , BF16 , and FP16 . These results validate the design of the EFloat format,295 and demonstrate that EFloats can be used for compressing and computing using vector embedding296 models.297 6 Related Work298 Model quantization is widely used to compress pre-trained models for the inference phase ( Gupta299 and Agrawal ( 2020 ) ) . Quantization covers two broad approaches : the first represents a full-precision300 ( e.g. , 32-bit ) floating point weight value using reduced ( e.g. , BF16 or FP16 ) or mixed precision301 floats , and the second converts full-precision floating point values into integer values with fewer bits302 ( e.g. , INT8 , INT4 , and INT1 ( Migacz ( 2017 ) ; Wu et al . ( 2020 ) ; TensorFlow Documentation ( 2020 ) ; 303 Jacob et al . ( 2017 ) ) ) . In conjunction with the model compression work , there has been significant304 work in devising reduced-precision floating point formats tuned for broader machine learning and305 HPC applications ( Sapunov ( 2020 ) ; Abdelfattah et al . ( 2020 ) ) . Unlike the inference-focused model306 compression work , reduced-precision floating points are designed to work for both model training and307 inference phases . The most common reduced-precision floating point formats use 16 bits . Current308 16-bit implementations include IEEE 754 half-precision ( FP16 ) ; Brain Floating Point , BFloat16309 ( Wang and Kumar ( 2019 ) ; Kalamkar et al . ( 2019 ) ) ; and Deep Learning Float ( DLFloat ) ( Agrawal310 et al . ( 2019 ) ) , with 1 sign bit , 6 exponent bits , and 7 fraction bits . TensorFloat-32 ( TF32 ) from311 Nvidia is a 19-bit format that combines 8 exponent bits from BFLOAT16 and 10 exponent bits from312 IEEE FP16 ( Kharya ( 2020 ) ) . Hybrid Block Floating Point ( HBFP ) ( Drumond et al . ( 2018 ) ) , Intel313 Nervana ’ s Flexpoint ( Koster et al . ( 2017 ) ) , and Microsoft MSFP ( Rouhani et al . ( 2020 ) ) formats314 combine the advantages of fixed point and floating point formats by splitting up the significand and315 the exponent part which is shared across multiple numeric values . Recent research proposals have316 described training of key deep learning models using even reduced precision floating point values317 ( 8- and 4-bit floats ) ( Sun et al . ( 2020 ) ; Wang et al . ( 2018 ) ; Cambier et al . ( 2020 ) ; Mellempudi et al.318 ( 2019 ) ) . Recently proposed AdaptiveFloat ( Tambe et al . ( 2020 ) ) is an inference-targeted floating-point319 format which maximizes its dynamic range at a network layer granularity by dynamically shifting320 its exponent range via modifications to the exponent bias and by optimally clipping ( quantizing ) its321 representable datapoints . Our proposed EFloat design practically achieves the same result without322 altering the exponent range and quantizing full-precision values.323 7 Conclusion324 We introduced EFloat , a novel entropy-coded variable length floating point format for deep learning325 applications . This format can be used for compressing a trained deep learning model , as well as for326 enabling more accurate model representations using reduced-precision floating point formats . While327 our intended use cases were initially for the database embedding ( db2Vec ) workloads , we demonstrate328 that the proposed format works effectively for other vector embedding models , and can be used for a329 much broader class of NLP models including transformer-based models . Broadly , EFloat may be330 used in deep learning applications where tradeoffs need to be made between range , precision , memory331 capacity and bandwidth savings . As a future work , we plan to explore the Benford distribution332 pattern ( Benford ( 1938 ) ; Newcomb ( 1881 ) ) exhibited by significands of vector embedding models333 ( Appendix A in the supplementary document ) and investigate its application in rounding EFloat334 values . A follow-up study on 8-bit floats and integers is being considered as well.335 References336 Bulent Abali , Bart Blaner , John Reilly , Matthias Klein , Ashutosh Mishra , Craig B Agricola , Bedri337 Sendir , Alper Buyuktosunoglu , Christian Jacobi , William J Starke , et al . 2020 . Data Compression338 Accelerator on IBM POWER9 and z15 Processors . In 2020 ACM/IEEE 47th Annual International339 Symposium on Computer Architecture ( ISCA ) . IEEE Computer Society , 1–14.340 Ahmad Abdelfattah , Hartwig Anzt , Erik G. Boman , Erin Carson , Terry Cojean , Jack Dongarra , Mark341 Gates , Thomas Grützmacher , Nicholas J. Higham , Sherry Li , Neil Lindquist , Yang Liu , Jennifer342 Loe , Piotr Luszczek , Pratik Nayak , Sri Pranesh , Siva Rajamanickam , Tobias Ribizel , Barry Smith,343 Kasia Swirydowicz , Stephen Thomas , Stanimire Tomov , Yaohung M. Tsai , Ichitaro Yamazaki , and344 Urike Meier Yang . 2020 . A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic.345 arXiv:2007.06674 [ cs.MS ] 346 A. Agrawal , S. M. Mueller , B. M. Fleischer , X . Sun , N. Wang , J. Choi , and K. Gopalakrishnan . 2019.347 DLFloat : A 16-b Floating Point Format Designed for Deep Learning Training and Inference . In348 2019 IEEE 26th Symposium on Computer Arithmetic ( ARITH ) . 92–95.349 Felipe Almeida and Geraldo Xexeo . 2019 . Word Embeddings : A Survey . arXiv:1901.09069 [ cs.CL ] 350 Emily M. Bender and Alexander Koller . 2020 . Climbing towards NLU : On Meaning , Form , and351 Understanding in the Age of Data . In Proceedings of the 58th Annual Meeting of the Association352 for Computational Linguistics . Association for Computational Linguistics , Online , 5185–5198.353 https : //doi.org/10.18653/v1/2020.acl-main.463354 Frank Benford . 1938 . The law of anomalous numbers . In Proc . American Philosophical Society,355 Vol . 78 . 551–572.356 Piotr Bojanowski , Edouard Grave , Armand Joulin , and Tomas Mikolov . 2017a . Enriching Word357 Vectors with Subword Information . Transactions of the Association for Computational Linguistics358 5 ( 2017 ) , 135–146.359 Piotr Bojanowski , Edouard Grave , Armand Joulin , and Tomas Mikolov . 2017b . En-360 riching Word Vectors with Subword Information . arXiv:1607.04606 [ cs.CL ] 361 https : //fasttext.cc/docs/en/english-vectors.html MIT License.362 https : //github.com/facebookresearch/fastText/blob/master/LICENSE.363 Rajesh Bordawekar and Oded Shmueli . 2016 . Enabling Cognitive Intelligence Queries in Relational364 Databases using Low-dimensional Word Embeddings . CoRR abs/1603.07185 ( 2016 ) . http:365 //arxiv.org/abs/1603.07185366 Rajesh Bordawekar and Oded Shmueli . 2017 . Using Word Embedding to Enable Semantic Queries367 in Relational Databases . In Proceedings of the 1st Workshop on Data Management for End-to-End368 Machine Learning ( Chicago , IL , USA ) ( DEEM ’ 17 ) . ACM , New York , NY , USA , Article 5 , 4 pages.369 https : //doi.org/10.1145/3076246.3076251370 Tom B . Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhari-371 wal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal,372 Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel M.373 Ziegler , Jeffrey Wu , Clemens Winter , Christopher Hesse , Mark Chen , Eric Sigler , Mateusz Litwin,374 Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford,375 Ilya Sutskever , and Dario Amodei . 2020 . Language Models are Few-Shot Learners . CoRR376 abs/2005.14165 ( 2020 ) . arXiv:2005.14165 https : //arxiv.org/abs/2005.14165377 Brown-corpus 2021 . Brown Corpus . http : //www.sls.hawaii.edu/bley-vroman/brown_378 nolines.txt379 Leopold Cambier , Anahita Bhiwandiwalla , Ting Gong , Mehran Nekuii , Oguz H Elibol , and Hanlin380 Tang . 2020 . Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep381 Neural Networks . arXiv:2001.05674 [ cs.LG ] 382 Daniel Cer , Yinfei Yang , Sheng-yi Kong , Nan Hua , Nicole Limtiaco , Rhomni St. John , Noah383 Constant , Mario Guajardo-Cespedes , Steve Yuan , Chris Tar , Brian Strope , and Ray Kurzweil.384 2018 . Universal Sentence Encoder for English . In Proceedings of the 2018 Conference on385 Empirical Methods in Natural Language Processing : System Demonstrations . Association for386 Computational Linguistics , Brussels , Belgium , 169–174 . https : //doi.org/10.18653/v1/387 D18-2029 Apache-2.0 License.388 Qingyu Chen , Yifan Peng , and Zhiyong Lu . 2019 . BioSentVec : creating sentence embed-389 dings for biomedical texts . In The 7th IEEE International Conference on Healthcare Infor-390 matics . https : //github.com/ncbi-nlp/BioSentVec License : https : //github.com/ncbi-391 nlp/BioSentVec/blob/master/LICENSE.txt.392 C. J . Date . 1982 . A Formal Definition of the relational Model . SIGMOD Rec . 13 , 1 ( 1982 ) , 18–29.393 https : //doi.org/10.1145/984514.984515394 Mario Drumond , Tao Lin , Martin Jaggi , and Babak Falsafi . 2018 . Training DNNs with Hybrid395 Block Floating Point . In Proceedings of the 32nd International Conference on Neural Information396 Processing Systems ( NIPS1́8 ) . Curran Associates Inc. , Red Hook , NY , USA , 451–461.397 William Fedus , Barret Zoph , and Noam Shazeer . 2021 . Switch Transformers : Scaling to Trillion398 Parameter Models with Simple and Efficient Sparsity . arXiv:2101.03961 [ cs.LG ] 399 Google . 2021 . TensorFlow Universal Sentence Encoder . https : //tfhub.dev/google/400 universal-sentence-encoder/4 Apache-2.0 License.401 Manish Gupta and Puneet Agrawal . 2020 . Compression of Deep Learning Models for Text : A Survey.402 arXiv:2008.05221 [ cs.CL ] 403 Geoffrey Hinton . 2013 . Learning Distributed Representations For Statistical Language Modelling.404 https : //www.cs.toronto.edu/~hinton/csc2535/note/hlbl.pdf.405 G. E. Hinton , J. L. McClelland , and D. E. Rumelhart . 1986 . Distributed representations . Parallel406 distributed processing : explorations in the microstructure of cognition 1 ( 1986 ) , 77–109.407 Torsten Hoefler . 2020 . OpenAI ’ s GPT-3 took more than 314 Zettaflop ( mixed fp16/fp32 ) to train ! 408 https : //twitter.com/thoefler/status/1268291652701077504 ? s=20.409 IBM . 2020 . Predict Customer Churn using Watson Machine Learning and Jupyter Notebooks410 on Cloud Pak for Data . https : //github.com/IBM/telco-customer-churn-on-icp4d411 Apache-2.0 License.412 IEEE . 2019 . `` IEEE Standard for Floating-Point Arithmetic '' . , 84 pages . https : //doi.org/10.413 1109/IEEESTD.2019.8766229414 Benoit Jacob , Skirmantas Kligys , Bo Chen , Menglong Zhu , Matthew Tang , Andrew Howard , Hartwig415 Adam , and Dmitry Kalenichenko . 2017 . Quantization and Training of Neural Networks for416 Efficient Integer-Arithmetic-Only Inference . arXiv:1712.05877 [ cs.LG ] 417 Kalervo Järvelin and Jaana Kekäläinen . 2002 . Cumulated Gain-Based Evaluation of IR Techniques.418 ACM Transactions on Information Systems 20 , 4 ( 2002 ) . https : //doi.org/10.1145/582415.419 582418420 Armand Joulin , Edouard Grave , Piotr Bojanowski , and Tomas Mikolov . 2016 . Bag of Tricks for421 Efficient Text Classification . arXiv:1607.01759 [ cs.CL ] 422 Dhiraj Kalamkar , Dheevatsa Mudigere , Naveen Mellempudi , Dipankar Das , Kunal Banerjee,423 Sasikanth Avancha , Dharma Teja Vooturi , Nataraj Jammalamadaka , Jianyu Huang , Hector Yuen,424 Jiyan Yang , Jongsoo Park , Alexander Heinecke , Evangelos Georganas , Sudarshan Srinivasan , Ab-425 hisek Kundu , Misha Smelyanskiy , Bharat Kaul , and Pradeep Dubey . 2019 . A Study of BFLOAT16426 for Deep Learning Training . arXiv:1905.12322 [ cs.LG ] 427 Paresh Kharya . 2020 . TensorFloat-32 in the A100 GPU Accelerates AI Training , HPC up to 20x.428 Nvidia Blog.429 Urs Koster , Tristan J. Webb , Xin Wang , Marcel Nassar , Arjun K. Bansal , William H. Constable,430 Oguz H. Elibol , Scott Gray , Stewart Hall , Luke Hornof , Amir Khosrowshahi , Carey Kloss , Ruby J.431 Pai , and Naveen Rao . 2017 . Flexpoint : An Adaptive Numerical Format for Efficient Training of432 Deep Neural Networks . arXiv:1711.02213 [ cs.LG ] 433 Quoc V. Le and Tomas Mikolov . 2014 . Distributed Representations of Sentences and Documents.434 CoRR abs/1405.4053 ( 2014 ) . http : //arxiv.org/abs/1405.4053435 Adam Lerer , Ledell Wu , Jiajun Shen , Timothee Lacroix , Luca Wehrstedt , Abhijit Bose , and436 Alex Peysakhovich . 2019 . PyTorch-BigGraph : A Large-scale Graph Embedding System.437 arXiv:1903.12287 [ cs.LG ] https : //github.com/facebookresearch/PyTorch-BigGraph438 BSD License . https : //github.com/facebookresearch/PyTorch-BigGraph/blob/master/LICENSE.txt.439 Naveen Mellempudi , Sudarshan Srinivasan , Dipankar Das , and Bharat Kaul . 2019 . Mixed Precision440 Training With 8-bit Floating Point . arXiv:1905.12334 [ cs.LG ] 441 Szymon Migacz . 2017 . 8-bit Inference with TensorRT . https : //on-demand.gputechconf.com/442 gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf.443 Tomas Mikolov , Kai Chen , Greg Corrado , and Jeffrey Dean . 2013 . Efficient Estimation of Word444 Representations in Vector Space . CoRR abs/1301.3781 ( 2013 ) . http : //arxiv.org/abs/1301.445 3781446 Simon Newcomb . 1881 . Note on the frequency of use of the different digits in natural numbers.447 American Journal of Mathematics 4 , 1/4 ( 1881 ) , 39–40.448 Jeffrey Pennington , Richard Socher , and Christopher D. Manning . 2014a . GloVe : Common Crawl449 840B Dataset . https : //nlp.stanford.edu/projects/glove License : Open Data Commons450 Public Domain Dedication and License.451 Jeffrey Pennington , Richard Socher , and Christopher D. Manning . 2014b . GloVe : Global Vectors for452 Word Representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural453 Language Processing . 1532–1543 . http : //aclweb.org/anthology/D/D14/D14-1162.pdf454 Bita Rouhani , Daniel Lo , Ritchie Zhao , Ming Liu , Jeremy Fowers , Kalin Ovtcharov , Anna Vino-455 gradsky , Sarah Massengill , Lita Yang , Ray Bittner , Alessandro Forin , Haishan Zhu , Taesik Na,456 Prerak Patel , Shuai Che , Lok Chand Koppaka , Xia Song , Subhojit Som , Kaustav Das , Saurabh457 Tiwary , Steve Reinhardt , Sitaram Lanka , Eric Chung , and Doug Burger . 2020 . Pushing the Limits458 of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point . In NeurIPS 2020.459 ACM.460 David Salomon . 2004 . Data compression : the complete reference . Springer Science & Business461 Media.462 Grigory Sapunov . 2020 . FP64 , FP32 , FP16 , BFLOAT16 , TF32 , and other members of the ZOO.463 https : //moocaholic.medium.com/ fp64-fp32-fp16-bfloat16-tf32-and-other-members- of-the-zoo-464 a1ca7897d407.465 Mohammad Shoeybi , Mostofa Patwary , Raul Puri , Patrick LeGresley , Jared Casper , and Bryan466 Catanzaro . 2020 . Megatron-LM : Training Multi-Billion Parameter Language Models Using Model467 Parallelism . arXiv:1909.08053 [ cs.CL ] 468 Xiao Sun , Naigang Wang , Chia-yu Chen , Jia-min Ni , Ankur Agrawal , Xiaodong Cui , Swagath469 Venkararamini , Kaoutar El Maghraoui , and Vijayalakshmi Srinivasan . 2020 . Ultra-Low Precision470 4-bit Training of Deep Neural Networks . In Advances in Neural Information Processing Systems,471 Vol . 33.472 T. Tambe , E. Y. Yang , Z. Wan , Y. Deng , V. Janapa Reddi , A . Rush , D. Brooks , and G. Y. Wei.473 2020 . Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep474 Learning Inference . In 2020 57th ACM/IEEE Design Automation Conference ( DAC ) . 1–6 . https:475 //doi.org/10.1109/DAC18072.2020.9218516476 TensorFlow Documentation . 2020 . TensorFlow Lite 8-bit quantization specification . https:477 //www.tensorflow.org/lite/performance/quantization_spec478 Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz479 Kaiser , and Illia Polosukhin . 2017 . Attention Is All You Need . arXiv:1706.03762 [ cs.CL ] 480 Naigang Wang , Jungwook Choi , Daniel Brand , Chia-Yu Chen , and Kailash Gopalakrishnan . 2018.481 Training Deep Neural Networks with 8-bit Floating Point Numbers . In Advances in Neural482 Information Processing Systems , S. Bengio , H. Wallach , H. Larochelle , K. Grauman , N. Cesa-483 Bianchi , and R. Garnett ( Eds . ) , Vol . 31 . Curran Associates , Inc. , 7675–7684.484 Shibo Wang and Pankaj Kumar . 2019 . BFloat16 : The secret to high performance on Cloud TPUs.485 Yining Wang , Liwei Wang , Yuanzhi Li , Di He , Tie-Yan Liu , and Wei Chen . 2013 . A Theoretical486 Analysis of NDCG Type Ranking Measures . CoRR abs/1304.6480 ( 2013 ) . http : //arxiv.org/487 abs/1304.6480488 Hao Wu , Patrick Judd , Xiaojie Zhang , Mikhail Isaev , and Paulius Micikevicius . 2020 . In-489 teger Quantization for Deep Learning Inference : Principles and Empirical Evaluation.490 arXiv:2004.09602 [ cs.LG ] 491 Yijia Zhang , Qingyu Chen , Zhihao Yang , Hongfei Lin , and Zhiyong Lu . 2019 . BioWordVec,492 improving biomedical word embeddings with subword information and MeSH . Scientific Data 6,493 52 ( 2019 ) . https : //github.com/ncbi-nlp/BioSentVec License : https : //github.com/ncbi-494 nlp/BioSentVec/blob/master/LICENSE.txt.495 Checklist496 1 . For all authors ... 497 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s498 contributions and scope ? [ Yes ] 499 ( b ) Did you describe the limitations of your work ? [ Yes ] 500 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] This work501 does not deal with societal impact of AI.502 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to503 them ? [ Yes ] 504 2 . If you are including theoretical results ... 505 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] Experimental506 work.507 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] Experimental work.508 3 . If you ran experiments ... 509 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-510 mental results ( either in the supplemental material or as a URL ) ? [ No ] The codes used511 in experiments are proprietary.512 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they513 were chosen ) ? [ No ] We used pre-trained models514 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-515 ments multiple times ) ? [ No ] Experiments used pre-trained models.516 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,517 type of GPUs , internal cluster , or cloud provider ) ? [ No ] Experiments used pre-trained518 models.519 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 520 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 521 ( b ) Did you mention the license of the assets ? [ Yes ] License information is provided in the522 citation.523 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 524 Information provided in the citation.525 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re526 using/curating ? [ N/A ] 527 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable528 information or offensive content ? [ N/A ] 529 5 . If you used crowdsourcing or conducted research with human subjects ... 530 ( a ) Did you include the full text of instructions given to participants and screenshots , if531 applicable ? [ N/A ] 532 ( b ) Did you describe any potential participant risks , with links to Institutional Review533 Board ( IRB ) approvals , if applicable ? [ N/A ] 534 ( c ) Did you include the estimated hourly wage paid to participants and the total amount535 spent on participant compensation ? [ N/A ] 536

Summary: The paper proposes a new tapered precision numerical format (EFloat) where the exponent of floating value is represented by Variable Length Huffman coding (The maximum of bit-width is limited). Since the Huffman coding was designed based on the probability of exponent occurrence, the most frequent exponent values are presented by fewer bits, which free some bits to be allocated for precision. The vector embedding benchmarks are used to evaluate the efficacy of the new numerical format compared to other numerical formats such as BFLOAT16 and FP16. Based on the metric represented in the paper, such as RMSE and NDCG, the EFloat is better than BFLOAT16 and comparable with FP16.