EFloat: Entropy-coded Floating Point Format for Deep Learning
Authors: Rajesh Bordawekar
Bulent Abali
Ming-Hung Chen
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Floating point representations
Huffman Encoding
Vector Embedding

Abstract: In a large class of deep learning models, specifically vector
embedding models in NLP, we observe that floating point exponent
values tend to cluster around few unique values, presenting entropy
encoding opportunities. The proposed EFloat floating point number
format encodes frequent exponent values and signs with Huffman codes
to minimize the average exponent field width while keeping the
original exponent range unchanged. Saved bits then become available to
the significand increasing the EFloat numeric precision on average by
4.3 bits compared to other low-precision floating point formats of
equal bit budget. We currently use the EFloat format for compressing
and saving memory used in large NLP deep learning models while I/O and
memory bandwidth savings in GPUs and AI accelerators are also
possible.  Using RMS-error as a precision metric, we demonstrate that
EFloat provides more accurate floating point representation than other
formats with the same bit budget. EF12 with 12-bit budget has less end-to-end application error than the 16-bit BF16.
EF16 RMS-error is 17 to 35 times less than BF16 RMS-error for a range of datasets.
Using the NDCG metric for evaluating ranked results of similarity and dissimilarity queries in NLP,
we demonstrate that EFloat matches the result quality of other floating
point representations with larger bit budgets.

29 The primary goal for this work explore compression strategies for large vector embedding models such that one can recover or minimize the loss in the original model, or use the same compressed EFloat: Entropy-coded Floating Point Format for Deep Learning
Anonymous Author(s)
Affiliation Address email
Abstract
In large class of deep learning models, specifically vector embedding models in NLP, we observe that floating point exponent values tend to cluster around few unique values, presenting entropy encoding opportunities. The proposed EFloat floating point number format encodes frequent exponent values and signs with Huffman codes minimize the average exponent field width while keeping the original exponent range unchanged. Saved bits then become available to the significand increasing the EFloat numeric precision average by 4.3 bits compared to other low-precision floating point formats of equal bit budget. The EFloat format makes 8-bit and smaller floats practical by preserving the full exponent range of a 32-bit floating point representation. We currently use the EFloat format for compressing and saving memory used in large NLP deep learning models while I/O and memory bandwidth savings in GPUs and Al accelerators are also possible. Using RMS-error as precision metric, we demonstrate that EFloat provides more accurate floating point representation than other formats with the same bit budget. EF12 with 12-bit budget has less end-to-end application error than the 16-bit BFloat16. EF16 RMS-error is 17 to 35 times less than BF16 RMS-error for range of datasets. Using the NDCG metric for evaluating ranked results of similarity and dissimilarity queries in NLP, we demonstrate that EFloat matches the result quality other floating point representations with larger bit budgets.
2
4
6
8
10
11
13
14
15
17
18
19
Introduction
20
21 As natural language processing (NLP) models expand their capabilities, complexity, and training costs,
the model sizes have been increasing dramatically. For example, state-of-the-art transformmrrbasedd NLP models such as BERT (Vaswani et al. (2017). Megatron-LMM (Shoeybi et al. (2020), Open
23
24
GPT-3 (Brown et al. 2020)), or Google Switch-C Transformers (Fedus al. (222)). contain from
25 hundreds of millions, to even trillion parameters (Hoefler (200)): Fedus et al. 2). Although
NLP model compression is very active area of research (Section 6). its current focus is model inference scenarios, in which reduced precision and integer quantization are commonly used, given that the original model need not be restored.
27
30
31 model in the inference phase. The database embedding (db2Vec), a vector embedding technique designed to develop semantic models from multi-modal relational database tables (Bordawekar and
33 Shmueli (2016) (017)), forms the impetus behind this exploration. Db2Vec differs from NL.P counterparts, such Word2Vec (Mikolov al. (2013)) and GloVe (Pennington et al. (2014b)), in 35 that its source data follows the relational data model (Date ( 1982)) (the source data is not a natural language document but relational database table). Considering the relational database tables can
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Number of bits
(2) IEEE SINGLE (FP32)
E
N M
[b] BELOAT16
[c] IEEE HALF (FP16)
FEEEE
(e) ERLOATIS (8-bit logical exponens)
bits
(15-N) bits
(I) EFLOKT16 Frequent Exponents
$ BEEE N
(g) EFLOATIS infrequent Exp.
FEEFEE
[h] EFLOATI6 Average (3 4-bit typical)
S SEEEN
EFLOAT11
Figure Floating point formats are compared. EFloat has fixed total width, but the boundary between the exponent and the significand is variable (e). The exponent is entropy coded, providing an average 4.3 extra bits of precision to the significand (e.g., (h)), while keeping the logical exponent range at 8 bits, same as that of FP32. EFloat has greater precision and range than the existing FP formats having the same budget.
be very large (e.g., billions of rows in table) with large number of unique tokens, it leads to 38 much larger vocabulary than a traditional natural language document. As a result, trained db2Vec
39 models be very large. Any trained vector embedding model is snapshot of it's weight matrices
40 and consists of weight values represented typically with IEEE 32-bit single-precision floating point (FP32) format. Therefore, we focus compression approaches that exploit different low-precision floating point formats.
42
Existing -ew-precision (FF)))))poin((FP)) formats make tradeoff between the number of exponent and significand bits. An FP number is the form
44
signbit x gexponent-bias x signi
45
The exponent largely determines the range of minimum and maximum values representable by the format and the significand width determines the precision (A constant bias is added to exponents make them all positive integers which simplifies magnitude comparisons.) For example, the BFloat (BF16) format with an 8-bit exponent and 7-bit significand has wide range but low precision when compared to FP32 (Wang and Kumar (2019)) and IEEE 754-2019 Half-precision (FP16) as illustrated in Figures D b,b,)). On the other hand, FP16 with a 5-bit exponent and 10-bit significand has a greater precision but tighter range than BF16 (TEE (019)).
46
47
48
49
50
51
In this work, introduce variable-width exponent and a variable-width significand with a total FP bit budget n, as illustrated in ee). Our design is motivated by a key pattern that observed across wide range of vector embedding models: post-training, these models use only few of the 2 256 unique exponents available FP32 and with bell-shaped distribution caused by certain class of non-linear activation functions used in model training. The EFloat design exploits this behavior and assigns the least number exponent bits most common exponent values, without losing the exponent range the original floating point value.
low-precision FP format, EFloat (EFn), that entropy-coded
53
54
55
56
57
58
59
The proposed EFloat format has the following benefits: Reduced-bi representation of any floating point format (e.g., FP32, FP16), by using fewer exponent bits map the same exponent range the original value.
61
83
For a given bit budget (c.g., 16), EFloat provides more accurate representation of the FP32 values than BF16 and FP16 by using fewer exponent bits capture the same range as before. and then using the remaining bits to increase significand precision.
84
85
86
The format suitable for both memory and bandwidth compression and reduced-bir computations over pre-trained vector embedding models. Software implementation trades

---

68
compute cycles with capacity and I/O bandwidth savings. A factor of reduction in memory footprint is achieved converting FP32 values EF11. Hardware conversion possible for FPn to EFn and vice versa with simple Static RAM based lookup tables.
89
70
71
For a given dataset, many different FP to EF conversion tables are possible. Tables may be optimized for maximum significand width (highest exponent compression) at the expense of worse precision for few floats with infrequent exponents (less significand bits for outliers) and vice versa.
73
74
75
Since vector embedding models are used in wide array of NLP transformer architectures, the EFloat format be used for much wider (and more space consuming) class of NL.P models.
77
we first present the analysis of various vector embedding models. The EFloat format 79 is presented in Section 3 Section describes key steps in conversion between EFloat and other FP formats. Section presents eπror analysis of various EFloat widths (EFn) against BF16 and FP16. 81 In Section 6 a review of related work on model compression and foating-poin formats for deep learning is presented. Finally, Section presents conclusions and outlines future directions.
In Section
2 Analyzing vector embedding models
84 Vector embedding models are extensively used in natural language processing (NLP) to capture and exploit semantic relationships of word entities (e.g., words, sentences, phrases, paragraphs, or documents). A trained vector embedding model consists of a set of vectors, each vector encoding distributed representation of inferred semantics of word entity, i... single vector captures different attributes of the inferred semantics (Hinton et al. (1866)) created, in part, by contributions by other word entities. Every vector embedding model implements some variant of the log-bilinear language (LBL) model that predicts the probability of the next word wi given the previous words (context) (Hinton (203)); Almeida and Xexeo (2019); Bender and Koller (2020). The LBL model 92 first predicts valued vector representation of word by linearly combining the veal-valued vector representations of its context words. Then the distributed representation of the word is computed based on the similarity between the predicted representation and the representations of all words in the vocabulary. This step is accomplished using the normalized exponential Softmax function the associated feature vectors. The output the Softmax function is the probability distribution over different possible outcomes, where V is the vocabulary size.
85
86
88
90
94
96
97
40%
gongle-umv-sent--one
pytorch-aggragh
pubmedi wordRvec enwili word2ves
Exporentt Popubtioo (normatonnnnooipeee (nomaalzedoo 10081
90%
fect hest sw wiwwowwwood
fast text craw
pubmedisenttzve
8408 gloe-glave
20%
enwrki-doc
100%
5%
0%6
Expogent
-17
worddee
credit drave-ddvvee
puppppuueeeeeeee senttve:
pupmeudeeddee word2ves crawi
enwili doc2vec
8408
churn
Figure 2: Histogram of the exponent fields of 32-bit floating-point (FP32) values found vector- embedding and related NLP models. Only the db2 Vec, word2vee, doc2vec, and sentence-encoder models were generated. Others were downloaded as publically available pretrained models.
38 Figure presents histograms of exponent values multiple pre-trained vector embedding models, where the X-axis represents exponent values (from the 8-bit exponent portion of 32-bit IEEE 754

---

100
floating point value). For each X-axis value, the Y-axis represents normalized number of occurrences of that exponent value, i.e., histogram. Vector embedding and related NLP models presented in Fig. 2 include word embedding (word2Vec) (Mikolov et al. (2013); /hang et al. (2019)) ) sentence (sent2
101
102
103
and document embedding (doc2Vec) (Le and Mikolov (2014 Chen
Chenn et al. (2019), GloVe (Pennington)
104
et al. (2014a)), subword embedding (FastText) (Bojanowski et al. (201 7a by), database embedding
105
(db2 graph embedding (PyTorch BigGraph Lerer et al. (2019)) ), and Google's ' transformer- based
universal sentence encoder (Cer et al. (2018); Google (202I)) using the Brown corpus (Brown- corpus (22T)). All these models implement different variations of the LBL model. The word2Vec Vec based models, e.g., word2 Sent2Vec, doc2Vec, Vec, and FastText, different versions of Softmax as the activation function. GloVe, on the other hand, is a count- based optimization approach that uses word co-occurrence matrix and weighted least-square as the optimization function. The FastText subword model (Joulin et al. (016);; Bojanowski et al. 7a)) assigns vector for every character n-gram, using an extended skip-gram model (Mikolov al. (2033) and then, words are represented as the sum of these representations. The universal sentence encoder generates embedding vectors for sentences using a standard Transformer architecture that
107
neural network with
109
111
112
113
115
Jakes word embedding vectors as input and uses a Softmax function to compute attention (Vaswani et al. (2017)) Irrespective of the model type, we observe that exponent values cluster around 116
117
certain range of values, and display a distinct peak. The only exception is the doc2 Vec model that exhibits two peaks as the Vec2 first builds fine-grained embeddings for words and then uses them
118
119
to build embeddings for coarser- coarser-grained entities such as paragraphs via concatenating and averaging
120
individual word vectors which results in a smaller second peak as observed in Fig.
106-55
Aandankid takutaa
506-25
706-55
JOir bardton
100-
03
01810)
0.02
2403
Figure 3: The Sigmoid (I)) curve and its gradi- ent. The floating-pont (FP32) exponent of few neural weights are overlaid on (x)).
Figure 4: Exponent population (FP32) of model weights during the training of churn-db2vec model, for iterations 0 to through 2481
121
The Softmax family of activations functions used in vector embedding models responsible for the clustering behavior of exponents (Figure 2). To understand the reasons, let us delve deeper into the training of an embedding model. For illustration purposes, we use database embedding (db2 of the Telcom Churn data (IBM (2020) as an example. db2Vec is an adaptation of the word2Vec approach, and has been designed to build an embedding model from structured database tables that adhere to the relational data model. Like Word2Vec, db2Vec also uses Skipgram with Negative Sampling (SGNS) as the training approach. The SGNS approach uses binary classifier based on the logistic (Sigmoid) function instead of using the coftmax-based predictor. The overall training process involves multiple paok-propagation iterations to update model weights using the gradient of the Sigmoid function. Weights get updated iteratively during the back-propagationn process by the error computed for that iteration. Practically, the error is computed using the gradient of the activation function. During model training, observe that the weights rapidly converge (Fig(4) to their final values. Their exponents are substantially clustered at the slope of the Sigmoid curve, the 2 88 to 20 evidenced by Figures and 4 Training eliminates smaller exponents ex from the model because the activation function output is practically zero for any input value when weights are small, and large exponents are non-existent of normalization of weights.
122
123
124
125
126
127
129
130
131
132
133
134
output range of Sigmoid,
135
136
137
3
The EFloat format: EFn.
138
The key idea behind the EFloat format the varithle-width encoding of exponents using the well- known Huffman algorithm. Frequency of unique exponent values in the dataset determine the coded-exponent widths which may vary between as small as 1-bit and some software configurable maximum, c.g., 8-bit (Figure [c.f.g)). Thanks to the entropy coding of the Huffman algorithm,
140

---

142
frequent exponent values are coded with fewer bits and infrequent exponents are coded with more bits as observed in 5
143
144
Bits saved from the exponent become part of the significand, therefore increasing the floating point precision compared other float formats with the same bit budget. An N-bit coded-exponent in an EFI6 float results in (15-N)-bit significand as shown in Fig c.). Since Elloats with frequent exponents have wide significands, the entire dataset has greater precision on average. Efloats with infrequent exponents have narrow significands. But, their contribution is relatively small in the common calculations used in model training and inferencing, such as dot-products, vector-sums, and dusine-similarity (EFloat precision is quantified and compared to prior formats in Section 5
148
147
148
149
150
EFloat average have greater precision and range than any other fxed-field FP format with the same bit budget. For example, EF16 with a 3-bit coded-exponent has 12-bits of significand compared the 7-bit significand found in BF16 (Figures] [ [h)))). EFloat exponent's logical width is always 8-bit, the same as for FP32 and BF16, irrespective of EFloat width. Even for extremely narrow floats such EF8, the logical exponent width can be 8-bit since encoding compresses the exponent field.
152
154
155
196
The EFloat format compresses special values of IEEE 754, such as signed zeros and infinities losslessly. NaN are semantically compressed losslessly: converting a NaN to and from FP32 to EFn and vice-versa still results a NaN. Denormal floats may round to zero since least significant bits of significands are truncated during encoding.
158
EFloat encoding and decoding
180
161
The Huffiman algorithm: a popular lossless compression algorithm used in many compression tools and compressed data formats (Salomon (2004). Data symbols are encoded with variable-length binary codes whose length are determined by the symbol probabilities algorithm builds binary tree with each leaf assigned symbol. Higher probability symbols closer to the tree root than others. The path from the tree root to the leaf the binary coding of the symbol. To demonstrate with a trivial example, the letters A, B, C occuring with probabilities of 0.5, 0.25, and 0.25 may be encoded with the bit patterns 0, 10, and 11, respectively. The algorithm yields 1.5-bit/symbol compression efficiency, better than 8-bits/symboo using ASCII representation
163
the data stream. The
165
166
167
168
189
or 2-bittssyymbol using a simplistic mapping of the letters to 2-bit integers. Huffman coding is optimal when symbol probabilities are negative powers of 2. However, it is an effective compression method even for non-optimal data distributions. Fig 5 shows the Huffman coded exponent widths as a function of exponent frequencies of a word2vec trained model.
170
172
173
Huffman codes have the prefix property which states that no code is prefix of a longer code (due its tree structure.) As result, the Huffman code not only encodes the original symbol but the code-length as well. We use this property to locate the bit position of the movable boundary between the exponent and the significand fields when decoding EFloats( [e)..
175
176
177
Length-L.ing Limiting: The basic Huffman algorithm, depending on probabilities, may produce extremely wide codes consuming the entire width of EFloats and more. We use the Length-Limiting variant of the Huffman algorithm to set a maximum coded-exponent width (Abali et al. (2020)). In Fig the maximum width is set to 8-bits resulting in infrequent exponents encoded with that maximum.
179
300,002
Crodked Cootent 2ldt Ben
RECUUSACY
Figure 6: EFloat uses Length- Length-Limited Limited Huff- man algorithm to set the maximum width of coded-exponents to 5, 8, or 10-bits
Figure 5: EFloat variable exponent widths are function of the exponent population (word2vec)
181 The software-defined limit presents an opportunity to tune the FFloat precision to particular NI.P application requirements. Figure/ the effect of limiting maximum width of coded-exponents to
5

---

183
5, 8, and 10-bits. As the limit is increased, the least frequent exponents are coded with the maximum width codes, therefore their respective significands lose precision. On the other hand, with increasing limits the most frequent exponents are coded with fewer bits reducing both the minimum and the average coded-exponent widths, therefore their respective significands gain precision. Therefore, EFFlat not only compresses the regular floats but for a given EFn. budget of n-bits the application can optimize the end-to-end precision by adjusting the maximum coded seded-exponent width.
184
185
186
187
188
189
That the minimum and average code widths are inversely proportional to the maximum code-width may appear counterintuitive (Figure O Let the maximum coded-exponent width be K-bits (K 5, K = 8, etc.). K-bits are sufficient to represent N unique exponents in the dataset provided [loge(M) [N)] S K holds. When K is chosen much larger than the minimum [lo92([V]], some codes
190
191
192
193
can have fewer than K-bits. Consider short L-bit code such that L S K (e.g., assume K = 8, = and the short-code = 00.) Due to the prefix property the I--bit code consumes 2K-L bit patterns out of the 2 maximum possible (e.g., all 8-bit patterns whose 2-bit prefix is 00, 64 patterns in total, are consumed by the code 00.) The remaining N exponents can still be encoded if
194
195
196
N - - 2 2 holds. We observed in practice that increasing the maximum code-width by 2 to bits over the minimum (logg(N)) gives a good compression ratio.
198
199
EF Encoding and the Code-Table: During the conversion from FP32 to an EFn (e.g.. EF16,,
200
exponents in the original dataset are histogrammed first, e.g.. 2ig The histogram representing probabilities of the exponents is the input to the
201
Huffman algorithm. The output is a 256-entry (2*) code-table indexed by the original 8-bit exponent. Each table entry contains a pair, the wiith coded-exponent and its width. Note that the code-table is quite small, tens of bytes in practice, since few unique exponents are present in most NIP datasets as Fig shows.
202
203
204
205
When the sign bits have a skewed distribution, e.g., if they are substantially positive, then the sign bit and the 8-bit exponent may be treated as a single 9-bit integer when histogramming. Thanks to Huffmman coding, a skewed sign bit distribution may provide up to one additional bit of precision to the significand.
208
207
208
209
Using the code-table, the entire dataset is converted from FP32 to the chosen EFn width (e.g., EF16) replacing original exponents with coded-exponents. Least significant bits of the FP32 significand are truncated to match the EFn width. For example, in Fig 5 the algorithm encodes the most frequent exponent with 2-bits. Accounting for the sign bit, this yields a 13-bit significand in 6F16 by truncating the bottom 10-bit of the 23-bit significand of FP32. We use the nund-to-nearestt method to provide on average 0.5 bits of additional precision: when the leading bit of the truncated part is I the upper part of the significand is incremented +1 provided it doesn't I overflow in to the exponent field.
211
212
213
215
For large datasets, a statistically representative subset may also be used to reduce histogram collection time. When the histogram is known in advance, a pre-built code-table may be used. Pre-built code-tables eliminate the overhead of executing the Huffman algorithm. During training exponents rapidly converge to their final values as observed in Fig The exponent distribution is practically identical for all iterations 11 to 2481, Therefore, single pre-built code-table optimized for final iterations may serve for all iterations start to finish. The same pre-built table, although suboptimal for early iterations, may be used because significand precision is not as important that point in time; model weight updates are dominated by exponent updates. Once exponents settled to their final values the significand precision becomes important since model weights updates progressively get smaller.
217
219
221
223
224
225
For EFloat FP32 conversion (i.e., decoding) table described carlier. A decoder-table indexed by the coded-exponent may be used to decode the original exponent value and the significand's 's leading bit position in constant time. Each table entry contains the original exponent and width of the coded-exponent. To index the decoder-table with
Elloat Decoding:
inverse mapping of the
227
229
230
variable-width codes many entries are filled with duplicates. For example, the 2-bit coded-exponen 00 is duplicated 64 times in the table at locations 00000000 through 0011 10111111 with each location
231
232
containing the pair (original exponent and wode-width=>> 2). Duplicating entries is equivalent to having
233
logical don't 't care bits in the index which is a useful in hardware based decoder implementations.
The second element of each table entry contains the EFloat significand width. Since the significand was truncated earlier during the FP32 to EFloat conversion, the missing least significant bits must be padded with zeros to match the original FP32 width.
235

---

Evaluating the EFloat representation
237
238
In this section, we evaluate the efficacy of the EFloat format using two sets of experiments. The first set measures the loss of precision in representing FP32 data in BF16, FPI6, and EFloat formats with bit budgets from 16 to 8 bits. The second of experiments compares the quality of ranked results for similarity and dissimilarity queries using the Normalized Discounted Cumulative Gain (NDCG) score for BF16, FP16, and various EFloat formulations. Table presentss the list of models
239
240
242
243
used in these experiments, along with their characteristics: model types, model size (stored using FP32), number of unique exponents, range of exponent bits generated by the Huffmman algorithm, the
244
average count of exponent bits, and minimum and maximum average count of significand bits. For 6F16, the average significand length is 4.3 bits more than BF16 (with 7-bit significand) and 1.2 bits more than FP16 (with 10-bit significand).
246
Table I: EFloat characteristics from EF16 to EF8 for different datasets
Unique exponents
EFn exponent bits
EFn significand bits (Avg.)
Model
Type
Size
Min
Max
Avg.
Max (EF16)
Min (EF8)
churn
db2vec
20 MB
23
3
3.6
11.4
3.4
crawl
fast-text
4.3 GB
30
3
5
3.4
11.6
3.6
enwiki
word2vec
9.6 GB
27
4
4.2
10.8
4.8
MDM
db2vec
14 GB
24
3
6
3.6
11.4
3.4
840B
GloVe
GB
35
3
3.5
11.5
3.5
wiki-sw
fast-text
2.2 GB
22
3
5
3.6
10.5
3,4
virginia
db2Vee
222 MB
24
3
3.7
11.3
3.4
The first of experiments compares the loss of precision due the least significant significand bits being truncated during conversion from FP32 to various low-precision format (c.g.. EF16 or BF16), the values are converted back to FP32, and the arithmetic difference, fo - fe, of the original FP32 value, fo, and the regenerated FP32 value, ft, is computed.
249
formats. Given a
250
251
252
This difference represents the precision loss due to conversion. Root Mean Square Error (RMSE) metric is then used to summarize the loss of precision across a dataset of N floats as:
253
N
RMSE
U22 fE) Σ
254
We then compare the errors of BF16/FP1 and EFn by dividing RM RMSEEBFI/FPP by RMSEEFF
255
in Table 2 Ratios greater than 1.0 indicate that the EFloat error is less than BF16 FP16 errors.
256
For EF16, across all models, we observe an average RMSE error ratio of 24.1 for BF16, and 1.5 for
257
FP16. Note that for these experimental results, the datasets were encoded with a minimum of 3-bit and maximum of 6-bit coded-exponente resulting in an average width in the range of 3.4 to 4.2-bits (Table D. Accordingly, for EF16, the minimum significand width is 10-bit which is 3-bit wider than BF1 and of the same length as FP16. Therefore, EF16 has significantly higher precision against
258
259
280
261
BF16 than FP1 Also, Table shows that EF12 has the same to slightly better RMSE than 6F16
262
since the RMSE ratios are in the 1.0 to 2.2 range. Thus, EF12 uses 25% less bandwidth and memory capacity than BF16 for similar floating sooating-point precision.
263
284
Note that the RMSE method amplifies larger emros due to the squaring of differences. EFloat coded floating point values with short significands (i.e., those with infrequent exponents) are disproportion- ately represented in the RMSE summation. However, the true measure of error for vector embedding models will be the evaluation of ranked results for similarity queries for different floating point formats. Unlike the binning in traditional classification inference tasks, ranked results from similarity queries are far more sensitive numerical precision. We use the Normalized Discounted Cumulative
265
266
287
288
289
Gain (NDCG) metric (Järvelin and Kekäläinen (2002); Wang et al. (2013)) to evaluate the quality of ranked results for different floating point formats. NDCG is widely used in information retrieval and web search to evaluate the relevance of retrieved documents. NDCG is normalization of the
270
271
272
273
Discounted Cumulative Gain (DCG) measure. DCG is calculated as weighted sum of the degree of
relevancy of the ranked items, where the weight is decreasing function of the position of an item.
275
NDCG computed by normalizing DCG by IDCG, which is the DCG measure for a perceived ideal ranking result. Thus, the NDCG measure always lies within [0.0,1.0].
7

---

Table 2: BFloatI6 (BF16), IEEE Half (FP16), and EF16-8 precision comparisons using RMSE-with-
FP32 ratio. Higher better.
EFI6
EFI4
EF12
EFI0
EF8
Model
BF16
FPI6
BFI6
FPI6
BF16
FP16
BF16
FPI6
BFI6
FPI6
churn
22.5
1.4
5.6
0.4
1.4
0.09
0.3
0.02
0.08
0.005
crawl
34.6
2.2
8.6
0.5
2.2
0.1
0.5
0.03
0.1
0.008
enwiki
16.9
1.0
4.2
0.3
1.0
0.07
0.3
0.02
0.06
0.004
MDM
27.9
1.8
6.9
0.4
0.1
0.4
0.03
0.007
840B
25.0
1.6
6.3
0.4
1.6
0.09
0.4
0.02
0.09
0.006
wiki-sw
22.0
1.4
5.5
0.3
0.08
0.3
0.02
0.08
0.005
virginia
19.6
1.2
4.9
0.3
1.2
0.08
0.3
0.02
0.07
0.004
277 For given vector embedding model, we choose q == 20 randomly selected distinct query points. For each query point, compute similar and dissimilar points by computing cosine similarities over the corresponding vectors. For similarity queries, the result contains a list of points sorted in decreasing order of their similarity scores (most similar pair of items will have
279
closer 1.0), and for dissimilarity queries, the result list is sorted in increasing order of their similarity scores (most closer -1.0). For each query point, dissimilarity queries for different floating point formats, and use the top k 10 results for each test compute the NDCG score, (NDCG@ @ 10). our evaluation, we use the ranked results for FP32 as the baseline for calculating the IDCG. For each model, we report the average NDCG@ @ 10 score computed over 20 query points using BF16, FP16, and various EFn from EF16 to EF8.
281
dissimilar pair of items will have
similarity and
283
285
NDCG@10 For Similarity Queries
MOCC Score
craw
MOM
virginia
wBloow
Models
Figure 7: Evaluation of similarity query accuracy using NDCG score across different floating point formats. Higher score (closer to 1.0) is better.
Figure presents NDCG10 results for similarity queries, and Figure presents NDCG10 results for dissimilarity queries. For both similarity and dissimilarity queries. EF16 matches or exceeds the quality of BF16 FP16 (in particular, among the three formats. BF16 provides the worst qaulity results), Furthermore, EF 14 and EF12 provide similar quality results as EF16 in many instances. The two lower-precision EFn, EFIO 1 and EF8, consistently generate the least quality results.
288
290
292
In summary, results from the two sets of experiments (Table and Figures and g conclusively demonstrate that: (1) Given a bit budget, EFloat has higher accuracy than other formats, (2) In many scenarios, EFn with reduced budget (e.g.. EF14 or EF12) provides results of quality comparable to higher precision formats, e.g., BF16, and FP16. These results validate the design of the EFloat format, and demonstrate that EFloats can be used for compressing and computing using vector embedding models.
294
296

---

NDCG@10 for Dissimilarity Queries
1002
Vigieia
Medels
Figure 8: Evaluation of dissimilarity query aceuracy using NDCG score across different floating
point format. Higher score (closer to 1.0) is better.
6
Related Work
298
299
Model quantization is widely used to compress pre-trained models for the inference phase (Gupta and Agrawal (2020)). Quantization covers two broad approaches: the first represents full-precision (c.g., 32-bit) floating point weight value using reduced (e.g., BF16 or FP16) or mixed precision floats, and the second converts full-precision floating point values into integer values with fewer bits
300
302
303
(e.g., INT8, INT4, and INT1 (Migacz) (2017): 7): Wu et al. 2020); : TensorFlow Documentation 2020);
Jacob et al. (217))). In conjunction with the model compression work, there has been significant
304
work in devising reduced-precisee floating point formats tuned for broader machine learning and
306
HPC applications (Sapunov (2020); Abdelfattah et al. (2020)). Unlike the inference-foouue compression work, reduced-precisiine floating points are designed to work for both model training and inference phases. The most common reduced- prreccciipeciiooonnn floating point formats use bits. Current
model
308
16-bit implementations include IEEE 754 half- prefisrcciiion (FP16); Brain Floating Point, BF Wang and Kumar (2019); Kalamkar et al. 2019)); and Deep Learning Float (DLFloat) (Agrawal et al. (2019)), with sign bit, 6 exponent bits, and 7 fraction bits. TensorFloat-33 (TF32) from Nvidia is 19- format that combines exponent bits from BFLOAT16 and 10 exponent bits from
310
312
IEEE FP16 (Kharya (2020)) Hybrid Block Floating Point (HBFP) (Drumond et al. (2018). Intel
Nervana's 's Flexpoint (Ksster et al. (2017), and Microsoft MSFP (Rouhani et al. (2020) formats combine the advantages of fixed point and floating point formats by splitting up the significand and the exponent part which is shared across multiple numeric values. Recent research proposals have described training of key deep learning models using
314
315
316
reduced precision floating point values (8- and 4-bit floats) (Sun et al. (2020); Wang et al.] (2018); Cambier et al. 2020); Mellempudi et al. siferrence-argette floating-pon point format which maximizes its dynamic range at a network layer granularity by dynamically shifting its exponent range via modifications the exponent bias and by optimally clipping (quantizing) its representable datapoints. Our proposed EFloat design practically achieves the same result without altering the exponent range and quantizing full-pre cell--recision values.
318
319
) Recently proposed AdaptiveFloat (Tambe al. (2020)
320
322
323
Conclusion
324
We introduced EFloat, novel entropy-coded variable length floating point format for deep learning applications. This format can be used for compressing a trained deep learning model, as well as for enabling accurate model representations using reduced-precision floating point formats. While our intended use cases were initially for the database embedding (db2Vec) workloads, we demonstrate that the proposed format works effectively for other vector embedding models, and can be used for models. Broadly, EFloat may be used in deep learning applications where tradeoffs need to be made between range, precision, memory capacity and bandwidth savings. As a future work, we plan to explore the Benford distribution
326
328
330
much broader class of NL. models including transformer-
332
pattern (Benford (9938) Newcomb| (188I) exhibited by significands of vector embedding models (Appendix A in the supplementary document) and investigate its application in rounding EFloat values. follow-up study on 8-bit floats and integers is being considered well.
334
335

---

References
336
337
Bulent Abali, Bart Blaner, John Reilly, Matthias Klein, Ashutosh Mishra, Craig B Agricola, Bedri Sendir, Alper Buyuktosunoglu, Christian Jacobi, William J Starke, al. 2020. Data Compression Accelerator on IBM POWER9 and z15 Processors. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE Computer Society, 1-14.
338
339
340
341
Ahmad Abdelfattah, Hartwig Anzt, Erik G. Boman, Erin Carson, Terry Cojean, Jack Dongarra, Mark Gates, Thomas Grüizzacherr, Nicholas J. Higham, Sherry Li, Neil Lindquist, Yang Liu, Jennifer Loe, Piotr Luszezek, Pratik Nayak, Sri Pranesh, Siva Rajamaniekam, Tobias Ribizel, Barry Smith, Kasia Swirydowicz, Stephen Thomas, Stanimire Tomov, Yaohung M. Tsai, Ichitaro Yamazaki, and Urike Meier Yang. 2020. A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic. AXii:2007.06667 [cs.MS]
343
345
A. Agrawal, S. M. Mueller, B. M. Fleischer, X. Sun, N. Wang, J. Choi, and K. Gopalakrishnan. 2019. DLFloat: A 16-b Floating Point Format Designed for Deep Learning Training and Inference. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH). 92-95.
348
Felipe Almeida and Geraldo Xexco. 2019. Word Embeddings: A Survey.
[es.CL]
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computationall Linguistics. Association for Computational Linguistics, Online, 5185 5198. https: org/ 10. 18653/v1/2020 acl-main. 463
352
354
355
Frank Benford. 1938. The law of anomalous numbers. In Proc. American Philosophical Society, Vol. 78. 551-572.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017a. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5 (2017), 135-146.
358
359
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017b. riching Word Vectors with Subword Information. html
En-
361
DXXiv:160.004600
[cs.CL]
382
http://fftttettcccccc//////////s http://giullooooloooeeee////ssst
MIT License.
363
364
Rajesh Bordawekar and Oded Shmueli. 2016. Enabling Cognitive Intelligence Queries in Relational
365
Databases using Low-dimensional Word Embeddings. CoRR abs/1603.07188. (2016). http:
386
//arxiv.org wwtpg//aaa//...../.// 07185
387 Rajesh Bordawekar and Oded Shmueli. 2017. Using Word Embedding to Enable Semantic Queries in Relational Databases. Proceedings of the 1st Workshop on Data Management for End-to-End Machine Learning (Chicago, IL, USA) (DEEM'17). ACM, New York, NY, USA, Article 5, 4 pages. https://dot org/ 10. 1145/3076246. 3076251
388
389
370
371 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. CoRR abs 2205 14165
372
373
374
375
376
377
abs/2005. 14165 (2020). arXiv:2005. 14165
378
Brown-corpus 2021. Brown Corpus. nolines.
sls. www.wawwwi.....................a www.:l/l...aiii..eeeeeeeevraaaaa
Leopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz H Elibol, and Hanlin Tang. 2020. Shifted and Squeezed 8-bit Floating Point format for Pre-iieci Training of Deep Neural Networks. 4rXiv:2001.05674 [cs.LG]
381
10

---

383
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario
384
Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Brussels, Belgium, 169-174. https://doo. /tttp:://oi.orr/100 1....8866vv11
386
387
388
D18- 2029 Apache-2.0 License.
389
Qingyu Chen, Yifan Peng. and Zhiyong Lu. 2019. BioSentVec: creating sentence embed- dings for biomedical texts. In The 7th IEEE International Conference on Healthcare Infor- matics. https:///githu: com cww/ncbi wwwcciiilpppiee hialp//BooeetVeee License: https//giihub.com//cbr
390
381
392
393 C. J. Date. 1982. A Formal Definition of the relational Model. SIGMOD Rec. 13, 1 (1982), 18-29.
https://doi org/10 10. 1145/ 9955544444444411
395
Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. 2018. Training DNNs with Hybrid Block Floating Point. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPSí8). Curran Associates Inc., Red Hook, NY, USA, 451-461.
396
387
398
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv:2101.03961 [cs.LG]
399
400
Google. 2021.
TensorFlow Universal Sentence Encoder. universal- sentence- www.necceeecodrr//// Apache-2.0 License.
https://tfhub.de//gge
401
402
Manish Gupta and Puneet Agrawal. 2020. Compression of Deep Learning Models for Text: A Survey. arXiv:2008.05221 [cs.CL]
404
Geoffrey Hinton. 2013. Learning Distributed Representations For Statistical Language Modelling.
405
https://www Cs toronto...........
www.wotottooooooooooommmomooooo. edu/ eeeeeeduuuiieeeeee "hantonncccc2235//o/////ll pdf
G. E. Hinton, J. L. McClelland, and D. E. Rumelhart. 1986. Distributed representations. Parallel
407
distributed processing: explorations in the microstructure of cognition (1986), 77-109.
408
Torsten Hoefler. 2020. OpenAl's GPT-3 took more than 314 Zettaflop (mixed fp16/fp32) to train! httts://twiteee http:///ww....ooooo/o/////////ar
409
410
IBM. 2020. Predict Customer Churn using Watson Machine Learning and Jupyter Notebooks
Cloud Pak for Data. htt.://tt////ttoooo.oooooooooooa www.:ooeeee--uuuo-o---oo-e- icp4d Apache-2.0 License.
412
413
JEEE 2019. "IEEE Standard for Floating-Point Arithmetic".. 84 pages. https://doi.cnn org/10
414
(109/IEEESTD
415
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. Quantization and Training of Neural Networks for Efficient Miteger-Arttmmetic-O0 Inference. arXiv:1712.05877 [cs.LG]
416
417
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems 20, 4 (2002). https://doi. 1145/ 582415 1145/582415
419
582418
421
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of Tricks for Efficient Text Classification. arXiv:1607.01755 [cs.CL]
422
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das. Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Ab- hisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. 2019. A Study of BFLOAT16 for Deep Learning Training. arXiv:1905.1 12322 [cs.LG]
424
425
426
428 Paresh Kharya. 2020. TensorFloat-32 in the 1100 GPU Accelerates AI Training, HPC up to 20x. Nvidia Blog.

---

430
UIs Koster, Tristan J. Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William H. Constable, Oguz H. Elibol, Scott Gray, Stewart Hall, Luke Hornff, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, and Naveen Rao. 2017. Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. arXiv:1711.0221: [cs.LGG]
431
435
434
Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. 34055444403
CoRR abs/1405 (2014).
436
Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019. PyTorch-BigGraph: A Large-scale Graph Embedding System. arXivv1903. 12287 http:///iiiiiiiiooooaaoc/oo/o/aa
437
439
BSD License. https://gituu..cmm/aaaai
440
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. 2019. Mixed Precision Training With 8-bit Floating Point. arXiv:1905.1220 [cs.LG]
Szymon Migacz. 2017. 8-bit Inference with TensorRT. https://on-oeei -tthccggtttttttcoono com/ gtc///01//preeenttaciooo///ee sttpe/aatiiio//11--811/ htt.s/o----bb---iffeeeeeneeeee with tensorrt. pdf
gpwtechcont
443
444
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word
Representations in Vector Space. CoRR abs/1301.3788 (2013). http::/arxiv.org//20
446
3781
447
Simon Newcomb. 1881. Note on the frequency of use of the different digits in natural numbers. American Journal of Mathematics 4, 1/4 (1881), 39-40.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014a. GloVe: Common Crawl 840B Dataset. stttps://nlp stw.fsaafoorroo wdw/projects License: Open Data Commons Public Domain Dedication and License.
450
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014b. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 1532-1543. http:/ http///awlleeeebee//eee/eee/eee/ wwwpwb.orgg//ooo.ooeooooooooolll anthologyytaaaaaaaaaaaaaow //gggy/D/// /ttp///D/1//1111111 11622. pdf
453
Bita Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vino- gradsky. Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, and Doug Burger. 2020. Pushing the Limits of Narrow Precision Inferencing Cloud Scale with Microsoft Floating Point. In NeurIPS 2020. ACM.
456
458
459
480
461
David Salomon. 2004. Data compression: the complete reference. Springer Science & Business Media.
462
463
Grigory Sapunov. 2020. FP64, FP32, FP16, BFLOATI6, TF32, and other members of the ZOO. https://moocaholic.mediii
464
fp64-fp32
fp32-fpp212pppp- fp122-ff66--11 16-t2t16-tf32 -ad--antrr---embeere of-the-zoo-
485
alca78978407.
466
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Meeattonn-MM: Training [cs. CL]
467
Parameter Language Models Using Model
468
Parallelism. arXiv:1909.0800
469
Xiao Sun, Naigang Wang, Chia-yu Chen, Jia-min Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkararamini, Kaoutar El Maghraoui, and Vijayalakshmi Srinivasan. 2020. Ultra-Low Precision 4-tit Training of Deep Neural Networks. In Advances in Neural Information Processing Systems,
470
471
472
Vol. 33.
473
T. Tambe, F. Y. Yang, Z. Wan, Y. Deng, V. Janapa Reddi, A. Rush, D. Brooks, and G. Y. Wei. 2020. Algorithm-Hardee Hardware Co-Design of Adaptive Floating-Poin Encodings for Resilient Deep Learning Inference. In 2020 57th ACM/IEEE Design Automation Conference (DAC). 1-6. https /doii.org/00.110/DAA1113 2020.9218516
474
475
476
477
TensorFlow Documentation. 2020. TensorFlow Lite 8-bit quantization specification.
https
12

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention All You Need. arXiv:1706.03762 [cs.CL]
480
481
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. 2018. Training Deep Neural Networks with 8-bit Floating Point Numbers. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc., 7675-7684.
483
Shibo Wang and Pankaj Kumar. 2019. BFloat16: The secret to high performance Cloud TPUs.
486
Yining Wang, Liwei Wang, Yuanzhi Li. Di He, Kan Liu, and Wei Chen. 2013. Theoretical Analysis of NDCG Type Ranking Measures. CoRR abs/ 1304680046480 (2013), http://arxiv org/
487
488
abs/1304.6493
489
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020. In- teger Quantization for Deep Learning Inference: Principles and Empirical Evaluation.
491
arXiv:2004.09602 [cs.LG]
492
Yijia Zhang, Qingyu Chen, Zhihao Yang. Hongfei Lin, and Zhiyong Lu. 2019. BioWord improving biomedical word embeddings with subword information and McSH. Scientific Data 6, 52 (2019). htttps://github com wwciinccinnnnn wwwciiinl/Biioen/Veee License: https://gihuu..co//ng
494
13

---

Checklist
496
487
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
499
contributions and scope? [Yes]
500
(b) Did you descrihe the limitations of your work? [Yes]
501
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work does not deal with societal impact of AI.
502
503
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
505
2. If you are including theoretical results...
506
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Experimental work.
507
508
(b) Did you include complete proofs of all theoretical results? [N/A] Experimental work.
509
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [No] The codes used in experiments
511
proprietary.
513
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [No] We used pre-trained models
514
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Experiments used pre-trained models.
516
517
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [No] Experiments used pre-trained models.
519
520
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
521
(a) If your work uses existing assets, did you eite the creators? [Yes]
522
(b) Did you mention the license of the assets? [Yes] License information provided the
citation.
(c) Did you include any assets either in the supplemental material as URL? [Yes] Information provided in the citation.
525
528
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
527
528
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
530
5. If you used crowdsourcing or conducted research with human subjects...
531
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
532
533
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
534
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
536
14


Reviews:
[{"review_id": "yXEV64T2Q1T", "reviewer": "Reviewer_AEbL", "summary": "This paper introduces EFloat, a domain-specific entropy-encoding numerical format targeted for NLP applications. EFloat takes advantage of clustered values in parameter distributions to allow Huffman coding to compress the average bitwidth. The authors propose this method as a means of compressing parameters to save memory and IO bandwidth in deep learning models. They further introduce the vector embedding model, describe the algorithms for encoding and decoding EFloat, and evaluate EFloat against IEEE FP16 and BFloat. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, any method that decodes parameters before inference should be accompanied with a strong explanation of its hardware efficiency or a careful discussion of its limitations and intended domain. The current paper does not sufficiently motivate EFloat over standard table look-up quantization methods. Furthermore, the experimental section lacks standard deep learning evaluation metrics, and the RMSE values and NDCG scores given are not impressive enough in my opinion to deal with the extra complexity of EFloat.\n\nStrengths:\n1. The design of the encoding and decoding methods using code tables seems well-designed with efficiency in mind.\n2. Figure 2 is a strong motivation for compressing the exponent bits.\n3. This seems like a perfect application for length-limited Huffman coding.\n\nWeaknesses:\n1. \"Deep Learning\" title seems to overstate the domain discussed in the paper, which largely focused on vector embedding models. \n2. Section 4. clearly states the encoding and decoding procedures and the small size of the code tables, but it would be helpful to have a better understanding of how this additional conversion affects the target hardware or applications. The paper focused primarily on the IO and memory bandwidth savings, yet the decoding and encoding task must have some significant performance tradeoffs even on specialized hardware backends.\n3. The concept of entropy-encoded bitwidth seems to be general enough to map to any application once the data distribution is known. Since the relevant topics have been around for a half a century, this implies to me that there are some significant considerations that must be tackled to make such a method practical in hardware.\n4. There should be some evaluation on standard accuracy metrics. Since this paper focuses its discussion on NLP models, perhaps there could be some comparisons using modern NLP models and datasets. The NDCC goes beyond the average bitwidth and RMSE scores but still falls short for me in evaluating the whole system.\n5. The results in Figures 7 and 8 seem not strong enough to warrant the extra encode / decode complexity and uncertainty in the lossy compression.\n6. There have been previous attempts to use Huffman coding on model parameters, e.g. Deep Compression, that should probably be included in the related works.\n7. This is almost a hybrid method between lookup-based quantization for the exponent and floating point for the significant. There should be some at least high level comparisons to lookup-based quantization methods. \n\nQuestions\n1. Were other compression schemes with the prefix property considered? \n2. Are other more accurate compression methods for the exponent possible given a representative dataset? This method seems to be designed for pre-trained models. \n3. Why the particular focus on the vector embedding models?\n4. In my experience, most deep learning models can be quantized to 8 bits today with degradation in accuracy. Why the particular focus around 16 bits? \n\n\nMinor Issues\n1. L31 \"it's\" -> \"its\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No extra negative societal impacts compared to those found typically with machine learning.", "review_text": "", "extra_scores": {"ethical_concerns": "No extra ethical concerns except those found typically with machine learning.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}, {"review_id": "VKOVnH8htz1", "reviewer": "Reviewer_f6Ju", "summary": "The work proposes a new floating-point data type with entropy encoded exponent to store large embedding models. These data types are studied under several bit widths and compared with 16-bit brain float and IEEE 16-bit float data types and show favorable results.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Entropy encoding of the exponent is a well-studied method to compress floating-point data, but, to my knowledge, has not seen application in deep neural networks which usually have different data distributions than other domains of computer science and also have systematically shifting distributions over the course of training. This work provides some much-needed data on the effectiveness of exponent compression and how the underlying exponents shift during training. I see this work as an important contribution that will inform many other eep-learning-focused floating-point data types that are yet to come.\n\nQuality: The manuscript has a high quality. It is easy to read and contains extensive explanations of core concepts which will make it easy to read for anyone outside of this sub-field. The quality of experiments is satisfactory, but the quality could be improved substantially by including other data types beyond the standard brain-float and IEEE data types. While other specific data types are a pain to implement, I would have wished to see 14 and 12-bit versions of IEEE and brain float data types in the comparison. This would have allowed one to make out trends in how these data types diverge in usefulness as the available bits are reduced.\n\nWhat I am missing is a more careful analysis of approximation errors. Instead of RMSE ratio, I would have liked to see more standard absolute errors and relative/approximation errors. Just reporting one form of relative errors might be deceiving since values with very small magnitude usually have high relative errors but small absolute errors. Additionally, a break down of absolute errors per exponent would yield rich information of behavior across EFloats and other data types. Insights from such information would greatly benefit future work in this area.\n\nAdditionally, the generality of EFloats would be much clearer if it would be applied to different networks and tasks. Although the comparison with other data types that work seamlessly for training are favorable, from my experience new data types can include certain kinks which can introduce new problems during training. Such problems might not be apparent for the task explored in this paper since these networks are shallow compared to other mainstream neural networks.\n\nClarity: As mentioned before, the manuscript is easy to read and the details are perfectly clear. The work should be easily readable for researchers in different sub-areas.\n\nSignificance: I believe this work will inform any future work on floating-point data types for machine learning. As such, it is a significant contribution. However, compared to other work at NeurIPS, I can see that this work currently has a rather narrow focus and it is unclear if it would generalize to effective training of other networks, such as transformers or convolutional networks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are not discussed by the authors. I think the work has no negative societal impact, but a discussion of limitations would have been helpful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "T89Vxvw3mu", "reviewer": "Reviewer_7rZw", "summary": "The paper proposes a new tapered precision numerical format (EFloat) where the exponent of floating value is represented by Variable Length Huffman coding (The maximum of bit-width is limited). Since the Huffman coding was designed based on the probability of exponent occurrence, the most frequent exponent values are presented by fewer bits, which free some bits to be allocated for precision. The vector embedding benchmarks are used to evaluate the efficacy of the new numerical format compared to other numerical formats such as BFLOAT16 and FP16. Based on the metric represented in the paper, such as RMSE and NDCG, the EFloat is better than BFLOAT16 and comparable with FP16. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1- Using Huffman coding for the exponent is an interesting and novel idea. \n\n2- Benford’s low is interesting and is used in approximate computing, and I would suggest the author add this analysis instead of considering it as future work. \n\nWeakness and suggestion:\n\n1- Comparision with other numerical formats such as DFLOAT16, Posit [1], and Adaptive Posit [2] (Regime size is limited) is not discussed in the paper. The author needs to compare the new approach with these numerical formats experimentally. \n\n2- NDGC and RMSE are not comprehensive metrics for comparing the new numerical format with others. One way of comparing the numerical format is decimal accuracy [1]. Furthermore, the author can also report log loss [3]. \n\n3- The trade-off between accuracy and hardware complexity is not discussed in this paper compared to other numerical formats.  The LUT-based conversion from EFLOAT to FP32 is explained in the paper; however, the complexity of this conversion does not quantify ( Experimentally or Theoretically )   \n\n4- It is not obvious why the FP16 showed better performance in terms of RMSE and NDGC metrics than EFloat (in some cases), and this limitation of EFloat is not discussed in the paper. \n\n5- How the author ignoring the overflow in significant if all the bit is 1. Based on the explanation in the paper, if we increase the least significant bit, then we have an overflow. This part is unclear on the paper, and it would be suggested to be explained better. Moreover, why did the author not use round-to-nearest-even?\n\n6- The collection of histograms is a complicated task, and as mentioned in the paper, the statistic of histograms needs to be considered. However, the author did not discuss this limitation in detail and the solution to overcome it. \n\n7- The author suggests the log n +2 for the length of the exponent. However, this might be true for this application, but in general, the length depends on the application. What is the solution of the author for this problem? Is it needed to store the size of the exponent in a register? \n\n8- Suggestion: Length-Limited Huffman algorithm is the suboptimal and theoretical analysis needed to be performed to understand the effect of this sub-optimal design on vector embedding performance. This theoretical analysis, such as theoretical upper-bound on convergence rate.\n\nReferences:\n\n1- Gustafson, John L., and Isaac T. Yonemoto. \"Beating floating point at its own game: Posit arithmetic.\" Supercomputing Frontiers and Innovations 4.2 (2017): 71-86.\n\n2- Langroudi, Hamed F., et al. \"Adaptive Posit: Parameter aware numerical format for deep learning inference on the edge.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\n3- Zhang, Jian, Jiyan Yang, and Hector Yuen. \"Training with low-precision embedding tables.\" Systems for Machine Learning Workshop at NeurIPS. Vol. 2018. 2018.\n\n\nPost Rebuttal=====\nI appreciate the authors' responses to my comments. After carefully reading your responses and other reviewers’ comments, I have decided to increase my score to 5. However, still, I believe the experiment section lacks standard deep learning evaluation metrics as also mentioned by other reviewers. One suggestion is to use a hybrid EFloat and float16 for transformer or other modern NLP applications and report the standard NLP metric. Besides this, I still recommended comparing with DFloat16 and Posit (Adaptive Posit). I believe in a research study we try to compare the ideas that conceptually are the same but the approaches are different (It does not matter the approach is commercialized or which approach shows better performance). The Efloat and Posit (Adaptive Posit) conceptually are the same and both try to represent bell-shaped curve efficiently with the variable encoding of the exponent.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address the limitations and potential negative societal impact of their work. The suggestion is presented in the Weakness section. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a promising paper that studies an interesting idea. The reviewers' scores were mixed, but on balance the concerns about the experimental evaluation caused most reviewers to lean towards rejection. The paper would be substantially improved by more standard end-to-end performance comparison. For example, a strong empirical result would show that, for some hardware and some standard network, EFloat can run inference for a pretrained model faster than standard compression approaches (due to the reduced memory load). As written, we don't know when the trade-off made by EFloat will be useful, both because of the lack of standard end-to-end accuracy benchmarks and the incomplete evaluation of systems/hardware metrics.
Decision: Reject