SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Atli Kosson Martin Jaggi
firstname_lastname@epfl.ch
Multiplications are responsible for most of the computational cost involved in neural network training and inference. Recent research has thus looked for ways to reduce the cost associated with them. Inspired by Mogami (2020), we replace multiplication with a cheap piecewise affine approximation that is achieved by adding the bit representation of the floating point numbers together as integers. We show that transformers can be trained with the resulting modified matrix multiplications on both vision and language tasks with little to no performance impact, and without changes to the training hyperparameters. We further replace all non-linearities in the networks making them fully and jointly piecewise affine in both inputs and weights. Finally, we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion.
The computational cost of training state-of-the-art neural networks has risen rapidly in recent years [1]. Aside from the increased price of training, this also requires more energy which often comes with an environmental impact. Large language models, such as foundation models [2], are particularly expensive to train. These models are typically based on the transformer architecture [34].
Neural network training consists largely of matrix multiplications that generally account for the vast majority of the computational cost for standard architectures such as transformers. These matrix multiplications are performed by multiplying and accumulating the scalar elements of the matrices which are generally encoded as floating point values during training. As floating point multiplication is significantly more complicated than addition, it requires more
Figure 1: Elementary neural network operations (solid lines) along with cheaper piecewise affine alternatives (dashed lines). The piecewise affine functions periodically match the originals and the derivative of each segment is an exact power of 2.
logic gates and energy to perform in hardware. Horowitz (2014) estimates that for float32 operands, multiplication requires  $4 \times$  more energy than addition. Approaches for computationally efficient training thus often focus on reducing the cost of the multiplications. One such method is mixed precision training [23], where the multiplications are performed in a lower precision format such as float16 or bfloat16 [15] while the accumulation is still performed in the standard float32. Mod-
ern hardware accelerators for deep learning, such as TPUs [14] and tensor core GPUs [26], have specialized hardware support for matrix multiplication in mixed precision.
A very different approach is to modify the neural network architecture itself, such as replacing multiplications with computationally cheaper but fundamentally different operations. This is the idea behind AdderNet [4] that uses the absolute difference of two numbers instead of their product in matrix multiplications. AdderNets involve few multiplications during inference but require special tricks during backpropagation that involve multiplications and/or exponentiation to achieve optimal performance. Another alternative is tropical algebra that replaces the products with additions and the accumulation with a maximum, but has not shown to give performance competitive with standard neural networks [22, 9]. For both of these approaches, the main hardware advantage is replacing multiplications with additions that tend to be much cheaper in hardware.
The core idea we pursue in this paper falls in-between the two aforementioned approaches. We replace multiplications with operations that roughly approximate them but have a cost as cheap as addition. Since the operations are not fundamentally different, they should serve as a drop-in replacement for multiplications—similar to mixed precision—without requiring architectural changes, while potentially being significantly cheaper. One such approximation is Mitchell's logarithmic multiplication [24] which approximates the product of  $A > 0$  and  $B > 0$  as:
$$
A B \approx \operatorname {p a e x p} _ {2} (\operatorname {p a l o g} _ {2} (A) + \operatorname {p a l o g} _ {2} (B)) \tag {1}
$$
where  $\mathrm{paexp}_2$  and  $\mathrm{palog}_2$  are cheap approximations of  $\exp_2$  and  $\log_2$  shown in Figure 1 along with an example output. All the approximations are piecewise affine (sometimes called piecewise linear) in their input arguments. The core of this operation can be performed efficiently in hardware using int additions which are around  $9 \times$  cheaper than float addition for 32-bit operands [12]. We roughly estimate the cost of piecewise affine multiplication to be around  $30 \times$  less than that of standard float32 multiplication in terms of the area and energy used by the operation itself (Appendix B)<sup>1</sup>. Approaches based on logarithmic multiplication have also been explored for neural network inference and found to enable around  $5 \times$  energy saving compared to bffloat16 multiply-and-accumulate operations [16].
Mogami [25] showed that Equation (1) can be performed by directly adding the floating point number representations together as integers, with some extra handling of the exponent accounting for the exponent bias, underflows and overflows. They further show that this trick can be used to train convolutional neural networks (in particular ResNet50 [11]) with a minor or no degradation compared to standard multiplications. In this work we further explore the application of piecewise affine approximations to neural network training, summarized below:
- We show that transformers can be trained with piecewise affine matrix multiplications on both vision and language data with little to no performance impact. We compare this to AdderNet based transformers [30] demonstrating better accuracy while replacing more multiplications.
- We define additional piecewise affine functions for all other operations in a neural network and demonstrate fully multiplication-free training, with only a minor performance impact. No multiplications are used in any part of training, including the forward pass, backpropagation and optimizer computation. To the best of our knowledge, this is the first time a neural network has been trained entirely without standard multiplications.
- We show that the resulting neural networks are fully piecewise affine (sometimes called piecewise linear) both in their input and in their weights, a property that traditional transformers are very far from satisfying.
- We publicly release our code $^2$ , including custom kernels, in the hopes of aiding further research into multiplication-free neural networks.
<sup>1</sup> Note that special hardware support is needed to realize these gains, see discussion in Section 5.
$^{2}$ Code available at https://github.com/epfml/piecewise-affine-multiplication
2.1 Floating Point Numbers
Since the exact representation of floating point numbers is important to our method we will give a brief overview here. A floating point number is represented as:
$$
(- 1) ^ {S} \cdot 2 ^ {E} \cdot (1 + M) \tag {2}
$$
where  $S \in \{0,1\}$  is a sign bit,  $E$  is an integer encoding the exponent, and  $0 \leq M < 1$  is the mantissa fraction. Note that this is analogous to the normalized scientific notation for decimal numbers (for example  $+1.23 \cdot 10^{-2}$ ).
The encoding used for  $E$  and  $M$  varies between floating point formats. The commonly used IEEE 754 float32 uses the following bit representation:
$$
[ S, \underbrace {e _ {0} , \dots , e _ {7}} _ {=: \bar {E}}, \underbrace {m _ {0} , \dots , m _ {2 2}} _ {=: \bar {M}} ] \tag {3}
$$
where the exponent is encoded as an unsigned integer  $\bar{E}$  with a bias of 127 i.e.  $E = \bar{E} - 127 = e_0e_1e_2e_3e_4e_5e_6e_7 - 127$  and the mantissa as an unsigned integer  $\bar{M}$  that is divided by  $2^{23}$  to get  $M = \frac{\bar{M}}{2^{23}} = \frac{m_0\ldots m_{22}}{2^{23}}$ .
Floating point numbers can also encode special values that do not fit within the representation given in Equation (2). In float32  $\bar{E} = 255$  is used to represent NaN (not a number) or infinity depending on the value of  $\bar{M}$ . For  $\bar{E} = 0$  the float is interpreted as a denormal number  $(-1)^{S} \cdot 2^{E} \cdot M$  which also gives the representation of 0, which is  $\bar{E} = 0$  and  $\bar{M} = 0$ . Some floating point formats, e.g. bfloat16 do not allow denormal numbers other than 0, since those can be more expensive to handle in hardware.
2.2 Piecewise Affine Multiplication
We define the piecewise affine multiplication (or PAM for brevity) of two floating point numbers:
$$
A = (- 1) ^ {S _ {A}} \cdot 2 ^ {E _ {A}} \cdot (1 + M _ {A}), \quad B = (- 1) ^ {S _ {B}} \cdot 2 ^ {E _ {B}} \cdot (1 + M _ {B}) \tag {4}
$$
as  $A\hat{\cdot} B$
$$
A ^ {\hat {}} B := (- 1) ^ {S _ {A ^ {\hat {}}} \cdot B} \cdot 2 ^ {E _ {A ^ {\hat {}}} \cdot B} \cdot (1 + M _ {A ^ {\hat {}}} B) \tag {5}
$$
$$
S _ {A \hat {\cdot} B} := S _ {A} + S _ {B} \tag {6}
$$
$$
E _ {A} \circ_ {B} := E _ {A} + E _ {B} + \mathbf {1} \{M _ {A} + M _ {B} \geq 1 \} \tag {7}
$$
$$
M _ {A: B} := M _ {A} + M _ {B} - \mathbf {1} \left\{M _ {A} + M _ {B} \geq 1 \right\} \tag {8}
$$
where  $\mathbf{1}\{a\}$  is one when  $a$  holds and zero otherwise.
We observe that  $A \hat{\cdot} B$  is roughly achieved by adding the signs, exponents and mantissas of  $A$  and  $B$ . In fact, this becomes precise if we allow  $M_A \hat{\cdot} B$  to overflow into  $E_{A \hat{\cdot} B}$ . This happens if we add the float32 bit representations of  $A$  and  $B$  together as int32.
A technicality is that the exponent requires special handling, as the resulting  $\bar{E}$  is only offset by  $-127$  but should be offset by twice that amount requiring us to subtract 127 from the exponent of the sum. We also need to make sure the resulting  $\bar{E}$  has not overflowed or underflowed, instead clamping it to the min and max values. Possible denormal values (mantissa starting with zeros) can be flushed to zero for simplicity, as done in e.g. bfloat16. Finally we check that the incoming  $A$  and  $B$  do not represent special values such as NaN or infinity and handle those cases explicitly.
For float32, PAM can thus be simply implemented with two int additions and a couple of checks for the exponent. For current hardware accelerators such as GPUs, this is expensive and requires multiple instructions. However the logic is much simpler than for standard multiplication, so new hardware with native support could perform this operation cheaply. With a custom floating point format without an exponent bias, we could further get rid of one of the int additions as well. This could also be done for configurable hardware such as FPGAs. In Appendix B we roughly estimate the hardware cost of PAM, estimating the savings to be around 5-30x over standard multiplication in float16 and float32.
2.3 Other Piecewise Affine Operations
As mentioned in the introduction, piecewise affine multiplication can be expressed in terms of  $\mathrm{paexp}_2$  and  $\mathrm{palog}_2$  which we can write as follows for  $A, B$  from Equation (4):
$$
\operatorname {p a e x p} _ {2} (A) = 2 ^ {\lfloor A \rfloor} \cdot (1 + A - \lfloor A \rfloor) \tag {9}
$$
$$
\operatorname {p a l o g} _ {2} (A) = E _ {A} + M _ {A}, \text {f o r} A > 0 \tag {10}
$$
where  $\lfloor A\rfloor$  denotes rounding down to the nearest integer. For positive inputs Equation 1 then becomes:
$$
\begin{array}{l} A B \approx \operatorname {p a e x p} _ {2} (\operatorname {p a l o g} _ {2} (A) + \operatorname {p a l o g} _ {2} (B)) (11) \\ = 2 ^ {\left\lfloor E _ {A} + E _ {B} + M _ {A} + M _ {B} \right\rfloor} \cdot \left(1 + E _ {A} + E _ {B} + M _ {A} + M _ {B} - \left\lfloor E _ {A} + E _ {B} + M _ {A} + M _ {B} \right\rfloor\right) (12) \\ = 2 ^ {E _ {A} + E _ {B} + \mathbf {1} \left\{M _ {A} + M _ {B} \geq 1 \right\}} \cdot \left(1 + M _ {A} + M _ {B} - \mathbf {1} \left\{M _ {A} + M _ {B} \geq 1 \right\}\right) (13) \\ \end{array}
$$
Which is equivalent to  $A \hat{\cdot} B$ , aside from the computation of the sign.
We define piecewise affine division as the inverse of  $A\hat{\cdot} B$
$$
A \dot {\hat {z}} B := (- 1) ^ {S _ {A} \hat {z} _ {B}} \cdot 2 ^ {E _ {A} \hat {z} _ {B}} \cdot \left(1 + M _ {A} \hat {z} _ {B}\right) \tag {14}
$$
$$
S _ {A \hat {\div} B} := S _ {A} - S _ {B} \tag {15}
$$
$$
E _ {A \hat {\div} B} := E _ {A} - E _ {B} - \mathbf {1} \{M _ {A} - M _ {B} \leq 0 \} \tag {16}
$$
$$
M _ {A} \div_ {B} := M _ {A} - M _ {B} + \mathbf {1} \left\{M _ {A} - M _ {B} \leq 1 \right\} \tag {17}
$$
which corresponds to  $\mathrm{paexp}_2(\mathrm{palog}_2(A) - \mathrm{palog}_2(B))$  for positive numbers.  $A\hat{\div} B$  can also be efficiently performed in hardware float32 using int32 subtraction where a bias offset of 127 needs to be added to the result.
With  $\text{paexp}_2$  and  $\text{palog}_2$  we can also define general piecewise affine functions for powers, including the natural logarithm and exponential as well as square roots.
$$
\operatorname {p a e x p} (A) := \operatorname {p a e x p} _ {2} \left(\log_ {2} (e) ^ {\hat {\cdot}} A\right) \tag {18}
$$
$$
\operatorname {p a l o g} (A) := \operatorname {p a l o g} _ {2} (A) \hat {\div} \log_ {2} (e) \tag {19}
$$
$$
\operatorname {p a s q r t} (A) := \operatorname {p a e x p} _ {2} \left(\operatorname {p a l o g} _ {2} (A) \hat {\div} 2\right) \tag {20}
$$
With these functions we can implement various neural networks operations such as normalization layers, softmax, cross-entropy and other losses as well as optimizer updates.
2.4 Piecewise Affine Neural Networks
Modern artificial neural networks are compositions of the operations discussed in the previous subsections. We thus replace each traditional operation in a network with its piecewise affine approximation. Unlike traditional networks, the resulting network will now be a piecewise affine function, both in its weights and also in its input values. This follows directly since the composition (and sum) of two piecewise affine functions (with one or many arguments) remains piecewise affine. We iterate the composition through all layers and the loss computation.
This global property of being piecewise affine (and thus having piecewise constant gradients) is remarkable as it is does not hold for traditional e.g. ReLU networks, due to the composition with the most common losses, and also is far from true for traditional transformers. Despite this drastic change in the gradient structure and loss landscape, we will show that commonly used optimizers (and their multiplication-free counterparts) still enable successful training out of the box without additional hyperparameter tuning.
2.5 Derivatives
Following Mogami [25], we consider two ways to compute the derivatives of the piecewise affine functions. The first option is to compute the true derivatives of those functions, which are piecewise constant and discontinuous. The second option is to implement the analytical derivative of the original function we are approximating, for example standard multiplication, with the piecewise affine functions. We will refer to the first option as the exact derivative and the second as the
Table 1: Derivatives for Piecewise Affine Operations for A, B defined in Equation 4.
OPERATION	EXACT DERIVATIVE	APPROXIMATE DERIVATIVE
Y = AˆB	δA=2EB+1{MA+MB≥1}δY	δA=BˆδY
Y = Aˆ÷B	δA=2-EB-1{MA-MB≤0}δY	δA=δYˆ÷B
Y = Aˆ÷B	δB=-(AˆδY)ˆ÷(BˆB)	δB=-(AˆδY)ˆ÷(BˆB)
Y = paexp2(A)	δA=2[⊥A]δY	δA=2Aˆ: ln(2)ˆδY
Y = palog2(A)	δA=2-EAδY	δA=δYˆ÷(Aˆln(2))
approximate derivative. Table 1 lists both kinds of derivatives for each operation described previously. We use delta notation for backpropagation, i.e.  $\delta_A = \frac{\partial L}{\partial A}$  and  $\delta_Y = \frac{\partial L}{\partial Y}$  for some unspecified scalar function  $L$  that only depends on  $A$  through the output  $Y$ .
We note that even though the exact derivative sometimes contains a multiplication, one of the numbers is an exact power of two which means that the multiplication can be performed exactly via PAM, see Section 2.7. Both types of derivatives can therefore be computed in a multiplication-free manner, only using PAM and related operations. This also extends to derived operations such as paexp, palog, pasqrt, softmax and normalization. We can backpropagate through the derived functions via the computational graph that defines them, using either the exact or approximate derivatives given above. By extension, we can perform the entire forward and backward pass through a neural network consisting of these primitives without any traditional multiplication operations.
2.6 Optimizers
Networks with piecewise-affine components can be trained using standard gradient based optimization using the derivatives given in the previous section. However, the implementation of the optimization algorithm itself might involve additional multiplications (such as with varying learning rates, adaptive learning rates or for computing other optimizer statistics). We thus also experiment with replacing all optimizer multiplications and divisions with their piecewise affine analogs, and quantify the impact of this change in our experimental results.
2.7 Approximation Error
Figure 2 compares PAM and standard multiplication. It also shows the relative error of  $x_{1} \hat{\cdot} x_{2}$  compared to  $x_{1}x_{2}$ , which depends on the exact mantissas of the inputs. PAM gives the exact result when the mantissa of either input is zero, i.e. when it is a power of 2. The maximum relative error of  $\frac{(1 + 0.5 + 0.5) - 1.5^2}{1.5^2} = -1 / 9 = -11.1\%$  is obtained when  $M = 0.5$  for both numbers. When either input is a fixed, e.g. when multiplying a variable by a constant, the maximum error can be lower.
Figure 2: Comparison of the piecewise affine multiplication and standard multiplication of two numbers  $x_{1}, x_{2}$  in the range [1, 2]. These patterns repeat per octave. Left: Piecewise affine multiplication. Middle: Standard multiplication. Right: Relative error  $\frac{x_{1} \cdot x_{2} - x_{1}x_{2}}{x_{1}x_{2}}$  measured in percents.
The relative approximation error of PAM can be reduced by an additional PAM operation. For an appropriately chosen value  $\alpha$ , the expression  $x_{1} \hat{\cdot} x_{2} \hat{\cdot} \alpha$  will better approximate  $x_{1} x_{2}$  on average and/or the worst case. If either  $x_{1}$  or  $x_{2}$  is a constant that is known ahead of time, e.g. when computing paexp and palog, this can be taken into account when choosing  $\alpha$  for a specific operation. For PAM matrix multiplications, scaling the output instead of the inner scalar products may be sufficient to eliminate bias in the output magnitude. In general this sort of approximation will be more expensive than a single PAM operation and we leave it to future work to explore where and when this additional error compensation is beneficial.
The exact and approximate derivatives have different error characteristics when viewed as approximations of the derivative for a standard multiplication. The exact derivative is unbiased on average, since PAM is continuous and matches multiplication at certain points. However, the exact derivatives are discontinuous and have a high variance in the sense that for a specific input the error is often large. On the other hand, the approximate derivative usually has a lower error at a given point (i.e. low variance) but may be biased on average. These characteristics of the two derivative types also hold for the other functions. Figures 3 and 4 in the appendix show the exact and approximate derivatives for several functions.
We use PyTorch [28] for our experiments and run them using either Nvidia A100 (40GB) or V100 (32GB) GPUs. Elementwise piecewise affine multiplication, division,  $\mathrm{palog}_2$  and  $\mathrm{paexp}_2$ , as well as PAM matrix multiplications are performed using custom CUDA kernels. For verification, we also run alternative native torch implementations of these operations that run significantly slower. Other functions such as normalization and softmax are implemented by composing these operations. The training setup (e.g. hyperparameters) for both the standard and piecewise affine networks is identical. All hyperparameters are obtained from other works and have not been tuned specifically for piecewise affine training.
In our experiments we consider two training tasks, one on text data and one in computer vision. The first one is German to English translation on the IWSLT14 DE-EN dataset [3] consisting of roughly 160K sentence pairs from translated TED talks. The model is a small transformer with 6 encoder and decoder layers each, 4 attention heads an embedding dimension of 512, a feed forward hidden dimension of 1024 and ReLU activations. The training code for this task is based on the FairSeq [27] codebase. Our baseline setup trains for 20 epochs, using a cosine decay schedule with 4000 warmup steps and a peak learning rate  $5 \cdot 10^{-4}$  with a maximum batch size of 4096 tokens. We use AdamW [21, 17] for optimization with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$  and weight decay of  $10^{-4}$ . During training we apply a dropout with drop probability 0.3 and use cross entropy with a label smoothing of 0.1. For evaluation we use a beam search with 5 beams and report the BLEU score on the test set.
The second task is training a DeiT-Tiny [32] Vision Transformer [5, 7] without distillation. This model has 5M parameters, 12 layers with 3 heads and an embedding dimension of 192. We train on either CIFAR10 [19] or the ImageNet-1k [6] dataset. CIFAR10 consists of 50K training and 10K test images of size  $32 \times 32$  corresponding to 10 classes. When training on CIFAR10 we upscale the images to  $224 \times 224$  and use the same data augmentation procedure described in DeiT [32] for ImageNet. ImageNet-1k [6] contains 1.28M training images and 50K validation images in 1000 classes. The images have different sizes but are processed at a resolution of  $224 \times 224$  for training. These experiments are based on the PyTorch Image Models project [36]. Optimization is performed using AdamW with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and weight decay of 0.05. Our base learning rate of  $5 \cdot 10^{-4}$  which is scaled linearly by the total batch size (1152 for ImageNet, 1024 for CIFAR10) divided by 512. We use a cosine decay learning rate schedule with a warmup of 5 epochs and train for a total of 300 epochs on ImageNet or 600 epochs on CIFAR10.
3.2 Replacing Matrix Multiplications
As mentioned before, matrix multiplications account for the majority of the computational cost of training neural networks and are therefore provide the largest opportunity for cost reduction. We experiment with replacing all matrix multiplications on both the forward and the backward passes with matrix multiplications where scalar multiplications are replaced by piecewise affine
Table 2: Top-1 Test/Validation Accuracy for DeiT-Tiny Training. The brackets denote the change from the baseline. *Shu et al. 2021 reports a different baseline accuracy of  $92.6\%$  on CIFAR10.
DATASET	BASELINE	PA-MATMUL	ADDR
CIFAR10	94.8%	95.0[+0.2]%	92.4[-0.2]%*
IMAGENET	72.2%	72.2[+0.0]%	70.5[-1.7]%
multiplications. This includes all linear layers, the batched matrix multiplication used in the attention and the convolutional layer used for the patch embedding in the vision transformer. Here we use the same type of matrix multiplications on the backwards pass which corresponds to the approximate derivatives described in Section 2.5.