Integer Networks for Data Compression with Latent-Variable Models

We consider the problem of using variational latent-variable models for data compression . For such models to produce a compressed binary sequence , which is the universal data representation in a digital world , the latent representation needs to be subjected to entropy coding . Range coding as an entropy coding technique is optimal , but it can fail catastrophically if the computation of the prior differs even slightly between the sending and the receiving side . Unfortunately , this is a common scenario when floating point math is used and the sender and receiver operate on different hardware or software platforms , as numerical round-off is often platform dependent . We propose using integer networks as a universal solution to this problem , and demonstrate that they enable reliable cross-platform encoding and decoding of images using variational models . 1 INTRODUCTION . The task of information transmission in today ’ s world is largely divided into two separate endeavors : source coding , or the representation of data ( such as audio or images ) as sequences of bits , and channel coding , representing sequences of bits as analog signals on imperfect , physical channels such as radio waves ( Cover and Thomas , 2006 ) . This decoupling has substantial benefits , as the binary representations of arbitrary data can be seamlessly transmitted over arbitrary physical channels by only changing the underlying channel code , rather than having to design a new code for every possible combination of data source and physical channel . Hence , the universal representation of any compressed data today is the binary channel , a representation which consists of a variable number of binary symbols , each with probability 12 , and no noise ( i.e . uncertainty ) . As a latent representation , the binary channel unfortunately is a severe restriction compared to the richness of latent representations defined by many variational latent-variable models in the literature ( e.g. , Kingma and Welling , 2014 ; Sønderby et al. , 2016 ; van den Oord et al. , 2017 ) , and in particular models targeted at data compression ( Theis et al. , 2017 ; Ágústsson et al. , 2017 ; Ballé et al. , 2018 ) . Variational latent-variable models such as VAEs ( Kingma and Welling , 2014 ) consist of an encoder model distribution e ( y | x ) bringing the data x into a latent representation y , and a decoder model distribution d ( x | y ) , which represents the data likelihood conditioned on the latents . Given an encoder e , we observe the marginal distribution of latentsm ( y ) = Ex [ e ( y | x ) ] , where the expectation runs over the ( unknown ) data distribution . The prior p ( y ) is a variational estimate of the marginal ( Alemi et al. , 2018 ) . By choosing the parametric forms of these distributions and the training objective appropriately , many such models succeed in representing relevant information in the data they are trained for quite compactly ( i.e. , with a small expected Kullback–Leibler ( KL ) divergence between the encoder and the prior , ExDKL [ e‖p ] ) , and so may be called compressive in a sense . However , not all of them can be directly used for practical data compression , as the representation needs to be further converted into binary ( entropy encoded ) . This conversion is typically performed by range coding , or arithmetic coding ( Rissanen and Langdon , 1981 ) . Range coding is asymptotically optimal : the length of the binary sequence quickly converges to the expected KL divergence in bits , for reasonably large sequences ( such as , for one image ) . For this to hold , the following requirements must be satisfied : • The representation must be discrete-valued , i.e . have a finite number of states , and be noiseless – i.e . the conditional entropy of the encoder must be zero : H [ e ] = Ex Ey∼e [ − log e ( y | x ) ] = 0 . ( 1 ) • All scalar elements of the representation y must be brought into a total ordering , and the prior needs to be written using the chain rule of calculus ( as a product of conditionals ) , as the algorithm can only encode or decode one scalar random variable at a time . • Both sides of the binary channel ( i.e . sender and receiver ) must be able to evaluate the prior , and they must have identical instances of it . The latter point is crucial , as range coding is extremely sensitive to differences in p between sender and receiver – so sensitive , in fact , that even small perturbations due to floating point round-off error can lead to catastrophic error propagation . Unfortunately , numerical round-off is highly platform dependent , and in typical data compression applications , sender and receiver may well employ different hardware or software platforms . Round-off error may even be non-deterministic on one and the same computer . Figure 1 illustrates a decoding failure in a model which computes p using floating point math , caused by such computational non-determinism in sender vs. receiver . Recently , latent-variable models have been explored that employ artificial neural networks ( ANNs ) to compute hierarchical or autoregressive priors ( Sønderby et al. , 2016 ; van den Oord et al. , 2017 ) , including some of the best-performing learned image compression models ( Ballé et al. , 2018 ; Minnen et al. , 2018 ; Klopp et al. , 2018 ) . Because ANNs are typically based on floating point math , these methods are vulnerable to catastrophic failures when deployed on heterogeneous platforms . To address this problem , and enable use of powerful learned variational models for real-world data compression , we propose to use integer arithmetic in these ANNs , as floating-point arithmetic can not presently be made deterministic across arbitrary platforms . We formulate a type of quantized neural network we call integer networks , which are specifically targeted at generative and compression models , and at preventing computational non-determinism in computation of the prior . Because full determinism is a feature of many existing , widely used image and video compression methods , we also consider using integer networks end to end for computing the representation itself . 2 INTEGER NEURAL NETWORKS . ANNs are typically composite functions that alternate between linear and elementwise nonlinear operations . One linear operation followed by a nonlinearity is considered one layer of the network . To ensure that such a network can be implemented deterministically on a wide variety of hardware platforms , we restrict all the data types to be integral , and all operations to be implemented either with basic arithmetic or lookup tables . Because integer multiplications ( including matrix multiplications or convolutions ) increase the dynamic range of the output compared to their inputs , we introduce an additional step after each linear operator , where we divide each of its output by a learned parameter . Concretely , we define the relationship between inputs u and outputs w of one layer as : v = ( Hu+ b ) c , ( 2 ) w = g ( v ) . ( 3 ) In order , the inputs u are subjected to a linear transform H ( a matrix multiplication , or a convolution ) ; a bias vector b is added ; the result is divided elementwise by a vector c , yielding an intermediate result vector v ; and finally , an elementwise nonlinearity g is applied to v. The activations w and all intermediate results , as well as the parametersH , b , and c are all defined as integers . However , they may use differing number formats . For v to be integral , we define here to perform rounding division ( equivalent to division followed by rounding to the nearest integer ) . In programming languages such as C , this can be implemented with integer operands m , n as m n = Q ( mn ) = ( m+ n // 2 ) // n , ( 4 ) where Q rounds to the nearest integer and // is floor division ; here , the addition can be folded into the bias b as an optimization . We constrain the linear filter coefficients H and the bias vector b to generally use signed integers , and the scaling vector c to use unsigned integers . We implement the accumulators of the linear transform with larger bit width than the activations and filter coefficients , in order to reflect the potentially increased dynamic range of multiplicative operations . We assume here that the bias and scaling vectors , as well as the intermediate vector v , have the same bit width as the accumulators . The elementwise nonlinearity g must be saturating on both ends of its domain , because integers can only represent finite number ranges . In order to maximize utility of the dynamic range , we scale nonlinearities such that their range matches the bit width of w , while their domain can be scaled somewhat arbitrarily . Depending on the range of the nonlinearity , the activations w may use a signed or unsigned number format . For instance , a reasonable choice of number formats and nonlinearity would be : H : 8-bit signed b , v : 32-bit signed ( same as accumulator ) c : 32-bit unsigned w : 8-bit unsigned gQReLU ( v ) = max ( min ( v , 255 ) , 0 ) In this example , the nonlinearity can be implemented with a simple clipping operation . Refer to figure 2 , left , for a visualization ( for visualization purposes , the figure shows a smaller bit width ) . Another example is : H : 4-bit signed b , v : 16-bit signed ( same as accumulator ) c : 16-bit unsigned w : 4-bit signed gQtanh ( v ) = Q ( 7 tanh ( v15 ) ) Here , the nonlinearity approximates the hyperbolic tangent , a widely used nonlinearity . It may be best implemented using a lookup table ( see figure 2 , right , for a visualization ) . We scale its range to fill the 4-bit signed integer number format ofw by multiplying its output with 7 . The domain can be scaled somewhat arbitrarily , since v has a larger bit width than w. When it is chosen too small , w may not utilize all integer values , leading to a large quantization error . When it is chosen too large , overflow may occur in v , or the size of the lookup table may grow too large for practical purposes . Therefore , it is best to determine the input scaling based on the shape of the nonlinearity and the available dynamic range . Here , we simply chose the value of 15 “ by eye ” , so that the nonlinearity is reasonably well represented with the lookup table ( i.e. , we made sure that at least two or three input values are mapped to each output value , in order to preserve the approximate shape of the nonlinearity ) . 3 TRAINING INTEGER NEURAL NETWORKS . To effectively accumulate small gradient signals , we train the networks entirely using floating point computations , rounded to integers after every computational operation , while the backpropagation is done with full floating point precision . More concretely , we define the integer parameters H , b , and c as functions of their floating point equivalentsH ′ , b′ , and c′ , respectively : H = Q ( h ′ 1/s ( h ′ 1 ) ) ) ... Q ( h′N/s ( h ′ N ) ) , ( 5 ) b = Q ( 2K b′ ) , ( 6 ) c = Q ( 2K r ( c′ ) ) . ( 7 ) Here , we simply rescale each element of b′ using a constant K , which is the bit-width of the kernel H ( e.g . 8-bits in the QReLu networks ) , and round it to the nearest integer . The reparameterization mapping r is borrowed from Ballé ( 2018 ) : r ( c′ ) = max ( c′ , √ 1 + 2 ) 2 − 2 . ( 8 ) When c is small , perturbations in c can lead to excessively large fluctuations of the quotient ( i.e. , the input to the nonlinearity ) . This leads to instabilities in training . r ensures that values of c are always positive , while gracefully scaling down gradient magnitudes on c near zero . Effectively , the step size on c is multiplied with a factor that is approximately linear in c ( Ballé , 2018 ) . Before rounding the linear filter coefficients in H ′ = [ h′1 , . . . , h ′ N ] > , we apply a special rescaling function s to each of its filters h′ : s ( h′ ) = max ( ( −2K−1 ) −1 min i h′i , ( 2 K−1 − 1 ) −1 max i h′i , ) . ( 9 ) s rescales each filter such that at least one of its minimum and maximum coefficients hits one of the dynamic range bounds ( −2K−1 and 2K−1−1 ) , while keeping zero at zero . This represents the finest possible quantization of the filter given its integer representation , and thus maximizes accuracy . To prevent division by zero , we ensure the divisor is larger than or equal to a small constant ( for example , = 10−20 ) . In order to backpropagate gradient signals into the parameters , one can not simply take gradients of the loss function with respect to H ′ , b′ , or c′ , since the rounding function Q has zero gradients almost everywhere , except for the half-integer positions where the gradient is positive infinity . A simple remedy is to replace the derivative of Q with the identity function , since this is the smoothed gradient across all rounded values . Further , we treat the rescaling divisor s as if it were a constant . That is , we compute the derivatives of the loss function with respect toH ′ , b′ , and c′ as with the chain rule of calculus , but overriding : ∂h ∂h′ : = 1 s ( h′ ) , ∂b ∂b′ : = 2K , ∂c ∂c′ : = 2Kr′ ( c′ ) , ( 10 ) where r′ is the replacement gradient function for r as proposed by Ballé ( 2018 ) . After training is completed , we compute the integer parametersH , b and c one more time , and from then on use them for evaluation . Note that further reparameterization of the kernelsH ′ , such as Sadam ( Ballé , 2018 ) , or of the biases b′ or scaling parameters c′ , is possible by simply chaining reparameterizations . In addition to rounding the parameters , it is necessary to round the activations . To obtain gradients for the rounding division , we simply substitute the gradient of floating point division . To estimate gradients for the rounded activation functions , we replace their gradient with the corresponding nonrounded activation function , plotted in cyan in figure 2 . In the case of QReLU , the gradient of the clipping operation is a box function , which can lead to training getting stuck , since if activations consistently hit one of the bounds , no gradients are propagated back ( this is sometimes called the “ dead unit ” problem ) . As a remedy , we replace the gradient instead with ∂gQReLU ( v ) ∂v : = exp ( −αβ ∣∣∣ 2v 2L − 1 − 1 ∣∣∣β ) , ( 11 ) where α = 1βΓ ( 1 β ) , and L is the bit width of w. This function corresponds to a scaled generalized Gaussian probability density with shape parameter β . In this context , we can think of β as a temperature parameter that makes the function converge to the gradient of the clipping operation as β goes to infinity . Although this setting permits an annealing schedule , we simply chose β = 4 and obtained good results . The integral of this function is plotted in figure 2 ( left ) in cyan , along with other choices of β in gray .

Summary: The paper presents a very important problem of utilizing a model on different platforms with own numerical round-offs. As a result, a model run on a different hardware or software than the one on which it was trained could completely fail due to numerical rounding-off issues. This problem has been considered in various papers, however, the classification task was mainly discussed. In this paper, on the other hand, the authors present how the numerical rounding-off issue could be solved in Latent-Variable Models (LVM).