# Predictive Quantization and Symbolic Dynamics
## 4 Representation
### Bit-Rate-Limited Encoding  
As mentioned above, in this study we consider a particular case of reduced representation that is based on rate-distortion theory. Rate-distortion theory offers an optimal solution for finding the most compact (least-rate) encoding for a given limit on the distortion or reconstruction error. Equivalently, distortion-rate finds the best encoding in terms of least distortion for a given rate. Algorithms for optimal bit allocation according to the rate-distortion theory are so called bit allocation methods that we describe below. By using a rate-limited channel between the encoder and decoder of the VAE, we are able to control the complexity of the encodings using a bit allocation algorithm. In our case, we use rate as the free parameter to find the least-distortion codes under the assumption that latent codes in VAE are distributed as multi-variate uncorrelated Gaussians. The rate-distortion function that provides the lower limit on the achievable rate \(R\) as a function of the maximal-allowed distortion level \(D\), is given by  
\[R(D)=\begin{cases}\frac{1}{2}log_{2}\frac{\sigma^{2}}{D},&\text{if }0\leq D\leq \sigma^{2}\\ 0,&\text{if }D>\sigma^{2}.\end{cases} \tag{15}\]  
where \(R\) is the rate and \(D\) is the distortion value. This rate-distortion function can be converted into a distortion-rate function \(D(R)=\sigma^{2}2^{-2R}\) that gives the lower limit on distortion D that is achievable for a given rate R. This ideal lower limit (i.e., least distortion)can be efficiently achieved for a particular type of signal that is known in communication theory as a \"multivariate Gaussian channel\". We adopt this channel model for our experiments without further justification. What is special about this type of channel is that an optimal bit reduction can be achieved by using the so-called reverse water-filling algorithm [18]. This algorithm starts with a predefined bit-rate R and successively allocates one bit at a time to the strongest component, repeating the process until all bits in the bit pool are exhausted. One should note that channels (i.e., latent variables in our case) with variance less than allowed distortion, or channels that run out of bits for a given rate, are given zero bits and, thus, are eliminated from the transmission.  
A schematic representation of channel inclusion in the autoencoder architecture is given by Figure 7.  
Encoding the latent components at a rate \(R\) changes the mean and variance of the VAE as follows [19]  
\[Q(z_{d}|z_{e}) =Normal(\mu_{d},\sigma_{d}^{2}) \tag{16}\] \[\mu_{d} =z_{e}+2^{-2R}(\mu_{e}-z_{e})\] (17) \[\sigma_{d}^{2} =2^{-4R}(2^{2R}-1)\sigma_{e}^{2} \tag{18}\]  
This process requires some explanation: for a given rate \(R\) we obtain the bit-rate for each of the latent variables according to the reverse water-filling procedure. This gives us different numbers of bits for each latent dimension, where the high-variance dimensions tend to grab the bits first, often leaving the weak (small-variance) latent variables with zero bits. Next, we proceed by sampling a value from the encoder distribution according to the original VAE mean and variance parameters, \(\mu_{e}\) and \(\sigma_{e}^{2}\). Then, using the rate \(R\) and the original mean and variance parameters for each latent variable, we derive a new mean and variance \(\mu_{d}\) and \(\sigma_{d}^{2}\). We use these probability parameters to sample a reduced-bit-rate value and use it as our new input to the decoder. One can see from Equation (18) that latent variables that are allocated zero bits need not be transmitted. More precisely, the value that the decoder needs is the mean value of that latent variable that is independent of the particular instance being transmitted. This mean value can be obtained a priori and thus can be \"hard coded\" into the decoder ahead of time, with no need to transmit it. Channels allocated a very high rate will transmit an (almost) unaltered value of the latent variable that was sampled in the VAE encoder.