Neural Communication Systems with Bandwidth-limited Channel
Authors: Karen Ullrich
Fabio Viola
Danilo J. Rezende
Venue:  (2020)
Keywords: variational inference
joint coding
bandwidth-limited channel
deep learning
representation learning
compression

Abstract: Reliably transmitting messages despite information loss due to a noisy channel is a core problem of information theory.  One of the most important aspects of real world communication is that it may happen at varying levels of information transfer. The bandwidth-limited channel models this phenomenon.  In this study we consider learning joint coding with the bandwidth-limited channel.  Although, classical results suggest that it is asymptotically optimal to separate the sub-tasks of compression (source coding) and error correction (channel coding), it is well known that for finite block-length problems, and when there are restrictions to the computational complexity of coding, this optimality may not be achieved. Thus, we empirically compare the performance of joint and separate systems, and conclude that joint systems outperform their separate counterparts when coding is performed by flexible learnable function approximators such as neural networks.  Specifically, we cast the joint communication problem as a variational learning problem. To facilitate this, we introduce a differentiable and computationally efficient version of this channel.  We show that our design compensates for the loss of information by two mechanisms: (i) missing information is modelled by a prior model incorporated in the channel model, and (ii) sampling from the joint model is improved by auxiliary latent variables in the decoder. Experimental results justify the validity of our design decisions through improved distortion and FID scores.

Under review as a conference paper at ICLR 2020
NEURAL COMMUNICATION SYSTEMS WITH HHNDIDDT-LLMITED CHANNEL
Anonymous authors Paper under double-blind review
ABSTRACT
Reliably transmitting messages despite information loss due to a noisy channel is a core problem of information theory. One of the most important aspects of real world communication that it may happen at varying levels of information transfer. The hendwidth-liiiieee channel models this phenomenon. In this study we consider learning joint coding with the bandwidth-limited channel. Although, classical results suggest that it is asymptotically optimal to separate the sub-tasks of compression (source coding) and error correction (channel coding), it is well known that for finite block-length problems, and when there are restrictions to the computational complexity of coding, this optimality may not be achieved. Thus, we empirically compare the performance of joint and separate systems, and conclude that joint systems outperform their separate counterparts when coding is performed by flexible learnable function approximators such as neural networks. Specifically, we cast the joint communication problem as a variational learning problem. To facilitate this, we introduce a differentiable and computationally efficient version of the
channel. We show that our design compensates for the loss of information by two mechanisms: (i) missing information is modelled by a prior model incorporated in the channel model, and (ii) sampling from the joint model is improved by auxiliary latent variables in the decoder. Experimental results justify the validity of our design decisions through improved distortion and FID scores.
INTRODUCTION
The 21st century is often referred to as the information age. Information is being created, stored and sent at rates never before seen. To cope with this deluge of information, it vital to design optimal communication systems. Such systems solve the problem of reliably transmitting information from sender to receiver given some form of information loss due to transmission errors (i.e. through a noisy channel). As the size of the transmitted messages goes infinity for memory-less communication channels, the joint source-channel coding theorem (Shannon 1948) states that it is optimal to split the communication task into two sub-tasks: (i) removing redundant information from the message (source coding) and (ii) re-introducing some redundancy into the encoded message to allow for message reconstruction despite the channel information loss (channel coding). As result, separate systems have been studied extensively in the literature and in fact are the standard way of coding for many scenarios. However, is also well known that there are limits the optimality of separate systems in practical settings. Most importantly for this work, limitations arise when we seek to encode finite length messages (Kostina & Verdu 2013). These limits lead to two consequences: (i) When there is a budget on transmission bits, source and channel coding errors need to be balanced for best reconstruction results. (ii) Decoding via
principle becomes an hard problem (Berlekamp et al. 1978). Thus approximations need to be made that can lead to highly
sub-optimal solutions (Koetter & Vontobel 2003; Feldman et al.) 2005 Vontobel & Koetter 2007).
Recent work (Choi et al.) 2019 Farsad et al.) 2018), has thus looked at the problem of learning to jointly communicate. This includes systems that learn to do source and channel coding jointly from data. Practically this can be achieved by learning neural network encoders and decoders, where channels are simulated by adding noise to encoded messages. Several desirable properties of such systems were shown, including improvements in decoding speed and code length. Complementary to this body of work, we focus this study on the investigation of neural joint models with the bandwidth- limited channel. Specifically, we direct our experimentation the bandwidth- limited channel due

---

Under review as a conference paper at ICLR 2020
to it's ubiquity as a fundamental component in the real world communication systems. The main contributions this work include:
1. We cast the problem of learning joint communication variational learning problem, parallel to other work (Choi et al. 2019).
2. We justify the importance of jointly learned systems by empirically evaluating the gap between neural systems for joint and separate communication.
3. We design standard channels such as the Gaussian and Binary channel as differentiable proba- bilistic nodes, which serve as base for our design of the bandwidth-limited channel.
4. We investigate two core design choices of our neural joint model and bandwidth-liminn
channel:
(i) how transmission rate can be improved through learned prior models (ii) how we may improve image reconstructions in the low bandwidth regime through the introduction of auxiliary latent variables in the decoding process.
2 NOTATION AND PRELIMINARIES
We mark sets as calligraphic letters (i.e. X') random variables as capital letters (i.e. X) and their values as lower case letters (i.e. I). We use capital letters denote probability distributions (i.e. P(X)) and lower case letters for the corresponding densities (i.e. p(z)). We will refer to a property of stochastic processes, the entropy. It describes the average rate at which a process emits information. Formally,
(X) -ox P(X)),
where IE' is the expectation. Further, we expect the reader to be familiar with the distortion-rate theory. Appendix summarizes these shortly and makes connections to neural compression systems.
SOURCE AND CHANNEL CODING FOR COMMUNICATION SYSTEMS
X
(a)
sender A
encoder
channe
decoder
receiver B
Y= E(X)
CIZIY)
X D(Z)
X
source
channel
Y
channe
Z
charnel decoder Z'= (Z)
source
(b)
sender A
encoder
encoder
decocir D'I2')
receiver B
Y' = (X()
== ("()")
CIZIY)
Figure I: (a) Joint communication system: message X passes through joint source and channel encoder before it passes a channel and is subsequently decoded. (b) Separate communication system: This system distinguishes source encoding and decoding (red) from channel encoding and decoding (blue). Red and blue system are designed independently of each other.
In this section the reader will be introduced to communication systems and in particular to the challenge of joint coding when a finite bit-length budget is given.
Communication is defined by entity A called the sender, source, that induces state X, the message, in another entity B, the receiver. We call this transfer of information successful if A and B agree about the message being sent: X = X, or if the message distortion ||X - X|| does not exceed a certain level D. Real-world communication is an inherently noisy physical process where many uncontrollable or unpredictable factors may interfere with a sent message before it reaches its receiver. To account for this interference, communication typically organized into three distinct components which we illustrate in Figure I (i) The encoder Y E(X), whose role is to compress its inputs (i.e. to remove redundant information) and subsequently prepare them for transmission through the channel with minimal distortion. (ii) The channel Z = C'Zl over which we have no control, and represents the unpredictable distortions caused by the physical transmission process. (iii) The decoder X- = DZZ), whose goal is to reverse the process to the original datum X from the received code Z.
2

---

Under review as a conference paper at ICLR 2020
Channel capacity The most important characteristic of channel is its capacity. In order to evaluate it, we may compute the number of distinguishable messages that we can send through the channel given encoder. The logarithm of that number is referred to information capacity the channel. is given by the maximum of the mutual information of Y and 2, T(Y; 2), taken over all possible input distributions P(Y),
C = max I(Y; 2). P(Y)
Other relevant properties of channel models include (i) bandwidth, the number of information units passing a channel per time unit, (ii) memory, the independence of joint probabilities of transmitted sequence, where a channel with fully independent joint probabilities is called 'meooryless, and (ii) feedback, the ability for the sender side of the system to know what bits have arrived at the receiver side, resulting in Y E(X. 2). Note that this work, we will constrain our research question to feedback and memoryless channels.
Joint source-channel coding The channel ca- pacity fundamentally restricts the ability of communication system to transfer messages. The source-channel coding theorem (SCCT) specifies this restriction as follows. For i.i.d. variables and memoryless channels, given a cer- tain tolerated message distortion D, must send codes with length R(D). The data may recovered by the receiver at distortion D if and only if R(D) < C.
Y
X
CZZI)
E
E
Z
(2
(x(%,((8)
C(ZIY)
Y'
M
Furthermore, can be shown that there exists a two stage method that is as good as any al- ternative transmit information over noisy channel reliably: source coding and channel coding. These two steps can accomplished by two distinctly designed systems, referred to as the source encoder and decoder, ES( and DS(.). and the channel encoder and decoder, EC() and DC (.), respectively. It is easy to why this result had great impact on the de- sign of communication systems in practice, A communication problem is essentially defined by its source and its channel. Any such tuple defines individual problem, resulting in enormous problem space. By separation, it is possible to independently reuse good source or channel solutions for other problems.
Y' 1 Z
D IZ)
( (X) 12
8
X - 8
(b) Figure 2: (a) Graphical model of the jointly learned communication system. The message X is passed by the encoder and the channel, to be reconstructed into X. Note that because marginal- ization is not possible we apply a variational ap- proximation Q aid inference. (b) Graphical models of the separately learned communication system. Two systems are learned independently: VAE for source compression (red) and an AE for channel transmission (blue).
(a)
However, there also restrictions the appli- cability of the SCCT. For finite length messages, we have to trade bits for compression and chan- nel coding against each other. This is not trivial (Pile 1967 Csiszar 1982, Kostina Verdū 2013).
On top of this, encoders and decoders
are being idealized to any function. In practical settings however, we may not be able to identify optimal encoders. Further, they are computationally restricted. In the era of machine learning, however, hypothesis spaces can be searched increasingly quickly in an automated fashion, allowing researchers to search over the space of joint solutions for the first time. For these reasons, propose to learn joint communication systems using flexible function approximators such as deep neural networks.
3

---

Under review as a conference paper at ICLR 2020
4 LEARNED COMMUNICATION S SYSTEMS
In this section, we outline how to learn neural encoders and decoders for a given joint communication problem defined by a channel and source. Our approach requires differentiable path through the communication system. For this, we design appropriate channel models. Additionally we introduce a new design for the hendwwith-limitee channel, adapted from classical models, and explain how to do marginalization of bands in practice. Consequently, we frame learning in the joint and in the separate model as a variational optimization problem. Our approach is related to auto-encoders (Vincent et al.) 2010) and variational auto-encoders (Rezende et al., 2014] Kingma & Welling) 2013). We will outline the connection here. Finally, we introduce auxiliary latent variables (ALV) to the decoding process as means to combat low reconstruction quality when little information is transmitted though such model.
4.1 CHANNEL MODELS
To enable back- proppaaaaantionn through communication system, we shall introduce the most common channel models in the literature and explain how to build them in differentiable fashion.
Gaussian Channel We start this discussion with the Gaussian channel model, the most important continuous alphabet channel. It is time discrete channel that distorts incoming signal Y by i.i.d. Gaussian noise W.
Z Y W
W; x²)
(3)
However, this particular definition of limited use. When the noise is fixed but the power of the input is not, one can easily design channel encodings that essentially ignore that noise. It is thus common to power constrain the input, this is equivalent to keeping constant signal-to-noise ratio (SNR), 8. It can be shown that the channel capacity of power limited Gaussian channel is equal to C 1/2 log (I + s) bits per transmission (Cover & Thomas 2012). For a differentiable Gaussian channel with constant SNR, we assume the channel inpul to be isotropic Gaussian distribution
Y, ~ [Ny,, o. We propose to use the reparameterization trick (Kingma & Welling 2013
Rezende et al. [2014), where a probabilistic node is separated into a parameter independent stochastic node and a deterministic one. By using the trick twice we can rewrite the channel to
2, = Y, + Wi:
W; ~ NO,,1) 1).
(4)
Bandwidth -imited channel Related to the Gaussian channel, and one of the most important models for communication, e.g. over radio and wifi, is the bandwidth-limine channel. The channel capacity for Gaussian bandwidth limited channel is known to be linearly related to the bandwidth C' ~ B. In the classical literature, this is described as a continuous time, white noise and bandwidth- -asss filtered channel I however, in this work, we adopt the concept to be a discrete time channel, for which we introduce the bandwidth B as a discrete latent variable,
T
C(ZIY)= = CCZ.B Σ P CIZ.Y (2, B
II Py, (Z)) t=
(5)
B
C(ABBY
where / is a discrete time step. In words, signal X gets encoded into a sequence Y (Y)) The sequence gets transmitted up to B by sending Y = {Yi) though a channel C(Z, such as the Gaussian channel. Other information Y (Y B+1 is lost. This information is replaced by samples from a prior over Y, Py; (Z)). The full integration over the input domain required to compute the integral Pr, (Yt) f e(Yi (1z)) dr is expensive. Thus, we will introduce an approximation to it, i.e., a standard Gaussian prior or a more elaborate model such as the ConvDraw prior (Gregor et al. (0166).
To summarize, we have introduced a differentiable and computationally efficient version of the bandwidth- bandwidth-limited limited channel. For this, we turn it into a time discrete channel by introducing the discrete latent variable B. To marginalize over the latent variable we may either do Monte Carlo sampling or
Cover & Thomas 2012 chap. 10 for more details
4

---

Under review as a conference paper at ICLR 2020
complete marginalization. The model also requires model for codes that have been dropped. This is similar to the prior a variational auto-encoder and can be learned to arbitrary complexity.
Other differentiable models include the erasure channel, first considered in Kim et al. (2018a). However, this channel is mainly relevant for feedback systems, we will thus not discuss it in this context. Another relevant channel is the Binary channel, which we detail in the appendix. For real-world channels there is the option to learn a parametric model that emulates them by sending random information units. Subsequently this model can be utilized channel model. If only black-box model of the channel is available, our proposed framework may be extended by using discrete optimization schemes. For example VIMCO (Mnih Rezende (016) has been used in Choi et al. (2019). The implementation of the channels we consider here can be found online, gi github.. cwm/ anwaannuunnssooooo code
4.2 SEPARATE SOURCE-CHANNNE CODING
As described in section 3] the joint communication problem can be broken down into two independent problems; the source coding and channel coding problem. Here, we demonstrate how to apply the variational auto-encoder as source coder and an auto-encoder as channel coder. Note that, there is no exchange of information between those two systems. We provide a visual aid for this section in Figure
Source- VAE In recent years, neural networks have been shown to be useful source compressors.
Specifically, variational auto-encoders (VAEs) have been pointed to as natural source coding systems
(Kingma& & Welling 2013 Alemi et al., (017). showing great practical success (Ballé et al. 2016
2018b Minnen et al. 2018 Zhou al. 2018 Tschannen et al.] 2018). Such a source- VAE
essentially a learned probabilistic model. Based set of samples emitted by the source X { aim learn the source encoder E XY") and the source decoder D ()((Z)), both parameterized functions of 1 and 8, respectively. The learning objective thereby originates from looking at the model as latent-variable model (with the encoding Z being the latent variable) for which we aim to do maximum marginal likelihood learning of the parameters. The involved marginalization, however, forces the introduction of a variational approximation, the encoder, to construct lower bound on the marginal log likelihood, known as variational inference. For this, we set the source encoder to be the variational approximation to the source decoder, such that Y' _ z' and X 1 8:
Ep(x) (log P(X 2 Ex(x)
E Eg (lo (S(()))) (A0Zz)JJKL((EBB) (L(E((())))P
(6)
This bound is known as the evidence lower bound (ELBO). Optimizing ELBO is equivalent to optimizing a rate( (R)-distortion()) problem. We adjust the rate-distortion trade-off to desired rate or distortion by introducing parameter 3 into the objective, this framework is well known as 3-VAE (Higgins et al. 2017: Alemi al. 2017).
Generally, it is possible to optimize decoder and encoder independently. This however would only make sense if we consider channel coding systems that do not try to reconstruct their inputs. Note that in contrast to the original formulation in section 3 encoder and decoder have been turned into probabilistic mappings rather than deterministic ones. This allows one to find an ideal compression rate given certain distortion-rate trade-off B. The rate can practically be achieved with the so called bits-back coding (Hinton Van Camp/ 1993] Townsend al.) 2019). For inference it became common that the parameters for the encoder distribution may be predicted by neural network parameterized by 0. This is called amortized inference. The parameters of this inference model and the generative model, the decoder, are trained jointly though stochastic maximization of ELBO. To do this efficiently, it is common to use the reparameterization trick (Kingma & Welling 2013 et al. 2014).
Rezende
Finally, it is important to note that the prior distributions in the context compression may not be learned (Z) P(Z). This would conflict the independence of the source and channel.
5

---

Under review as a conference paper at ICLR 2020
Channel-AE For training a neural channel coding system, will use samples from the source independent prior (y'm 1.M send Y though the probabilistic channel Z C(Z after which try to recover the inputs by channel decoding Z' = D (Z) The system is trained by minimizing a measure of distortion between Y and Z'
Vnn ~ P(Z). After using a deterministic encoding Y = En (Y Y") 1m=1- 1.
Note, that a for a simple additive withe Gaussian noise channel there exists a near optimal channel coding scheme: LDPC. However, in more general scenarios they do not perform as well anymore and can be beaten by neural network architectures (Kim et al.) 2018a). Further, it has been shown that neural networks can decode them efficiently (Nachmani al. 2016). For the sake of generalizing to more complex channels we thus propose general purpose neural network channel coding.
4.3 JOINT BOURCE-CHANNEE CODING
For the jointly optimized system. we translate the communication system as described in section 3 into a generative model (x( f ( y (X(xx) dy dz. Similar to the previous section, we think of the encoder and decoder as parameterized mappings, while the channel model taken as given. We are interested in performing maximum likelihood learning of the model parameters o, by optimizing
E E((x log Po(((()) = Ep(x) log / e=(0s/)c(xx))d dy dz
The required marginalization in equation frequently applied solution is to introduce a variational approximation QY, Z(X) to the posterior, construct a lower bound on the marginal likelihood.
7 however, leads to generally intractable integrals. One
log (X 2 E (Y.22)))))) (log D& (&12)
(X, E
(8)
As before, this represents an ELBO. Note though that, the first term in equation g refers to the quality of the message reconstruction and the second to how closely the receiver understands the sender. This is different to the previous section where the message never actually passes the communication system. The variational posterior plays very different role there where it is assumed to be the encoder. In the joint scenario the posterior only serves to train the system, at test time, however is of no interest. To it up, proposed framework optimizes the actual objective of communication, the message reconstruction. For channels that do not allow for information transfer this model turns into a VAE.
Auxiliary latent variable Decoders When the information transmitted by the channel is variable, i.e. for the bandwidth- limited channel, model has adapt low and high infor- mation transmission rates. To contest information loss due to noisy channel, we propose introduce auxiliary latent variables V to the decoder model. This model choice ac- knowledges the implicit marginalization over lost information. Although expected message distortion is unchanged, when sampling from such a model, message reconstructions should occur more in distribution with the true source distribution (e.g. one would expect sharper images).
Z
V
2,
D2.8
X
QAVI&
N
We can enforce this change to the decoder by adapting the distortion term in equation g As before we would need to marginalize over V but choose the variational approach in- stead,
Figure 3: Excerpt of the graphical model in ( We show how the decoder changes when introduc- ing auxiliary latent variables V.
-D 2 Qv (v.2 E QV (D
DKL (KL((2(1) X,,,,,,,))) -V ((V))
Again, we introduced an approximate posterior to circumvent the intractable task, where P(V) is a prior these newly introduced latent variables. Just as before, the parameters of the variational

---

Under review as a conference paper at ICLR 2020
distribution shall be inferred by deep neural network with parameters Ç. We indicated the compo- nents introduced to the communication model in yellow in Figure We note that we could execute the same idea using other conditional generative models, and corresponding inference methods, such as conditional GANs. We leave this exploration to future research.
5 RELATED WORK
The field of learned image and video compression has enhanced rapidly over the past few years. While Ma et al. (2019) give a recent concise overview of the field, here we focus on probabílistic auto-encoding approaches first proposed by Theis et al. (2017). The main focus of the field of image compression is to close the gap between theoretical ideas and well performing systems. One block of efforts focuses on learning representations. While VAEs tend work better in the continuous regime, most codes and channels can best be described by binary representations. To bridge this, it has been proposed to (i) quantize continuous representations by convolving them with a uniform distribution (Ballé et al.] 2016 2018b Minnen et al. |01S/ssssssoon et al.) 2017), (ii) learn discrete representations directly (van den Oord et al. 2017| Ballé al. [0188a] Shen et al. 2019) or even (iii)
learn to generate common codecs e.g. JPEG (Jiang et 2017 Liu et al. 2018). Note that some of these systems rely on learned priors; however, these are actually not suitable for separate coding. Other work is focused on biasing compression towards image features important for perception or system security (Li et al., 2018: Agustsson et al. 2018). For situations where sequences of source inputs are communicated, neural buffers have also been explored to allow reordering of elements to improve code length (Graves et al., 2018). Another branch research focuses on the architecture of encoder and decoder models (Gregor et al.] [016)] De Fauw al., 2019] Zhoou et al.] 2018). Additionally, there is work looking into performing tasks on compressed representations directly (Torfason al. 2018). Important mention also efforts make the often expensive encoder and decoder more computationally efficient (Ballé et al. 2015).
In contrast to neural source coding, neural channel coding has yet to be explored so extensively. However, first studies (Nachmani et al. 2016, Gruber et al. (2017: Cammerer et al. 2017 Dörer et al., 2017) demonstrate great success with neural encoder/decoder architectures. For example, was shown that neural model can find a solution to the Gaussian feedback channel which benefits from the feedback, a result known before but not demonstrated by any channel code yet (Kim et al., 201 8018ap b).
Most related to our work in spirit is a range of end-to-end learned joint communication systems. Farsad et al. (018)) apply joint source channel system to text: Boourssoulazee et al. (2019) use auto-encoders to transmit messages over the AWGN channel; and Zarcone et al. (2018) use joint systems for data compression. Closest to our work is the study by Choi et al. (2019) where they learn the communication system in a variational fashion as well, but exclusively look at the binary erasure channel. The discrete channel leads to another variant of the learning scheme.
EXPERIMENTS
We focus our experiments on the fandwidth-limite channel with additive white Gaussian noise (AWGN) and power restricted inputs. The latter ensures limited channel capacity. First, we verify the importance of joint coding in contrast to separate channel coding in this context. For this, we compare neural joint and separate models, finding the joint model consistently outperforms it's separate counterpart. These findings echo other recent work. We therefore expand upon this by focusing the remainder of our experiments on the bandwidth-limited channel model proposed in this work. Specifically, we investigate the performance of neural joint model with the AWGN bandwidth-limine channel for different prior and decoder choices.
All results are evaluated on CelebA (Liu et al. 2015). All images were re-scaled to resolution of 32 x 32 pixels. Encoders and decoders have generally been chosen to be Residual Networks (He et al. 2016), due to their wide usage in range of generative modelling tasks, e.g. in Gregoo et al. (2016).
6.1 COMPARING JOINT AND SEPARATE NEURAL MODELS WITH GAUSSIAN CHANNEL
7

---

Under review as a conference paper at ICLR 2020
As previously discussed in section S not predict precisely how a separate model would compare in contrast to a joint one. We hence compare separate and joint neural models as de- scribed in section 4 for the AWGN various SNRs. For both models we choose the same pos- terior distribution for latent encoding: isotropic Gaussians. We additionally choose the obser- vation model to be Gaussian since it is quite common to measure distortion in L2-space. En- coder and decoders of both models share the same architecture configuration. For the sepa- rate system, we choose a standard Gaussian to be the prior for the source- VAE and simultane- ously the data source for the channel-AE. We note that this cannot be data dependent prior this would leak information to the channel cod- ing system. For both models we hyper-optimize over range of beta values on log-scaled grid
10a
Joint Model
Separate Model
Distortion
10
10a
10"
10*
Signal-to- Aignal-toNNoise ratio
Figure Results of comparing distortion for joint and separate neural communication systems at vari- ous signal-to-noise ratios for the Gaussian channel. The joint model outperforms the separate one con- sistently.
We optimize both models with an SGD algorithm.
We evaluate both systems by sending message through the encoder, channel and eventually the decoder, subsequently measuring the L2-distortion between sent and received messages. The quanti- tative results are presented in Figure oon the left. For any of the 5 SNRs that we run our experiment we find the joint model outperforms the separate one. We observe, though, that the difference between the systems shrinks towards either end of the range of the SNRs presented. This effect can be explained: For very high SNRs the channel model becomes somewhat redundant, thus both systems resemble a source- VAE. At very low SNRs both systems fail to communicate as they approach the channel capacity. Our findings are in line with other recent work: Choi et al. (2019) show that joint systems outperform hybrid models (ncural source coding, hand-designed channel coding) and Kim et al. (2018a) show, for some feedback channels, learned neural models outperform hand-crafted channel codes.
6.2
COMMUNICATION MODEL. DESIGN FOR BANDWIDTH-LIMITED CHANNEL
250
Auro-Regressive
200
0.30
150
0.35
100
0.30
Gemerator
0.05
Decodor: with zux latent variables (CMV)
Decoder: with aus latent variebles (Canoket)
2.n
1.0
Bandwicth
Bandwicth
Figure 5: We consider joint models trained based on the AWGN bandwidth-limited channel with a fixed SNR of 1. In both figures we contrast message quality with bandwidth. The higher the bandwidth the more information is transmitted to the receiver. Left: We measure message quality by distortion in L2-space. We compare two approximations to the channel encoding distributions. Our complex prior outperforms simpler one. Further, observe a linear relationship between bandwidth and distortion. Right: We measure message quality by FID score. Lower FID score is better. We compare decoders without auxiliary latent variables decoders with auxiliary latent variables.
After verifying the importance of joint modelling for Gaussian channels, we will now investigate the performance a joint model on the AWGN mondwidth-limiteee channel design we introduced in section 4.1 In this experiment fix the SNR of the AWGN to 1.
Too get an understanding of the sensitivity this parameter we put all results in the appendix.
8

---

Under review as a conference paper at ICLR 2020
Two choices for the model are relevant, the prior that models channel codes and the decoder. Both deal with a lack of information in the low bandwidth regime.
Prior As mentioned in the section 4.1 we require approximation to Py; In our first experiment, we investigate how much the complexity of this approximation influences the quality of message reconstruction. Here we shall compare spherical Gaussian and ConvDraw prior (see Gregor et al. (2016)) to contrast a simple with a complex approximation. We consider a 100 dimensional latent space. The space is partitioned into parts. Each part representing another band. Other specifications of the experiment are equivalent to the previous section. We present our findings in Figure 5 (Left). We observe that, as expected, the message distortion decreases when we transmit more information, for both approximations, We additionally observe that the quality of reconstruction increases when the more complex prior model used, and the distortion gap between priors increases when less information units are being transported through the channel. Furthermore, for both prior choices, the distortion decreases almost linearly with the bandwidth increasing. This result is in line with classic findings that show a linear relationship between channel capacity and bandwidth of an input power restricted AWGN. Finally, we shall give a visual impression the reconstructions at various bandwidth in Figure 6 in the appendix.
Decoder For small bandwidths, we find that loss of information leads to blurry reconstructions even with learned priors. To combat this, we contrast model without auxiliary latent variables with our proposed auxiliary latent variable model. Specifically, for these two models, we use an unconditional ConvDraw decoder and conditional Conv decoder (Giregor al., 2016) respectively. As measure of in-distribution affiliation we use the well established FID measure (Radford et al. 2015). This measure has mainly served evaluate the quality of GAN samples. Smaller FID measures are better. In this experiment, we use the more complex auto-regressive prior model. Other experiment details remain the same as before. The results of this experiment are presented in [ For both decoders, as expected, the sample quality drops for smaller bandwidth. However, the model with auxiliary latent variables significantly outperforms the one without across the full range of bandwidth presented here. We thus conclude auxiliary latent variable decoders can significantly improve the quality of communicated messages in some respects, and therefore encourage their continued exploration.
7 DISCUSSION
In this paper, we derived a generative model for joint coding with the bandwidth-limiter channel and showed how to perform learning based on variational inference. For this, we introduced differentiable and efficient model of the channel. Since vack-propagation through the channel is now possible, we demonstrate how we can learn flexible function approximators for coding by Monte Carlo sampling.
To justify the usage of joint coding instead of channel coding, first compared joint with separate communication models. Joint models were shown to consistently and significantly outperform their separate counterparts. Given joint coding channel transfers little or variable amounts of information, the decoder might be helped by under- standing the source distribution. We put this idea into practice by focusing on two modelling choices. First, when there is no information transferred, the decoder may draw a sample from the encoding distribution Py (Z.) to get source-typical encoding. We test how the complexity of the distribution model influences reconstruction performance. We find the more complex model to improve the distortion especially in the low transmission regime. Second, when sampling message reconstructions from the communication system, missing information leads to averaged reconstructions (i.e. blurry images). We prevent this by introducing auxiliary latent variable decoders. In experiments, we show that these decoders improve message reconstruction considerably in terms FID score.
basis, we investigate our main hypothesis that when
Further, this models serves as a simple method to learn a latent encoding that is sorted according to information content and channel noise, eliminating the need to pass the latent code through lossless compressor before transmitting the data. This is an essential property for sequential information transfer. In future work, we want to explore this aspect more extensively. Future efforts in this field would focus on reinforcing our finding further by investigating the same hypothesis in other data domains and with other channels.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. In Advances in Neural Information Processing Systems, pp. 1141-1151. 2017.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. arXiv preprint 1rXiv:180..022 2018.
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy, Fixing broken elbo. arXiv preprint arXiv: 1.1111011144, 2017.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling images using a generalized normalization transformation. arXiv preprint arXiv:1511.0623 2015.
Johannes Ballé, Valero Laparra, and Ecro P. Simoncelli. End-to-end optimized image compression. CoRR, abs///11l........./.. 7,017044 2016.
Johannes Ballé, Nick Johnston, and David Minnen. Integer networks for data compression with latent-variable models. 2018a.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. CoRR, abs// 880.01146, 2018b.
Elwyn Berlekamp, Robert McEliece, and Henk Van Tilborg. On the inherent intractability of certain coding problems (corresp.). IEEE Transactions on Information Theory, 24(3):384-386, 1978.
Eirina Bourtsoulatze, David Burth Kurka, and Deniz Gündüz. Deep joint source-channel coding for wireless image transmission. IEEE Transactions on Cognitive Communications and Networking, 2019.
Sebastian Cammerer, Tobias Gruber, Jakob Hoydis, and Stephan ten Brink, Scaling deep learning- based decoding of polar codes via partitioning. In GLOOBEOMM 2017-2017 IEFF Global Commu- nications Conference, pp. 1-6. IEEE, 2017.
Kristy Choi, Kedar Tatwawadi, Aditya Grover, Tsachy Weissman, and Stefano Ermon. Neural joint source-channel coding. In Proceedings of the 36th International Conference Machine Learning, JCML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 1182-1192, 2019. URL httt::/proceedii: ngs .ml press / wwt.sppre/scccotoeoo c9t./hhoil...hmmm
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Imre Csiszar. Linear codes for sources and source networks: Error exponents, universal coding. IEEE
Transactions on Information Theory, 28(4):585-592, 1982.
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. arXiv preprint 4rXiv:1903.04933, 2019.
Sebastian Dörner, Sebastian Cammerer, Jakob Hoydis, and Stephan ten Brink. Deep learning based communication over the air. IEEE Journal of Selected Topics in Signal Processing, 12(1):132-143. 2017.
Nariman Farsad, Milind Rao, and Andrea Goldsmith. Deep learning for joint source-channel coding of text. In 2018 IEEE International Conference on Acouslics, Speech and Signul Processing (ICASSP), pp. 2326-2330. IEEE, 2018
Jon Feldman, Martin J Wainwright, and David R Karger. Using linear programming to decode binary linear codes. IEEE Transactions on Information Theory, 51(3):954-972, 2005.
Alex Graves, Jacob Menick, and Aaron van den Oord. Associative compression networks. arXiv preprint 1xXvv:884..024777 2018
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In NIPS, 2016.
10

---

Under review as a conference paper at ICLR 2020
Tobias Gruber, Sebastian Cammerer, Jakob Hoydis, and Stephan ten Brink. On deep learning-basee channel decoding. In 2017 51st Anmual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, volume 3, 2017.
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the description length of the weights. In in Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory. Citeseer, 1993.
Feng Jiang, Wen Tao, Shaohui Liu, Jie Ren, Xun Guo, and Debin Zhao. An end-to-end compression framework based on convolutional neural networks. IEEE Trunsactions on Circuits and Systems for Video Technology. PP:1-1, 08 2017. doi: 10.1 10.1109/CSSV.220.22734433
Hyeji Kim, Yihan Jiang. Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Deepcode: Feedback codes via deep learning. In Advances in Neural Information Processing Systems, pp. 9458-9466 2018a.
Hyeji Kim, Yihan Jiang, Ranvir Rana, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Communication algorithms via deep learning. arXiv preprint arXiv:/805.090) 00311, 2018b.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Ralf Koetter and Pascal O Vontobel. Graph-covers and iterative decoding of finite length codes. In Proc. 3rd Intern. Symp. on Turbo Codes and Related Topics, pp. 1-5. Citeseer, 2003.
Victoria Kostina and Sergio Verdú. Lossy joint source-channel coding in the finite blocklength regime. IEEE Transactions Information Theory, 59(5):2545-2555 2575, 2013.
Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao, and David Zhang. Learning convolutional networks for content- weiihted image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3214-3223, 2018.
Zihao Liu, Tao Liu, Wujie Wen, Lei Jiang, Jie Xu, Yanzhi Wang, and Gang Quan. Deepn-jpeg: deep neural network favorable jpeg- based image compression framework. In Proceedings of the 55th Anmual Design Automation Conference, pp. 18. ACM, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Siwei Ma. Xinfeng Zhang, Chuanmin Jía, Zhenghui Zhao, Shiqi Wang, and Shanshe Wanga. Image and video compression with neural networks: A review. IEEE Transactions on Circuits and Systems for Video Technology, 2019.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. CoRR, abs/161 1. 100712, 2016.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems, pp. 10771-10780, 2018
Andriy Mnih and Danilo Rezende. Variational inference for monte carlo objectives. arXiv preprint 1002::1602.06675, 2016.
Eliya Nachmani, Yair Be'ery, and David Burshtein. Learning to decode linear codes using deep learning. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 341-346. IEFF, 2016.
11

---

Under review as a conference paper at ICLR 2020
Randolph John Pilc. Coding theorems for discrete source-channel pairs. PhD thesis, Massachusetts Institute of Technology, 1967.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.06454. 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and
approximate inference in deep generative models. arXiv preprint arXiv: 1401.4082 2014.
Claude E Shannon. A mathematical theory of communication, part ii. Bell Syst. Tech. J., 27:623-656. 1948.
Yuming Shen, Li Liu, and Ling Shao. Unsupervised binary representation learning with deep variational networks. International Journal of Computer Vision, pp. 1-15, 2019.
Lueas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arXiv: 1o.iv::7703003355, 2017.
Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Towards image understanding from deep compression without decoding. arXiv preprint arXiv:1803.06131,, 2018.
James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding, arXiv preprint 17Xiv:1901.04866 2019.
Michael Tschannen, Eirikur Agustsson, and Mario Lucic. Deep generative models for distribution- preserving lossy compression. In NeurlPS, 2018
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 1(Dec):3371-3408 2010.
Paseal Vontobel and Ralf Koetter. On low-complexity linear- preear-progammiin decoding of ldpe codes. European transactions on telecommunications, 18(5):509-55) 20000
Ryan Zarcone, Dylan Paiton, Alex Anderson, Jesse Engel, HS Philip Wong, and Bruno Olshausen. Joint source-channel coding with neural networks for analog data compression and storage. In 2018 Data Compression Conference, pp. 147-156. IEEE, 2018.
Lei Zhou, Chunlei Cai, Yue Gao, Sanbao Su, and Junmin Wu. Variational autoencoder for low bit-rate image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2617-2620, 2018
12

---

Under review as a conference paper at ICLR 2020
Appendices
A
MESSAGE RECONSTRUCTIONS FOR THE BANDWIDTH-LIMITED CHANNEL
X
X(II)
)((Y))=)
X((X))=)
X(1Y))
X((Y))
Figure 6: First row: Message samples from the source distribution. Other rows from top to bottom: Samples of the reconstructed message at all considered bandwidths. The top row has least information.
B RATE-DISTORTION PERSPECTIVE
Originally, information theory would study how message can be communicated to receiver without errors. It is often more realistic scenario, though, to think of the receiver to tolerate a certain amount of distortion. Intuitively, the more we allow for distortion of message the smaller the number of bits we need to communicate. Rate-distortion theory is a major field of information theory that studies how these modifications to the original set-up effect fundamental theorems such as data compression or transmission. The limitations of the classical view become clear when considering continuous random variables. Continuous random variables require infinite precision to represent exactly. Hence it is not possible to send finite rate codes. Assume X to be the continuous random variable to be represented by X')). Say we are given R bits to send X. X'(X) than can take 2d values. The goal of rate-distortion coding is to distribute these 2 codepoints such that a minimal distortion, measured by a distortion function d,
noisy channel
d: X' X + R+
(10)
where source alphabet, is being achieved.
B. SOURCE CODING IN A RATE-DISTORTIE SENSE
Source coding in the context of rate-diston rate-distortion theory entails two steps: quantization X'(X and traditional source encoding Y (X')= = ( ). In both steps the goal is to keep the loss of information minimal given rate that shall be achieved, / (X;; Y) S R where
Y); = Ex(x,,) [log P(X, Y) log P(X)
(11)
is the mutual information. The goal is keep this bound tight. However, computing the mutual information is hard since we do not have access to the true data density. Following Alemi et al. (2017), instead find variational approximation,
H - D-D5 ((x
(12)
with
= Bp DS (X[('))
(13)
R = E (Y' - log (Y))),
(14)
13

---

Under review as a conference paper at ICLR 2020
where they introduce (Y) the variational approximation to P(Y) = (X This reestablishes that the data entropy bounds feasible (compression rate R. distortion D) pairs: (X) < R + D. This ties together with the result from optimal coding that the source entropy bounds the optimal code length. Hinton & Van Camp (1993) show that via bits-back coding this code length (rate) can actually be achieved. This argument has further been hardened by Townsend et al. 2019) who design actual compression algorithm in this manner.
B.2 JOINT
CODING IN A LATE-DISTORIE SENSE
In the previous section we discussed how relaxing the requirement to sending a message exactly, to sending a message under a certain distortion, effects source coding. The optimal code length would thus be R(D) bits/symbol rather than H (X) 2 D) in the error-free scenario. We can connect this information Shannon's 's channel coding theorem. We know that the channel capacity C restricts the number of bits that can be send. Thus there exists a solution for a maximum distortion communication system only if RDD C. When in the previous section R helps to describe the number of bits that represent a random variable X, we can similarly find variational approximation to (X, Z) the amount of bits representing X after passing the channel,
T = Ep(x)E(YIX) [EC(ziT) [log C(Z\Y) - log N(
(15)
where equivalent to the discussion in the previous section (2) is the variational approximation to P(Z) = (r00)001x)/(((11) We shall refer to T as the transmission rate.
C RELAXING THE BINARY CHANNEL
C.I RELAXING THE BERNOULLI: BINARY CONCRETE DISTRIBUTION
Learning of systems with stochastic nodes (x) in Machine Learning is often synonymous with optimizing an objective function L( Φ) Ex (xx) (x)))((X) w.r.t the parameters 0,6 via some gradient descent based scheme. The challenge lies in computing the parameters that belong to the stochastic node. A popular approach to this problem is the application of the so called reparameterization trick (Kingma & Welling 2013 Rezende et al.) (2044) in which a stochastic node Po with parameter dependency ( is turned into stochastic node (2) without parameter dependentcy and a determined function g+(-).
[(0, )) = Ex Exx (XX) = Ez~Q(z)(e((so(2)
(16)
with x = go (z). The remodelled stochastic node now allows for gradient based stochastic optimiza- tion via Monte Carlo sampling
VgL(0, 6) = (B2))))
(17)
For example, consider sampling from the Gaussian distribution, ~V((X σ) can be replaced by sampling from a standard Gaussian Z ~ (2(200, 0, 1) and applying x = 9(m.o) (2) = t Reparam- eterizing a discrete distribution such as the Bernoulli B(p) is not as straight forward. Maddison et al. (2016) propose reparameterization with Gumbel-Max trick. Specifically, the reparameterization is based on a logistic random variable L ~ Logistic(L) (L) as parameter-fres stochastic node, the relaxed Bernoulli sample can than be attained by
Y= (L+
(18)
X x=o(Y)
where a corresponds to the location parameter p in the Bernoulli distribution, T the temperature adjusts the amount of relaxiation and is the sigmoid function. The density corresponding this sampling procedure is given by,
Pa. (x) = Tox (T-
(19)
(a--T(1-=)-1
We shall call this the Binary Concrete distribution or relaxed Bernoulli distribution (a)). It has several desirable properties.
'If there is no possibility of reparametization, one can retain to the score-function estimator, also known as REINFORCE or fikelihood-ratio estimator, which allows to compute the gradient via Monte Carlo sampling. This however leads higher variance gradients.
14

---

Under review as a conference paper at ICLR 2020
1. P(X > 0.5)
2. P(lim =1 = Ita
T-0
3. If S than Pa.r (x) ) is log-convex in 2.
One problem, however, we may often be faced with is computing the log-likelihooe of such a stochastic node. For example when computing the KL--divergence of variational auto-encoder. Due to the saturation of the sigmoid function computing the log-likelihood empirically may lead to underflow issues. This why it has been proposed compute the log-likelihood based on the samples Y before applying the sigmoid function since this is an inevitable function. The corresponding log- likelihood is given by,
log 9a, (y) = log - Ty + log a 2log(1 + exp( + log a)).
(20)
As an alternative we may clip the log- lggliikliioodd This variant is easier to apply when there is no direct access to the stochastic node, we shall see what this means precisely in the next section.
C.2 BINARY CHANNEL
The symmetric binary channel is discrete channel with input and output alphahet of size 2, Y' E 10, z E {0, 1)D. The channel can be realized with Bernoulli noise on each input pixel,
(2W 1)
7 = 2. (2Y, 1) +
W. ~ Pw (W,)) = B(W.)p)
(21)
where B(p) is a Bernoulli with the likelihood of keeping an input bit. The channel is called a symmetric channel because the probability of changing bit does not depend on its state.
Following, we relax the channel as defined above to allow for training of differntiable communication models. For this we will utilize the relaxed Bernoulli distribution described in the previous section. As for the Gaussian Channel we assume that that Y; can be constructed from a learned Bernoulli itself with Y, Bry, (Y.lay.)- In order compute equation & we need to evaluate the channel density given its input C(Z(Y). Since Z depends deterministicalll on W; and Y, the channel density equals the noise density with transformed input argument C(ZlY) = Pw ((22 1)(2Y / + 1). For Bernoulli noise, such as the original channel formulation, the system could thus not learn. We thus propose to also adapt the noise to be relaxed Bernoulli with probability density as in Eq. (19).
Finally, we may restrict our channel to specific SNR. This can be computed to
2ppy, + 0.5 p- PY,
(22)
-2pr,p 0.5 p p-py
To train neural models, we will use the relaxed Bernoulli for both Y and W, however the SNR is computed assuming both as Bernoulli distributions. This assumption only correct for Initial experimental results, with the relaxed binary channel have been showing that optimization with this parameterisation somewhat challenging. Our finding supports a finding Choi et al. (2099) that focus on VIMCO instead of the re- re-parameterisation trick.
D SENSITIVITY OF THE COMMUNICATION SYSTEMS TO THE B
In optimizing communication systems, perhaps the most important hyper parameter. This is why present the complete set of results for experiment one in Figure For the source VAE 3 trades compression rate vs distortion. At maximum compression, the channel source distribution would be emulated perfectly and thus the channel AE input distribution. However, this scenario would also eliminate the mutual information between X and Y. Thus a balance must found by tuning B.
15

---

Under review as a conference paper at ICLR 2020
10
10
Signal-to-Noiser www.-ooo--oseeRliii Ratio
Signal-too to Noise Ratio Ratio
10
Distortion
-10
-10
10
10
10a
10
10
10
10
10
10a
10
10
10
10 Beta
109
10
10a
Beta
(a) Joint Model
(b) Separate Model
Figure 7: The results in Figure ?? show the distortion the SNR for optimized B. Here present all results. Note that, the joint model is more sensitive to changes in 3 in the range we have chosen.
16


Reviews:
[{"review_id": "ryeWu1pp9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is out of my research area. I could understand that the paper studies the message transformation with bandwidth-limited channels. It seems naturally the message transformation could be represented as a autoencoder model. The paper proposed variational model for this problem and it seems to me the paper employs the popular models in neural networks for example VAE, etc. Technically, what's new of this paper? Was it the auxiliary variable decoders? Is it that this class of algorithms/models firstly applied to this problem domain? To be honest the paper mentioned most of the terminologies in ML and seems that the paper wanted to connect to them, for example, ELBO, VAE, GAN, re-parameterization, etc. The paper provides experimental results on the designed model for bandwidth-limited channel.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "H1gA7XOEhS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPaper Summary:\n\nThe paper proposes to use ML methods, specifically neural networks, to learn source and/or channel coding systems, either jointly or separately. Specifically, they investigate these systems under the bandwidth-limited channel.  They investigate their models applied to the task of transferring images across the channel.\n\nPros:\n\n- The paper is in a difficult area, and needs to communicate ideas about communications and information as well as various deep learning models. The initial exposition does this well.\n\n- The paper discusses ML-based communications models applied to the bandwidth limited channel and performs experiments to investigate joint vs. separate coding. \n\nTwo main things of concern:\n\n- This area (communications, compression, coding) is highly developed, yet there are no comparisons to practical, state of the art techniques in this paper. The experiments investigate using the authors' models to send images across a noisy channel. So shouldn't we see a comparison to an equivalent non-ML-based system that currently accomplishes this task, if only for perspective? \n\n- The paper contains too much exposition and the structure makes it difficult to read.  The authors' work seems to mostly be focused in sections 4 and 6, but prior work is summarized in section 5.  The authors want to communicate a lot of ideas which the audience might not be familiar with, and this is a difficult task. However, the paper could be reorganized such that the ideas are clearer. Rate distortion is delegated to an appendix, but two pages are spent on basic communications. Overall it could be much more focused. \n\nSome smaller concerns:\n\n- Pg. 3 \"hypothesis spaces can be searched increasingly quickly in an automated fashion, allowing researchers to search...\" \nI think this statement may be over-reaching. The authors also frequently use the term \"flexible function approximation\". Neural networks do have interesting approximation properties. But this is not a complete picture of deep learning, and the situation is much more nuanced than this. Where is the role of optimization and data? In order to transmit data with the author's algorithms, do we need to go out and collect a large body of examples in the specific domain, like CelebA? Because you don't need to do that to e.g. compress any image with JPEG, code the data using LDPC to send across a channel. Is this comparing apples and oranges?\n\n- Pg. 6 \"Note, that a simple additive white Gaussian noise.... LDPC. However, in more general scenarios they do not perform as well and can be beaten by neural network architectures.....\"\n\nI think the claims in this paragraph need to be toned down a bit. LDPC does not perform well in regards to what? Can be beaten under what conditions? Decode efficiently with regard to what block length? I don't think the picture is as clear as painted here.\n\n- Pg. 4 The statements here regarding the bandwidth-limited channel appear to be the focus of the author's work. This section should be expanded and explained. Reading the first paragraph then going to the second paragraph (\"To summarize...\"), there's sort of a disconnect. How do we know these two things are equivalent? What exactly is novel that was introduced?\n\nSmall typos:\n- Pg. 2 final paragraph, some  norm is used for the distortion, but this is not defined (either on pg. 2 or in App. B)\n- Pg. 6 white is misspelled \"withe\"\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g4UQY3Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on transmitting messages reliably by learning joint coding with the bandwidth-limited channel. The authors justify joint systems outperform their separate counterparts when coding is performed by flexible learnable function approximators. Their experiments show the advantage of their design decisions via improved distoration and FID scores.\n\nPros:\n\n1. This paper is clearly written and well-structured in logic. For example, the authors use Figures 1 and 2 assist readers to catch the difference between joint communication system and separate communication system.\n\n2. This paper gives a reliazation of joint source-channel coding, especially to give auxilary latent variable decoders.\n\n3. This paper has been verified in both Gaussian channel and bandwidth-limited channel. The empirical results show the advantage of joint coding.\n\nCons:\n\n1. Intuitively, you Section 4.3 should be better than Section 4.2. However, I don't see any difference or major items to justify this kind of benefits. Could you please explain why techniques in Section 4.3 can outperform these in Section 4.2.\n\n2. Although the authors verified their work on CelebA, it seems that the proposed method has very limited applications. If possible, the authors should do more datasets to verify their proposed method, which will be more useful to boarder readers.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was some support for this paper, but it was on the borderline and significant concerns were raised. It did not compare to the exiting related literature on communications, compression, and coding. There were significant issues with clarity.
Decision: Reject