Question: Is the adaptation method applicable to other e2e auto encoder models, e.g., GAN or VAE. In another words, is there any specific advantage of selecting MDN?
Evidence: Consider the same problem setup as § 3, where we observe a small dataset of samples from the target channel distribution. Suppose we have adapted the MDN channel by optimizing over the parameters ψ, we can use the inverse-affine feature transformations (defined in Eq. ()) to transform the channel output y from a component of the target Gaussian mixture to the same component of the source Gaussian mixture. However, this transformation requires knowledge of both the channel input x and the mixture component i, which are not observed (latent) at the decoder. We address this by first determining the most-probable pair of channel input and mixture component for a given y (using the MAP rule), and applying the corresponding inverse-affine feature transformation as
The joint posterior over the channel input x and mixture component i, given the channel output y is based on the adapted (target) Gaussian mixture, given by
.
The adapted decoder based on the above affine feature transformation (see Fig.) is defined as  In the previous method, an input transformation layer is introduced at the decoder only during adaptation, but not during training of the autoencoder. Alternatively, we propose an input transformation layer at the decoder that takes the channel output y and produces a best estimate of the encoded symbol x, which is then given as input to the decoder as shown in Fig.. This input transformation layer is included during the autoencoder training as a fixed non-linear transformation that does not have any trainable parameters. Since the decoder is trained to predict using x instead of y, it is inherently robust to changes in the distribution of y.
Given a generative model of the channel conditional density using Gaussian mixtures, we can estimate the plug-in Bayes posterior distribution of x given y, P θc (x | y) (ref.
Eq. ()). From this, we can find the MAP estimate of x given y as
The adapted decoder based on this input transformation, referred to as the MAP symbol estimation (SE) layer, is defined as
and illustrated in Fig.. Whenever the MDN model is adapted to changes in the channel distribution, resulting in a new MDN with parameters θ c , the MAP SE layer is also updated using θ c . This input transformation shields the decoder from changes to the distribution of the channel output y.
Since the MAP SE layer is also included in the autoencoder during training, the non-differentiable argmax function presents an obstacle to training the autoencoder using backpropagation. We address this by using a temperature-scaled softmax approximation to the argmax, which is differentiable and provides a close approximation for small temperature values. This approximation is used only during training, whereas the exact argmax is used during inference. Details on this approximation, and a modified autoencoder training algorithm with temperature annealing are discussed in Appendix C.5.
Comments. The proposed input transformation methods at the decoder have some similarities to equalization methods used in communication receivers. However, our problem setting considers a memoryless channel, and does not deal with intersymbol interference (ISI), which is the main focus of equalization methods. A key advantage of the proposed adaptation is that it is very computationally efficient to implement at the receiver of a communication system. A discussion of the computational complexity of the proposed methods is given in Appendix C.4.
Suppose we have adapted the MDN channel by optimizing over the parameters ψ, we can use the inverse-affine feature transformations (defined in Eq. (4)) to transform the channel output y from a component of the target Gaussian mixture to the same component of the source Gaussian mixture.
Answer: Just to clarify, the choice of a GAN, VAE, or MDN pertains only to the generative model of the channel, and not the end-to-end autoencoder. The proposed method is specific to a mixture density network channel model. Adaptation of other generative models such as GAN and VAE using a small number of samples is an interesting but challenging problem. The choice of MDN as the channel model for this problem comes with the following advantages:
The conditional density corresponding to the MDN is a mixture of Gaussians, whose parametric form is easy to adapt using symbol and component-conditional affine transformations. The number of affine transformation parameters is much smaller than the number of MDN parameters, making the adaptation less susceptible to overfitting on the small dataset from the target distribution.
The MDN choice makes it possible to derive a closed-form KL-divergence between the source and target Gaussian mixtures under some mild assumptions. This is used as a regularization term in the adaptation objective, which allows to directly constrain the change in distribution.
The affine transformation property between the mixture of Gaussians allows us to design an input transformation method at the decoder (Sec 4.1) that can compensate for changes in the distribution of the channel output. This same property also allows us to propose a validation metric that is used for selecting the hyper-parameter lambda (Appendix C.3).