Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for domain adaptation, where we have access to only a small number of labeled target samples and aim to adapt a model with a small training cost. They propose a fast and light-weight method for adapting a Gaussian mixture density network
(MDN). Specifically, they transform the parameters of source Gaussian mixture models to those of the target domain by applying affine transformations.  They make the assumption of one-to-one pairing between the components of the mixture models. To optimize a model, they regularize training by the Kullback-Leibler divergence between two paired mixture models. This method is specialized for e2e learning for a communication system using an auto-encoder.  They evaluate the performance of the model on several adaptation scenarios in wireless communication and show a good advantage over baselines when enough target samples are not available. 
main_review: *Strong Points*
1. Their proposed method is simple, but shows solid gain over baselines when target samples are limited. Applying affine transformation to get the parameters of the Gaussian mixture model in the target domain is a convincing approach. And, their simple approach is showing good gains in several cases. 
2. Judging from their description of the related work, this problem is not much studied in the wireless communication area. Exploring a new and practical scenario is a strong point in this paper. 

*Weak Points*
1. The method is basically specialized for wireless communication with an autoencoder. Although the first several sentences of the abstract seem to describe more general scenarios, their methods are tailored for the wireless communication experiments, and unclear whether the proposed method is widely applicable in other recognition problems, e.g., image classification and sentiment analysis. 
2. It is hard to judge the novelty of the proposed method since their baseline is very naive. There is no baseline borrowed from the literature of few-shot classification or domain adaptation work for limited target samples, e.g., "Few-Shot Adversarial Domain Adaptation, Neurips 2017". I understand that it may be hard to directly borrow such baselines since they are not designed for wireless communication. 
3. Related to 2, they are not covering several few-shot domain adaptation works, such as  "Few-Shot Adversarial Domain Adaptation, Neurips 2017". Their work is not the first to study the problem. It was confusing. 

summary_of_the_review: From the weakness I described above, I recommend this paper below the acceptance threshold. The work may have high importance in the application to wireless communication, but lacks discussions related to existing domain adaptation work and comparison to them.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose an MDN based adaptation model to e2e communication system. Specifically, it transforms the inputs to the decoder such that their class-conditional distributions are close to that of the source domain. Experiments are done on both synthetic and real-world datasets. 
main_review: Strength:
(1)	The paper proposes a new MDN based adaptation model for e2e communication system. 

(2)	The proposed method is verified by both synthetic and real-world datasets.

Here are some detailed comments:

(1)	Is the adaptation method applicable to other e2e autoencoder models, e.g., GAN or VAE. In another words, is there any specific advantage of selecting MDN?

(2)	Related works on transfer learning miss important references on semi-supervised domain adaptation, e.g., [ref1] and [ref2]. Please also find more semi-supervised transfer methods online. 

[ref1] Adaptive Knowledge Transfer based on Transfer Neural Kernel Network

[ref2] Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

(3)	Do you also assume the marginal distribution of x, (or s) is also different between the source and target domains?

(4)	Diagonal covariance implies the inputs are independent, which is a quite strong assumption, it is necessary to work on the general covariance case.

(5)	It is unclear how to ensure $\mathbf{C}_i$ to be diagonal. In fact, I think it is infeasible to make $\mathbf{C}_i$ diagonal according to the derivation in appendix C.1. 

(6)	What’s difference between $\hat{\pi_i}$ and $\hat{\alpha_i}$.

(7)	The objective eq.(6) calculates the KL-divergence of two gaussian mixtures. This requires the data size of the source and target to be the same. Is the source data size reduced to match the small number of target data points. In this way, it is not very convincing to estimate the true conditional probability using limited data points.

(8)	The comparison methods lack the state-of-the-art semi-supervised domain adaptation methods, which is less convincing. 

(9)	Experiments on simulated and real-world datasets are imbalanced. More real-world experiments are expected.  Moreover, the current setting of section 5.3 is not very clear. How is the transfer setting formulated, i.e., please give the source and target clearly. 


summary_of_the_review: Overall, the paper needs further improvements to reach ICLR standard. See detailed comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an end-to-end learning framework for communication systems using autoencoders. The basic idea is to only retrain a small adaptation layer between the MDN channel model and the decoder. Compared to retraining the autoencoder or the entire MDN channel model, the proposed solution runs faster and fits the dynamic communication channel environment better.
main_review: 1. The authors only mentioned adapting the MDN model in the abstract and introduction. Readers will be confused why it can be faster than retraining the autoencoder. They revealed that they only train a very small adaptation layer rather than the MDN model on page 3. In my opinion, they should put figure 1 on page 1 to avoid confusion about the speed.

2. When discussing the shortcoming of existing solutions, the author mentioned that the rapidly changing channel leaves not enough time to collect the data to train the autoencoder. But they have the same problem here, they only have a tiny dataset to train the adaptation layer, why overfitting will not happen if they only have such a small dataset?

3. Where does the small labeled dataset come from, and how often to recollect such a dataset and re-run the adaptation?  As the authors mentioned, the channel is changing rapidly, if the channel has changed, the adaptation layer needs to be updated. But collecting the dataset too frequently will leave not enough time to transmit actual data. How to solve this problem?

4. The regularization constant lambda needs to be set carefully for a given pair of source and target distributions. The dependency of lambda choice on the source-target distribution pair makes the proposed solution very impractical. Why not use a "dynamic'' lambda like in ADMM? Is it because a fixed lambda converges faster?

5. No benchmark comparison. There are existing solutions that retrain the entire MDN. The authors should compare with them to show that retraining a small adaption layer does not cause too much accuracy loss compared to retraining the entire MDN.
summary_of_the_review: Overall, the proposed idea looks OK. But it is unclear why training with such a small dataset will not lead to overfitting. Also, the overhead of recollecting the labeled dataset and retraining under a rapidly changing channel is not discussed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers an application of deep-learning to communication systems. Due to the nature of deep neural networks that always learn to work on training distribution, the current deep-learning based methods often perform badly when there is a domain shift between training and testing data. And there is no surprise that this problem occurs in communication systems using neural networks. The authors propose  a fast and light-weight method for adapting a Gaussian mixture density network (MDN) using only a small set of target domain samples. This method is well-suited for the communication setting, where the distribution of target data changes rapidly (e.g., a wireless channel), making it challenging to collect a large number of samples and retrain. The authors propose a method for adapting the auto-encoder without modifying the encoder and decoder neural networks, and adapting only the MDN model of the channel. The method utilizes feature transformations at the decoder to compensate for changes in the channel distribution, and effectively present to the decoder samples close to the source distribution. Some experimental results are provided to evaluate the proposed approach.
main_review: Strengths

The need for domain adaptation in communication systems is practically important and the paper does a good job motivating it. The paper is reasonably well written and provides a good background on applications to wireless communications.

Weaknesses
- On the theory side: the proposed domain adaptation method involves maximizing a regularized log-likelihood and is optimized using known methods such as BFGS quasi-newton methods. 
- On the application side:  the block error rates of 0.3 etc in Fig. 5 are too high to be of practical interest in communications applications. I would argue that none of the schemes presented are practical for the models of interest.
- The authors notes that for AWGN to Ricean fading, the performance of the adapted network is sometimes worse than the baseline. The authors’ explanation for this phenomenon is that the “distribution of the two domains are not very different. ”. However, if the distribution of the two domains are close, we should expect the performance of the adapted system stays the same as or slightly better than the baseline. Furthermore it should be noted that the AWGN to Ricean fading is of practical interest as both models involve a dominant line-of-sight component. 

summary_of_the_review: The paper has significant weaknesses in the proposed applications side and has limited novelty on the algorithmic side. I therefore do not recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addressed the problem of domain adaptation when the target domain has only few labeled samples. It proposes to use an MDN with Gaussian components to represent the source distribution and then adapt the MDN parameters to match the target distribution to the source distribution.
This approach is adopted to address the problem of adapting a communication autoencoder (trained on the source distribution) to samples drawn from the target distribution without worsening the error rate of the autoencoder. The paper includes results on simulated and real data.

main_review: The addressed problem is of practical relevance and the authors carry out experiments on both real and simulated data, with good results.
However, I have some reservations.

What is the focus of the paper? Is it the domain adaptation approach or the specific problem of domain adaptation (DA) of autoencoders using an MDN as the channel model. I am asking because from the abstract it seems that the main focus is on the adaptation technique and the autoencoder case is just an application examples, while from the introduction section it seems that the main focus is on the autoencoder. 

The authors assume that the source and target distribution have the same number of components with a one-to-one correspondence. How realistic is the working hypothesis? In the appendix the authors report the results if the number of components is mis-matched. I think it should be included in the main body of the paper instead.
Also, how realistic is the working hypothesis that labelled samples from the target domain an be collected before the channel changes again? Wireless channels can vary very rapidly, so I presume that the proposed approach applies only relatively stable channels.

It would be interesting to include cases in which the domain adaptation fails. I presume that if the channel changes significantly then the proposed adaptation might fail.

In (section 4.1), the authors adopt the MAP rule to determine the channel input and the Gaussian component. How complex is the MAP rule? I presume that if the number of Gaussian components and the constellation size become large, it might be an issue. (The same question applies to section 4.2 in which the MAP rule is applied as well). I am not sure if in Appendix C.4 the authors included the complexity of MAP rule. I think they should as it is part of the overall approach.

I am confused about the way results are reported in Table 1. The authors report the estimated conditional likelihood p(x/y) estimate by the various methods, but what is the true p(x/y)? How to evaluate if the estimated conditional likelihood is close to the true one? Also, it seems that for large number of samples the “Transfer-last-layer” outperforms the proposed approach. Why is this the case?

Why in Fig 4 and 5 the 16 QAM performs less than the no adaptation?

Under “observations and takeaways” the authors state that “Between the two adaptation methods, MAP SE performs marginally better than the Affine method.” Which one of the two would the author recommend? Are they equivalent in terms of complexity?

Since the authors claim that the proposed approach is fast and light-weight, the authors should include a study of the complexity analysis in the main body of the paper (I suggest move it there from the appendix), including also the training and running times of the proposed approach. I see that they include the number of trainable parameters, which is very important, but it’s only partial information.

The baselines included in the numerical results are very similar to the proposed approach. It would be interesting if the authors could include other methods as well, maybe even more complex, to analyze the trade-off between performance and complexity offered by  the proposed approach.

Maybe the authors could include a future work section in the conclusion (if they assume to continue research on this topic) and discuss the generality of this approach. What are other possible application areas?

summary_of_the_review: The addressed problem is of interest and the authors test the proposed technique both on simulate and real data. However, the main technical contribution of the paper relies on the well-known property of Gaussian distributions (hence the paper provides only marginal novelty) and the two working hypothesis (same number of components between the two Gaussian distributions and a one-to-one
correspondence between the components) are not properly discussed from a practical viewpoint. Finally, considering the fact that the authors promote their approach because it is fast and light-weight, a proper complexity analysis is missing in the paper (the one in the appendix is partial and does not seem to include the cost of the MAP rule).
review_title: null

Meta-review: The paper proposes a domain adaptation method that is specific to auto-encoder and wireless domain. The proposed method shows solid gain over baseline and is simple. There were multiple complaints on reviewer's side regarding clarity of the abstract and application of the work and related work that was responded to by the authors during the rebuttal period. However, the modifications were not enough to address all concerns. Mainly, the assumptions for the method to work such as source and target having the same number of components and diagonal covariance. It would help the paper to discuss the cases where the model fails. In addition, the paper will benefit from a stronger baseline.
Decision: Reject