Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: - The paper addresses the problem of handling domain-shifts that arises in generative learnt channel models in E2E communication systems in a few-shot setting.
- The proposed domain adaptation approach is tailored around a Mixture Density Network (MDN) representing the channel model. In here, the approach:
    - learns an adapter layer, which models an affine transform of the original conditional channel distribution
    - introduces an additional regularization objective to ensure the adapter doesn't converge to bad/degenerate solutions
    - presents a feature transformation formulation on the decoder side to aid learning on the domain-shifted distributions
- The approach is evaluated extensively, covering multiple types of distribution changes in both synthethic settings as well on a high-resolution mmWave testbed.
strength_and_weaknesses: ### Strengths

**1. Extensive evaluation**
- The approach is evaluated rigorously with well-suited baselines and a range of scenarios (e.g., multiple types of domain shifts, real-world evaluation). I especially appreciate evaluations studying when the (reasonable) assumptions are violated.

**2. Motivation and relevant problem**
- While there has been a lot of attention on generative channel modelling recently, most works in my knowledge largely (and somewhat incorrectly) assume a stationary distribution. This paper takes a step in the right direction by addressing this pain-point.

**3. Insightful approach**
- The approach overall is insightful and makes sense. By learning an adapter network and learning parameters relevant for the domain shifts (e.g., like FiLM modules), it makes few-shot domain-adaptation more tractable.
- Furthermore, I find the choice of the channel model representation (MDNs) to also be sufficiently appropriate for the task (as opposed to GANs) for this study. 

### Concerns

**1. "labeled set obtained for free"**
- The paper at multiple times claims that few-shot learning is especially possible since we can get labeled dataset for free -- I find this slightly confusing.
- Wouldn't the labeled dataset be split between the encoder (transmitter) and decoder (receiver) devices? As a result, for a party to have the full labeled dataset, isn't a prerequisite communicating labels back to the other party?

**2. Evaluation: Some observations unclear**
- I found some patterns in the evaluation was somewhat unclear and would appreciate the authors' answers on the questions below:
- (a) Oracle-approach gap in Figure 4/5: I'm slightly surprised that proposed approach's symbol error rate does not converge to the oracle with a reasonable number of additional examples (50 * 16-QAM classes = 800), given that there are 50 learnable parameters. Are the authors aware if convergence is possible with even higher examples? Morevover, what is the size of the source dataset?
- (b) Unchanged error rates in Figure 4/5 for many baselines: Are the authors aware of why the error rates of many baselines do not improve at all in spite of more training examples? Were the "finetune" baselines finedtuned only on the new data or a combination? In the case of combination, are domain-invariant features learnt? 
- (nitpick) Please summarize the performance degradation discussions in Ricean fading experiments in the main paper.

**3. Evaluation: Performance under no distribution change**
- I appreciate that the authors also evaluate under a non-domain shifted dataset in Figure 10. Can the authors clarify why results drop in performance when there is no distribution change?
- Specifically, it appears that the adapter layers' parameters are initialized such that it produces a identity mapping (page 18), so I'm surprised that this nonetheless degrades performance.

**4. SNR=14-20 dB**
- Can the authors comment whether a SNR of 14-20dB (which to me appears really large) is a reasonable setting? Did the authors also evaluate SNR vs. error rates for the approach and baselines? I wonder if the results shown here apply only in high SNR regimes.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Good. It was generally easy reading the paper, thanks to really crisp text and a comprehensive background section. The minor issue I found is that some patterns in the results are not discussed (see concern 2, 3) The only nitpick I have are the figures (esp. Figures 4-6) where legends are highly illegible.

**Quality**: Good. While there are minor discrepancies the approach (e.g., performance slightly deteriorates when there is no distribution change, does not translate well to certain distribution changes), I think it can be overlooked in light of the remaining contributions.

**Novelty**: Very good. The authors tackle a very well motivated problem (see strength 2) and propose an insightful approach to tackle it (see strength 3).

**Reproducibility**: Very good. The main paper (esp. the large appendix) appears to contain many details of the approach. Additionally, the code is provided as well. I'm not sure if the authors plan to release the channels from the mmWave FPGA testbed.
summary_of_the_review: The paper tackles a relevant bottleneck in generative channel modelling for E2E communication systems (i.e., they are trained assuming a stationary distribution, but this isn't the typical case). The approach is novel and intuitive in my opinion, and is further evaluated extensively in both simulated and real conditions. While I have some minor concerns (e.g., can one really have a labelled dataset for this task in practise?), I don't think they significantly affect the paper's claims and contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper models the changes of channel in a communication system as a few-shot domain adaptation problem. They employ the Gaussian mixture density network to specifically model the channel and propose a transformation to compensate for changes in the channel distribution. They perform experiments both on simulated channel distributions and FPGA.
strength_and_weaknesses: Pros:

1. This paper considers the frequent change of channel in communication systems. They treat the change of channel as distribution shift and link this practical problem with few-shot domain adaptation. I think this is novel and advanced enough in the field of communication.

2. The proposed solution is easy to follow and can be used in more general few-shot DA scenarios. In addition, this method has better real-time performance compared with previous works.

Cons:

1. This paper lacks the few-shot domain adaptation methods as baselines, e.g., [1]. Current baselines are all the basic FDA solutions, and I worry about their competitiveness. 

2. The evaluation metric they use is only the SER. As the application research article, the performance of the proposed method in practical communication problems is essential. They need to show the advantages of their learning-based method over conventional methods.

3. I find that the number of target data per class is more than 10 in this paper, and maybe such an amount is beyond the scale of few-shot learning. Additional experiments with less than 7 samples per class are important.

[1] Mottian et al. Few-Shot Adversarial Domain Adaptation. NeurIPS, 2017.

clarity,_quality,_novelty_and_reproducibility: These are all seem good. The presentation is clear, and the writing quality is above the bar. For the communication field, I think the novelty is sufficient. They also provide the source code in supplementary materials.
summary_of_the_review: This paper aims to address the varying channels in communication with the help of few-shot DA. Their solution is simple and easy to follow, and the corresponding theoretical and empirical analysis are thorough. However, some experimental settings (see above) are need to be further optimized.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors provide a few-shot domain adaptation method to address the channel distribution changes of communication systems. Specifically, using the properties of Gaussian mixtures, they propose a solid domain adaption process for the generative channel model (MDN). Besides, they propose a input-transformation method which transform the input of decoder from target domain into source domain, without modifying the encoder-decoder networks. They also derive experiments on a mmWave FPGA platform and show the strong performance improvements of the proposed method. 
strength_and_weaknesses: Strength
1.	The paper is well organized and has rich details.
2.	The advantage of the proposed method is clearly stated and demonstrated.
3.	The adaptation approach is based on appropriate assumptions and is well supported by the properties of Gaussian mixtures.
4.	The effectiveness of the method is evaluated by both simulated and real experiments, and there are also experiments when the assumptions could not hold.

Weaknesses
1.	Some confusions.
In Parameter Transformation part, you state that “The number of adaptation parameters is given by k (2 d2 + d + 2). This is typically much smaller than the number of MDN parameters (weights and biases from all layers)”. 
In previous part you state that “The MDN output with all the mixture parameters has dimension p = k (d(d + 1)/2 + d + 1).” 
Why the adaptation parameters is much smaller than the number of MDN parameters?
2.	Some figures are not self-explanatory. For instance, in Figure 4, the line of No adapt or Finetune are covered by other lines, without additional explanation. 
3.	More experiments. How the unsupervised domain adaptation performs based on the baseline model and how it compares with the proposed approach?

clarity,_quality,_novelty_and_reproducibility: The paper is well organized and clearly stated. The advantage of the method is well stated.
summary_of_the_review: The paper is well organized and has rich details. The work is based on appropriate assumptions and the properties of Gaussian mixtures, and the effectiveness is demonstrated by experiments on both simulated and real experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I Summary:

I.1 Investigated Problem:
- The authors investigate the frequent changes of channels in communication systems and treat these changes as shifts in distribution (domain). The framework of few-shot domain adaptation is then considered to tackle the problem in question. 

I.2 Proposed Solution:
- Leveraging the properties of Gaussian mixtures, the proposed domain adaptation approach is tailored around a Mixture Density Network (MDN) representing the channel distribution. 
- A regularized, parameter-efficient adaptation of the MDN using a set of affine transformations is proposed.
The learned affine transformations are then used to design an optimal transformation at the decoder input to compensate for the distribution shift, and effectively present to the decoder inputs close to the source distribution.

I.3 Validity Proof of the Proposed Solution:
- An extensive evaluation is conducted, covering multiple types of distribution changes in both synthetic settings as well on a high-resolution mmWave testbed.

II Strengths:

- II.1 From a structural point of view:
    - The paper is well-written. The method is clearly presented and ideas are easy to follow. 

- II.2 From an analytical point of view: 
    - The approach is:
        - novel;
        - evaluated rigorously;
        - compared with well-suited baselines;
        - considers a range of scenarios with different types of domain shifts, real-world as well as synthetic evaluations;
        - takes a step in the right direction by considering the more realistic assumption of the non-stationary distribution of a channel in communication systems.

- II.3 From a perspective of soundness (development, unity, and coherence) and completeness (correctness):
    -  The strength points mentioned above are sufficient evidence of the soundness and completeness of the paper. The transparency aspect of the submission is also appreciated as open-source code is provided for reproducibility purposes and other details are contained in the appendix.
 
III Addressing what can be thought of as weaknesses:

- The authors tried to address all the concerns raised by the reviewers (all of them were minor). It is worth mentioning that concern about the experimental setting was raised as a state-of-the-art few-shot adversarial domain adaptation method (FADA [1]) was not considered in the comparison conducted with existing baselines. The authors provide an explanation for the reason why FADA was not included in the comparison as it does not provide a way of adapting the mixture density network which is an important component to keep track of changes in channel distribution.

- Reviewers also mentioned the necessity of a full evaluation of the proposed solution that takes into account metrics reflecting the practical aspect of the method. The authors agreed that other metrics such as the throughput and adaptation speed relative to the coherence time of the channel could be considered as part of a practical communication system. Yet, this would require a significant systems-level effort, which can be left for future work.

IV Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community): The proposed solution has great potential to be of benefit to the whole community, especially researchers interested application of deep learning in communication systems.

V References:

- Motiian, Saeid, et al. "Few-shot adversarial domain adaptation." Advances in neural information processing systems (2017).

justification_for_why_not_higher_score: it would be unfair to the authors of the paper to state an arbitrary reason for not attributing a higher score. The paper has a significant potential to be impactful as it addresses a practical problem and provide an efficient solution by tackling an existing problem in communication systems with realistic assumption. 

- The paper received a very high grade (7.33) from reviewers with a decent confidence level (3.33) from reviewers. The following reason can be considered for not attributing the paper the mention of oral:
    - Not considering a full evaluation of the proposed solution which takes into account metrics reflecting its practical aspect (To be objective and fair, it is important to mention that this would require a significant systems-level effort, which can be left for future work as stated by the authors);
    - The method is presented as a solution to a communication system problem. From the title of the paper, one can think that it is targeting a narrower audience of researchers while the proposed method is generic (which is of benefit to a greater audience and should be considered as a main strong point).

Determining if a paper should receive an award or a mention in a venue according to some arbitrary criterion could be meaningless. In the history of machine learning, great impactful foundational works, such as Generative Adversarial Networks (Goodfellow et al, 2014), were not accepted as spotlights or orals and yet their long-term impact was the true reward that authors could reap. 
Other works presenting and proposing a solution designed to a very specific problem have been adopted later by a more general audience of researchers and applied to a broader spectrum of applications. To name a few, the UNet architecture (Ronneberger et al, 2015) was originally designed for biomedical image segmentations and has been adopted largely in the computer vision and speech community for different tasks.

References: 
- Goodfellow, Ian, et al. "Generative adversarial networks." (NeurIPS 2014).
- Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

justification_for_why_not_lower_score: Reviewers unanimously recommended acceptance of the paper and most reviewers agreed that the paper is marginally above the threshold. The paper has the potential to be impactful as it introduces the first differentiable neural ray tracer for wireless channel modelling and new datasets for benchmarking purposes are provided.
Decision: Accept: notable-top-25%