Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to address the posterior collapse issue that occurred during training sequence VAEs. To this end, the authors first provide theoretical support for the effectiveness of word-level Dropout regularization to mitigate this problem. Then, they propose a new non-uniform, data-dependent Dropout scheme by introducing the adversarial training strategy with proper regularization. Finally, the empirical effectiveness of the proposed method is demonstrated with three different datasets by comparing it with other sequence VAEs.


strengths_and_weaknesses: ### **Pros.**

- **Clarity**. Overall, the writing is clear and easy to follow. In addition, the organization of the main draft is well-established.
- **Well motivated problem and intuitive solution**. Resolving the posterior collapse issue is an important and interesting problem to solve. Also, the provided theoretical support and the proposed method are intuitive.

### **Cons.**

- **Insufficient empirical demonstration**. Although the clear advantage of the provided theoretical support and intuitive method, there are clear limitations in the current empirical results. 
  1. *Fairness with the hyper-parameter search*: as presented in lines 271-274, the authors conduct an extensive hyper-parameter search for the proposed method. However, such a hyper-parameter search is not conducted for the baselines, e.g., word Dropout with a fixed-rate 0.4. As the optimal hyper-parameter largely varies on each dataset, the authors should conduct the same hyper-parameter search procedure to baselines for a fair comparison. 
  2. *Limited improvement*: even though the adversarial training scheme might increase the overall computations during the training, the improvement is relatively marginal compared to the vanilla Word Dropout. 
  3. *Absence of the results with SNLI*: the authors mention that they have conducted experiments on three different datasets (lines 237-239). However, the results on only two datasets are presented in Table 1. Although they present some qualitative results on SNLI, quantitative results (like Table 1) should be provided to demonstrate its effectiveness in various scenarios. 
  4. *Quantitative support in addition to qualitative results in Tables 2 and 3*: although the provided qualitative results are intuitive and helpful to demonstrate the benefit of addressing the posterior collapse issue with the proposed method, they also have some concerns for the generalization to other samples. Therefore, if the authors can provide additional quantitative results, such as alignment with the saliency map for Table 2 or the similarity between original samples and interpolated ones, then it would significantly enhance the merit of the proposed method.

### **Other Comments.**

- Minor comments
    - Mis-allocation; line 185 and line 190.
questions: Please carefully read the above cons and address the related questions and suggestions. 
limitations: The authors have adequately addressed their work's limitations and potential negative social impact at the end of the conclusions.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper aims to investigate the problem of posterior collapse that variational autoencoders often suffer from. To tackle this issue, the authors propose an adversarial training strategy. To be specific, an adversary is employed to learn the probabilities of the decoder input dropout. The method is first demonstrated theoretically to remove pointwise mutual information and then evidenced by positive experimental results on Yahoo, Yelp, and SNLI against several baselines with double LSTMs as the decoder. A quantitative analysis follows to demonstrate the method further (e.g., the trade-off between exploration and exploitation).
strengths_and_weaknesses: Strengths:1). The authors touch on the issue shared by most autoregressive decoders, that is, posterior collapse; thus, works in this direction, I believe, will benefit the whole community due to the popularity of the autoregressive decoders. 2). The method is well-motivated by the problem of posterior collapse and well-defined theoretically. The idea is fairly simple for an easy understanding and application. 3). The introduced method is successfully examined to outperform baselines by thoughtful experiments on several benchmarks. A quantitative analysis is conducted, showing specific examples of the adversary.

Weaknesses:1). Technically speaking, the contribution of this work is incremental, and its technique deep is shallow. The proposed probabilistic word dropout is not that impressive or novel. To me, it sounds like a probabilistic teacher forcing. The massive notations and formulas appear to be not that necessary for me to understand the idea. 2). The improvements on three tasks over the previous works and self-implemented baselines are marginal. Further analysis beyond the main experiments is not sufficient. 3). The backbone is constrained to be the double LSTM, while popular transformers is not involved. Although this seems to work for different encoders, pre-trained language models are not covered as well. The application of this method to more extensive model structures remains a potential concern.
questions: 1). Is double lstm a part of the contributions? This misleads the point of this work.

2). Since the idea itself is fairly simple, are these notations and formulas necessary for its presentation?

3). The ablation study regarding the double lstm is necessary. In other words, experiments against various types of the decoder are required.

4). (line 152) The problems described lack the support of previous works.

5). (line 185) A wrong reference to the appendix.

6). (figure 2) What does the y-axis correspond to?

7). Tables 2 & 3 can be placed in the appendix to make some space for more experiments and studies.

limitations: Limitations are mentioned in Section 6, while it seems to be not sufficient.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors tackle the problem known as "posterior collapse" which arises while applying variational autoencoders (VAEs) to sequential data in an autoregressive manner. The authors extend a previous method known as "word dropout" which uniformly drops words during the autoregressive decoding process. The authors suggest to formulate the problem as a minimax optimization task. They present an adversary, that predicts the most important words (in term of information gain) and mask them. To regularize this training scheme, the authors suggest to mask only the top K words, where K is an hyperparameter of the method. Another regularization which adds exploratory factor to the process is that the adversary estimates a Gaussian distribution parameters as scores for the masking process, then the scores are sampled from this distribution. Another regularization added to the method is that the estimated adversary Gaussian distribution score for each of the words  is also being regularized using KL-divergence term to the normal Gaussian distribution.
strengths_and_weaknesses: **Strengths**
- The paper is written in a clear and easy to follow manner, high clarity.
- The approach is novel and original, although extended a previous uniform sampling method.
- The work provides some new insights to the "posterior collapse" in terms of theoretical analysis.
- The work present experimentation study which support the claims and proposed method in this paper

**Weaknesses**
- The significance of the work is not completely clear to me, I am not confident that the improvements over the baseline are significant.
- The ablation study is missing a more in-depth analysis of the parameter \lambda. It is not clear why the authors chose \lambda=1 as their choice. 
- In Table 1, we do not see improvement in KL of MI over uniform sampling on the Yahoo dataset, the paper does not discuss it. 
questions: - The paper suggests the section 4.1 permits alternative approaches for generating per word dropout probabilities. It would have been helpful to discuss approaches that the authors tried and did not work for them.
limitations: I believe there is no potential negative societal impact to this work.
review_title: null

Meta-review: metareview: The work analyzes the use of decoder input dropout in training sequence VAEs and proposes an adversarial dropout scheme in place of the typical uniform dropout. The paper is clearly written (reviewers Lidz, WbUQ), well-motivated (all reviewers), and provides insights into posterior collapse and word dropout (Lidz, WbUQ). All reviewers were concerned that the empirical performance improvement seems small, but in light of the writing, analysis, and careful experiments I nonetheless recommend acceptance.

confidence: Less certain
Decision: Accept