Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a data augmentation strategy for a class of problems that the amount of labelled data is limited while the evaluation procedure is easier. Specifically, they are able to incorporate some of the model’s output into training data to guide the training procedure. The idea is quite simple and effective according to empirical results.

Pros:
    The idea is quite simple and easy to implement. It can be applied into a broad problem. Also, it’s not tied to specific neural architecture
Section 4 provide an interpretation from view of EM algorithm, which is quite interesting.
    The improvement of empirical studies are desirable, validating the effectiveness of the proposed method on two complex tasks, including molecule optimization and program synthetic.
Cons:
    K in Algorithm 1 plays an important role. I expect more discussion of how to select K. Is K fixed during training procedure in different epochs? Is K the same magnitude as size of original training dataset.
In molecular optimization task, i suggest authors to add more details on setup. For example, on DRD2 and QED, what’s QED(X) and DRD2(X)? In success metric, what is your required constraint on both similarity and property score?
Extra:
    I don’t understand why the earlier collected data pairs are thrown away. Have you tried the strategy that incorporate all the augmented data?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: (post rebuttal) I appreciate the authors for detailed rebuttal. I stay with the original score based on the following reason.

In RAML you sample from exponentiated reward distribution and do maximum likelihood (minimizing forward KL in classic control as inference framework). How you sample depends on the problem assumption. In the original paper, they used a heuristic based on edit distance from ground truth, but in theirs they did not have assumption that you can evaluate correctness (as used in external filter). In this paper's context, the sampling naturally comes down to rejection sampling. Therefore I keep the original stance regarding the similarity between this work and RAML. 

---

The paper proposes an iterative data augmentation approach based self-generation and filtering with success criteria. The authors justify the algorithm as an EM procedure of maximizing \log p^*(y|x), where p^*(y|x) \propto p(y|x) * p(c=1|x,y). They demonstrate that the iterative data augmentation procedure can provide significant gains to SOA models in molecule generation and outperform MLE+RL method in program synthesis datasets.

Strengths:
- A simple approach that can be added to any seq2seq translation where success metric can be evaluated efficiently
- Demonstrated results on multiple applications

Weaknesses:
- The method requires being able to evaluate constraints  
- The approach is simple and does not seem to present significant novelty over prior methods. Particularly, the approach could be considered as nesting RAML [1] updates with (1) low temperature, (2) self-generated trajectories. 

Other comments:
- Some missing references [1, 2]. [2] also studies molecule generation and proposes approaches similar to RL + MLE. 

[1] Norouzi, Mohammad, et al. "Reward augmented maximum likelihood for neural structured prediction." Advances In Neural Information Processing Systems. 2016.
[2] Jaques, Natasha, et al. "Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a training scheme to enhance the optimization process where the outputs are required to meet certain constraints. The authors propose to insert an additional target augmentation phase after the regular training. For each datapoint, the algorithm samples candidate outputs until it find a valid output according the an external filter. The model is further fine-tuned on the augmented dataset. In experiments, the authors evaluated the proposed training scheme on two datasets: molecular optimization and program synthesis. Results show that the proposed training algorithm improves the success rate and the diversity among generated targets.

The improvement over the baselines is impressive and the paper seems to be well written. However, I have some concerns related to the novelty of the algorithm and some details in the experiments.

1) After reading Algorithm 1, it looks like a knowledge distillation step where the teacher model is the regular model enhanced by external filtering.

2) If you are able to train neural-based property evaluator F1, why don't just directly use F1(y) as a value network and use simple REINFORCE to create additional teacher signal for the model.

3) Another concern is that the baselines may be too weak. There may be simpler decoding strategies to guide the sequence generation with the property evaluator. For example, you can reject all invalid choices of next token in the sequence by checking the property evaluator score. A related work can be https://openreview.net/forum?id=HJIHtIJvz . Is this feasible on the datasets used in this paper?

review_title: Official Blind Review #3

Meta-review: This paper proposes a training scheme to enhance the optimization process where the outputs are required to meet certain constraints. The authors propose to insert an additional target augmentation phase after the regular training. For each datapoint, the algorithm samples candidate outputs until it find a valid output according the an external filter. The model is further fine-tuned on the augmented dataset. 

The authors provided detailed answers and responses to the reviews, which the reviewers appreciated. However, some significant concerns remained, and  due to a large number of stronger papers, this paper was not accepted at this time.
Decision: Reject