Iterative Target Augmentation for Effective Conditional Generation
Authors: Kevin Yang
Wengong Jin
Kyle Swanson
Regina Barzilay
Tommi Jaakkola
Venue:  (2020)
Keywords: data augmentation
generative models
self-training
molecular optimization
program synthesis

Abstract: Many challenging prediction problems, from molecular optimization to program synthesis, involve creating complex structured objects as outputs. However, available training data may not be sufficient for a generative model to learn all possible complex transformations. By leveraging the idea that evaluation is easier than generation, we show how a simple, broadly applicable, iterative target augmentation scheme can be surprisingly effective in guiding the training and use of such models. Our scheme views the generative model as a prior distribution, and employs a separately trained filter as the likelihood. In each augmentation step, we filter the model's outputs to obtain additional prediction targets for the next training epoch. Our method is applicable in the supervised as well as semi-supervised settings. We demonstrate that our approach yields significant gains over strong baselines both in molecular optimization and program synthesis. In particular, our augmented model outperforms the previous state-of-the-art in molecular optimization by over 10% in absolute gain. 

Under review as a conference paper at ICLR 2020
ITERATIVE TARGET AUGMENTATION FOR EFFECTIVE CONDITIONAL GENERATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Many challenging prediction problems, from molecular optimization to program synthesis, involve creating complex structured objects as outputs. However, avail- able training data may not be sufficient for generative model to learn all possible complex transformations. By leveraging the idea that evaluation is easier than gen- eration, show how a simple, broadly applicable, iterative target augmentation scheme can be surprisingly effective in guiding the training and use of such mod- els. Our scheme views the generative model prior distribution, and employs a separately trained filter as the likelihood. In each augmentation step, we filter the model's outputs obtain additional prediction targets for the next training epoch. Our method is applicable in the supervised as well semi-supervised settings. We demonstrate that our approach yields significant gains over strong baselines both in molecular optimization and program synthesis. In particular, augmented model outperforms the previous state-of-the-art in molecular optimization by over 10% in absolute gain.
INTRODUCTION
Deep architectures are becoming increasingly adept at generating complex objects such as images, text, molecules, programs. Many useful generation problems be seen as translation tasks, where the goal is to take a source (precursor) object such as molecule and turn it into a target satisfying given design characteristics. Indeed, molecular optimization of this kind is key step in drug development, though the adoption of automated tools remains limited due to accuracy concerns. We propose here simple, broadly applicable mega-algorithm to improve translation quality.
Translation is a challenging task for many reasons. Objects are complex and the available training data pairs do not fully exemplify the intricate ways which valid targets can created from the precursors. Moreover, precursors provided at test time may differ substantially from those available during training a scenario common in drug development. While data augmentation and semi- supervised methods have been used to address some of these challenges, the focus has been on either simple prediction tasks (e.g., classification) or augmenting data primarily on the source side. We show, in contrast, that iteratively augmenting translation targets significantly improves performance on complex generation tasks in which each precursor corresponds to multiple possible outputs.
Our iterative target augmentation approach builds the idea that it is casier to evaluate candidate objects than to generate them. Thus a learned predictor target object quality (a filter) can be used to effectively guide the generation process. To this end, construct it to the complex generative model's sampled translations of training set precursors. Candidate translations that pass the filter criteria become part of the training data for the next training epoch. The translation model is therefore iteratively guided to generate candidates that pass the filter. The generative model can be viewed as an adaptively tuned prior distribution over complex objects, with the filter as the likelihood. For this reason, it is helpful to apply the filter at test time as well, or to use the approach transductively[] to adapt the generation process to novel test cases. The approach is reminiscent of self-training or reranking approaches employed with some success for parsing (MeClosky et al. 2006 Charniak et al. (0166). However, our case, it is the candidate generator that is complex while the filter is relatively simple and remains fixed during the iterative process.
external filter and apply
'Allowing the model access set precursors (but targets) during training.

---

Under review as a conference paper at ICLR 2020
We demonstrate that meta-algorithm is quite effective and consistent in its ability improve translation quality in the supervised setting. On a program synthesis task (Bunel et al. 2018). under the same neural architecture, our augmented model outperforms their MLE baseline by 8% and their RL model by 3% in top-1 generalization accuracy (in absolute measure). On molecular optimiza- tion (Tin et al., 20T9a). their sequence to sequence translation baseline, when combined with our target data augmentation, achieves a new state-of-the-art result and outperforms their graph based approach by over 10% in success rate. Their graph based methods are also improved by iterative tar- get augmentation with more than 10% absolute gain. The results reflect the difficulty of generation in comparison to evaluation; indeed, the gains persist even the filter quality is reduced somewhat. Source side augmentation with unlabeled precursors (the semi-supprrised setting) can further im- prove results, but only when combined with the filter the target data augmentation framework. We provide ablation experiments to empirically highlight the effect of our method and also offer some theoretical insights for why it is effective.
2 RELATED WORK
Molecular Optimization The goal of molecular optimization is to learn to modify compounds so as to improve their chemical properties. Jaquus et al. (2017); You et al. (2018); Popova et al. (2018) used reinforcement learning approaches, while,Jin et al. 2019a b) formulated this problem as graph- to to-graph translation and significantly outperformed previous methods. However, their performance remains imperfect due to the limited size of given training sets. Our work uses property prediction models to check whether generated molecules have desired chemical properties. Recent advances in graph convolutional networks (Duvenaud et al. 2015 Gilmer et al.] (017) have provided effec- tive solutions to predict those properties in silico. In this work, we use an off-the-shelf property prediction model (Yang et al. 2019) to filter proposed translation pairs during data augmentation.
Program Synthesis Program synthesis is the task of generating program (using domain-specific language) based on given input-output specifications (Bunel et al. [0188] Gulwani 2011 Devlin al. (017). One can check a generated program's 's correctness by simply executing on each input and verifying its output. Indeed, /Zhang et al. (2018); [Chnn et al. (2019) leverage this idea in their respective decoding procedures, while also using structural constraints on valid programs.
Semi-supervised Learning Our method is related to various approaches in semi-supervised learn- ing. In image and text classification, data augmentation and label guessing (Berthelot et al.) 2019, Xie et al. 2019) are commonly applied to obtain artificial labels for unlabeled data. In machine translation, Norouzi et al. (2016) sample new targets from a stationary distribution in order to match the model distribution to the exponentiated payoff distribution centered at a single target sentence. Back-translation (Sennrich et al. 2013 Edunov et al. 2018)) creates extra translation pairs by using backward translation system translate late unlabeled sentences from target language into source language. In contrast, our method works in the forward direction because many translation tasks are not symmetric. Moreover, our data augmentation is carried out over multiple iterations, in which we use the augmented model to generate new data for the next iteration.
In syntactic parsing, our method is closely related self-training (McClosky al., 2006). They generate new parse trees from unlabeled sentences by applying an existing parser followed by reranker, and then treat the resulting parse trees as new training targets. However, their method is not iterative, and their reranker is explicitly trained to operate over the top & outputs of the parser; in contrast, our filter is independent of the generative model. addition we show that our ap- proach, which can be viewed as iteratively combining reranking and self-training, is theoretically motivated and can improve the performance of highly complex neural models in multiple domains. Co-training (Blum & Mitchell 1998) and tri-training (Zhou & Li 20055 Charniak et al. 2016) also augment a parsing dataset by adding targets on which multiple baseline models agree. Instead of using multiple learners, our method uses task-specific constraints to select correct outputs.
3 ITERATIVE TARGET AUGMENTATION
Our iterative target augmentation framework can be applied any conditional generation task with task-spe constraints. For example, molecular optimization (Jin et al.] b019a is the task of transforming a given molecule X into another compound Y with improved chemical properties,
2

---

Under review as a conference paper at ICLR 2020
Input-Output Specification
Candidates
Filtered Candidates
Program A
Program A
dar man (1
Generation
Program B
Program C
Correctness Filtering
Program A Program
repeat (4)
putMackez
Program D
turnbett ()
Figure Illustration of data generation process in the program synthesis setting. Given an input-output specification, we first use our generation model to generate candidate programs, and then select correct programs using our external filter. Images input-output specification and the program A are from Bunel et al. (018).
Algorithm Augmentation by iterative target augmentation
Input: Original training set D (K), (X,, Y,)] procedure AUGMENTDATASET(O.
2: Di+1 =
>> Initialize augmented dataset.
3:
for (X,, in do
4:
for attempt in I,.............. ........... C.......o. do
5:
Apply model M, to X; to sample candidate Y' if Y' passes external filter then
6:
7:
Add (X,, Y') to D++1
8:
if K successful translations added then break from loop
9:
10:
return augmented dataset Dz+1
11:
procedure TRAIN(D)
12:
for epoch in I,..., do Train model D. for epoch in 1.... 12 do
>> Regular training
14:
b Iterative target augmentation
D1+1 AUGMENTDATASETTO.
16:
M++1 +- Train model M, on Dz+1.
while constraining Y remain similar to X. Program synthesis (Bunel et al. 2018 Chen et al., 2019) is the task of generating program y satisfying input specification X; for example, X may be a set of input-output test cases which Y must pass.
Without loss of generality, we formulate the generation task as translation problem. For given input X, the model learns to generate an output satisfying the constraint c. The proposed aug- mentation framework can be applied to any translation model M trained on existing dataset = {(,, Y.) As illustrated in Figure []] our method is an iterative procedure in which each steps:
iteration consists of the following
Let Dr be the training set at iteration t. To construct each next training set D441, we feed each input X; E D (the original training set, not D,) into the translation model up to C' times to sample C' candidate translations Y Y, We take the first K distinct translations for each X; satisfying the constraint e and add them D:+1- When we do not find K distinct valid translations, we simply add the original translation Y, to De+1-
Augmentation Step:
Training Step: We continue to train the model M2 over the new training set D:41 for one epoch.
The above training procedure summarized in Algorithm As the constraint e known a priori, we can construct an external filter to remove generated outputs that violate e during the augmentation step. At test time, we also use this filter to screen predicted outputs. To propose of a given input X, we have the model generate up to L outputs until we find one satisfying the constraint e. If all L attempts fail for a particular input, we just output the first of the failed attempts.
final translation
One could augment D. instead of D and continuously expand the dataset, but the empirical effect is small (see Appendix B.3). We note this augmentation step be trivially parallelized if speed is concern.
3

---

Under review as a conference paper at ICLR 2020
Finally, additional improvement, observe that the augmentation step be carried out for unlabeled inputs X that have no corresponding Y. Thus we can further augment our training dataset in the transductive setting by including test set inputs during the augmentation step, or in the semi-supervised setting by simply including unlabeled inputs.
MOTIVATION FOR ITERATIVE TARGET AUGMENTATION
We provide here some theoretical motivation for our iterative target augmentation framework. For simplicity, we consider an external filter cx,y that is a binary indicator function representing whether output satisfies the desired constraint in relation to imput X. In other words, we would like to generate Y such that Y E B(X) {Y') serves as a reasonable prior distribution over outputs, we could simply "invert" the filter and use
1}. If the initial translation model p(0) (Y[X)
(Y x pl) (Y CX.Y
(1)
as the ideal translation model. While this posterior caleulation is typically not feasible but could be approximated through samples, it relies heavily on the appropriateness of the prior (model prior to augmentation). Instead, we go step further and iteratively optimize parametrically defined prior translation model (X))) Note that the resulting prior can become much more concentrated around acceptable translations.
We maximize the log-likelihood that candidate translations satisfy the constraints implicitly encoded in the filter
Ex [log Pa = [
(2)
In many cases there are multiple viable outputs for any given input X. The training data may provide only one (or none) of them. Therefore, treat the output structure Y latent variable, and expand the inner term of 2q.2) as
log Po 1 X)
log Σ Po(Y, =1 (X)
(3)
log Σ
P(ex,y (XX))
(4)
Y
= log Σ
cx,Y Po(YIX)
(5)
Since the above objective involves discrete latent variables Y, we propose to maximize 5 using the standard EM algorithm (Dempster et al., 1977), especially its incremental, approximate variant. The target augmentation step in our approach a sampled version of the E-step where the posterior samples are drawn with rejection sampling guided by the filter. The number of samples K controls The additional training step based on the augmented targets corresponds to generalized M-step. More precisely, let p)4) (X) be the current translation model after í epochs of augmentation training. In epoch t + 1, the augmentation step first samples C' different candidates for each input X using the old model p(t) parameterized by g(t), and then removes those which violate the constraint c, interpretable as samples from the current posterior (x x Pato (Y]X)xxxy_ As a result, the training step maximizes the EM auxiliary objective stochastic gradient descent:
the quality of approximation to the posterior 3
J(O) ((0) = Ex Σ (X(X) log Po(Y
(6)
We train the model with multiple iterations and show empirically that model performance indeed keeps improving we add more iterations. The EM approach is likely converge to different and perforrin translation model than the initial posterior calculation discussed above.
EXPERIMENTS
We demonstrate the broad applicability of iterative target augmentation by applying it to two tasks of different domains: molecular optimization and program synthesis.
See Appendix B.3 for details and experiments on the effect of sample size K.
4

---

Under review as a conference paper at ICLR 2020
NH
NH
Constraints:
Translate
Molecular Similarity sim(X. 2 0.4 Drug-likeness QED(Y) 0.9
Input Molecule X QED(X) 0.78
Target Molecule Y QED(Y) = 0.92
Figure 2: Illustration of molecular optimization. Molecules can be modeled as graphs, with atoms as nodes and bonds as edges. Here, the task is to train translation model to modify given input molecule into target molecule with higher drug-likeness (QED) score. The constraint has two components: the output Y must be highly drug-like, and must be sufficiently similar the input X.
5.1 MOLECULAR OPTIMIZATION
The goal of molecular optimization is to learn to modify molecules as improve their chemical properties. As illustrated in Figure this task formulated as a graph-to-graph translation problem. of molecular pairs Y)). X is the input
Similar to machine translation, the training set molecule (precursor) and Y is similar molecule with improved chemical properties. Each molecule in the training set D further labeled with its property score. Our method well-suited this task because the target molecule is not unique: each precursor molecule can be modified in many different ways to optimize its properties.
The constraint for this task contains two parts: 1) the chemical property of Y must exceed a certain threshold 3, and 2) the molecular similarity between X and Y must exceed sim certain threshold 8. The molecular similarity sim(X, X, Y) is defined as Tanimoto similarity on Morgan
External Filter
fingerprints (Rogers Hahn 2010). which measures structural overlap between two molecules.
In real world settings, ground truth values of chemical properties are often evaluated through experi- mental assays, which are too expensive and time-consuming to run for iterative target augmentation. Therefore, we construct an in silico property predictor F1 to approximate the true property evalu- ator Fo. To train this property prediction model, we use the molecules in the training set and their labeled property values. The predictor Fj is parameterized trained using the Chemprop package (Yang et al.) 2019). During data augmentation, we use F1 to filter out molecules whose predicted property is under the threshold B.
graph convolutional network and
5.1.1 EXPERIMENTAL SETUP
We follow the evaluation setup of Jin et al. (2019)) for two molecular optimization tasks:
1. QED Optimization:
The task is improve the drug-likeness (QED) of given compound X. The similarity constraint is sim (X, Y) 2 0.4 and the property constraint is QED(Y) 2 0.9, with QED(Y) [o, 1] defined by the system of Bickerton et al. al. (012).
2. DRD2 Optimization: The task is to optimize biological activity against the dopamine type 2 receptor (DRD2). The similarity constraint sim(X, Y) 2 0.4 and the property constraint
DR) > 0.5, where DRD2(Y) E [0, 1 is the predicted probability of biological activity given by the model from Olivecrona et al. (2017).
We treat the output the in silico evaluators from Bickerton et al. 2012 and Olivecrona et al. 2017] as ground truth, and we use them only during test-time evaluation A
Evaluation Metrics. During evaluation, we are interested both in the probability that the model will find successful modification for a given molecule, well as the diversity of the successful modifications when there are multiple. We translate each molecule in the test set Z = 20 times, resulting in candidate modifications Yi .....y Yz (not necessarily distinct). We use the following two evaluation metrics:
'Athhough the Chemprop model we use in our filter is quite powerful, it fails to perfectly approximate the ground truth models for both QED and DRD2. The test set RMSE between our Chemprop model and the ground truth 0.015 the QED task and 0.059 on DRD2, where hoth properties range from to 1.
5

---

Under review as a conference paper at ICLR 2020
Model
QED Suce.
QED Div.
DRD2 Suec.
DRD2 Div.
VSeq2Seq
58.5
0.331
75.9
0.176
VSeq2Seq+ (Ours)
89.0
0.470
97.2
0.361
VSeq2Seq+, superviid (Ours)
95.0
0.471
99.6
0.408
VSeg2Seq+, transductive (Ours)
92.6
0.451
97.9
0.358
HierGNN
76.6
0.477
85.9
0.192
HierGNN+ (Ours)
93.1
0.514
97.6
0.418
Table 1: Performance of different models QED and DRD2 optimization tasks. Italicized mod-
els with + are augmented with iterative target augmentation. We emphasize that iterative target augmentation remains critical to performance in the semi-supervised and transductive settings; data augmentation without an external filter instead decreases performance.
1. Success: The fraction of molecules X for which any of the outputs Y1 ... Yz meet the required similarity and property constraints (specified previously for each task). This is our main metric. 2. Diversity: For each molecule X, we measure the average Tanimoto distance (defined as 1 sYm(Yi, Yj)) between pairs within the set of successfully translated compounds among Y1 ..... Yz. If there are one fewer successful translations then the diversity is O. We average this quantity across all test molecules.
Models and Baselines. We consider the following two model architectures from Jin et al. (0019)) to show that our augmentation scheme is not tied to specific neural architectures.
1. VSeq2Seq. a sequence-to-seqqeeee translation model generating molecules by their SMILES string [Weininger] 1988).
2. HierGNN, hierarchical graph-to-graple architecture that achieves state-of-the-art performance on the QED and DRD2 tasks, outperforming VSeq2Seq by a wide margin.
We apply our iterative augmentation procedure to the above two models, generating up to K new targets per precursor during each epoch of iterative target augmentation. Additionally, we evaluate augmentation of VSeq2Seq in transductive setting, well as in semi-supervised setting where we provide 100K additional source-side precursors from the ZINC database (Sterling & Irwin 2015). Full hyperparameters are in Appendix A
5.1.2 RESULTS
As shown in Table our iterative augmentation paradigm significantly improves the performance of VSeq2Seq and HierGNN. On both datasets, the translation success rate increases by over 10% in ab- solute terms for both models. In fact, VSeq2Seq+, our augmentation of the simple VSeq2Seq model, outperforms the non-aus au-umeteeee version HierGNN. This result strongly confirms our hypothesis about the inherent challenge of learning translation models in data sparse scenarios. Moreover, find that adding more precursors during data augmentation further improves the VSeq2Seq model. On the QED dataset, the translation success rate improves from 89.0% 92.6% by just adding test set molecules as precursors (VSeq2Seq+, transductive). When instead adding 100K presursors from the external ZINC database, the performance further increases 95.0% (VSeq2Seq+, semi- supervised). We observe similar improvements for the DRD2 task as well. Beyond accuracy gain, our augmentation strategy also improves the diversity of generated molecules. For instance, on the DRD2 dataset, our approach yields 100% relative gain in terms of output diversity.
Importance of Property Predictor Although the property predictor used in data augmentation is different from the ground truth property evaluator used at test time, the difference in evaluators does not derail the overall training process. Here we analyze the influence of the quality of the property predictor used in data augmentation. Specifically, we rerun predictors in the croperty-predicting component of our external filter. We obtain these less accurate predictors by undertraining Chemprop and decreasing its hidden dimension. For comparison, also report results with the oracle property predictor which is the ground truth property evaluator.
experiments using less accurate
As shown in Figure 3 the DRD2 dataset, using predictors that deviate significantly from the ground truth. This implies that our framework can potentially be applied to other properties that are harder to predict. On the QED dataset, our
able to maintain strong performance despite

---

Under review as a conference paper at ICLR 2020
100
100
QED Success
DRD2 Success
50
50
0
0
0.02
0.04
0.06
0.08
0.1
0.2
0.3
0.4
Predictor RMSE
Predictor RMSE
Figure 3: Left: QED success rate Chemprop predictor's RMSE with respect to ground truth on test set. The red line shows the performance of the (unaugmented) VSeq2Seq baseline. Right: Same plot for DRD2. In each plot, the far left point with RMSE is obtained by reusing the ground truth predictor, while the second-from-lefi point is the Chemprop predictor we use to obtain our main results. Points further the right are weaker predictors trained for fewer epochs and with less capacity, simulating scenario where the property is more difficult to model.
Model
Train
Test
QED Succ.
QED Div.
DRD2 Suec.
DRI>2 Div.
VSeq2Seq
X
58.5
0.331
75.9
0.176
VSeq2Seq(test)
x
77.4
0.471
87.2
0.200
VSeq2Seq(train)
1
x
81.8
0.430
92.2
0.321
VSeq2Seq+
/
/
89.0
0.470
97.2
0.361
VSeegSeg(nn-filtr)
X
x
47.5
0.297
51.0
0.185
Table 2: Ablation analysis of filtering at training and test time. "Train" indicates a model whose training process uses data augmentation according to our framework. "Test" indicates a model that uses the external filter at prediction time to discard candidate outputs which fail to pass the filter. The evaluation for VSeq2Seq(nooiilee is conducted after 10 augmentation epochs, as the best validation set performance only decreases over the course of training.
method less tolerant to inaccurate property prediction because the property constraint is much tighter - it requires the QED score of an output Y to be in the range [0.9, 1.0].
Importance of External Filtering Our full model (VSeq2Seq+) uses the external filter during both training and testing. We further experiment with Vseq2seq(test), version of our model trained without data augmentation but which uses the external filter to remove invalid outputs at test time. As shown in Table 2 VSeq2Seq(test) performs significantly worse than full model trained under data augmentation. Similarly, model VSeq2Seq(train) trained with the data augmentation but without the prediction time filtering also performs much worse than the full model.
In addition, we run an augmentation-only version of the model without an external filter. This model (referred to as VSeqQSee(nn-filten in Table augments the data in each epoch by simply using the first K distinct candidate translations for each precursor X in the training set, without using the external filter all. In addition, we provide this model with the 100K unlabeled precursors from the remi-supervised setting. Nevertheless, we find that the performance of this model steadily declines from that of the bootstrapped starting point with each data augmentation epoch. Thus the external filter is necessary to prevent poor targets from leading the model training astray.
5.2 PROGRAM SYNTHESTS
In program synthesis, the source is a set of input-output specifications for the program, and the target a program that passes all test cases. Our method is suitable for this task because the target program not unique. Multiple programs may be consistent with the given input-output specifications. The external filter straightforward for this task: we simply check whether the generated output passes all test cases. Note that at evaluation time, each instance contains extra held-out input-output test cases; the program must pass these in addition to the given test cases in order to be considered correct. When perform prediction time filtering, do not held-out test cases in our filter.

---

Under review as a conference paper at ICLR 2020
Model
Top-1 Generalization
Top- Generalizaaion
82
80
MLE (Bunel et al. 2018)
71.91
MLE + RL + Beam Search
77.12
(Bunel al. 2018)
MLE+ (Ours)
80.17
10
30
40
Augmentation Epoch
Table 3: Model performance on Karel program syn- thesis task. MLE+ is our augmented version of the MLE model (Bunel et al. 2018).
Figure 4: Top-1 generalization accuracy of MLE+ model on validation set of Karel task across different epochs.
5.2.1 EXPERIMENTAL SETUP
Our task is based on the educational Karel programming language (Pattis 1981 used for evaluation in Bunel et al. (2018) and [Cen et al. (2019). Commands in the Karel language guide robot's actions in 2D grid, and may include for loops, while loops, and conditionals. Figure Tigure[contain contains an example. We follow the experiment setup of Bunel et al. (2018).
The evaluation metric is top-1 generalization. This metric measures how often the model can generate a program that passes the input-output test cases on the test set. At test time, we use our model to generate up to L candidate programs and select the first one to pass the input-output specifications (not including held-out test cases).
Evaluation Metrics.
Models and Baselines. Our main baseline is the MLE baseline from Bunel al. (0018). This model consists a CNN encoder for the input-output grids and LSTM decoder along with a handcoded syntax checker. is trained maximize the likelihood of the provided target program. Our model is the augmentation of this MLE baseline by our iterative target augmentation framework. As with molecular optimization, augmentation step. Additionally, we compare against the best model from Bunel et al. (2018), which finetunes the same MLE architecture using an RL method with beam search estimate gradients We use the same hyperparameters as the original MLE baseline; see Appendix A for details.
generate up to K 4 targets per precursor during each
5.2.2 RESULTS
Table 3 shows the performance of model in comparison to previous work. Our model (MLE+) outperforms the base MIE model in Bunel et al. (2018) model by a wide margin. Moreover, our model outperforms the best reinforcement learning model (RL Beam Search) in Bunel et al. (2018), which was trained to directly maximize the generalization metric. This demonstrates the efficacy of our approach in the program synthesis domain. Since our augmentation framework is complementary to architectural improvements, we hypothesize that other techniques, such as exe- cution based synthesis (Chen et al. 2019), can benefit from our approach well.
6 CONCLUSION
In this work, have presented an iterative target augmentation framework for generation tasks with multiple possible outputs. Our approach is theoretically motivated, and we demonstrate strong empirical results on both the molecular optimization and program synthesis tasks, significantly out- performing baseline models on each task. Moreover, we find that iterative target augmentation is complementary to architectural improvements, and that its effect can quite robust to the quality of the external filter. Finally, in principle our approach is applicable to other domains as well.
"More recently, Chen et al. 2019) achieved state-of of of-thh-art art performance on the same Karel task, with top- generalization accuracy of 92%. They use different architecture highly specialized for program synthesis as results in this
well specialized ensemble method. Thus their results not directly comparahle paper. We did not apply our method to their model their implementation is publicly available.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
1xXiv:1905.02299 2019. 2019.
G Richard Bickerton, Gaia Paolini, Jérémy Besmard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90, 2012.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceed- ings of the eleventh annual conference on Computational learning theory, pp. 92-100. Citeseer, 1998.
Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Lever- aging grammar and reinforcement learning for neural program synthesis. arXiv preprint arXiv: 000..08855...6 2018.
Eugene Charniak et al. Parsing language modeling. In Proceedings of the 2016 Conference Empirical Methods Natural Language Processing, pp. 2331-23366 2016.
Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. Interna-
lional Conference on Learning Representations, 2019.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series (Methodological), 39(1): 1-22, 1977.
Jacob Devlin, Rudy R Bunel, Rishabh Singh, Matthew Hausknecht, and Pushmeet Kohli. Neural program meta-induction. In Advances in Neural Information Processing Systems, pp. 2080-2088, 2017.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aapuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in Neural Information Processing Systems, pp. 2224-2222222 2232,, 2015.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back tack-translation at scale. arXiv preprint arXiv: 09381, 2018.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning. 2017.
Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. In ACM Sigplan Notices, volume 46, pp. 317-330. ACM, 2011.
Natasha Jaques, Shixiang Gu Dzmitry Bahdanau, José Miguel Hermández-Lobato Richard Tumner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- els with kl-control. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1645-1654. JMLR. org, 2017.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multirresolution autoregressive graph-to-graph translation for molecules. arXiv preprint 1rXiv: 11223, 2019a.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecular optimization. International Conference on Learning Representa- fion, 2019b.
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. Pro- ceedings the main conference human language technology conference the North American Chapter of the Association of Computational Linguistics, pp. 152-159. Association for Compu- tational Linguistics, 2006.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems, pp. 1723-1731, 2016.

---

Under review as a conference paper at ICLR 2020
Mareus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):48, 2017.
Richard E. Pattis. Karel the Robot: A Gentle Introduction to the Art of Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1981. ISBN 0471089281.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeaap7885, 2018.
David Rogers and Mathew Hahn. Extended-conneccivity fingerprints. J. Chem. Inf. Model., 50(5): 742-754, 2010.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv: 1,06709, 2015.
Teague Sterling and John J Irwin. Zinc 15 15- 5igadd discovery for everyone. Journal of chemical information and modeling, (5(11):2324-3333 2015.
David Weininger. Smiles, chemical language and information system. 1. introduction method- ology and encoding rules. J. Chem. Inf. Model., 28(1):31-36 1988.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv:/9088 10.9904.22800 2848, 2019.
Kevin Yang. Kyle Swanson, Wengong Jin, Connor W Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perees Tim Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecu- lar representations for property prediction. Journal of chemical information and modeling, 2019.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems, pp. 6410-6421, 2018.
Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, William E Byrd, Raquel Urtasun, and Richard Zemel. Leveraging constraint logic programming for neural guided program synthesis. 2018.
Zhi-Hua Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. 1EEE Transactions on Knowledge Data Engineering, (11):1529-1541, 2005.
10

---

Under review as a conference paper at ICLR 2020
A
MODEL HYPERPARAMETERR
Our augmented models share the same hyperparameters as their baseline counterparts in all cases.
A.1 MOLECULAR OPTIMIZATION
For the VSeq2Seq model we use batch size 64, embedding and hidden dimension 300, VAE latent dimension 30, and an LSTM with depth (bidirectional in the encoder, unidirectional in the de- coder). For models using iterative target augmentation, 71 is set to 5 and m2 is set 10, while for the baseline models we train for 20 epochs (corresponding to 11 20, 122 0). The HierGNN model shares the same hyperparameters as in Jin et al. (019a)).
For the training time and prediction time filtering parameters, set K 4, C 200, and L 10 for both the QED and DRD2 tasks.
A.2 PROGRAM SYNTHESIS
For the Karel program synthesis task, use the same hyperparameters the MLE baseline model in Bunel et al. (2018). We use beam size of 64 at test time, the same as the MLE baseline, but simply sample programs from the decoder distribution when running iterative target augmentation during training. The baseline model is trained for 100 epochs, while for the model employing iterative target augmentation we train as normal for n 15 epochs followed by n2 = 50 epochs of iterative target augmentation. Due to the large size of the full training dataset, in each epoch of iterative augmentation we use to of the dataset, so in total we make passes over the entire dataset.
For the training time and prediction time filtering parameters, we set K 4, C = =50, and L 10.
B ADDITIONAL EXPERIMENTAL DETAILS
B. DATASET SIZES
In Table we provide the training, validation, and test set sizes for all our tasks. For each task we the splits our baselines.
Task
Training Set
Validation Set
Test Set
QED 88306
360
800
DRD2
34404
500
1000
Karel 1116854
2500
2500
Table Number of source-target pairs training, validation, and test sets for each task.
B.2 MOLECULAR OPTIMIZATION LEARNING CURVES
In Figure 5 we provide the validation set performance per iterative target augmentation epoch for VSeq2Seqe model both the QED and DRD2 tasks. The corresponding figure for the MLE+ model on the Karel task is in the main text in Figure 4
11

---

Under review as a conference paper at ICLR 2020
95
100
95
QED Success
90
DRD2 Success
85
80
80
75
75
2
0
2
8
10
Data Augmentation Epoch
Data Augmentation Epoch
Figure 5: Left: QED success rate for VSeq2Seq+ on validation set for each epoch of iterative target augmentation. Right: Same plot for DRD2. For each plot, the far left point indicates the performance of the bootstrapped model.
B.3 FURTHER MOLECULAR OPTIMIZATION EXPERIMENTS
In molecular optimization tasks, experiment with the effect of modifying K, the number of new targets added per precursor during each training epoch. In all other experiments we have used unsurprising that
K 4. Since taking K 0 corresponds performance may suffer when K is too small. However, as shown in Table 5 at least in molecular optimization there is relatively little change in performance for much larger than
the base non-augmented model,
Model
QED Suce.
QED Div.
DRD2 Suce.
DRD2 Div.
VSeq2Seqt, K=2
85.1
0.453
95.9
0.327
VSeq2Seqt, K=4
89.0
0.470
97.2
0.361
VSeq2Seqt. K=8
88.4
0.480
97.6
0.373
Table 5: Performance of our model VSeq2Seq+ with different values of K. All other experiments use K 4.
We also experiment with version of method which continually grows the training dataset by keeping all augmented targets, instead of discarding new targets at the end of each epoch. We chose the latter version for our main experiments due its closer alignment to our EM motivation. However, we demonstrate Table 6 that performance gains from continually growing the dataset small to insignificant in molecular optimization tasks.
Model
QED Suce.
QED Div.
DRD2 Suce.
DRD2 Div.
VSeq2Seq+
89.0
0.470
97.2
0.361
VSeq2Seq+, keep-targets
89.8
0.465
97.6
0.363
Table 6: Performance of proposed augmentation scheme, VSeq2Seq+, compared alterna- tive version (VSeq2Seq+, keep-targets) which keeps all generated targets and continually grows the training dataset.
B.4 PROGRAM SYNTHESIS ABLATIONS
In Table provide the same ablation analysis that provided in the main text for molecular optimization, demonstrating that both training time iterative target augmentation as well as predic- tion time filtering beneficial to model performance. However, note that even MLE(train), our model without prediction time filtering, outperforms the best RL method from Bunel et al. (0118).
12

---

Under review as a conference paper at ICLR 2020
Model
Train
Test
Top-1 Generalization
MLE*
x
x
70.91
MLE(test)"
X
74.12
MLE(train)
1
x
77.92
MLE+
/
80.17
Table 7: Ablation analysis of filtering training and test time. "Train" indicates model whose training process uses data augmentation according to our framework. "Test" indicates a model that uses the external filter prediction time to discard candidate outputs which fail to pass the filter. Note that MLE and MLE(test) are based on an MLE checkpoint which underperforms the published result from Bunel et (2018) by point, due training for fewer epochs.
13


Reviews:
[{"review_id": "r1x2uWNr9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a data augmentation strategy for a class of problems that the amount of labelled data is limited while the evaluation procedure is easier. Specifically, they are able to incorporate some of the model’s output into training data to guide the training procedure. The idea is quite simple and effective according to empirical results.\n\nPros:\n    The idea is quite simple and easy to implement. It can be applied into a broad problem. Also, it’s not tied to specific neural architecture\nSection 4 provide an interpretation from view of EM algorithm, which is quite interesting.\n    The improvement of empirical studies are desirable, validating the effectiveness of the proposed method on two complex tasks, including molecule optimization and program synthetic.\nCons:\n    K in Algorithm 1 plays an important role. I expect more discussion of how to select K. Is K fixed during training procedure in different epochs? Is K the same magnitude as size of original training dataset.\nIn molecular optimization task, i suggest authors to add more details on setup. For example, on DRD2 and QED, what’s QED(X) and DRD2(X)? In success metric, what is your required constraint on both similarity and property score?\nExtra:\n    I don’t understand why the earlier collected data pairs are thrown away. Have you tried the strategy that incorporate all the augmented data?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylmFKK6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(post rebuttal) I appreciate the authors for detailed rebuttal. I stay with the original score based on the following reason.\n\nIn RAML you sample from exponentiated reward distribution and do maximum likelihood (minimizing forward KL in classic control as inference framework). How you sample depends on the problem assumption. In the original paper, they used a heuristic based on edit distance from ground truth, but in theirs they did not have assumption that you can evaluate correctness (as used in external filter). In this paper's context, the sampling naturally comes down to rejection sampling. Therefore I keep the original stance regarding the similarity between this work and RAML. \n\n---\n\nThe paper proposes an iterative data augmentation approach based self-generation and filtering with success criteria. The authors justify the algorithm as an EM procedure of maximizing \\log p^*(y|x), where p^*(y|x) \\propto p(y|x) * p(c=1|x,y). They demonstrate that the iterative data augmentation procedure can provide significant gains to SOA models in molecule generation and outperform MLE+RL method in program synthesis datasets.\n\nStrengths:\n- A simple approach that can be added to any seq2seq translation where success metric can be evaluated efficiently\n- Demonstrated results on multiple applications\n\nWeaknesses:\n- The method requires being able to evaluate constraints  \n- The approach is simple and does not seem to present significant novelty over prior methods. Particularly, the approach could be considered as nesting RAML [1] updates with (1) low temperature, (2) self-generated trajectories. \n\nOther comments:\n- Some missing references [1, 2]. [2] also studies molecule generation and proposes approaches similar to RL + MLE. \n\n[1] Norouzi, Mohammad, et al. \"Reward augmented maximum likelihood for neural structured prediction.\" Advances In Neural Information Processing Systems. 2016.\n[2] Jaques, Natasha, et al. \"Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lkPZB2tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a training scheme to enhance the optimization process where the outputs are required to meet certain constraints. The authors propose to insert an additional target augmentation phase after the regular training. For each datapoint, the algorithm samples candidate outputs until it find a valid output according the an external filter. The model is further fine-tuned on the augmented dataset. In experiments, the authors evaluated the proposed training scheme on two datasets: molecular optimization and program synthesis. Results show that the proposed training algorithm improves the success rate and the diversity among generated targets.\n\nThe improvement over the baselines is impressive and the paper seems to be well written. However, I have some concerns related to the novelty of the algorithm and some details in the experiments.\n\n1) After reading Algorithm 1, it looks like a knowledge distillation step where the teacher model is the regular model enhanced by external filtering.\n\n2) If you are able to train neural-based property evaluator F1, why don't just directly use F1(y) as a value network and use simple REINFORCE to create additional teacher signal for the model.\n\n3) Another concern is that the baselines may be too weak. There may be simpler decoding strategies to guide the sequence generation with the property evaluator. For example, you can reject all invalid choices of next token in the sequence by checking the property evaluator score. A related work can be https://openreview.net/forum?id=HJIHtIJvz . Is this feasible on the datasets used in this paper?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a training scheme to enhance the optimization process where the outputs are required to meet certain constraints. The authors propose to insert an additional target augmentation phase after the regular training. For each datapoint, the algorithm samples candidate outputs until it find a valid output according the an external filter. The model is further fine-tuned on the augmented dataset. 

The authors provided detailed answers and responses to the reviews, which the reviewers appreciated. However, some significant concerns remained, and  due to a large number of stronger papers, this paper was not accepted at this time.
Decision: Reject