Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors describe a method to improve the performance of generative adversarial networks in the task of generating structured objectives that have to satisfy complicated constraints. The proposed solution involves using an additional term in the GAN objective that penalizes the generation of invalid samples. This term, called the semantic loss, is given by a multiple of the log probability of the model generating valid samples.

Clarity:

The paper is not very well written and several parts need to be clarified. In particular, in equation 3. What is theta in this equation?  how is it obtained? The authors mention briefly how their method could be used to deal with intractable constraints, but they're almost no specific details or examples of how this is done in practice. The proposed approach relays on the knowledge compilation method, but they're very few details of it in the document. Is it used at all in the experiments?

I am concern about the lack of reproducibility of the paper. I believe, from the paper as it is, it will be impossible to reproduce the results. There are no details about public code release, hyper-parameters settings, etc. For example,
in section 4.3 the authors mention that they condition the constraint on 5 latent dimensions without giving details about which dimensions exactly.

Significance:

It is hard to quantify the significance of the contribution. The constrained images problem is very toy and simple and the experiments with molecules do not include any baseline (only the GAN model without the constraint). There have been
many recent contributions improving the validity of generative models for molecules and the authors do not compare with any of them.

The authors also fail to cite relevant work such as

Jaques, Natasha, et al. "Sequence tutor: Conservative fine-tuning of sequence
generation models with kl-control." Proceedings of the 34th International
Conference on Machine Learning-Volume 70. JMLR. org, 2017.

Seff, Ari, et al. "Discrete Object Generation with Reversible Inductive
Construction." arXiv preprint arXiv:1907.08268 (2019).

Novelty:

The proposed approach is rather incremental and lacks novelty. It consists in just applying the semantic loss approach of Xu et al. 2018 to GAN training, with very limited new methodological or algorithm contributions.

Quality:

The experiments performed are not strong enough to validate the proposed method. The authors do not consider strong baselines in their evaluations.

Summary:

I find that the problem addressed by the authors is highly relevant and the proposed approach has the potential to be useful in practice. However, the paper needs to be improved regarding its clarity, reproducibility and strength of experiments before it can be accepted for publication.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposed Constrained Adversarial Networks (CAN), which incorporates structural constraints by augmenting a penalty term in the training object. The penalty term is formulated as the semantic loss proposed in [1] which can handle any logical constraints. Experiments are demonstrated to show the advantage of CAN over standard GAN in terms of whether the generated samples satisfy the hard constraints, and whether they are novel and unique.

First, I'd like to thank the authors for making this paper easy to follow. I like the idea of encouraging constraints for generative models, which is useful and interesting. However, given the published paper [1], this work seems to be a bit incremental.

The semantic loss for incorporating constraints and the knowledge compilation techniques for efficient evaluation are both introduced and discussed in [1]. The novelty of this paper is to apply these techniques to generative models, which seem to be a bit straightforward. A similar idea is proposed in [2], where the authors also discussed logical constraints and generative models, but they call the augmented penalty as 'posterior regularization'. I will be interested in a comparison to their method in terms of both methodology level and experiment level.

Overall the contribution of this paper does not seem to be strong enough. I would personally vote for weak rejection.

[1] Xu, Jingyi, et al. "A semantic loss function for deep learning with symbolic knowledge." arXiv preprint arXiv:1711.11157 (2017).
[2] Hu, Zhiting, et al. "Deep generative models with learnable knowledge constraints." Advances in Neural Information Processing Systems. 2018.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper the authors present a Generative Adversarial Neural Networks with Xu et al.’s semantic loss applied to the generator. They call this GAN a Constrained Adversarial Network or (CAN) and identify it as a new class of GAN. The authors present three different problem domains for their experiments focused on the generation of constrained images, chunks of Super Mario Bros.-style levels, and molecules. For each domain they include particular constraints for the semantic loss, which biases the generator towards creating valid content according to these constraints. 

The paper at present has a number of issues holding it back. First, I am not convinced by the author’s claims that the application of an existing loss function to the generator is sufficient to identify a new class of GAN. Second, there is a lack of technical detail in the experiments necessary to replicate them. Third, there is a lack of discussion of the experimental results to place them in context for readers. Finally following from the earlier points, there seems to be a lack of technical contributions in the paper. 

I certainly agree with the authors about the inability of GANs to learn structural constraints with insufficient training data, as this has been demonstrated in many examples of prior work. I also agree that particular problem domains, as identified by the authors, have stronger structural requirements. However, it is unclear to me why in these instances one would use GANs and not some alternative approach such as constraint-based solvers. Or even if one wanted to employ GANs, what the benefit of adapting the constraints into a loss function is compared to say constraining generated output in a post-hoc process.

The descriptions of the two of the three experiments do not include any discussion of the GAN architectures or hyperparameters. While this is not strictly necessary in the paper text some discussion in an appendix or a citation to a prior application of the architecture(s) would be appropriate. Without this, it is impossible for future researchers to replicate these results. Further, it is difficult for readers to place the results in context. For individual experiments, such as the Super Mario Bros. experiment, it is unclear why certain choices were made. For example, why train a GAN on just level 1-3 or 3-3, and not train a single model on multiple levels as is common in the field of procedural content generation via machine learning. 

There is a lack of discussion in the paper on the results of each experiment. For example, the output of the GANs for all the experiments seems quite low, and the differences in terms of the results between the GAN and the CAN across the experiments do not seem to be substantial. Some discussion to put this into context for readers would be helpful.

As far as I can understand the primary technical contributions of the paper are: (1) the application of Xu et al.’s semantic loss to GANS, (2) the constraints developed for the three experiments, and (3) the results of the three experiments. I am unconvinced of the utility of these contributions to a general machine learning audience.

---

Updated my review as the authors included extra detail regarding the experiments in a new draft, which helped with the reproducibility issue. However, I am still unconvinced in the contributions of the paper outside of what I previously listed. While I am also unfamiliar with any prior example demonstrating that GANs produce invalid structure, this is not a surprising result. Especially as validity can be defined in an arbitrary, domain-specific manner.

review_title: Official Blind Review #3

Meta-review: This paper develops ideas for enabling the data generation with GANs in the presence of structured constraints on the data manifold. This problem is interesting and quite relevant to the ICLR community. The reviewers raised concerns about the similarity to prior work (Xu et al '17), and missing comparisons to previous approaches that study this problem (e.g. Hu et al '18) that make it difficult to judge the significance of the work. Overall, the paper is slightly below the bar for acceptance.
Decision: Reject