Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a novel way to train variational auto-encoders and apply it to generate small molecules. The paper is well presented and the method easy to follow. Overall I think the goals of the article are somewhat disjoint, and it feels like the proposed method (LOVAE) should be applicable to standard datasets as well (e.g. CIFAR10), which would have helped the “story” of the paper. This is especially important since I find the motivation for the LOVAE reasonably weak, and there’s not much in terms of theoretical backing.

Some further comments are as follows:
* Glow isn’t really a good “general reference” on reversible generative models, many successful versions exist prior to it.
* The QED metric seems to top out at 0.948, why does this happen? It doesn’t look very informative as is.
* All images, notably Figure 5, should be in vector format. It’s currently hard to see what’s in there and it does not scale.
* There are some language errors, e.g. “the training of decoder”. It might be worth having another look at language throughout.

review_title: LATENT OPTIMIZATION VARIATIONAL AUTOENCODER FOR CONDITIONAL MOLECULAR GENERATION

final_score: rating: 3: Clear rejection
review_content: The paper proposes a way to use conditional variational autoencoder setups to design materials conditioned on properties (labels). The problem is of some significance in the materials design space using generative models, using machinery from neural network modeling - RNNs, VAEs, GANs and optimization. 

The writing is clear, and material is readable. However, I note the following issues with the content:

1. Problem motivation not very clear
I felt that the paper does not provide good enough context into the problem and had to piece it together from related work cited (e.g. [1], [2]). In particular, the work seems to be quite related to the work [1] by Kang and Cho, where they use a graphical model fashioned as a VAE, with the dependencies being the label or property y, the latent variable z and the molecule x - i.e. ELBO(log p(x, y)) in equation (2) of the paper. 

The paper's claim is that they address problems with the vanilla conditional VAE setup. However, it seemed difficult to figure out what exactly the paper was trying to fix in the original VAE setup. They mention "posterior collapse", although there does not seem to be much evidence given (please clarify) to show these deficiencies in the setup - I assume that we should look at section 5.2, Table 1, but no mention of posterior collapse is made in these likelihood numbers. 

I would also like to quote the following lines from the beginning of section 4. It is unclear again what is meant by saying that the decoder is sensitive to the encoder results and the accuracy might vary ...

"Since the latent variable z generated from the encoder is inputted to the decoder to calculate Ltotal,
the encoder and decoder are closely related. If the encoder generates non-informative or highly
volatile z, then the decoder is difficult to reconstruct x from z. In this case, it may not be optimal to
update the encoder and decoder simultaneously. The decoder is very sensitive to the encoder results,
z, and the accuracy of the decoder training may vary according to the encoder results. In other words,
the result of the encoder can be optimized in advance in direction in which the decoder trains well.
That is, better encoder and better latent variable can make the decoder even better."

2. The optimization process: Aside from the issues regarding the paper's motivation, the optimization procedure adopted does not seem to be very clear and could perhaps need clarification. The paper mentions a 'two step' optimization process, of first updating the encoder (keeping decoder fixed) and then updating the decoder. This is strongly suggestive of the wake-sleep algorithm by Hinton et al [2], but if that is so, the exposition would need considerable elaboration with equations, optimization objectives, and clarification on how the optimization machinery used solves the actual problems at hand (which I also had some difficulty following, as explained in the previous point). In other words, the connection between the technique adopted and the problem solved is not clear. 

I think the paper needs additional work in explaining the motivation of the problem with examples on why the original VAE setup is deficient and what the proposed fixes do to solve the problem at hand. 

[1] Kang and Cho: https://arxiv.org/pdf/1805.00108.pdf

[2] Wake-sleep algorithm https://www.cs.toronto.edu/~hinton/csc2535/readings/ws.pdf



review_title: Motivation and technical content unclear - reject

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose extensions to VAE training to stabilize the learning process. First, the encoder is trained while the decoder is fixed. This is followed by an additive optimization of the latent variable from the encoder. Finally, the decoder is updated based on the updated latent variable from the encoder. They conduct experiments on molecule generation tasks to show the efficacy of their method. The paper is well written and easy to follow.

My main concern with this paper is novelty. Their method of pretraining the encoder is not particularly novel and has been applied in Natural Language Processing. For example, in He et al (ICLR 2019) [2], the encoder and decoder steps of the VAE are separated and more steps of the encoder are performed in each iteration. 
Also in  another related work, Li et al (EMNLP 2019), they combine this method with another trick (KL-thresholding). The difference between the proposed method and these existing methods are not clear (and I believe are minor). In fact the method of He at al, can be considered as a generalization of the proposed method, with minor differences.

If most of the techniques proposed in the paper are well known in the NLP community, then the only novelty is the application of this technique to SMILES strings. It is not surprising at all that it is better than vanilla VAE training. Even in this case, I think the comparisons with existing work is not thorough. 

Is it possible to compare the generated molecules on a broader set of criteria, for example using the MOSES evaluation framework? [1]. This allows one to evaluate the efficacy of the method along different criteria like validity, novelty, uniqueness, FCD, etc and compare it with a broader set of methods. The code is available at https://github.com/molecularsets/moses

I would request the authors to carefully position their work with related work mentioned above.

[1] Polykovskiy et al. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models

[2] He et al. Lagging Inference networks and posterior collapse in Variational Autoencoders (ICLR 2019)

[3] Li et al. A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (EMNLP 2019)

review_title: Application of existing work

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The paper proposes a latent optimization VAE (LOVAE) to insert an optimization step of the latent variable and alternately update the encoder and decoder of conditional VAE. The additional latent optimization step optimizes phi with respect to the reconstruction loss. It makes sense the reconstruction loss is low in table 1. 


1: Although the adaptation of latent optimization to cVAE is interesting and can improve the performance, the idea or the contribution is not significant enough, especially considering many existing VAE variants (including those already discussed in the paper). 

2. One of the key results -- to generate molecules with similar property as the conditional property. Table 3 only compares with vanilla VAE, which doesn’t take condition property as input. The baseline is not persuasive. It will be more convincing if the authors can compare with other VAE types (i.e. include, but not limited to cVAE) and other non-VAE but reverse molecule design methods (i.e. GAN type, or not generative models).  Additionally, it is less convincing that the authors compare with different sets of existing methods on different tasks, for example Table 3, Table 4, Table 5. What are the results look like if say we add Table 4's other methods as a comparison for  table 3?  for example. JT-VAE (Jin et al., 2018)  , ORGAN (Guimaraes et al., 2017)  ,GCPN (You et al., 2018a) . The paper may need some more coherent experiments to be more persuasive. 



3. It will be more clear if the authors can mark the best values in tables in bold. 

review_title: review

Meta-review: The reviewers agreed that the paper can be improved in several aspects such a motivation, novelty and framing of the contribution. The reviewers also liked the clarity of the presentation and some of the ideas. The reviewers constructive input may be used to improve this work. 
Decision: Reject