Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to address challenging stable crystal materials generation problems via diffusion variational autoencoder with graph representation learning. Several recent advances in generative models and GNN are combined together to develop the entire workflow from data distribution learning, prediction to sample generation and property optimization. The authors demonstrate their method by using three datasets and compare with three baseline methods. 
main_review: **Strength**
- This paper proposes an entire workflow to tackle crystal materials generation. It is an important step towards the grand challenge of materials design. 
- The authors demonstrate the proposed method by using several datasets and baselines. Although the performance can be improved and some metrics are not very reasonable, these efforts can be identified as benchmarks for further exploration in the AI/ML community.  

**Weakness**

General comments: 
- The technical contribution on AI/ML side is probably marginal. Multiple recent models, like SE(3) with PGNNs, diffusion models (NCSN), CrystalNN, DimeNet++, GemNet-dQ, are simply used and combined without improvements. Although building the workflow is not trivial, I did not get an overall insight, e.g., why do you choose such models?  

- Even though this paper is probably classified as an application paper, the contribution to the domain area may not be very substantial. First, the problem of crystal generation has been posted before and several studies already exist. Second, It is difficult to evaluate the success of the paper in crystal generation as it lacks good absolute metrics or a clear performance goal for a successful model. This approach does outperform existing methods in certain areas, but these existing methods are either too limited or too underperforming themselves to serve as a sufficiently challenging benchmark.

Detailed comments:

- The authors claim to generate stable structures due to some physical or chemical motivations. However, I did not get the insight that why stable data distribution will guarantee a stable structure generation via noising and denoising. Does it depend on how do you define the stable structure, it is statistical stable or physical stable?  I suppose that is the reason why you use the NCSN model for generation tasks but it might be easy to follow the intuition.  Why not using energy-based models [1] or flow-based models [2, 3], which are widely used for molecular generation? 

[1] Liu, Meng, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. "GraphEBM: Molecular graph generation with energy-based models." arXiv preprint arXiv:2102.00546 (2021).

[2] Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M. and Tang, J., 2020. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382.

[3] Luo, Youzhi, Keqiang Yan, and Shuiwang Ji. "GraphDF: A discrete flow model for molecular graph generation." arXiv preprint arXiv:2102.01189 (2021).

- If NCSN works better than the other generative models, how do you define the sensitive noise parameters? The suggested noise levels are specifically designed for image generation, but do they work well for materials generation as well? If not, do you provide an ablation study to show the difference? Since now I do not see a code submission, I am curious about the reproducibility because the workflow looks pretty complex consisting of multiple-step training, rather than a straightforward end-to-end generation. 

- Another weakness is the reconstruction loss from the VAE architecture. Visually, it is difficult to tell in Figure 3 if the reconstruction structures have shifted significantly or not. How big an impact does this reconstruction loss have on properties such as formation energy, for example? Is it also possible for elements to change upon reconstruction? In Figure 3 on the MP-20 dataset, it looks like the element has shifted for the second structure; is this really the case or a plotting error?  

- A characteristic of this method is that the crystal parameters (composition and # of atoms, crystal lattice) are decoupled and predicted separately from the atomic structure. In addition, the generation is from random initiation, how do you define the upper and lower bound of your initialization, for different parameters? We know composition, # of atoms, lattice parameter share different scales and they are a mixture of continuous and discrete, do you normalize them? Have you considered dequantization?  It would be helpful to understand how to set up the dataset if these details can be provided. 

- Validity criterion is likely too weak and likely not useful in evaluating generation performance, as most structures should not have interatomic distances anywhere near 0.5 Angstroms. Charge neutrality may also not be a good criterion due to the existence of defect structures or materials such as electrides. 

- Authors use three existing datasets from the literature, containing (1) perovskites, (2) carbon, (3) and inorganic crystals. From what I can tell, the main contribution to these datasets by the authors is truncating the inorganic crystal dataset to a maximum of 20 atoms. Is 45000 for the inorganic crystal dataset a sufficiently large number of data for this type of problem, and to serve as a benchmark dataset? For the perovskite dataset, is it really necessary to use generative models here since they have a fixed ABO3 composition and do not have structural diversity, which defeats the purpose of a 3-D crystal generation dataset.

- Property optimization is not a fair comparison since the author’s method chooses the best out of ten, whereas other methods like Cond-DFC-VAE only use a single sample from conditioned generation. More importantly, property optimization based on optimized latent space should be able to generate realistic materials, but it might not be guaranteed due to several optimization issues. For example, gradient-based optimization in latent space (similar to JTVAE 2018 paper) may be trapped into local minima, do you consider a global optimizer, e.g., Bayesian optimization or CMA-ES?  I understand it also has scalability issues if the dimension of latent space is relatively high. In addition, some papers [GraphAF and conformation generation] use RL methods to generate a goal-oriented design. 

- For property metrics, there is a high possibility of generating very out-of-distribution crystal samples from the training data; since the property is predicted by an independent GNN trained on the same data, there is a strong concern about the property predictions will be inaccurate and meaningless.

- It looks like G-SchNet is a strong contender in this paper, however the authors implement it without considering periodicity. This is not necessarily a failure of the G-SchNet method, but rather it is used improperly and cannot perform well, and allows the work in this paper to perform favorably compared to it.

- Scalability issue. The dataset used here is still a small-scale graph, does it work well for a large-scale crystal generation, such as more than 20 atoms in Materials Project?  Do you anticipate any limitations of the current workflow? If so, could you add more discussion about the existing limitations of CDVAE? 


summary_of_the_review: The paper proposes to tackle challenging problems and provide a reasonably workable framework with several benchmarks. However, the core contributions from either the technical side or domain science side are not substantial and significant. Multiple pieces of work need more clear justification and explanation, and the metrics need to be improved with a more physically reasonable definition. I also expect to see a better comparison with the current baselines. Probably, the work is still ongoing so I recommend to reject it in the current form but expect to see more improvements in the near future.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a score-based Diffusion Variational Autoencoder to generate the periodic structure of stable materials (crystal structure generation). 

**Main contributions include**
* Creating a set of standard tasks, datasets and metrics to unify the evaluation process of inorganic crystal structure generation
* Proposing a **SE(3) equivariant, periodic boundary condition-sensitive** decoder to generate crystal structure structures based on a latent vector.


main_review: **strengths**
* The proposed score-based generative model for crystal structure generation, i.e., CDVAE, is novel. Although similar techniques have already been explored in other structure generation problems [1,2], I still believe the proposed model is quite novel as it addresses several important challenges in material structure generation such as **periodic boundary condition and composition prediction.**
* The proposed model is SE(3) equivariant and periodic boundary condition-sensitive, based on the recent advance on equivariant graph neural networks. It also considers the Periodicity influences on denoising target.
* The experimental design seems comprehensive, which may serve as standard benchmarks for inorganic crystal structure generation. The experimental results seems quite promising.
* The authors promise to provide the source code of the project after the review process.

**weaknesses**
* All the datasets included are inorganic. It would be very nice to see some initial results on organic crystal structure prediction.
* The lattice constant **L** is predicted from the latent vector and seems it is fixed during the decoding process.  You say that the structure of materials shoudl satisfy $\partial E / \partial L = 0$. How is this condition modeled in CDVAE? Can we update the lattice constant during decoding?
* The crystal structures often exhibit some symmetrcity, e.g. space group. I do not see any discussion related to this concept. Can CDVAE tackle the space group, or is this concept not applicable to the current datasets?

**Reference**
1. Learning gradient fields for molecular conformation generation
2. SE(3)-Equivariant Energy-based Models for End-to-End Protein Folding

summary_of_the_review: Overall, I think the submission is of high quality and reaches the bar of ICLR.
It would be nice if the authors could address the concerns listed above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new generative model for 3D periodic material molecular structure. They designed a special variational autoencoder, where the decoder is parameterized by the denoising score-matching framework. Experiments demonstrate the model can successfully generate valid and realistic material, and optimize desired properties.
main_review: Strengths:
- The tackled problem is specific but new and interesting for the machine learning research community, and the paper nicely provided 3 benchmarks to standardize the material generation problem.
- The paper is well-written and easy to follow. Both technical and empirical parts are clearly illustrated.
- The experiments are extensive to demonstrate the effectiveness across different datasets and metrics.

Weaknesses:
- Some detail of the "conditional score matching decoder" is confusing for me: how you sample the noisy atom type $\tilde{A} ~ (p_A +\sigma_A p_c)$. In my understanding, $p_A +\sigma_A p_c$ is not a valid categorical distribution, so I'm concerned with the sampling. Is there any softmax function imposed on it?
- Another concern is about the experiments. I appreciate the authors' efforts to provide several meaningful benchmarks, but the metrics in Sec 5.2 seem still very weak by just comparing validity, property, and fingerprints. There are some recent papers working on molecular conformation generation (samples are listed below), where the two conformation sets are compared by **coverage** and **matching** score. The main differences are the periodic and unconditional nature without graph input, but I think these two evaluations can still be easily adopted since you have also reported similar metrics in Sec 5.1. Could you provide more empirical justifications on the coverage and matching score?

 References for weakness 2:
- Xu, Minkai, et al. "Learning neural generative dynamics for molecular conformation generation." arXiv preprint arXiv:2102.10240 (2021).
- Ganea, Octavian-Eugen, et al. "GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles." arXiv preprint arXiv:2106.07802 (2021).

summary_of_the_review: The problem is specific but new to the general machine learning audience, and the proposed VAE model with equivariant score matching decoder is interesting for me. However, I still hold some technical confusion (weakness 1), and I think some important metrics from the highly related topic are missed in the material generation task, which is the most important empirical part of this paper. So currently I vote for a weak rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach to generate stable crystal structures using variational autoencoders without having to train on out-of-equilibrium data, which is scarce for materials. Beyond that, benchmark datasets and tasks for materials generation are proposed.
main_review: The approach is well motivated and novel. Being able to train a generative model without requiring large datasets of non-equilirbium configurations is very relevant considering the computational cost of electronic structure calculations. 

The proposed CDVAE incorporates the necessary invariances and a connection to the harmonic approximation is shown. CDVAE shows state-of-the-art results in validity and diversity of the generated materials. The comparison G-SchNet is not ideal, as a proper adaption to materials would have included periodic boundary conditions. These should be relatively straightforward to implement since they are already available for SchNet. The evaluation could be improved by performing at least a limited number of QM calculations on a small random subset of the generated structures.

The optimization of the formation energy is another strong point of the paper.
summary_of_the_review: The proposed approach is a novel and relevant contribution to the domain of materials generation.
review_title: null

Meta-review: This paper introduces a model, named Crystal Diffusion Variational Autoencoder (CDVAE), that can learn to sample valid material structures. It accounts for known symmetries (SE(3), permutation) of the structure via SE(3)-equivariant GNNs.

The proposed model is a complicated combination of many existing models / modeling techniques (VAEs, NCSNs, diffusion models) but it is not entirely ad hoc; the revised paper does a reasonable job in justifying the many different modeling choices made. 
Existing model components, often designed in the context of molecule generation, do not account for the periodicity of the crystal's lattice structure; so this paper introduces modifications to account for this periodicity.

The paper evaluates the model on several datasets and also introduces new benchmarks that can be used for further research. The experimental results look promising but there are a few remaining clarity issues with the metrics used (cf. reviews).
Decision: Accept (Poster)