Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  The paper studies representation learning for periodic objects such as crystal. The paper formally defines the periodic invariance property and periodic pattern encoding. To achieve both periodic invariance and periodic pattern encoding, the author proposes two type of constructions of local graphs, modified from existing methods (CGCNN and Graphnormer). The author further develops a proper message passing based transformer over the proposed graph construction method. Overall the paper is well-written and informative for representation learning on periodic objects. The experimental result shows the effectiveness of their designs.  
strengths_and_weaknesses: Strengths:
1. The written is extremely clear and informative in terms of analysis and literature review. Reading the paper is a kind of joy. The author also clearly state the difference with respect to several similar previous works.
2. The proposed method achieves all invariant properties desired. The design is simple, clean, and meaningful. 
3. The proposed properties for periodic objects are formal and informative. if these properties are proposed the first time here, this is a good contribution. 

Weaknesses:
1. Even stated several times, the proposed graph construction methods are similar to previous works. In this perspective, the novelty is a bit reduced. 
2. The performance improvement comparing with the strong baseline ALIGNN is small. Given ALIGNN uses angular information, it would interesting if the author can adapt angular information to the designed Matformer. If the author can further improve the current Matformer, this paper will become extremely good. 
questions: Please see previous comments. 
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the author/s proposed a graph transformer architecture, named Matformer, to infer the physical properties of crystal materials in which their atomic representation is encoded as graphs. The Matformer architecture encodes repeating patterns and is invariant to periodicity; these two properties are fundamental in modeling crystal materials. The performance are assessed on standard benchmarks, considerably outperforming the baselines. 
strengths_and_weaknesses: **Strengths:** 

*Originality*
Proposing a model that is invariant to pattern periodicity and that it explicitly captures repetitive patterns in a graph is a novelty.


*Quality*
The quality of this work is good; almost all proofs are present in the paper or in the appendix. The experimental part is clear and the reference adequate.

*Significance*
I think this work is significant for the community since it addresses specific properties of graphs that are not commonly studied.

**Weaknesses:** 
* One of your direct competitors is Graphormer, but there are no comparisons, why?
* In line 183 you said "... rigorously prove ..." , where is the proof?
* In line 46 you used $\mathbb{N}$ for the categorical case... the natural number is infinite, do you have infinite classes?
* In Eq 2 the definition of $q_{ij}^h$ consists of the concatenation of $q_i$ three times... is it correct?
* In line 215, you said that one of the differences with respect to Graphormer is that " ...  Graphormer treats every atom as a single node.", but you are doing the same following the definition of $a_i \in \mathbf{A}$. Can you elaborate more?
questions: My questions are reported in the weaknesses
limitations: I wondered about the model's performance when no periodicity or repetitive patterns in the graph are present.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers the property prediction tasks for crystal materials. This problem can be viewed as predicting a global target for a graph. The uniqueness is that the graph is periodic. The graph crystal structure repeats in the 3D space. Past works didn't really emphasize on this periodic invariance, or handle the periodic pattern encoding. The most commonly seen strategy is to set a fixed radius for an atom, and top $t$ closest neighbor atoms to this atom are connected to this atom with an edge. Such design implicitly considers the periodicity of the crytals. However, in some corner cases, such invariance may not be guaranteed. To this end, this paper proposes to explicitly capture the periodicity through self-connecting edges. Intuitively, the self-connecting edges connect one atom with its nearby duplicates. Since it is in the 3D space, there are multiple duplicates in different directions. So, multiple self-connecting edges are added to model the periodicity. The resulting model, Matformer, outperforms other GNN-based methods on two datasets, materials project and JARVIS. 
strengths_and_weaknesses: Pros:
1. The paper is written clearly. Though there are some typos, the general idea and methods are easy to follow. And the real-world problem is also important in materials science.
2. The ablation study is presented. I think the key argument of this paper is the periodic invariance. So proving the importance of the periodic module is critical.
3. Multiple datasets are included for validation. 

Cons:
1. Essentially, the paper extends the existing graph based methods for crystal property prediction. However, another direction is Euclidean neural networks. These models are not compared in this paper, such as E3NN.
2. The authors decide to drop the angle information due to high complexity. But ablation study should be provided to support this claim.
3. As I mentioned, the ablation study is critical for performance and operations. It would be better to expand the ablation study section and move the dataset descriptions to appendix.
4. From the comparison numbers, the improvements over others like ALIGNN are not that significant. 
5. Experiments only evaluate MAE. More metrics can be added.
questions: 1. I belielive $\ell_1, \ell_2, \ell_3$ are 3d vectors with only one dim non-zero?
2. It's better to include $\ell$ as a superscript in your denoting $f^*_i$ for the $\ell$-th layer
3. What do you store in edge feature vector?
4. In Table 1, it seems that the numbers from your reproduction and the original paper differ by a lot. Why is that?
5. Why GATGNN etc. do not show up in Table 3?
6. Whether to add the angle information is always debatable in this field. ALIGNN often becomes the second best numbers and it encodes angles. But you claimed angles "induce high complexity", could you clarify on this?
7. For each material property, do you train a separate model? or you use one learnt representaion for all?
limitations: 1. I do believe it would be better for authors to include the discussion for the comparison between Euclidean neural nets and graph neural nets in encoding the crystals.
2. Some interpretations on the results can help me to better understand the perforamnce gain.
3. More evaluation metrics would make it more convincing.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Periodic graph is ubiquitous in the real world, such as crystal material. Representation learning on periodic graphs is a significant task for downstream tasks such as property prediction. This paper proposes a new transformer-based framework for the representation learning on periodic graphs. Specifically, it comes up with a new strategy to construct graphs and a way to encode lattice matrix L. The experiments section shows the effectiveness of the method.
strengths_and_weaknesses: Strength
1. The paper formularizes the periodic graph in a clear way using A, P and L matrix.
3. The experimental results show the effectiveness of the proposed method.

Weakness:
1. The motivation of the paper can be further improved:
    a) Why do we need to construct a graph from the unit-cell coordinates rather than directly using coordinates to conduct representation learning? What's the benefit to borrow the graph structure here? I believe there are a few works for representation learning on 3D positions such as [1], [2] and [3]
    b) What is the motivation to form self-connecting edges to contain the information of lattice matrix? Why not just concatenate it with the representation of a single unit cell?
2. It seems that section 3.1 and 3.2 can be moved to section 2 or they can just serve as a single section, since it seems that they are the motivation/theoretical ground of the proposed model not the model itself.
3. Line 45, the paragraph is titled as "Crystal property prediction" but it seems that the whole paragraph is about the definition of peridic lattice. The title might be modified accordingly.
4. It seems that E(3) invariance has been mentioned even in the abstract but it is formally defined in section 3.1.
5. Line 87, it mentions that "the structure f a cell remains the same when ...". Does the structure here mean the the output of the "f" function mentioned in the Definition 1 above?
6. Line 90, It says "periodic invariance is also shown necessary for generating valid crystal representation". Why?
7. In table 2 the paper compares the time complexity of the proposed method with ALIGNN. It would be great if some complexity analysis is provided here.
8. Some relevant works [4, 5] regarding the periodic graphs generation can be discussed in the related works section.
9. The code is not published for now to reproduce the results.


[1] Xu, Minkai, et al. "Geodiff: A geometric diffusion model for molecular conformation generation." arXiv preprint arXiv:2203.02923 (2022).
[2] Kim, Seohyun, Jaeyoo Park, and Bohyung Han. "Rotation-invariant local-to-global representation learning for 3d point cloud." Advances in Neural Information Processing Systems 33 (2020): 8174-8185.
[3] Court, Callum J., et al. "3-D inorganic crystal structure generation and property prediction via representation learning." Journal of chemical information and modeling 60.10 (2020): 4518-4535.
[4] Xie, Tian, et al. "Crystal diffusion variational autoencoder for periodic material generation." arXiv preprint arXiv:2110.06197 (2021).
[5] Wang, Shiyu, Xiaojie Guo, and Liang Zhao. "Deep Generative Model for Periodic Graphs." arXiv preprint arXiv:2201.11932 (2022).
questions: I would suggest the author address the questions mentioned under the "Weakness" in the "Strengths And Weaknesses" section of the review. 
limitations: The limitations of the model are not mentioned, the negative societal impacts of your work are marked as N/A.
review_title: null

Meta-review: metareview: This paper had borderline reviews. While the reviewers felt that the method was novel and the presentation very good, they also cited weaknesses such as limited performance gains over baselines and limited novelty. The authors responded in detail to many of the concerns, including with additional experiments, causing several reviewers to increase their scores. We encourage the authors to incorporate aspects of these responses in the final version.


Note: As the AC I will note that I did not find the authors’ very long summary of the discussion helpful in making my decision. I briefly skimmed but did not fully read it. Such a summary/discussion from the authors will obviously be presented through a very biased lens, and so it is much more helpful for me as an AC to directly look at the discussion myself in making a decision, and to ask follow up questions to directly to the reviewers if any clarification is needed.
confidence: Less certain
Decision: Accept