Towards continuous machine learning on periodic crystals by ultra-fast invariants

Authors: Daniel Eric Widdowson, Vitaliy Kurlin
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: scientific integrity, periodic crystal, isometry invariant, continuous metric

Abstract:
Periodic point sets model all solid crystalline materials (crystals) whose atoms can be considered zero-sized points with or without atomic types.

This paper addresses the fundamental problem of checking whether claimed crystals are novel, not noisy perturbations of known materials obtained by unrealistic atomic replacements. Such near-duplicates have already skewed ground truth because past comparisons relied on discontinuous cells and symmetries.  
 
The proposed Lipschitz continuity under noise is a new essential requirement for machine learning on any data objects that have ambiguous representations and live in continuous spaces. 

For periodic point sets under isometry (any distance-preserving transformation), we designed the invariants that distinguish all known counter-examples to the completeness of past descriptors and detect thousands of (near-)duplicates in the world's five largest databases in a few minutes on a modest desktop computer.

Decision: reject
Reviewer scores: 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730639799913

thread_type:
official_review

<thread_object>
creator: reviewer_jeka

<content>
summary:
This paper presents a novel approach to periodic crystal representation by addressing limitations in existing methods for distinguishing unique crystalline structures. The authors propose a new invariant for periodic point sets, which is designed to improve the identification of unique crystals. This invariant provides a systematic approach to eliminating near-duplicate entries in material databases.

strengths:
The paper seems to offer a substantial contribution to evaluating novelty in crystalline materials, presenting a practical and (as far as i can tell) theoretically sound method for ensuring data quality in large-scale material databases. This seems to address an important challenge in materials science research.

weaknesses:
The relevance to ICLR is not particularly clear, as the work does not introduce a new ML method or focus on representations of ML models in the typical sense. The authors should address the relevance for iclr or machine learning in general to improve understanding by the ml community. A less technical introduction would also benefit understanding and readability. Overall, the paper may be better suited for a more specialized audience or a theory-focused computer science conference.

questions:
- Consider rephrasing or omitting the last sentence of the discussion, as it seems somewhat out of place.
- The paper is dense with technical details; consider moving some to the appendix (e.g. some examples) to allow more space for intuitive explanations and discussion.
- Invest more effort in improving the illustrations, particularly graphs. Figure 6’s text is small; consider adjusting the color scheme in Figure 1 to make it clearer and remove some of the elements to make it less crowded.
- The work may be suitable for a theory-focused ml conference, as it introduces a well-justified quality metric for material datasets rather than a new ML method.
- The language at times seems overly opinionated, with phrases like “notoriously hard problem” and “avoid such embarrassment.” These could be rephrased to maintain a more objective tone.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653043158

thread_type:
official_review

<thread_object>
creator: reviewer_qapk

<content>
summary:
This paper addresses identifying near duplicate crystal, using invariants computed from point distance distributions PDD. Apparently the authors found a fast method for detecting and describing near duplicate crystals, tested on retrieval experiments with several crystal databases.

strengths:
The development of invariants to rapidly identify near duplicate structures is interesting.

weaknesses:
This paper seems to use a lot of language, assumptions references only clear to those studying crystal structures, and thus it is unclear if or how it links to the broader machine learning community. For example, "points" are discussed throughout the paper, it’s only in discussion where it is mentioned these points are in 3D space. There is no mention of other work using point distance distributions PDDs, e.g. distance transforms in classic computer vision, invariants such as the scale invariant feature transform SIFT used to identify near duplicate photos on a repeating pixel lattice. 

No discussion of other invariants, e.g. 3D SIFT used to characterize and detecting near duplicates based on non-repeating structures in human brain images, which would seem to work quite well in the case of redundant crystal structures, using precisely the same KD search tree structure mentioned in this work, however appear unknown to the authors, e.g.
[a] Chauvin, Laurent, et al. "Neuroimage signature from salient keypoints is highly specific to individuals and shared by close relatives." NeuroImage 204 (2020): 116208.

questions:
Why is there no mention of invariants from well-known work outside of work repeating crystal structures? Particularly given identical technologies used for retrieval, e.g. KD-tree search trees, using invariants derived from non-repeating structures (e.g. human body anatomy) which would appear to be a more difficult problem? If these are irrelevant, it would be good to know why.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713319356

thread_type:
official_review

<thread_object>
creator: reviewer_daqf

<content>
summary:
The work presents a novel representation for periodic crystals that allows for efficient and effective identification of de-facto duplicates in current crystal databases. In particular the representation is Lipschitz-continuous under noise, i.e., tiny variations of crystal structures, e.g. caused by measurement noise, only yield tiny distances in representation space. The representation allows the efficient identification of duplicates or (noisy) near-duplicates in databases by pairwise comparisons in databases with in total almost two million entries, thus revealing a large number of previously unknown duplicates as well as labelling errors (same structure with different chemical composition, which is infeasible) in the databases. Last but not least, the proposed approach is able to distinguish all known counter-examples, i.e. examples indistinguishable by some previous representations.

strengths:
The work presents novel representations for periodic crystals that allows for efficient and effective identification of (near-)duplicates in existing databases. Thus the work does not only yield highly relevant practical findings, but also prevents "adversarial attacks" via claimed-new "discoveries" of materials which are however de facto already known and were only possible by exploiting deficiencies of previously proposed representations.

weaknesses:
The authors proposes a procedurally constructed representation, and "machine learning" in their work is limited to pairwise distance computation. In this regard, having "machine lerning" in the title is somewhat misleading. The contribution may be well suited also to be placed in a more application-oriented publication venue (Science?). However, it is clear that a representation (albeit engineered) with the stated properties that, furthermore, allows very fast distance computation is also highly relevant for any kind of machine learning on crystal structures, as manifested by the fact that previous works the authors build upon have appeared at NeurIPS and CVPR.

Note, I revised my score from 8 to 5 after contemplating the other Reviews and discussing with CLkj; see comment further down for details

questions:
-- In Problem 1.2, I assume (c) (2) should be symmetry, i.e., I guess d(b,c) is a typo and should be d(b,a)?

-- In 1.2. (d), lambda is not defined

-- p.2 l. 93f "gigantic loophole": It would be helpful to the reader if you could again cite the relevant reference (given earlier in the introduction)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730955558625

thread_type:
official_review

<thread_object>
creator: reviewer_x6eg

<content>
summary:
This is a new method for searching crystals. Multiple databases were used for evaluation.

The authors claim large improvements from years to hours.

The paper does make some large claims. For example, in the abstract there is a free standing claim saying
"The proposed Lipschitz continuity under noise is a new essential requirement for machine learning on any
data objects that have ambiguous representations and live in continuous spaces." This is for some types of
neural networks not for all algorithms. Correct? Please clarify by providing evidence or citations supporting the claim that this is an "essential requirement" for all ML on ambiguous data in continuous spaces. Alternatively, please clarify if this applies only to certain types of ML algorithms. 

The paper is written for an extremely specialized and narrowly focused domain and not for the general 
ML crowds found in ICLR.

The authors will need to add more explanations for general purpose ICLR audience. Otherwise this paper should be submitted to a Crystallography journal.

The main contribution is that the authors can  convert any PDD{h} into a fixed-size matrix,
which can be flattened into a vector for easy comparisons, while keeping the continuity and almost
all invariant data.

The authors also directly talk to the reviewers which is unexpected. For example, "We thank all reviewers for supporting scientific integrity, now guaranteed by the proposed invariants." or "Thank you for reading all the proofs!". Such claims are informal and not relevant for a publication in ICLR.

strengths:
This works showcases large speedups in comparing crystalline structures.
The works extends Widdowson & Kurlin, 2022.
The paper shows and extensive amount of proofs and theorems.

weaknesses:
The paper is not written for ML audience.
The paper is more relevant for Acta Crystallographica or similar journal.
The work is unnecessarily theoretical.  Giving the impression that there is not much novelty.

The title says "Towards continuous machine learning .. " but no ML example has been provided. Only similarity metrics.

questions:
It is not clear what is the theoretical vs algorithmic contribution of the work. Please explain.

It is not clear what is the difference with the paper Widdowson & Kurlin, 2022. Please explain.

It is not clear what is the underlying representation that allows for the acceleration. Please provide a detailed explanation of the data structure you use, and how it enables faster computations compared to previous methods.

It is not clear what is the relationship with other existing machine learning methods. Please explain.
For example, can you compare your method to particular ML techniques used in materials science or crystal structure analysis. Can this method be integrated with graph neural networks or other ML models used for predicting crystal properties? Such as resistance to temperature changes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269290145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
message from the area chair

comment:
Dear Area Chair, thank you for contacting us (now we know your acronym CLkj) and for confirming that 

"Theoretical contributions to machine learning and AI are acceptable at ICLR".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732597832221

thread_type:
official_comment

<thread_object>
creator: program_chairs

<content>
comment:
The program committee has reviewed the discussion on this forum. All discussion about papers should focus only on the intellectual merit of the paper. We remind all participants to review and follow the code of conduct for the conference and professional standard. Further exchange breaching the norm of the professionalism will cause participants to be removed from the conference, such as desk rejection of the paper without further review.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732612555855

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
focus only on the intellectual merit

comment:
Dear reviewers and area chair,

We hope you agree with the advice of the program chairs to "focus only on the intellectual merit", so we do not need to discuss again that the paper fits the topic of "applications to physical sciences (physics, chemistry, biology, etc.)" listed at https://www.iclr.cc/Conferences/2025/CallForPapers

In fact, the keywords "continuous machine learning" in the title are highly relevant to all machine learning because most real data are given by real-valued (not discrete) coordinates. Even if we forget about atomic structures, scans of macroscopic objects include positions of corners, salient points and other features with continuous real values, which are all noisy. Experiments on finite (often manually labeled) datasets of discretizations, e.g. pixel-based images, only approximate real ground-truth data in the infinitely continuous world.

If a machine learning algorithm was not proved to be stable under noise, its output can substantially change under a slight perturbation of input. Because all gradient-based optimization implicitly assume that target functions are smooth (differentiable at  least once), the continuity is a necessary requirement for any degree of smoothness, see a hierarchy of related concepts at https://en.wikipedia.org/wiki/Smoothness  

That is why Problem 1.2 formalizing the continuity in the Lipschitz form rigorously states the future requirements for learning continuous (stable-under-noise) representations for all machine learning on real data with continuous values and hence is highly relevant to the ICLR.

We are happy to include the recommended references on SIFT features and other descriptors of finite data, though we are still waiting for exact quotes or specific references to any past results that formally prove their Lipschitz continuity needed for detecting near-duplicates.  

We were a bit surprised by the reference " Chauvin, Laurent, et al. "Neuroimage signature from salient keypoints is highly specific to individuals and shared by close relatives." NeuroImage 204 (2020): 116208" on the topic of brain imaging, which seems a bit away from our work on periodic materials.

More generally, Figure 1 illustrated why the continuity in the periodic case is much harder than for any finite data. The key reason is the discontinuity of traditional representations based on a minimal (by volume) periodic cell (also called primitive or reduced). Any such cell can be doubled (tripled or scaled by any factor) and then almost any perturbation of a single point makes this extended cell again minimal.

This discontinuity derails all past attempts to build continuous descriptors of periodic sets by using finite subsets or bounded cells.  That is why new higher-order invariants in Definitions 3.1 and 3.4 are highly non-trivial, also justified by Theorems 4.1, 4.3, 4.4, 4.5.

The discontinuity of past invariants allowed anyone to easily disguise a known periodic structure (by a slight perturbation that significantly changes a traditional cell-based representation) and claim a "new material" without justification. Sections 5-6 demonstrated the scale of the resulting duplication in the world's largest databases of experimental materials, obtained in hours instead of years (estimated for past tools, which are all discontinuous anyway). 

Because any experimental structure can be easily perturbed without changing its physical properties, all such databases should be first cleaned by continuous invariants to avoid machine learning on skewed and unrealistic data. 

The new higher-order invariants PDD^h distinguished all (infinitely many) known counter-examples to the completeness of past invariants. The computations in Example 3,.3 and B.1 were actually done by hand, which shows how easy the new invariants can be computed.

Thank you again for focusing on the intellectual merits of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new representation for periodic crystals and a way to identify duplicates. This is an important problem, with applications in medicine. 

In its current form, the paper is unaccessible to the general machine learning audience, as found unanimously by all reviewers. Addressing this point will require a major revision which will require another round of reviews. 

It is claimed in the title that the framework is useful for all of continuous machine learning, however the paper purely focuses on a very specific application on crystals. To better support these claims, other machine learning applications will make the paper stronger and more accessible to a general audience. 

Due to the above reasons, I recommend to reject the paper. I encourage the authors to take the reviewers feedback into account for a revision.