Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an MCMC-based framework that introduces conditional transformations that respect certain 3d structural properties of proteins with the motivation to learn conformation invariant representation. 
strength_and_weaknesses: Strengths:

+ MCMC-based strategy for learning representations that are invariant to conditional transformations

Weaknesses:

- Most technical details are moved to Appendix to save space and the extra space is used to explain details of these details moved to Appendix, which makes it difficult to follow certain sections of the paper.

- Experimental results are included with not much insight. It was not clear whether the improvement over the baseline techniques for four tasks were due to proposed conditional sampling or other aspect of the learning changed by MCGD algorithm. 

clarity,_quality,_novelty_and_reproducibility: The paper was too heavy in introducing concepts and very light in explaining or illustrating these concepts.  Key details were moved to Appendix and the extra space is used to explain details of details, which makes it harder to follow the ideas in the paper. For example in Page 6 MCGD algorithm is not explained at all and the reader was referred to Appendix A.9 and that extra space is used to explain the convergence properties of MCGD. Even the Appendix A.9 does not discuss what MCGD does and only discusses other details about the algorithm. The reader now needs to check the paper from 2018 to see the key ideas of the MCGD algorithm. Only half a page is used for experiments due to page limitation and yet again the reader is referred to Appendix for additional experiments. 

There is some novelty in the MCMC based conformation invariant sampling strategy, but it is hard to judge the significance of this novelty given limited empirical evidence. 

Minor Corrections:

Please correct cyro-EM as cryo-EM

Remove a in "using directed a forest"

Remove extra to in "converges to to a unique stationary"  

How true it is to say that V is the "set" of atoms given that this set does not contain distinct elements, so it can't be a set? V is the set of nodes (not atoms). 

"n" was first used to denote the number of amino acids but later on it was used to denote the number of nodes in V, and thus the number of non-distinct atoms.  If m is the number of atoms, then what is n?



summary_of_the_review: Although the proposed MCMC based conditional sampling strategy was well thought out, some key concepts were not well explained in the main text. Experiments suggest the proposed sampling strategy improves over several baseline from the literature when these baselines were trained with conformation invariant samples generated with the proposed sampling strategy, but experimental evidence is not very compelling about whether this improvement is due to the networks learning better representations or due to change in some other aspects of the learning process involving MCGD.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a "conformer invariant representation learning" for protein property prediction in this submission. The basic idea is to sample viable conformation based on the given input protein and conformation. MCMC for sampling conformer distribution is integrated to representation learning via the existing Monte-Carlo Gradient Descent (MCGD) algorithm. The authors have also reported empirical results for four protein property prediction tasks on the ATOM3D data set. 
strength_and_weaknesses: The authors developed a learning framework that may help protein representation learning considering possible conformation transformations for given input proteins. The procedure of generating and sampling viable conformations is integrated by MCGD to protein representation learning that may help achieve better protein property prediction tasks. 

This reviewer has some concerns: 

1. The theoretical analyses for the ergodicity and the convergence to the steady-state conformer distributions seems to be problematic. Based on the description of the procedure, the sampled conformers will be validated by "structure validation tools such as Molprobity". Introducing such procedures may violate the desired ergodicity critical in propositions 3.2 and 3.3. Based on the presentation, this reviewer was not sure whether the proposed directed forest construction procedure will always guarantee that all the viable conformers can indeed be included. If not, the procedure does not really have the guarantee to be "conformer invariant" exactly. 

2. The theoretical analysis appears to be dependent on Definition 2.2  of "rigid backbone protein conformations" while in the main text and supplement, the authors emphasized that the proposed work is for non-rigid transformations. The authors may clearly state the limitations of the proposed procedure. 

3. The presented empirical results are limited with some tasks that the proposed procedure under-performing the baseline without considering "conformer invariance". Also, the proposed procedure appears to share some similarity with augmentation tricks. The authors may want to move the results in the appendix with augmentations into the performance comparison experiments of the main text to check whether the proposed procedure is meaningful or significant. Also, if the authors emphasized that considering more flexible conformers is critical, the performance comparison with other models considering rigid transformations may need to be checked. 
clarity,_quality,_novelty_and_reproducibility: The efforts of considering possibly non-rigidity of protein conformations besides other group invariance in protein representation learning can be important. However, there are several major concerns on the presented theoretical results. More comprehensive empirical results may be needed to show the significance of the developed procedure. There are also problems in presentation quality as detailed below: 

1. The authors should pay careful attention to math notations. For example, on page 3, when introducing the math representation of protein conformers, it is not clear whether the node set is for the number of amino acids or atoms in each amino acid. What did the authors many by  "m-atom" proteins? If the authors directly modeled m atoms, why V={1, ..., n} before Definition 2.2? 

2. The notations in the last paragraph and Figure 2 on page 4 are not consistent either. For example, The group of actions is denoted as $G$ in the last paragraph but $\mathcal{G}$ in Figure 2 and its caption. Some notations and acronyms in the paper were not defined clearly, which could be improved for better readability. 

summary_of_the_review: The authors developed a learning framework that may help protein representation learning considering possible conformation transformations for given input proteins. This reviewer has concerns on both reported theoretical and empirical results in the current version as detailed above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript discusses the need to consider a broader class of invariances in protein modelling than the rotation and translational invariances typically considered. The paper introduces the concept of conditional invariance, defined by transformations that modify the sidechain degrees of freedom of a protein, while leaving the backbone degrees of freedom unaltered. The authors propose using a Markov chain Monte Carlo procedure to generate alternative sidechain conformations during training such that the internal representation in a supervised learning task becomes invariant to the sidechain conformation of the input structure.
strength_and_weaknesses: The paper does a good job of formally describing the problem and the details of the proposed algorithm. The manuscript seems technically sound.  However, I struggle with the motivation for the method. The authors introduce the conditional invariances to sidechain conformations as a fundamental property on par with rotational and translation invariances of the global structure of the molecule, but this equivalence seems shaky. The standard desire for invariance to rotation and translation of a molecular structure has a clear motivation: we wish any result to be independent on the arbitrarily chosen coordinate system of our 3D coordinate system. Or in other words, the protein structure that we care about has 6 degrees of freedom fewer than when we parameterize it using Cartesian coordinates of all its atoms. From a physical perspective, if we are considering intrinsic properties of a protein, for instance its stability, the position and rotation of a protein is fundamentally irrelevant. All physical interactions between atoms in the molecule are independent of the global orientation and translation.

In contrast, the invariance towards sidechain conformations is less clear cut. The authors state that "for most proteins, regardless of their side chain conformation (as long as viable) under consideration – their protein fold class/ other scalar properties remain the same, their mutation (in)stability remains unaltered, protein ligand binding afﬁnity (apart from changes at the ligand binding site) remain the same, etc.". While it is true that the fold class stays the same because it is typically defined only based on the backbone conformation of a protein, the remaining properties (stability, binding affinity), are certainly dependent on side-chain conformations. The keyword here seems to be "as long as viable", which implicitly establishes the equivalence of different structures. But what is deemed "viable" will depend on the physical quantity we wish to predict, and the resolution at which we are modelling the system. The desired level of "invariance" will thus depend on the modelling task. For some tasks, the exact sidechain orientations will not be important - for other tasks, they will be critical.

Rather than consider the set of "equivalent" sidechain conformations, the standard approach in statistical physics would be to consider the *distribution* over sidechain structures. Any property of interest (e.g. prediction of a downstream task) could then be considered an expectation under this distribution. For instance, if we observe the physical system at some temperature in a fixed volume, the Boltzmann distribution will gives us such a distribution. In the case of this manuscript, the authors are interested in keeping the backbone degrees of freedom fixed - and would thus consider this conditional distribution of sidechain given backbone. The above would be valid for a molecular forcefield parameterizing the energy of a system - but one could presumably make a similar argument for MolProbity: that it induces a probability distribution over valid sidechain conformations, over which you then calculate the expectation. As far as I can see, this would lead to a similar procedure as the one the authors are proposing, but where you use the validity scores of molprobility as an energy rather than considering all sidechains conformations as equivalent. This approach would be easier to defend, because it does not rely on exact equivalences between structural states.

Alternatively, the method could be sold purely as a principled way to do data augmentation, for increased performance in a downstream task. 

I would suggest that the authors rewrite "1. introduction" and parts of the "2 Conditional Invariances for Proteins" so that the rotational and translation equivariances of the rigid body case are not presented as equivalent as the conditional invariances that they discuss here. As sketched out above, it would probably be most fruitful if the authors get rid of the concept of "conditional invariance" alltogether, because these invariances are task specific and only approximate - and that they instead present the results in terms of expectations over distributions over sidechain distributions. If the authors stick with "conditional invariance", they should make the limitations of this concept clear - and clarify that the underlying goal is to do meaningful data augmentation in protein systems, and perhaps let that be the driving motivation in the paper.
In short, my main problem with the paper is that the conditional invariances are elevated to something fundamental about the physical system - when in reality it is merely a consequence of the fact that certain details are irrelevant for a particular downstream task.  

Finally, for completeness, small fluctuations of the *backbone* structure will have similar properties as the sidechain "invariances" - that they will in many cases be "equivalent" with respect to a downstream prediction task.  It would make sense that the authors mention that they make the simplifying choice of considering only the sidechain conformers in this paper, ignoring this other source of fluctuation.

clarity,_quality,_novelty_and_reproducibility: ## Clarity
The paper is well written with clear figures to support the story.

## Quality
The manuscript seems technically sound. The method is supported with proofs in the appendix.

The result section should include a baseline where the MCMC is done offline, rather than as part of the training procedure. If a pool of samples could be generated prior to training and used as standard data augmentation (e.g. sampling randomly from the pool in each epoch), it would simplify the training procedure considerably. The Additional Result section has a "gold standard" baseline, which might actually be the baseline I request, but is not very clearly explained. The authors should describe how the "gold standard" is created - how long the offline MCMC has been run to create the gold standard (is it fully converged?), and what they mean when they write that the "MCMC is restarted in every epoch during training". If the method is indeed better than the offline method, it would be helpful if the authors could describe explicitly why they think that their approach is better than simple data augmentation (currently, it says "which can be attributed to the guarantees it provides to the learning framework" - which was not clear to me. Which guarantees are these?)

## Reproducibility
As far as I could see, no code was shared as part of this submission, and there was also no Code Availability statement in the manuscript. The authors should state if they intend to share the code. 


## Minor comments

Caption, Figure 1. "Just the alpha carbon and the side chain atom". "atom" -> "atoms"

"Traditionally, protein structures are solved by X-ray crystallography or cyro-EM". "cyro-EM" -> "cryo-EM"

"and the obtained structure is normally considered a unique 3D conformation of the molecule."
It is unclear what "unique" means here". Do you mean "representative"?

Proposition 3.3 and Proposition 3.4. The authors should state if these results are any different from the general results known for MCMC simulations of proteins. I would think that it is well known that transitions kernels such as the one described would lead to a Markov chain with a unique steady state distribution. If this is indeed the case, the authors should state that they are simply restating known results. If not, they should state how their result is different.

"Then, a simple way to obtain conformer invariant representations of protein x_j (apart from using trivial functions such as a constant function or function independent of X_v)"
What is X_v here?

summary_of_the_review: While the paper is technically sound, the motivatation for the proposed method is not convincing. Furthermore, the results section does not fully convince me that the complexity of the method (i.e. the MCMC sampling during training) is necessary to achieve the reported gains.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary of the work
The paper studies protein representation learning from the 3D structure. Recent works have shown that although the back-bond of protein 3D structure is fixed, the side chains associated with each back-bone atom are flexible and may correspond to different conformations.

The authors proposed a method to augment the current 3D structures with randomly sampled conformations from the gold 3D structure. The samples are used as additional data for both training and inference in downstream learning tasks where the representation of the proteins is approximated by the average of the representation of the conformations.

They show that this simple data augmentation approach improves the performance of supervised downstream tasks that only relies on 3D CNN. 3D GCN GNN to transform the gold conformations in the Atomic3D benchmark dataset.

Even though methods for sampling confirmation using MCMC exist in the literature, a novelty of the work relies on the new sampling approach that constructs for each back-bond atom a directed tree by breaking the loops in the side chains associated with the back-bond atom. Sampling was done by the sampling point set of the nodes in the trees starting from the root nodes down to the leaves, where the later point set is conditionally dependent on the parent's conformations. An MCMC approach is used to sample conformations of the side chains from these point sets.



strength_and_weaknesses: Strength
This is an interesting and novel idea on conformation sampling.

Good experimental results compared to the baseline that does not leverage augmented data.

Weaknesses


I see a lot of strong assumptions without thorough validation and supported evidence both theoretically and empirically

Step 4 in Algorithm 1: trees were sampled independently. Is there any evidence that the conformation of side chains of different amino acids are independent of each other?

Representing an amino acid as a tree requires breaking circles (if exists) at a random point, different breaking points result in different tree structures. The current conformation sampling approach assumes conditional independence between a node and all of its grand-parent given the parent nodes. This assumption seems a strong assumption that may lead to ill-approximation of the true conformation distribution. Is there any way to validate that the distribution of the sampled data fits the right distribution of conformation?


Since evidence about the approximation of the true conformation distribution is not provided, to demonstrate that the proposed conformation sampling approach based on the atom forest is a more effective data augmentation approach than other simple data augmentation approaches, the following simple  data augmentation should be considered as a baseline to compare to:
+ For each back-bond atom, keep the back-bond atom fixed, and randomly transform all the side-chain nodes associated with that back-bond atom using a random rotation matrix.

This baseline will validate that the data augmentation process proposed in the paper is an effective data augmentation approach compared to this simple data augmentation baseline.


A lot of relevant existing works in protein representation using amino acid sequences are ignored. When protein 3D structure data is scarce because of expensive acquisition, a lot of work on protein representation learning relies on linear amino acid sequences. Although this representation of protein does not tell much about the 3D conformations of protein explicitly, the availability of a large database of an amino acid sequence is very relevant for self-supervised learning to be trained on. The discussion of these works in the related work and comparison to these works should be conducted carefully. I would suggest the authors compare to at least the following baseline representation trained on large amino acid sequence databases, https://github.com/facebookresearch/esm:
+ ESM-1b
+ ESM-MSA-1b  

clarity,_quality,_novelty_and_reproducibility: The idea in the paper is original. No source code was included in the submission, it is hard to reproduce the results.
The paper was well written. However, it makes too many assumptions about the distribution of the conformation without clear support both theoretical and empirical evidence (see my comments on weaknesses). A large number of related works in protein representation learning using amino acid sequences were ignored. 

 

summary_of_the_review: 
I see this work as interesting and may have a good impact on the field. But some points need to improve especially stronger support for the proposed conformation sampling approach. I proposed additional experiments that the authors should consider to strengthen the support evidence, I will be happy to change my score once these additional experiments are added to the papers and the results are still significant concerning the new baselines.  see this work interesting and may have a good impact in the field. But there are points that need to improve especially a stronger support for the proposed conformation sampling approach. I proposed additional experiments that the authors should consider to strengthen the support evidence, I will be happy to change my score once these additional experiments are added to the papers and the results are still significant with respect to the new baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to learn a conditional invariant representation for protein structure by introducing a data augmentation approach. The approach augments the protein 3D conformations by applying conditional invariant transformations to sub-structures in protein with a novel Markov chain Monte Carlo (MCMC) sampling. The paper demonstrates improved performance over a GNN baseline that inputs the initial 3D structure of protein without data argumentation.
strength_and_weaknesses: Strength:
- The directed forest construction and conditional transformation of sub-structures provide a principled approach to sample conformations given a protein structure.
- The authors formally show that the MCGN optimization ensures the convergence of conformer invariant learning procedure.

Weakness:
- The approach assumes that the protein molecule can be decomposed into a directed forest structure. This assumption is often not true for realistic proteins. The authors use some arbitrary rules to deal with cycles/rings: “cycles/rings which are present in the molecule are broken on the basis of bond length, i.e., larger bonds are chosen before smaller bonds. Ties between same-length bonds are broken arbitrarily.”The resulting conformations are likely unrealistic due to these arbitrary rules.
- Consequently, the authors use a validity checker (Molprobity) to filter invalid conformations. It means that the sampled structures largely depend on the validity checker. It is unclear if the conditional invariant sampling approach provides benefits compared with a prior MCMC sampling approach. It will also be beneficial if the authors can report the acceptance rate of the MCMC sampling.
- The performance improvement over a GNN baseline is marginal (Table 1). It is unclear if the proposed approach can outperform a much simpler data argumentation baseline, e.g., random Gaussian noise + validity filtering.
- Have the authors considered leveraging substructures similar to the JTVAE paper by Jin et al. (ICML 2018) for the construction of the directed forest? 

clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and easy to follow.
- The approach to sample protein conformations is novel at the extent of the reviewer’s knowledge.
- The authors didn’t provide a code to reproduce their results.

summary_of_the_review: In summary, the authors introduce a novel conditional invariant approach to argument the protein conformations and improve the performance in property prediction tasks. Despite the nice theory, the performance improvement seems marginal and there are some realistic assumptions in the direction forest construction approach. I feel this work requires further improvement and more baselines to demonstrate its usefulness in realistic tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers agreed that there are many critical aspects concerning the underlying model assumptions (such as the usefulness of the proposed graph decomposition for molecule graphs), the usefulness of the generated structures and the process of validating these structures, the over-all novelty and the experimental validation and comparison. Even the most positive reviewer pointed out that there were "strong assumptions without thorough validation and supported evidence both theoretically and empirically". This over-all impression did not change significantly after the rebuttal and discussion phase, and I also largely agree with most raised points of criticism. Therefore I vote for rejection.
justification_for_why_not_higher_score: Too many (sometimes severe) points of criticism have been raised and still remained after the rebuttal.
Decision: Reject