Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the task of unsupervised learning of protein structures. They do so by maximizing the mutual information between two diffusion trajectories that start with a protein and a perturbed copy of the protein. The authors prove a lower bound of the mutual information which resembles an ELBO and specifically depends on the likelihood of one trajectory given the other.
As such one trajectory is used to compute the noises in the other trajectory.
By doing so, the procedure encodes both the structure and sequences jointly.
strength_and_weaknesses: ### Strengths
- The paper comes up with a lower bound for the mutual information between two diffusion sequences and uses it to lean an encoder for protein structure. This is an interesting case of theory guiding practice and makes the work interesting.
- The ablation study shows that all the components of the procedure contribute to improving the model.

### Weaknesses
- What are correlated views? These show up in the abstract and introduction without a simplified explanation and throw off a reader who is (even very slightly) outside the field.
- Table 1 and 2 are not color blind friendly, the community seems to use bold and underline to denote first and second, One can also use markers like $\dagger$ and $\circ$. Standard deviations are missing. Provide context for $F_{\text{max}}$, and $\rho$ in the caption for ease of skimming. In fact I was unable to find a definition of $F_{\text{max}}$.
- The abstract talks about comparing with motif based methods, this comparison doesn't show up in the experiments. 
- What do you mean by `We do not include the results of GVP on EC and GearNet-Edge on RES tasks due to their poor performance.`?
- [Personal nitpick] Page 2 first sentence - shortening Tables to Tabs is non-standard.
- The writing was pretty dense and I required multiple reads to understand the text. However the figures helped and the fact that this is not my primary field of research is not the authors' fault.

clarity,_quality,_novelty_and_reproducibility: - The method is novel and the methods derived in the paper are interesting.
- The writing is dense and assumes strong domain knowledge in proteins and is not really accessible to the general ML audience. Multiple concepts are not defined and this reviewer would find it hard to reproduce the work. The theoretical proofs are complete and fully reproducible to the point that results from cited papers are re-proved for convenience.
summary_of_the_review: - I am giving this paper a weak reject due to the minor concerns noted in the above two sections, specifically about reporting on the experimental results and making the writing more accessible to the general audience at ICLR. I am willing to upgrade my rating based on my interaction with the authors during the discussion phase.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applied diffusion model to learn the protein structure from protein sequence. Compared with previous method, they introduced contrast learning and utilized mutual information. The experiment revealed that the proposed method showed marginal performance increase. 
strength_and_weaknesses: Strength:
The proposed method tried to achieve better protein residual presentation by introducing contrast learning and maximize mutual information.
The proposed method showed marginal performance increase compared with existing methods.

Weakness:
As far as I regard, the proposed method is not quite intuitive. Diffusion model is already a method that apply graduate noise. Such that, introducing another level of random noise for contrast learning is not intuitive to me. The marginal performance increase also suggest some conflict between diffusion model and the contrast learning. I was wondering is there some theoretical analysis between diffusion model and the introduced random noise?
I also do not quite agree with the claim that added random noise could reflect the conformational fluctuation. Simply add the noise on the structure is not equivalent to random permutation. Sometimes, a random permutation could drastically change the protein structure. Simply add random noise, could never reflect such conformational change. 

 
clarity,_quality,_novelty_and_reproducibility: It is quite clear however the author did not provide code.
summary_of_the_review: The paper focus on the residual level prediction accuracy. It utilize diffusion model with added mutual information training. However, there lack some clear linkage between diffusion model and synthetic mutual information, which makes the method not intuitive. Moreover, the added noise is quite straightforward, which is not optimal to reflect conformation change.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose SiamDiff, a method that employs a multimodal diffusion process to simulate the structure-sequence co-diffusion trajectory. SiamDiff maximises the Mutual Information (MI) between paired correlated views of proteins, where one is the native protein and the other is obtained with a structure perturbation. Under this learning framework, the authors argue that the model can acquire the atom- and residue-level interactions underlying protein structural changes, as well as the residue type dependencies. Extensive experiments on diverse and standardised downstream tasks are shown to demonstrate the value of the learned protein representations.
strength_and_weaknesses: The authors introduce some interesting and novel ideas in this paper. They show that their Siamese multimodal diffusion process can obtain reasonable protein representations that are useful for several different downstream tasks. Modelling the joint distribution of protein sequences and structures through multimodal denoising allows them to capture SiamDiff to capture the atom and residue-level interaction. Another interesting point is their idea of maximising the MI between two Siamese diffusion trajectories, one based on the native view of the protein, and the other view being generated by a structure perturbation.

The mathematical framework presented is sound. The authors also present the proofs to every proposition they make. They also mention that the full code will be available for reproducibility. 

The experiments shown in the paper are very thorough, where all the tasks have standardised experimental settings in well-established benchmarks. The empirical results are strong enough to prove the effectiveness of the protein representations obtained by SiamDiff.

I feel that the paper is lacking clarity. While I understand that there are a lot of concepts to fit in the manuscript, I believe that this paper could benefit from high-level intuition behind the main ideas of the paper. The notation feels overloaded and it is hard to follow.

clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and the language is concise. However, as I mentioned before, I feel that the paper is not that clear. I had to go through several readings of the manuscript to fully comprehend the main ideas in the paper. Some better intuition and improving of the structure could be greatly beneficial to this paper. 

I believe this is a high-quality paper. The mathematical framework introduced is sound, the experiments are extensive and thorough, and the empirical results obtained are strong.

This method is a novel application of existing methods, however, the real novelty lies in the ideas applied to learn the protein representations. There are some clear distinctions from previous works, like the multimodal denoising and the Siamese paired views of proteins. 

As for reproducibility, the authors mention that the code will be made available in case the paper gets accepted. The representation learning is well-defined in the paper. However, the models used for downstream tasks, based on the representations, are not that clear. The information can be found in the supplementary material, but I believe that a couple of sentences providing more details about these models should be included in the main paper. In short, releasing the code for both the training of SiamDiff, as well as the pipeline for downstream tasks is critical for the reproducibility of this work.

summary_of_the_review: My recommendation is an accept. I believe that this is a good paper introducing some novel ideas that are very relevant to protein representation learning. The evaluation of the learned representations in downstream tasks was made following well-established benchmarks. Although, the performance achieved is not the highest for every task, the general value of these representations is evident from the empirical results. 
I do expect the authors to address the issues raised in this review, mainly about clarity.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use diffusion model as a pretrain model for several downstream tasks.

It decomposes the structure and uses a DDPM like diffusion for continuous variable and a multimodel diffusion for categorical variable. Experiment shows that the diffusion model serves as a good pretrain model.
strength_and_weaknesses: Strength:
Using diffusion model as a pretrain model seems to be a new application.

Weakness:
1. I am not an expert of the application area this paper considers. But simply examing the experiment, it seems the improvement is not large. There is no repeated experiment and it's unknown whether the performance is significant.

2. The overall technical novelty is low. It follows a standard design of diffusion model and the contribution seems restricted to diffusion as a pretrain model?
clarity,_quality,_novelty_and_reproducibility: Clarity: good.

Quality: good.

Novelty: Low.

Reproducibility: unknown. No example code was provided.
summary_of_the_review: See above.

My evaluation is a educated guess.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the task of unsupervised learning of protein structures. This is achieved by applying a diffusion model to learn the protein structure from the protein sequence. The approach maximizes the mutual information between two diffusion trajectories that start with a protein and a perturbed copy of the protein and use one trajectory to compute the noise in the other trajectory. The paper provides a lower bound of the mutual information, which resembles an ELBO and depends explicitly on the likelihood of one trajectory given the other.

**Strengths:**
* Reviewers acknowledged some novel ideas relevant to protein representation learning. For example, the new method tried to achieve better protein residual presentation by introducing contrast learning and maximizing mutual information. However, the method showed only a marginal performance increase compared with existing methods.
*  The evaluation of the learned representations followed well-established benchmarks. Although the performance is not the highest for every task, some value of these representations is evident from the empirical results. 


**Weaknesses:**
* Several reviewers pointed out that the method could be more intuitive. As diffusion models gradually apply noise to the input, introducing another level of random noise for contrast learning is not intuitive. The marginal performance increase also suggests conflict between the diffusion model and the contrast learning. 
* There was a consensus among the reviewers that the paper needs more clarity. The notation feels overloaded, and it is hard to follow. In addition, multiple concepts are not defined.
* The code is currently not available. Several reviewers raised concerns that it could be hard to reproduce the work which was not addressed. 
* Reviewers questioned the technical soundness of core method components. In particular, there is no clear linkage between the diffusion model and synthetic mutual information. Moreover, the added noise does not reflect changes in protein conformations.
justification_for_why_not_higher_score: * Several reviewers pointed out that the method could be more intuitive. As diffusion models gradually apply noise to the input, introducing another level of random noise for contrast learning is not intuitive. The marginal performance increase also suggests conflict between the diffusion model and the contrast learning. 
* There was a consensus among the reviewers that the paper needs more clarity. The notation feels overloaded, and it is hard to follow. In addition, multiple concepts are not defined.
* The code is currently not available. Several reviewers raised concerns that it could be hard to reproduce the work which was not addressed. 
* Reviewers questioned the technical soundness of core method components. In particular, there is no clear linkage between the diffusion model and synthetic mutual information. Moreover, the added noise does not reflect changes in protein conformations.
Decision: Reject