Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper proposes a structure-aware protein sequence embedding using an autoencoder. The authors use a neural network model (S4PRED) to generate structure-aware embeddings and compare this approach to purely sequence-based embeddings in a Protein-Protein Interaction (PPI)-prediction task. 
strengths_and_weaknesses: Strength:

- The authors show that structure-aware embeddings have potential advantages and provide grounds for future research directions.

Weaknesses:

- While the logic of the paper seems interesting, the proposed method failed to outperform or perform on-par with classical and existing methods.
- The authors should be able to explain why they think the structure-aware method did not perform better than the classical methods as expected. This information will be really helpful for other works.

suggested_changes: See weaknesses
review_title: null

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: The paper under review explores the feasibility of developing a structure-aware embedding for protein sequences. The authors used an autoencoder to generate the embeddings and employed a relatively simple secondary structure prediction model, S4PRED, as an oracle. The authors trained two types of embeddings - structure-aware and purely sequence aware - and compared their performance in a protein-protein interaction prediction task. 
strengths_and_weaknesses: ## Strengths

- The paper addresses an important problem in the field of deep learning for protein research - the incorporation of high-level structural information in protein representation learning.

## Weaknesses

-  The paper does not compare the performance of the proposed structure-aware embedding to state-of-the-art protein structure prediction models such as AlphaFold, which limits the scope of the findings. Since the proposed approach performs much worse than the baselines, I do not see why the authors do not use a more sophisticated structure learning model. 


suggested_changes: I would suggest the authors to investigate more sophisticated autoencoders or use the publicly available AlphaFold model. Moreover, some parts of the text are not written clearly (for example there is no definition of  S4PRED). The authors may need to clarify the definition of S4PRED or any other technical terms they use in their paper. This will help readers better understand the methods they used and the results they obtained. Additionally, if they are using a modified version of S4PRED, they should provide details on the modifications they made and the reasons behind them.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper tackles the question of improving protein sequence embeddings with pre-trained models that include knowledge of secondary structure. The method trains an autoencoder whose loss is based on MSE in a secondary structure encoded space using S4PRED.
strengths_and_weaknesses: Strengths:

- An interesting idea to improve protein embeddings using a pre-trained secondary structure model.
- The method is clear and (mostly reproducible)
- This paper meets the form requirements

Weaknesses:

- Would be nice to include widths of the 6 layers (as well as depth)
- The structure aware embeddings don’t seem to help on the PPI prediction task.
suggested_changes: - I’m not certain on the output of S4PRED, I assume its a graph? It would be great to clarify this.
- From reading the paper I thought the current results showed the structure aware model working better than a sequence only model. This seems to be not the case. It would be great to make the claims match the experiments. I still think this is an interesting direction, though other architectures or ways to incorporate structure information may be more fruitful.

Some suggestions to improve embedding

- I would encourage the authors to try other ways of encoding information contained in S4PRED into the embeddings. Such as using the output of an intermediate layer as information for the embedding. I think the current method is difficult as it loses any information unnecessary for secondary structure prediction. Thus if there is any sequence level information not included in the (predicted) secondary structure that is useful for PPI prediction, then this will be lost in the “Structure-Aware” embedding. I would call the current model a “Structure-Only” model and would be interested in a combination with the pure sequence model.
- The current autoencoder is very large for a dense model, I believe there are some training challenges with this structure. You may want to try convolutional structure or some more parameter efficient model.
review_title: null

Meta-review: metareview: This paper investigates combining pre-trained oracles with auto encoders for predicting protein structures.  In this, an auto encoder is used to generate structure-aware embeddings using the outputs of the pre-trained oracle.  The work seems original.  However, the text is not especially clear, and would greatly benefit from revising to really highlight what the inputs/outputs of each element of the model are, what embeddings correspond to and would enable etc.  The proposed method does not outperform the baselines, and while this isn't a requirement for publication here, it does raise the question of why the extra structure is justified.  Exploring the necessity and benefits of this method for downstream tasks in more detail in future revisions would greatly improve the paper.  
confidence: 3: The area chair is somewhat confident
Decision: Invite to revise