Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on the prediction of protein thermostability and the design of more thermostable proteins. After introducing a new benchmark to train and assess thermostability prediction models, authors describe a contrastive-learning framework to impart protein structure information on representations learned by large transformer networks usually trained on sequence alone (eg., ESM-1b). The paper also adapts ideas from other domains to carry out robust fine tuning and protein editing.
strength_and_weaknesses: **Strengths**
- The work is very thorough with many ablations to illustrate the benefits from the different modeling decisions suggested by the authors
- There are several novel contributions, in particular the Hotprotein benchmark and the structure-aware pre-training. The work also borrows ideas introduced in other fields to address (in a novel way) problems in protein prediction models and protein editing.

**Weaknesses**
- The main flaw I see in this paper is with respect to model performance validation. Unlike in many other application domains, protein sequences are not iid samples from the underlying data distribution as there is a lot of dependency between sequences that share common ancestors. This calls for particular cross-validation schemes to mitigate data leakage between training and test sets (eg., ensuring that sequences are not too similar), especially in several of the (semi-)supervised settings discussed in this work. See for example [1] for some ideas on how to do this. Similarly, it would be helpful if authors could comment on the similarities / overlap between the introduced Hotprotein dataset and the other datasets used in fine tuning / editing experiments (eg., FireProt and Meltome Atlas) -- if no substantial overlap, could you please specify that in supplementary?
- The paper is not very clear at times. For instance, the reasoning / rationale that gave rise to the equations 1 and 2 in section 4.1 is absent
- Not really a weakness, but rather a missed opportunity: since the SAP and fine-tuning frameworks are general and agnostic to thermostability, more extensive validation across protein families & fitness / protein attributes could be carried out to validate generalizability of the introduced ideas beyond thermostability. See for instance large-scale fitness benchmarks like ProteinGym [2].

------------------------------------------------------------------------------------------------------------

[1] Magńus Halld́or Ǵıslason,  Felix Teufel,  Jośe Juan Almagro Armenteros,  Ole Winther,  and Henrik  Nielsen.   Protein  dataset  partitioning  pipeline.   2021b.   URL https://github.com/graph-part/graph-part

[2] Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A.N., Marks, D.S., & Gal, Y. (2022). Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. ICML.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Overall the paper is well written and easy to follow, with an extensive related work section. Some sections could be made a bit clearer, eg., section 4.1 (as discussed above).

**Quality**: Very thorough sets of experiments and ablations. Besides some concerns on validation as mentioned above, the experimental design is well-thought through.

**Novelty**: 
- Several meaningful / novel contributions, in particular the HotProtein dataset and the structure-aware pre-training. The former is based on existing datasets (NCBI bioproject and Uniprot), but required curation to create the final dataset, and could be a valuable resource for the community. The latter is relatively general and its application could extend beyond the thermostability use case discussed in this work. 
- The fine-tuning approach and optimization approach for protein editing are borrowed from other works (respectively [3] and [4]). While authors properly cite the corresponding works, the language used in section 1 when listing contributions is a bit misleading (eg., “we propose …”, “we built …”). Would recommend that authors use a more accurate language there to properly reflect their contributions (eg., “we adapt…”)

**Reproducibility**: Main information needed for reproduction are provided. Authors mention that code and data will be made public in the abstract. 

**Minor points**: 
- References need to be cleaned up -- several papers do not have the proper venue / journal listed.
- You don't seem to specifically define what is a small vs large dataset anywhere (terminology used in section 5.2, first paragraph, 4th point)

------------------------------------------------------------------------------------------------------------
[3] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  Lora:  Low-rank adaptation of large language models.

[4] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085–1097, 2019.
summary_of_the_review: Overall a very thorough and interesting work. Some concerns on validation as discussed in the strengths / weaknesses section. I am leaning accept overall given the several meaningful contributions, in particular the HotProtein dataset and the structure-aware pre-training. I would be willing to recommend acceptance more enthusiastically if the aforementioned concerns are addressed during rebuttal. 

--------------------------------------------------------------------------------------------------------
[UPDATES POST REBUTTAL]

Thank you to the authors for their thorough responses during rebuttal. My main concerns have been adequately addressed (in particular regarding validation / potential data leakage). I have also read in detail the assessments from the other two reviewers, and believe their comments have also been properly addressed in reviews. I do believe that the newly introduced HotProtein benchmark, together with the methodological contributions (structure aware pre-training) and various adaptations from other domains to protein modeling (eg., factorized fine tuning and feature augmentations) represent a solid contribution. In my opinion this paper should be accepted to the conference, and I have increased both my score for correctness and overall appreciation accordingly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a new model for predicting protein thermostability based on a published dataset (Jarzab et al Nature Methods 2020) and create three derived different datasets annotating each of proteins with a class name that reflects the temperature that the organism lives at  (retrieved from NCBI). The task then becomes the ability to predict the correct class. They present two algorithmic ‘advances’: structure-aware pretraining and factorized sparse tuning  improve thermostability prediction. After evaluating compared other methods for prediction they also propose a method to generate sequences with increased thermostabilty. 

strength_and_weaknesses: A fundamental weakness of this paper is similar conceptually to many papers submitted to ML conferences on biological data -that is the way the train-test splits are made which is 10 fold cross validation. What this means is that they have ‘data leakage’ relative to what it seems they claim to predict -- maximally they show if you had 90% of the data with related  sequences in the split, you could predict the missing 10%. Not a very useful predictor Now imagine a test split that removes a family of sequences that has no sequence relation to training data - then one could say a load more about the ability of the model to generalize 

With thousands of proteins many of which are homologous i.e. similar  and very few classes in theri stes (2 or 5 )  - it's hard to see why one even needs a language model when a simple sequences similarity search as a baseline could do the same. Have the authors tested a simple baseline?

The sentence in the abstract “We present HotProtein, a large-scale protein dataset with growth temperature annotations of thermostability..” strongly suggests  they are reporting this dataset here for the first time - when in fact it was published 2 years ago and is publicly available. Since they do in fact clear the data for more facile use, the statement should make that distinction clear.  
Related work seems to be missing - especially where methods have collected datasets but also run models to predict mutation effects - some of which are thermostablity related ; since some of these are even unsupervised, using only evolutionary sequence alignments - minimally a  discussion of this body of work is warranted eg - see ProteinGym ( Tranception ICML 2022)  plus older papers that use VAEs ( Riesselman Nature Methods 2018)  and even Potts Models (eg Hopf et al 2017 Nature Biotech) plus MAVEsdb.  In addition FLIP ICML 2021 and 2022- Learning deep representations of enzyme thermal adaptation. Why no comparison  to  ESM1v ( Meier 2021), Tranception ( Notin 2022), FLIP 202, DeepET (2022) ?


Minor 
Explain why theory are not using the Meltome measurement as their thermostability label
Although it’s fine to explore whether using structural information improves existing approaches , the sentence on page 4  is misleading for different reasons -- “These models achieve considerable improvements on amino acid prediction tasks, e.g., contact prediction, mask prediction (Brandes et al., 2022), mutational effect prediction (Meier et al., 2021).” Compared to what? Meier’s results that the authors reference are great  BUT are only very marginally better than a VAE ( DeepSequence 2018 Nature Methods) and not as good as Tranception ( Notion et al,  ICML 2022) and are zero shot!! Using only sequence info. 
The structure barely improves the results - why do they think that is the case? 

clarity,_quality,_novelty_and_reproducibility: Whilst I quite like the model framework. I cannot evaluate it's advantage for the declared task as baselines are missing. 
summary_of_the_review: For the reasons given in the Weaknesses section I don't believe this paper achieves what the authors would indicate are their goals - thermostability prediction and design.  The fact that they do not test generalizability, limits the application of this work to where there are very large datasets  of experimental results on related proteins from environmentally related organisms. I would suggest that the authors rethink their evaluations, test with much more naive baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework for protein thermostability prediction. A large-scale protein dataset with organism-level temperature annotations is curated, and one pretraining and one tuning module are proposed for prediction.
The main contributions are:
1.	A dataset of protein sequences and folded 3D structures data with temperature annotations.
2.	Introducing a protein structure-aware pre-training module for protein-related tasks.
3.	Proposing a factorized sparse tuning module for thermostability prediction.

strength_and_weaknesses: Strengths:
1.	The amount of data for thermostability-related tasks is greatly expanded.
2.	The writing is generally clear, and easy to read.
3.	Editing proteins to improve thermostability is interesting, biological experiments can be added in the future.
4.	The experiments on the model are relatively comprehensive.
Weakness:
1.	Their pretraining module simply combines the sequence pretrained model and the 3-D structure-aware pre-trained model. No novelty on the loss function can be viewed.
2.	Two reasons mentioned for using factorized sparse tuning for this prediction task are not convincing. The sentence ‘while COVID-19 related proteins usually contain more than 1,000 amino acids’ has no strong connection with the studied problem.
3.	From the result, the framework seems just introduce some training strategies based on ESM-1B, and there is no design in biology for the thermostability prediction task. Besides, no significant improvement in results is observed.
4.	The description of biology in this paper is superficial, and the study lacks interpretability analysis.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the experiments are comprehensive. However, it's hard to say whether the model would really be useful in resolving biological problems. 
summary_of_the_review: The paper is clearly written, and the experiments are comprehensive. However, it's hard to say whether the model would really be useful in resolving biological problems.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has two parts: the creation of the dataset ("Hot protein") focusing on protein thermostability 
and the novel learning framework which uses structure-aware pretraining. 

The strength of the paper is both the dataset and the training framework, which borrows ideas from the other fields into the protein domain, which suffers from the lack of large datasets (typical for other fields). The part with factorized weights is really interesting, since it shows a practical way to learn generalizable models with limited data available.


The main concern that is present in the reviews is the model evaluation. The data is not i.i.d by default, and cross-validation can be tricky. However, the authors have given a complete answer to this question by clustering proteins and keeping one sequence in the cluster. The "reject" by other reviewer had the same concern, so I consider that this question is answered (the reviewer did not engage in the discussion).
The other possible weakness could be the missing baselines, but the authors answered this as well in their rebuttal. ]

Overall, I think it is a good paper, which is well-written.



justification_for_why_not_higher_score: The paper is on the borderline on the reviews, so the question is between accept / reject, but not higher.
justification_for_why_not_lower_score: This is a borderline; unfortunately, due to the limitations of the time zones, organizing the in-person meeting was close to impossible. 
I think, the authors have completely answered the concerns of the reviewers (even with the reject scores), while reviewers did not comment on the rebuttal. My personal reading (although I am not a biology expert at all)  gives the impression that this is a good paper. I personally liked the description of the dataset creation, which reads more like a novel. 
The part with the novel and factorized learning is really interesting.
All of these different parts make the model quite solid, and taking into account the amount of concurrent work I think it would be a pity if the authors did not not make it into the conference this time - this work may just loose to the competitors just on the random reject.
Decision: Accept: poster