Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper delves into the broad question of why generative sequence models work well in learning from phylogenetic data and points out some directions for improving them. It first starts showing that given simplifying assumptions made by popular models (which try to fit the data distribution $p^0$), it is not possible to infer fitness of a sequence given any amount of data. They suggest that instead, they should model the stationary distribution $p^{\inf}$. Authors utilize these assumptions and show that, in general, fitness is non-identifiable. Thus, they conclude models trying to fit p^0 are misspecified, however, this has some empirical benefits that can be evaluated in a hypothesis testing framework. Their experiments in synthetic data accept the hypothesis that models fitting $p^0$ can indeed get a close approximation to $p^{\inf}$.

strengths_and_weaknesses: Strengths:
- Draws interesting conclusions on the hardness of learning.
- It proposes future directions that are theoretically more plausible to work.

Weaknesses:
- Although this work has machine learning contributions, I am unsure that NeurIPS is the best venue to review this work. The assumptions that are made in the analysis are very domain specific and it is hard to evaluate if they are sound. For instance, the two exchangeability examples for assumption 3.1 (which is crucial to prove a key theorem of the paper, thm 3.2) date from before 2010. I am unable to tell if the most recent methods also make this assumption, therefore, it is hard to evaluate the generalizability of this assumption. 
- The claims made by authors may need to be qualified. See question 1.
questions: 1. One of the main arguments, if not the main one, for theorem 3.2 (non-identifiability) is that the model/prior at hand has to satisfy exchangeability (assumption 3.1). According to authors, in general, exchangeability is hold in models in the literature. Doesn't this suggest that we should attempt to build models that put non-exchangeable priors on the sequence? Also, if we remove this assumption, is the claim in line 194 correct? should it be reworded to "is in general impossible **with current models**"? 
2. Can you provide more recent examples that can help to understand how general the exchangeability assumption is?
limitations: - They recognize that the assumptions made in some parts of the theoretical analysis are not always guaranteed in real practice.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the issue of learning molecular fitness from evolutionary data. The studied data-generating process involves molecules that evolve according to a stochastic mutation process along with a fitness function that effects the survival of the mutations through selection. Due to phylogenetic effects, the true data-generating distribution convolves fitness with evolutionary history, meaning that we cannot estimate fitness just by learning the true data distribution. The authors provide a rigorous explanation of this effect and then go on to explain why despite this issue, empirical approaches that aim to learn the data distribution through MLE or Bayesian inference often can infer fitness quite well. The authors describe two hypotheses, the first of which appears to be the common assumption that previous practitioners had, whereas the second appears to be new hypothesis that the authors propose. Understanding which hypothesis is true has important implications for the field, as hypothesis 2 suggests that better density estimation will not necessarily improve fitness. The authors then construct a diagnostic test to test each hypothesis and validate it on simulated data. The authors then move to experimental data where they show that hypothesis two is predicted by this diagnostic test in the majority of scenarios. 
strengths_and_weaknesses: Overall, I think this is a great paper. The question studied is important, as estimating fitness from evolutionary data is of great interest in computational biology. The authors explain rigorously the issue of non-identifiability and then go on to provide a novel hypothesis for why existing approaches have empirically been able to estimate fitness nonetheless. The authors show under relatively loose conditions (Thm 4.1) why a mis-specified model may be beneficial for inferring fitness (hypothesis 2) and then go on to show under simulations how to test for hypothesis 2. Finally, they provide strong empirical evidence for hypothesis 2. 

For originality, quality, clarity, and significance I would rate this highly. 
questions: No questions
limitations: No issues
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors illuminate an issue with generative models fit to protein sequence data. Often, the likelihoods derived from these models are interpreted as a proxy for evolutionary fitness and applied on tasks like mutation pathogenicity prediction. However, theses models are mid-specified when they don’t explicitly consider the evolutionary process. The authors produce mathematical explanations of this mid-specification and explore various hypotheses with simulated and real data. The argument is clear and compelling. I appreciate the authors highlighting paths forward, for which research advances would improve protein sequence analysis.
strengths_and_weaknesses: Predicting the influence of mutations to proteins is an extremely important problem for both medical and evolutionary genetics. The author's make an important insight here that naively fitting models to available protein sequences, without consideration of the underlying generative process via a phylogeny driven by fitness, is problematic. They demonstrate the problem well, both theoretically and empirically. Finally, they identify compelling paths forward for future research.
questions: None
limitations: Yes, limitations have been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper asks two important questions in estimating molecular fitness using generative models for sequences: 1) Is fitness fully identifiable from the observation of sequence data alone? and 2) Is developing generative models that best emulate the data distribution necessary or sufficient to estimate fitness? Under certain assumptions and statistical model for sequences and fitness (which is later validated on real-data), the paper demonstrate that the answer to both questions (perhaps surprisingly) is no, i.e., 1) Fitness is not fully identifiable even from infinitely many number of observed sequences and 2) Generative models are successful in predicting fitness not necessarily because they fit the original data distribution the best, rather because they are trained over a function class (e.g., Potts models or Neural networks) that better fit to another so-called stationary distribution of sequence which originates from phylogenetic models and this captures fitness in the long run. 
strengths_and_weaknesses: Overall the questions posed by the authors are interesting and important and have clear messages that help the community of molecular inference. The approach to address the questions are clearly described, statistically sound, and opens new research questions. For these reasons I think the paper is above the acceptance line. Perhaps the key comment is that the analysis largely relies on certain assumptions on sequences and fitness which might restrict the scope of the paper. Furthermore, the final message of the paper is more of a statistical property of the biological sequence data and fitness; the paper can be strengthened by some constructive and operational algorithms inspired by the key message of the paper to help better model fitness. I will expand on these comments here.
questions: How large should the sequence dataset be for fitness estimation? The paper shows that better estimates of p0 using infinitely many sequence data is not the best strategy to model fitness since we like to estimate p\infinity instead. Obviously a very small amount of sequence data is also not useful for fitness estimation. This seems to suggest there is a sweet spot in terms of the amount of sequence data useful for fitness estimation after which there will get into a diminishing return regime. Is there any way to find what the sweet spot is perhaps using the analysis done in Fig 2? This will help since we will know how much sequence data is operationally enough for practical purposes and also give insights on how big and complex a model is sufficient for generative modeling towards the goal of fitness estimation.

Assumptions.  The analysis of identifiability of fitness naturally relies on some model for sequence and fitness function. Here, an Ornstein-Uhlenbeck tree model is employed in JFPM which evolves over a quadratic fitness landscape and a Kingman coalescent prior over the tree. While this particular choice completely shows an example of a case where fitness is not identifiable from the sequence data alone, is it fair to draw a universal conclusion for the rest of the models not considered here or fitness functions for which we don't have a good model for? Similarly another assumption seems to be the existence of a stationary p\infinity distribution that in the long run converges to an energy landscape corresponding to fitness. Is this always the case for all molecules and fitness functions? Can we imagine molecules with multiple fitness measures, some of which we are interested to estimate however are more neutral to the selection process and some others that we are not interested to measure however are more influential in selection? E.g., fluorescence (as objective fitness) versus thermostability (driving selection) of a protein.

Family of misspecified models for M. This is suggested in the future work but really seems to strengthen the exposition of the paper to give some demonstrations of potential models that are worse at density estimation and are better at fitness estimation. While the hypothesis testing procedure is a valid way to convey the message of the paper on misspecification, developing such M models would be the ultimate constructive way to improve the state of the art on fitness estimation while justifying the misspecification claim.

Transformers. Since different variants of transformers and other large-scale language models are now at the core of generative modeling of sequences it would be beneficial to bring a summary of the discussions from Appendix G to the paper. How much do we expect the two messages of the paper hold for protein language models?
limitations: Discussed before.
review_title: null

Meta-review: metareview: This paper presents the argument that molecular fitness is not identifiable from observational sequence data alone and that bigger models trained on bigger datasets will inevitably lead to better fitness estimation. The reviewers found strengths in the paper because "estimating fitness from evolutionary data is of great interest in computational biology", and because they "show under relatively loose conditions (Thm 4.1) why a mis-specified model may be beneficial for inferring fitness (hypothesis 2) and then go on to show under simulations how to test for hypothesis 2". All reviewers highlighted the importance of the problem and the theoretical and empirical contributions. There was a minor concern that this is the best venue for the work and how these results might translate across domains. There is interest across domains in the alignment between the true target of inference (here molecular fitness) and the upstream analysis targets (here density estimation). Overall, the reviewers evaluations were highly favorable and they considered this a valuable contribution to the field.
confidence: Certain
Decision: Accept