Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a generative sequence model, Bayesian embedded autoregressive (BEAR), which is non-parametric as opposed to parametric autoregressive models to address the misspecification problem in neural networks. The BEAR model is a Bayesian Markov model with an embedded autoregressive model into its prior. Using parametric model output as a prior allows BEAR to be used for non-parametric density estimation which makes asymptotic consistency possible. They also show that the learned concentration hyperparameter for a prior dirichlet distribution represents the amount of misspecification in the embedded autoregressive model (or other neural network that was embedded) that can be used as a warning when the model can not be trusted. Then, they prove that absolute performance of the distribution predicted by some model can be measured through testing whether it matches the optimal BEAR model. 

They theoretically prove that the BEAR model is more expressive than the naive AR model and that it converges to the true distribution as the size of the dataset increases. BEAR models are scalable and uncertainty-aware, which make them a useful generative model for full biological sequences. 
main_review: EDIT 2021-08-19: Changed score to 7.

The paper addresses the interesting problem of sequence generation with a convergence guarantee, and is scalable to whole genome sequences which make it practically used as a generative model. It is well-written and all claims are followed by theoretical proofs and simulation results. My main concern is that it does not reference or compare to the related literature on sequence compression.

Major comments:

- What is the utility of a generative model of biological sequences? The authors say that it would be useful to have a generative model, but not what it might be used for. The closest experiment to being practical is in section "Testing hypotheses", in which the authors design a statistical test for whether two sets of sequences are generated from the same distribution. Is this the main aim of BEAR?
- Density estimation is closely related to compression. There is a large literature of nucleotide sequence compression methods, such as the following, which the authors should cite and compare and contrast. 
	- https://academic.oup.com/gigascience/article/9/11/giaa119/5974977
	- https://academic.oup.com/bib/article/16/1/1/240815
- The manuscript includes a huge 62-page supplement. Much of the paper is unclear without reference to the supplement, including many of the experiments. The manuscript would be improved with a narrower focus, particularly on those tasks that are practically relevant. 
- In particular, the formulation of fk(θ) is not in the main manuscript, although it is very needed to know the parameters that are going to be learned during training. For example, what is B when β* > 0 in AR model if it is not learned during fitting? What is its value during sample generation? 
- In the “measuring misspecification” subsection in the results section, the relation between the misspecification and flexibility is not clear. The misspecification value has been shown to be proportional to a divergence between AR model and true distribution, however it is not clear how the divergence is related to the flexibility of the model.
- In the two-sample test experiment (evaluating a simulation model, ART), it has been shown that differences between real and simulation data can be observed when the test is done for individual lags. The behavior of bayes factor in terms of lag is not clear, for example why it increases and then decreases when comparing simulation and real data.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a generative Bayesian Markovian framework and a set of underlying models for DNA sequences to estimate the probability density of the current nucleotide given $L$ previous nucleotides $p(X_i|X_{i-L:i-1})$.
The authors investigate theoretically and empirically some applications of the framework such as density estimation of sequence data, parameter estimation and goodness-of-fit tests.
main_review: The paper is clearly written and well organized.

The novelty of the work lies in the theoretical description of the BEAR model and its applications to certain statistical problems for DNA modelling. The BEAR model is based on the Bayesian Markovian framework. It uses an embedded autoregressive (AR) model to estimate the hyperparamters of the Dirichlet prior distribution over the probability density of the kmers (nucleotide subsequences of length $L$). 
The claims are adequately, theoretically and experimentally, supported.

#### Comments

1) I think the paper is too long for a conference venue. The submitted manuscript respects the 9 pages limit but each main section constantly refers the reader to other sections of the supplemental material. The presented models and the applications cannot be well understood without the support of this material. The theoretical results presented in the main text are a summary of a set of theorems and propositions about models, density estimation, parameter estimation and hypothesis testing that are detailed in 45 pages in the supplemental material. It is the same case for empirical results, which are supported by 17 pages in the supplemental material. 

2) It is not clear whether the finite-lag Markov models described in Section D are new or does this name refer to an already known family of Markov models? From their definition, they are quite similar to variable-order Markov models. If not, what is the difference between both of these models? The variable-order Markov models have been used in nucleotide sequence analysis, e.g., in virus genome subtyping [1]. 

3) How does this work (embedding a parametric model in a Dirichlet prior) relate to the work of Tzu-Tsung Wong on Generalized Dirichlet distribution in a Bayesian analysis [2, 3]?

4) In the empirical evaluation, the authors designed “new” AR baseline models to embed them within the BEAR model and also to compare with. The results show that most of these baseline models are too weak and that their performance is lower or close to chance values. The comparison should also be done with other published methods or other deep architectures (RNNs, VAE) regardless of the AR framework (even RNNs could be considered as AR) [4, 5, 6]. In case the published methods cannot scale with terabyte-scale datasets (which I don't think is the case, e.g. metagenomics classification with linear methods [7] and UniRep for protein deep representation with mLSTM [8]), it will always be interesting to have comparison results on medium-size datasets. 

#### Minor comments

1) Lines 23-25: Genome modelling using genetic variant matrix is usually used in GWAS studies. However, there is large repertoire of studies that models the whole genome at nucleotide resolution in different genomics analysis applications [4, 9] 

2) Line 60: Rephrase: Crucial to our the theoretical and empirical analysis is the statistical setting.

#### References
* [1] Struck, D., et al. COMET: adaptive context-based modeling for ultrafast HIV-1 subtype identification. 2014.
* [2] Wong, T.T. Generalized Dirichlet distribution in Bayesian analysis. 1998.
* [3] Wong, T.T. and Liu, C.R. An efficient parameter estimation method for generalized Dirichlet priors in naïve Bayesian classifiers with multinomial models. 2016.
* [4] Im, Jinho, Byungkyu Park, and Kyungsook Han. A generative model for constructing nucleic acid sequences binding to a protein. 2019
* [5] Liang, Qiaoxing, et al. DeepMicrobes: taxonomic classification for metagenomics with deep learning. 2020.
* [6] Riesselman, Adam J., John B. Ingraham, and Debora S. Marks. Deep generative models of genetic variation capture the effects of mutations. 2018.
* [7] Vervier, K., et al. Large-scale machine learning for metagenomics sequence classification. 2016.
* [8] Alley, Ethan C., et al. "Unified rational protein engineering with sequence-based deep representation learning. 2019.
* [9] Lim, D., and Blanchette M. EvoLSTM: context-dependent models of sequence evolution using a sequence-to-sequence LSTM. 2020.

limitations_and_societal_impact: The limitations of the BEAR model are briefly discussed in Section 7.
The social impact of genetic-related models is highlighted in Section A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, authors propose a new generative sequence model, called the Bayesian embedded autoregressive (BEAR) model. BEAR is a nonparametric Bayesian Markov model which embeds an autoregressive (AR) model into its prior. Theoretical guarantees are provided for statistical problems including density estimation, misspecification detection, and hypothesis testing, and experimental results indicate BEAR's superior performance on a variety of datasets.
main_review: \textbf{Strength}: 
BEAR builds generative probabilistic models at the genome scale, which is able to detect and overcome misspecification problem. \\
In the paper, authors derive the asymptotic consistency guarantees and convergence rates for nonparametric density estimation, and prove consistency results for goodness-of-fit tests and two-sample tests, which provide theoretical guarantees of BEAR.\\
In the experimental evaluation, BEAR outperforms other autoregressive models on large-scale genome datasets.

\textbf{Weakness}: 
I have several concerns regarding to the proposed method. 
First, I wonder if BEAR can handle the long-range sequential data?\\ 
Next, I wonder if the parameters of BEAR are also interpretable? It would be great if the authors could discuss further about how to quantify the degree of misspecification introduced in the Section Robust parameter estimation, and how to select the optimal lag $L$. \\
Third, will the model be able to handle prior knowledge about some special elements on the genome sequence, such as transposable elements? 

limitations_and_societal_impact: The authors have discussed the limitations in the discussion section, and the proposed method does not have any negative societal impacts.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: In this study, authors have proposed a Bayesian embedded autoregressive model to tackle the problem of measuring and making DNA. They have implemented the model using a parametric auto-regressive model. The model is programmed in PYTHON language. They try to overcome challenges of  having enough flexibility to account for genomic complexity, interpretability to reach scientific conclusions, and scalability to train on billions of nucleotides. The authors have explored, theoretically and empirically, applications of BEAR models to a variety of statistical problems  including density estimation, robust parameter estimation, goodness-of-fit tests, and two-sample tests. 
main_review: The paper is well-written in English, and impressively organized. The major novelty of the model comes from nonparametrically perturb a parametric model, studying its theoretical properties, and developing algorithms and implementations for terabyte-scale inference. The model outperforms other standard models, however, it has some limitations such as not explicitly account for paired-
end read information, or other sources of long-distance information; But this model can be a starting point to address all these challenges. Authors have been honest about the limitations of their work. 


limitations_and_societal_impact: Not applicable
review_title: null

Meta-review: All four reviewers recommend accepting the submission, but three reviewers are not highly confident in their assessments and I am not highly confident in the other reviewer’s assessment; there is a disconnect between what this reviewer wrote and what rating this reviewer assigned. Therefore, I reviewed this submission carefully myself. The submission is well written, novel, and makes an important contribution to Bayesian modeling for genomic sequence data; I recommend accepting it.
Decision: Accept (Poster)