Title: Optimal prediction of Markov chains with and without spectral gap
Authors: Yanjun Han
Soham Jana
Yihong Wu
Venue: NeurIPS 2021
Keywords: Markov chains
prediction
redundancy
spectral gap
mixing time
Kullback Leibler risk

Abstract: We study the following learning problem with dependent data: Given a trajectory of length $n$ from a stationary Markov chain with $k$ states, the goal is to predict the distribution of the next state. For $3 \leq k \leq O(\sqrt{n})$, the optimal prediction risk in the Kullback-Leibler divergence is shown to be $\Theta(\frac{k^2}{n}\log \frac{n}{k^2})$, in contrast to the optimal rate of $\Theta(\frac{\log \log n}{n})$ for $k=2$ previously shown in Falahatgar et al in 2016. These nonparametric rates can be attributed to the memory in the data, as the spectral gap of the Markov chain can be arbitrarily small. To quantify the memory effect, we study irreducible reversible chains with a prescribed spectral gap. In addition to characterizing the optimal prediction risk for two states, we show that, as long as the spectral gap is not excessively small, the prediction risk in the Markov model is $O(\frac{k^2}{n})$, which coincides with that of an iid model with the same number of parameters. 

TL;DR: We study a prediction problem on Markov chains with finite state space and obtain optimal minimax rates.

Review discussion:
The paper is of high quality and the results are significant and interesting.
Nevertheless, while the authors do make efforts to explain their ideas and machinery, I find that the paper is hard to read and that several points are not clear or confusing.
For example, even after several reads I am not sure what "with and without spectral gap" means. Does it mean with and without ASSUMPTIONS on the spectral gap? Section 3 is called Optimal rates without spectral gap but there is no corresponding section for "with".
Furthermore, Section 3 is concerned merely with the lower bound.
Additionally, in Section 2 there are results that make assumptions on the spectral gap and results that do not.
Adding to the confusion, the connection between subsequent paragraphs is sometimes not clear and sometimes explanations for results come AFTER they appear. This makes reading difficult and hard to understand what exactly wad done by the authors and what is already known.
I would also make the following specific suggestions.
Whenever my remark is formulated as a question I mean that I find that an explanation IN THE TEXT is necessary.
1. [line 25] While it can be assumed that if $Y\sim P$ then $p_{i}=P(Y=i)$ and similarly for $Q$, this should be made clear.
2. [lines 36-38] I am not sure about the correctness of the claim that “if certain state has very small probability under the stationary distribution, consistent estimation of the transition matrix may not be possible”. Under irreducibility, every state is visited infinitely many times and consistent estimation is guaranteed. Under reducibility, some states may not be visited at all and the corresponding transition probabilities cannot be estimated.
3. [lines 50-51] Since highly related to this work, the authors should clarify what exactly “implicit assumptions on mixing time such as spectral gap conditions” that were made by HOP18 meant. (or at least point to where they are stated in HOP18, since I have not found them there).
4. [line 53] The authors should make the phrase “with and without spectral gap” more precise. Isn't “without assumptions on the spectral gap“ or “arbitrary spectral gap” clearer?
5. [line 87] It would be informative to explain the sudden restriction to reversible Markov chains. Is it an artifact of the analysis? Do reversibilization methods fail? What about the pseudo spectral gap?
6. [lines 92-93] From $\gamma_{0}>0$ it follows that the Markov chain is aperiodic. Together with irreducibility this give ergodicity. The authors might want to use this term explicitly.
7. [line 96] In what sense is Theorem 2 an extension of (2) of [FOPS16]?
6. [lines 124 and 131] “known as redundancy” and “is standard”. The authors might want to give references for the definition and for the result.
8. [line 142] To which reduction do the authors refer?
9. [line 213] Is there a reason for $X^{n+1}$ instead of $X^{n}$? What is $\Theta$?
10. [line 214] Shouldn't the definition $X^{n}\overset{\Delta}{=}(X_{1},\ldots,X_{n})$ come before line 213? Actually, $X^{n}$ was already defined in line 23.
11. [line 220] Here a more exact reference is given ([CT06, Chapter 13]). It would be nice if the authors could provide exact references for the well known definitions and results that are being used.
No
We thank the reviewer for the constructive and positive comments. We will try to implement the comments about wordings that have been pointed out. Specific responses to comments are given below.
1. “…
For example, even after several reads I am not sure what "with and without spectral gap" means. Does it mean with and without ASSUMPTIONS on the spectral gap?”
Response: We will change the title to “...with and without spectral gap conditions” for further clarification.
2. “Section 3 is called Optimal rates without spectral gap but there is no corresponding section for "with".”
Response: Due to page restrictions the main text has been dedicated to explaining the spectral gap independent results. The results based on spectral gap assumptions are more technical in nature and have been provided in supplemental file, section 7. We have mentioned this at the end of section 1.
3. “[line 25] While it can be assumed that if $Y\sim P$ then $p_i=P(Y=i)$ and similarly for $Q$, this should be made clear.”
Response: We will clarify this in the revision.
4. “[lines 36-38] I am not sure about the correctness of the claim that “if a certain state has very small probability under the stationary distribution, consistent estimation of the transition matrix may not be possible”. Under irreducibility, every state is visited infinitely many times and consistent estimation is guaranteed. Under reducibility, some states may not be visited at all and the corresponding transition probabilities cannot be estimated.”
Response:  Thanks for the question. The reviewer's reasoning is correct in the usual large-sample asymptotics, namely when the transition matrix is fixed and the length n of the trajectory tends to infinity. If the transition matrix depends on n (which is allowed in the worst-case analysis), then consistent estimation may not be possible.
5. “[lines 50-51] Since highly related to this work, the authors should clarify what exactly “implicit assumptions on mixing time such as spectral gap conditions” that were made by HOP18 meant.”
Response: In HOP18, the proof of the upper bound for prediction (Lemma 9 in the main paper and Lemma 7  in the supplement) and for estimation (Lemma 17 of the supplement) is based on Berstein-type concentration results of the empirical counts $N_i,N_{ij}$, which depends on the spectral gap. We will clarify this in the revision.
6. “[line 87] It would be informative to explain the sudden restriction to reversible Markov chains. Is it an artifact of the analysis? … What about the pseudo spectral gap?”
Response: The definition of spectral gap and absolute spectral gap requires irreducibility and reversibility.
Thus when presenting spectral gap dependent risk bounds, we restrict this class of Markov chains. We agree it is possible to use more general notions such as pseudo spectral gap to quantify the memory of the process, which is beyond the scope of the current paper.
7. “[line 96] In what sense is Theorem 2 an extension of (2) of [FOPS16]?”
Response:  FOPS16 gives the minimax rates over all binary Markov chains, which corresponds to the case of $\gamma_0=0$ in Theorem 2. Theorem 2 determines the minimax rate for all $\gamma_0$.
8. “[line 142] To which reduction do the authors refer?”
Response:  We referred to the reduction in equation (11). We will clarify this in the revision.
The paper is very-well written, and offers strong results. The problem is hard, and was until now only understood for the restricted 2-state case. The bounds are matching up to universal constants. The techniques are involved, enabling the authors to obtain bounds that are independent of the mixing properties of the chain (Th.1).
Although this reviewer did not verify the proofs carefully, they have found no reason to doubt them. This is a strong submission; this reviewer recommends acceptance.
* (L31-L38;L186-192)
In these two paragraphs, the authors seem to imply that there is something inherently simpler about the prediction problem than, say, estimation, illustrating this by saying that consistent estimation of the transition kernel is not possible if the stationary distribution comprises extremely small values. This last part, however, is a bit misleading and depends on the notion of distance one considers.
It is true that some notions of distance lead to a minimax rate that also captures a dependence in the stationary distribution.
But for example, by weighing the state-wise discrepancy in terms of the stationary distribution as in Hao et al. [2018, p.2], or considering a distance in terms of the edge-measures of the chains as  in Wolfer and  Kontorovich  [2021,  Theorem  3.3], it is possible to obtain rates that are independent on the stationary distribution for the estimation problem.
* (L79-81) Note that such interesting decoupling has also been observed in the Markov chain testing literature for some notion of distance (cited by the authors).
* Add a few words on algorithmic complexity.
* On page 1, when taking the infimum, indicate whether estimators must be (proper) Markov kernels.
* Since the results are expressed up to universal constants, one can remove squares from the logs in the bounds $\log (1 / k^2) = C \log 1/k$.
**REFERENCES**
* Y. Hao, A. Orlitsky, and V. Pichapati. On learning markov chains. In NeurIPS, 2018.
* Wolfer  and  A.  Kontorovich.    Statistical  estimation  of  ergodic  markov  chain  kernel  over  discrete  space. Bernoulli, 27(1):532–553, 02 2021.  doi:  10.3150/20-BEJ1248.
No
We thank the reviewer for the constructive and positive comments. Specific responses to comments are given below.
1. "(L31-L38;L186-192) In these two paragraphs, the authors seem to imply that there is something inherently simpler about the prediction problem than, say, estimation, illustrating this by saying that consistent estimation of the transition kernel is not possible if the stationary distribution comprises extremely small values. This last part, however, is a bit misleading and depends on the notion of distance one considers.
It is true that some notions of distance lead to a minimax rate that also captures a dependence in the stationary distribution.
But for example, by weighing the state-wise discrepancy in terms of the stationary distribution as in Hao et al. [2018, p.2], or considering a distance in terms of the edge-measures of the chains as in Wolfer and Kontorovich [2021, Theorem 3.3], it is possible to obtain rates that are independent on the stationary distribution for the estimation problem."
Response: Thanks for the suggestion. We will clarify in the revision that here consistent estimation is with respect to the usual loss function e.g. squared risk.
About the two references: the minimax rate of Hao et al. [2018, p.2] is obtained for chains with transition probabilities bounded away from 0 and 1 (say by delta), which implicitly restricts stationary distribution $\pi$ away from 0 and 1. The terms involving delta in their minimax rates are suppressed in the theorem statement.
On the other hand, the sample complexity produced in Wolfer and Kontorovich [2021, Theorem3.3] also depends on $\pi$ via $||\mu/\pi||_{2,\pi}$.
2. “Add a few words on algorithmic complexity.”
Response: The estimator achieving the minimax rates is given by $\tilde Q$ in equation (25) for m=2, which is average of the add-one estimators. Given any $X^{n-1}=x^{n-1}$, estimating the probability of $X_n=j$ entails calculating add one estimator $\hat{M}^{+1}(j|x_{n-1})$ for the sequences $x^{n-1}_{n-t+1}$, $t=2,\dots,n$, and then taking their average.
Calculating $\hat M^{+1}(j|x_{n-1})$ for sequence $x_1^{n-1}$ takes $O(n)$ time. Given the add one estmator based on
$x_{n-t+1}^{n-1}$ we need $O(1)$ time to calculate the same based on $x^{n-1}_{n-t+2}$. Summing over all $j$ this implies the total time complexity $O(nk)$. We will add this in the revision.
3. “On page 1, when taking the infimum, indicate whether estimators must be (proper) Markov kernels.”
Response: Thanks for pointing it out. We will indicate this in the revised version.
Many thanks to the authors for their detailed response.
This reviewer keeps the score unchanged: Accept.
The paper studies a clear and well motivated problem, and introduces universal compression techniques to the study of Markov chains. The result for Markov chains without spectral gap assumptions is nice.
No
We thank the reviewer for the positive comments.
Strengths:
1. This paper proves the optimal prediction risk with the number of states between 3 and $\sqrt{n}$ without the spectral gap, which requires more difficult proof technique than previous works analyzing markov chain with 2 states or with spectral gap.
2. To achieve optimality, the paper defines an average of add-on estimators to be a new estimator and proves its optimality.
3. The role of spectral gap is characterized clearly in the prediction risk.
Weakness / comments:
1. The most natural estimator for $M(j|i)$ might be the empirical frequency $N_{ij}/N_i$ instead of the add-one estimator in (5). It might be beneficial to discuss the reason why you use the add-one estimator: is it because empirical frequency is sub-optimal in nature, or just a technical issue.
2. The structure of this paper is a little strange. It might be better to move the main theorems and proof techniques (Section 1.1) outside the introduction (Section 1).
No
We thank the reviewer for the constructive and positive comments. We will try to implement the comments about wordings that have been pointed out. Specific responses to comments are given below.
“The most natural estimator for $M(j|i)$ might be the empirical frequency $N_{ij}/N_i$ instead of the add-one estimator in (5). It might be beneficial to discuss the reason why you use the add-one estimator: is it because empirical frequency is sub-optimal in nature, or just a technical issue.”
Response: For KL risk, directly using empirical distribution leads to infinite risk when certain symbols have non-zero probability but are not observed.
For this reason, smoothed versions  of empirical distribution (add-c estimators) are used. This reasoning is the same for both iid and for Markov data.
I thank the authors for responding to my questions. I will update my score.

Meta-review:
role: PC
data: comment: This paper is interested in predicting the next state, given $n$ observations from a single trajectory of a Markov chain. It is very well-written and offers strong, significant, interesting results.

Decision: Accept (Poster)