Title: The surprising efficiency of temporal difference learning for rare event prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: temporal difference learning

reinforcement learning

rare events

policy evaluation

prediction

perturbation bounds
Abstract: We quantify the efficiency of temporal difference (TD) learning over the direct, or Monte Carlo (MC), estimator for policy evaluation in reinforcement learning, with an emphasis on estimation of quantities related to rare events. Policy evaluation is complicated in the rare event setting by the long timescale of the event and by the need for \emph{relative accuracy} in estimates of very small values.  Specifically, we focus on least-squares TD (LSTD) prediction for finite state Markov chains, and show that LSTD can achieve relative accuracy far more efficiently than MC.  We prove a central limit theorem for the LSTD estimator and upper bound the 
  \emph{relative asymptotic variance}
  by simple quantities characterizing the connectivity of states relative to the transition probabilities between them. Using this bound, we show that, even when both the timescale of the rare event and the relative accuracy of the MC estimator are exponentially large in the number of states, LSTD maintains a fixed level of relative accuracy with  a total number of observed transitions of the Markov chain that is only \emph{polynomially} large in the number of states.

The surprising efficiency of temporal difference learning for rare event prediction

Abstract

We quantify the efficiency of temporal difference (TD) learning over the direct, or Monte Carlo (MC), estimator for policy evaluation in reinforcement learning, with an emphasis on estimation of quantities related to rare events. Policy evaluation is complicated in the rare event setting by the long timescale of the event and by the need for relative accuracy in estimates of very small values. Specifically, we focus on least-squares TD (LSTD) prediction for finite state Markov chains, and show that LSTD can achieve relative accuracy far more efficiently than MC. We prove a central limit theorem for the LSTD estimator and upper bound the relative asymptotic variance by simple quantities characterizing the connectivity of states relative to the transition probabilities between them. Using this bound, we show that, even when both the timescale of the rare event and the relative error of the MCestimator are exponentially large in the number of states, LSTD maintains a fixed level of relative accuracy with a total number of observed transitions of the Markov chain that is only polynomially large in the number of states.

1 Introduction

Prediction of the future behavior of a dynamical system from time series observations is a fundamental task in science and engineering. One need only remember the weather forecast consulted this morning to appreciate the substantial intersection this problem has with our everyday lives. Beyond its own obvious importance, prediction is also a key component in modern machine learning tasks like reinforcement learning, where the rules governing the dynamical system are adjusted to optimize some reward. Exciting applications in that field, like self-driving cars, have captured public imagination.

Prediction is made particularly difficult when the future behavior of interest involves some rare or extreme event. Examples include extreme weather or climate events, failure of reliable engineering products, and the conformational rearrangements that determine critical functions of biomolecules in our bodies. Precisely because of their out-sized impact on our lives and society, rare and extreme events are the most important to predict. Unfortunately, by definition, data sets often contain relatively few examples of these events. In this article we ask: can rare events be predicted accurately with time series data sets much shorter than the typical timescale of the event? The surprising answer is Yes. Our mathematical results support recent studies demonstrating accurate rare event prediction with limited data using methods related to those studied here, including in state-of-the-art application settings and for very rare events [Thiede et al., 2019, Strahan et al., 2021, Finkel et al., 2021, 2023b, Antoszewski et al., 2021, Lucente et al., 2021, Finkel et al., 2023a, Strahan et al., 2023a, Jacques-Dumas et al., 2023, Strahan et al., 2023b, Guo et al., 2024].

Within the context of reinforcement learning, 'prediction' refers to policy evaluation, that is, calculation of the value function of states given a policy [Sutton and Barto, 2018]. Prediction is the

foundation for policy improvement in control problems. In this work, we focus on prediction for Markov chains on a finite state space. While many dynamical systems of physical interest evolve in continuous spaces, this so-called 'tabular' setting is the most common for mathematical analysis. Moreover, some practical algorithms for prediction problems in continuous space begin with a projection onto a finite state Markov process [Thiede et al., 2019, Finkel et al., 2021, 2023a, Jacques-Dumas et al., 2023]. For those algorithms our analysis can be viewed as addressing estimation (but not approximation) error .

Specifically, we formulate a prediction problem in terms of a Markov reward process (MRP) ([ n ] , P, R ) , where [ n ] = { 1 , 2 , . . . , n } is the state space, P is the Markov transition kernel with P ( i, j ) = P [ X t +1 = j | X t = i ] , and R is a deterministic, non-negative reward function. Time dependence and randomness of R can be included by an enlargement of the state space. Our goal is to estimate a value function of the general form 1

where D is a subset of [ n ] , T = min { t ≥ 0 : X t / ∈ D } is the escape time from D , and the subscript i on the expectation indicates that X 0 = i . By appropriate choice of R , we can consider both cases in which the escape time from D is very large and cases in which only low probability escape paths accumulate significant reward. These are the typical situations for rare event statistics, where the reward R and the set D are specified by the application. In the example of self-driving cars, when investigating the rare event that the car crashes, D is the set of all states in which the car has not crashed.

1.1 Monte Carlo and temporal difference learning

The prediction problem has been extensively analyzed in the reinforcement learning literature [Sutton and Barto, 2018, Dann et al., 2014]. Our problem's main point of departure from that literature is the central role of the escape event from D in (1). We are specifically interested in cases in which E j [ T ] can be very large so that the length of a single escape trajectory can be very large, and/or cases in which large values of the reward functions are very unlikely so that u can be very small.

Our task is to learn an estimate of u from a set of M samples { X ℓ 0 , X ℓ 1 , . . . , X ℓ τ ∧ T ℓ } ℓ M =1 of X 0 , X 1 , . . . , X τ ∧ T with { X ℓ 0 } ℓ M =1 drawn from some (possibly unknown) initial probability distribution µ which is supported in D . Here and below we use the shorthand a ∧ b = min { a, b } . The deterministic positive integer τ , which we will refer to as a lag time, limits the length of the trajectories in the data set. In practical applications, the sample trajectories are often correlated, for example because multiple length τ trajectories can be harvested from a single longer trajectory. Here, to simplify the analysis, we assume that the trajectories are drawn independently.

The most direct estimator of u ( i ) is the Monte Carlo (MC) estimator

where M i is the number of samples with X ℓ 0 = i , i.e. M i = ∑ M ℓ =1 1 { i } ( X ℓ 0 ) . The estimator is undefined for i ∈ D with M i = 0 . The MC estimator does not allow finite values of τ (unless T is bounded by a constant). When the escape event is rare, T is very large and the MC estimator requires data sets containing very many observed time steps of X t . Moreover, estimators of rare event statistics are evaluated based on the amount of data required to achieve a desired relative error [Bucklew, 2004]. In the rare event setting, u ( i ) can be extremely small because of the low probability of the event to happen. In this case, assuming µ ( i ) > 0 , control of the relative variance, Var [ˆ u ( i )] /u 2 ( i ) , often requires M ∝ 1 /u ( i ) trajectory samples for the MC estimator. This scaling is fatal when u ( i ) is very small.

Temporal difference (TD) schemes, on the other hand, express u as the solution to a certain Bellman equation [Sutton, 1988]. They enforce temporal local consistency of the estimate of the value function.

As such, TD methods can accommodate any positive choice of τ , while MC methods require that each trajectory be observed until escape from D (i.e. τ = ∞ ). TD schemes are used heavily in reinforcement learning and they are often observed to out-perform MC [Sutton and Barto, 2018]. However, a quantitative understanding of the advantages of TD compared to MC remains elusive. In practice, there are several variants of TD schemes: parametric or tabular, online or batch. In this work, we stick to the simplest setting: the least-squares TD (LSTD) estimator [Bradtke and Barto, 1996] in the tabular setting of a finite state MRP. Our goal is to clearly characterize the benefit of the LSTD estimator over MC for predictions related to rare events in this setting.

TD schemes begin with the observation that, for any choice of τ , u in (1) solves the linear system

where

is the transition operator of X t ∧ T , S τ is the τ th power of S and R D ( i ) = R ( i ) for i ∈ D while R D ( i ) = 0 for i / ∈ D . In the limit τ →∞ , (2) becomes (1). In our theory and examples, we will see that the choice of τ can have a dramatic effect on the performance of the estimator.

TD methods use trajectory data to find an approximate solution to (2). Specifically, the LSTD estimator ˜ u in this tabular setting is the solution to a linear system where the transition matrix S in (2) is approximated with trajectory data. We set ˜ S 0 = I and, for t = 1 , 2 , . . . , τ ,

Unlike S t , ˜ S t are not powers of a single matrix. When it exists, the τ -step LSTD estimator, ˜ u solves

(Like the MC estimator, ˜ u ( i ) is undefined when M i = 0 ). In the τ →∞ limit, the τ -step LSTD and MCestimators are equivalent. We therefore occasionally refer to the Monte Carlo estimator as the τ = ∞ estimator.

1.2 Our contributions

We pause now to consider what classical perturbation theory for linear systems has to say about the feasibility of estimating u , the solution of (2), by ˜ u , the solution to (3). Clearly (3) only requires access to trajectories of length τ , but how sensitive is the solution of (2) to perturbations in S ? If it is very sensitive then any reduction in data requirements from shorter trajectories may be offset by a need for many more trajectories. Classical perturbation theory tells us that errors S t -˜ S t can, in the worst case, be amplified in u -˜ u by a factor of the condition number κ τ = ∥ ( I -S τ D ) -1 ∥∥ I -S τ D ∥ , where S τ D is the matrix obtained from S τ by setting to zero any row or column with index in D c [Demmel, 1997]. As the next lemma shows, when typical values of T are much larger than τ , ∥ ( I -S τ D ) -1 ∥ will be very large. The proof of Lemma 1 is in Appendix A.

Lemma 1. For any consistent matrix norm, if the restriction of S D to row and column indices in D is irreducible and aperiodic, then ∥ ( I -S τ D ) -1 ∥ ≥ E ν [ T ] /τ where ν ( i ) = lim t →∞ P [ X t = i | T > t ] is the quasi-stationary distribution and the subscript on the expectation indicates that X 0 is drawn from ν . (See Collett et al. [2012] for more about the quasi-stationary distribution).

While it is possible that a small value of ∥ I -S τ D ∥ can compensate for a large value of E ν [ T ] /τ and result in a moderate condition number, for many rare event problems this is not the case. For example, for the system studied in our numerical experiments in Section 3, ∥ I -S D ∥ is bounded away from zero even as the escape event becomes increasingly rare (see Appendix B).

Thus it would seem that we are forced to choose between two doomed options: choose τ small to control the length of trajectories we need to observe but observe a huge number ( M ∝ E ν [ T ] /τ ) of

them to drive down the error in ˜ S t , or observe very long trajectories ( τ ∝ E ν [ T ] ) to control κ τ . With either of these choices we would not expect TD to significantly outperform MC.

A primary goal of this article is to explain that, in many (perhaps most) cases, this is a false choice. The worst case analysis that characterizes the classical perturbation theory is, by design, pessimistic. But as we will explain, in our setting it is wildly pessimistic. With a few simple and practically relevant assumptions we will be able to show that τ -step LSTD can achieve remarkably accurate estimates with remarkably little data.

Throughout this paper we quantify the relative accuracy of ˜ u ( i ) by the relative asymptotic variance σ 2 i /u 2 ( i ) where σ 2 i is the variance of the limiting normal distribution in a central limit theorem for ˜ u ( i ) established in Theorem 1. Also in Theorem 1, we provide an upper bound on the relative asymptotic variance by simple quantities characterizing the connectivity of states relative to the transition probabilities between them. Crucially, neither the condition number κ τ , nor any expectation of T , appear explicitly in the bound. Next, we turn our focus to the rare event setting, which we characterize via the large n behavior of the estimator. In this setting both the typical escape time and the relative variance of the MC estimator can scale exponentially with n . As we show in Theorem 2, however, the relative asymptotic variance of the LSTD estimator scales, at worst like n 3 .

1.3 Related work

Since it first appeared in [Sutton, 1988], there have been numerous variants of TD [Bradtke and Barto, 1996, Dann et al., 2014, Lillicrap et al., 2019]. Early theoretical analysis of TD focused on asymptotic convergence with linear value function approximation [Jaakkola et al., 1993, Dayan and Sejnowski, 1994, Tsitsiklis and Van Roy, 1997], along with examples of divergence [Baird, 1995, Tsitsiklis and Van Roy, 1997]. More recently, nonasymptotic convergence analysis of TD has been performed [Bhandari et al., 2018, Dalal et al., 2018, Srikant and Ying, 2019, Cai et al., 2019].

Besides its own convergence, a long-considered question for TD is when and how it outperforms MC. While practical experience suggests that TD is more efficient than MC, a comprehensive theory is lacking. Grunewalder et al. [2007] and Grünewälder and Obermayer [2009] proved that LSTD is at least as statistically efficient as MC, but the improvement is not quantified. Cheikhi and Russo [2023] expresses the relative benefit of TD over MC by the inverse trajectory pooling coefficient, which facilitates interpretation of the ratio of asymptotic variances. But how that translates to quantitative improvements requires further elucidation. As far as we are aware, prediction for rare event problems has not been analyzed in the reinforcement learning literature.

TD enforces temporal consistency for trajectory data. Trajectory data has been utilized in applications more broadly. Markov state models [Husic and Pande, 2018] and dynamic mode decomposition methods [Schmid, 2022] were developed in the molecular dynamics and fluid dynamics communities respectively and have been successful at estimating eigenvectors and eigenvalues of Markov transition (or Koopman) operators from trajectory data in state-of-the-art applications. The dynamical Galerkin approximation [Thiede et al., 2019] method extends those approaches to the prediction problem studied here and has been used to study rare events in molecular dynamics and climate science [Strahan et al., 2021, Antoszewski et al., 2021, Finkel et al., 2021, 2023a, Guo et al., 2024]. Despite widespread application in the study of various rare events over several decades, the present article is the first theoretical evidence that trajectory analysis methods can be effective tools specifically for rare events.

1.4 Notation

X D - → N ( µ 0 , σ 2 0 ) indicates that a random variable X converges in distribution to a normal distribution with mean µ 0 and variance σ 2 0 . For any subset A ⊆ [ n ] , we define the hitting time T A = min { t > 0 : X t ∈ A } . When X t never hits A , T A = + ∞ . When the set A contains only one state i , we use T i for simplicity. Note that in (1), T counts from t = 0 instead. While S is the transition probability matrix for the Markov chain X t ∧ T , we define another Markov chain Y τ t whose transition probability matrix is S τ . For Y τ t , we define its hitting time to be T τ A = min { t > 0 : Y τ t ∈ A } . We define T τ = min { t ≥ 0 : Y τ t ∈ D } as a counterpart of T for X t . We use e i for the i th standard basis vector, i.e. e i ( j ) = δ ij . For any two quantities a and b , a ≳ b means that there exists a positive constant C independent of n such that a ≥ Cb . Similarly, a ≲ b means that a ≤ Cb .

2 An upper bound for relative asymptotic variance

As previous authors have done [Cheikhi and Russo, 2023], we will characterize the error of the LSTD estimator using a central limit theorem. Unlike previous work, we will also provide a simple upper bound on the relative asymptotic variance that clearly distinguishes the LSTD estimator from the MC estimator. Our subsequent analysis of the LSTD estimator in the rare event setting will be derived from this bound.

Theorem 1. Suppose µ ( i ) > 0 for all i ∈ D and P is irreducible. Then, as the number of samples M →∞ , with

Moreover, the relative asymptotic variance σ 2 i /u 2 ( i ) satisfies the upper bound

̸

where we have introduced the key quantity Q τ ( k, ℓ ) = P k [ T τ ℓ < T τ k ∧ T τ D c \{ ℓ } ] .

An important special case that will include one of the examples studied in Section 3, occurs when the reward function R is zero in D (but non-zero in D c ). In this case we can strengthen the bound in Theorem 1 by only including S τ in the numerator.

Corollary 1. Under the same assumptions as in Theorem 1 but with R ( i ) = 0 for any i ∈ D ,

̸

Let us take a moment to parse the bounds in Theorem 1 and Corollary 1. The bounds tell us that if the sum is bounded, our estimator achieves entry-wise relative accuracy, indicating that even when u ( i ) is extremely small, the error in ˜ u ( i ) can be much smaller (depending only on M ). But the denominator is the real star of the show. It is the sum of the probabilities of all paths connecting state k and state ℓ that do not return to k and do not exit D (except possibly through ℓ ). In particular Q τ ( k, ℓ ) ≥ S τ ( k, ℓ ) . While the probability Q τ ( k, ℓ ) can be very small for most pairs ( k, ℓ ) , evidently it is the size of Q τ ( k, ℓ ) 2 relative to S t ( k, ℓ ) for t ≤ τ (or only S τ ( k, ℓ ) ) that matters. Fortunately, in many cases, we can expect Q τ ( k, ℓ ) 2 to be much larger than S τ ( k, ℓ ) for a good choice of τ . We will quantify this statement in the rare event setting in Section 4.

A quantity very similar to Q τ ( k, ℓ ) appears in the matrix analysis literature [Thiede et al., 2015] as a measure of the sensitivity of the relative accuracy of the invariant distribution of a Markov chain to entry-wise perturbations in its transition probability matrix. The proofs of Theorem 1 and Corollary 1 are given in Appendix C. Lemma 2 in Appendix C, which is used to bound the variation in u between states, is key to the argument.

3 Experiments

We now consider two example problems that illustrate the challenges posed by rare event prediction. For both problems the statistical efficiency of the MC estimator degrades exponentially in the state space size n . Our goal is to see whether or not LSTD's performance degrades similarly as n increases, and, in so doing, to motivate our final bounds on the relative asymptotic variance of LSTD established later in Section 4. The examples and numerical results presented in this section are exactly consistent with the assumptions and results presented in Section 4.

Both problems are based on a one dimensional nearest neighbor chain X t ∈ [ n ] with transition rules

with P (1 , 0) = P ( n, n +1) = 0 on the boundary. Here p is the invariant probability vector of P ,

which has modes around i = 1 , i = ( n +1) / 2 , and i = n , that become more sharply peaked as n increases. The escape time from any of these modes scales exponentially with n . We will be interested in predictions related to the escape of X t from D = [ n ] \ { 1 , n } , and choose µ to be the uniform distribution on D .

Before moving on to the specific problem statements, we point out that despite the exponentially long waiting time to transition between the modes of p , transitions between nearest neighbor states remain stable as n increases. In fact, for i ∈ D , P ( i, i ± 1) ≥ 1 2(1+ e ) . Stable local transition probabilities such as these play a key role in our upper bounds in Section 4, where they are used to lower bound Q ( k, ℓ ) in (6).

The mean first passage time. The mean first passage time u ( i ) = E i [ T ] solves (2) with R ( i ) = 1 for i ∈ D and R ( i ) = 0 for i / ∈ D . We plot the mean first passage time for n = 20 , 40 , and 80 in the left panel of Fig. 1. Evidently the largest values of E i [ T ] scale exponentially with n . In Appendix B, we prove that, near i = ( n +1) / 2 ,

Therefore the trajectories required by the MC estimator will include a number of transitions of X t that scales exponentially with n . We compare the performance of the MC estimator of the mean first passage time to that of the LSTD estimator, ˜ u , found by solving (3) with τ = 1 . In the middle panel of Fig. 1 we plot the relative asymptotic variance, σ 2 i /u 2 ( i ) , for the same values of n . In the same panel we plot empirical estimates of the the (non-asymptotic) relative mean squared error (MSE), M MSE [˜ u ( i )] /u 2 ( i ) , with M = 10 n 3 . The empirical relative MSE estimates are computed by generating 30000 independent copies of the LSTD estimator. Remarkably, the relative MSE of the LSTD estimator grows more slowly than n 3 . Meanwhile, near the boundary of D , the relative MSE of the MC estimator (computed exactly) grows exponentially fast with n , as can be seen in the right panel of Fig. 1.

The committor. We now consider estimation of the committor function

̸

The committor corresponds to the choice R ( i ) = 0 for i = n and R ( n ) = 1 . Because E i [ T ] can be exponentially large, the MC estimator of the committor again requires a data set containing exponentially long trajectories of X t . Moreover, in Appendix B, we prove that u ( i ) can be exponentially small in n , as as is evident in the plot of the committor for n = 20 , 40 , and 80 , in the left panel of Fig. 2. As a consequence, the relative MSE of the MC estimator, plotted in the right panel of Fig. 2, grows

exponentially fast with n . As can be seen from the middle panel of Fig. 2, however, the relative asymptotic variance of the LSTD committor estimator with τ = 1 grows more slowly than n 3 . Again empirical estimates of the relative MSE of the LSTD estimator with M = 10 n 3 agree well with corresponding relative asymptotic variances and show the same trend.

Together, the mean first passage time and the committor estimation problems in this section strongly suggest a significant (polynomial versus exponential in n ) advantage for TD methods over MC. In the next section we will prove that the performance advantage observed on these two examples holds in significant generality.

Effect of the lag time. In the above experiments we have examined the error of the LSTD estimator with τ = 1 . In practice, the best choice of τ is long enough to avoid a nearly diagonal S τ , but not so long as to lose the efficiency advantage of TD over MC that we have just observed. Our bounds in Theorem 1 also change with τ . To better illustrate the effect of τ on the relative asymptotic variance of the LSTD estimator and on our bounds, we now consider a 'lazier' version of the Markov chain in (8). Specifically, we define P ( i, i ± 1) = p ( i ± 1) 10( p ( i )+ p ( i ± 1)) for i ± 1 ∈ D , with the same boundary conditions as before. We again consider estimates of the mean first passage time and the committor, now with fixed n = 40 . We plot the maximum of the relative asymptotic variance, σ 2 i /u 2 ( i ) , over i ∈ D in Fig. 3, along with the bound (6) for the mean first passage time in the left panel and (7) for the committor in the right panel. Though the plots do not extend to values of τ comparable with the largest values of E i [ T ] , the relative variance of the MC estimator corresponds to the asymptote of the true relative variance toward the right hand side of each plot. We observe that the true relative variance of the LSTD estimator is minimized in both cases for a choice of τ ≈ 30 and that, for this choice, the relative variance of the LSTD estimator is significantly smaller than the relative variance of the MC estimator. The bounds in (6) and (7) are designed to capture the high accuracy of LSTD for relatively small values of τ and we indeed see that they deteriorate as τ becomes very large. However, at least in the committor case, the bound accurately reproduces the initial decrease in error that occurs when τ is increased above τ = 1 .

Effect of the initial distribution. So far, in our experiments, we have chosen µ to be the uniform distribution on D . A natural alternative strategy is to harvest many short trajectories from a single, much

longer trajectory. Under ergodicity assumptions on P , these short trajectories are then approximately drawn from the invariant distribution of P . In Fig. 4, we present the relative asymptotic variance and the empirical estimates of the relative MSE of the mean first passage time and the committor, with P being the same as in (8), and µ chosen to be the invariant distribution p in (9) conditioned within D . For τ = 1 and n = 20 , 40 , and 80 , the maximum relative asymptotic variances of the LSTD estimators grow exponentially with n . With M = 10 n 3 , the empirical relative MSE estimators with 30000 independent copies of the LSTD estimators agree well with the relative asymptotic variance for n = 20 and 40 . For n = 80 , the empirical linear system (3) fails to define the LSTD estimators with high probability, and the empirical error is undefined. These results should be contrasted with the much smaller errors shown in the middle panels of Figs. 1 and 2, corresponding to uniformly chosen initial conditions. We expect that the results in Fig. 4 would be somewhat different if, instead of initializing trajectories independently from the conditional invariant distribution, we had harvested correlated initial conditions from a single ergodic trajectory. Nonetheless, the results indicate that the choice of µ can have a significant impact on the performance of LSTD for prediction of rare events.

4 Rare event assumptions and upper bound

Our final relative asymptotic variance bounds will rigorously establish the dramatic advantage of TD over MC that we observed in the experimental results of Section 3. As in that section, we focus on the behavior of relative variance as n increases. Our results rely on several basic assumptions. On the one hand, all of these assumptions are satisfied by the Markov chain examined in Section 3. On the other hand, the results in this section cover considerably more general Markov chains. The assumptions concern certain structural properties of the Markov chain as n increases, but crucially, they allow both the typical escape time and the relative variance of the MC estimator to grow exponentially in n .

To ensure that a unique ˜ u will exist for large enough M , we make the following minorization assumption on µ .

Assumption 1 (Lower bound on µ ) . For some constant α > 0 , independent of n , and all i ∈ D , µ ( i ) ≥ α n .

Recall that the uniform distribution on D was used to generate initial conditions for the numerical tests described in Section 3. The invariant distribution p in (9) conditioned within D , instead, violates this assumption, as its minimum is exponentially small in n .

As mentioned in Section 1.2 the perturbations, ˜ S τ -S τ , relevant for analysis of the LSTD estimator are far from the worst case perturbations characterizing classical perturbation bounds for Eq. (2). Each entry of the matrix ˜ S t is the empirical frequency of a specific transition. As a result, the variance of ˜ S t ( i, j ) is proportional to S t ( i, j ) (1 -S t ( i, j )) ≤ S t ( i, j ) , and we can characterize the perturbations by characterizing the entries themselves. Different entries of S t can have very different magnitudes, and many Markov processes exhibit higher transition probabilities between states that are 'close' according to some metric and small transition probabilities between states that are 'far away'. To express this notion in assumptions without resorting to any topology within which the state space may be embedded, we upgrade [ n ] to an unweighted directed graph G by including edges

only along transitions with significant probability. Our additional assumptions will be stated in terms of properties of this 'minorizing graph.'

Specifically, we introduce a directed edge from state i ∈ G to another state j ∈ G if S τ ( i, j ) ≥ c , where c is a positive constant independent of the number of states n .

Assumption 2 (Connected minorizing graph) . For some constant c > 0 independent of n , any two states i ∈ D and j ∈ [ n ] are connected by a path in G .

When P is irreducible, as assumed in Theorem 1, and n is fixed, we can always find a c > 0 so that any i ∈ D and j ∈ [ n ] are connected by a path in G for τ = 1 . Assumption 2 emphasizes the independence of n , asserting that the transition probabilities above the threshold c can, on their own, preserve the connectivity of the graph while n increases. Recall that the Markov chain studied in Section 3 satifies P ( i, i ± 1) ≥ 1 / (2(1 + e )) , implying that Assumption 2 is satisfied with the choice c = 1 / (2(1 + e )) .

Let d ( i, j ) be the length of the shortest path from state i to state j in G . When the graph is undirected, d is a distance on G , but it need not be. Finally, we characterize the decay of the transition probabilities of pairs of states that are 'far away' according to G .

̸

Assumption 3 ('Sub-Gaussian' transition probability tails) . For some constants β > 0 and C , independent of n , for all i = j , and t ≤ τ , S t ( i, j ) ≤ Ce -βd 2 ( i,j ) .

Assumption 3 stipulates that transition probabilities decay fast enough for distant states in G . For the Markov chain studied in Section 3, when τ = 1 this assumption is satisfied because transition probabilities are zero for states that are not nearest neighbors. Assumptions 2 and 3 essentially characterize the locality of the transitions. These assumptions are practically reasonable even when, as in the example in Section 3, the global landscape of the transitions exhibits challenging traits such as multi-modality.

Illustration. The left panel of Fig. 5, depicts a general (symmetric) Markov chain on a graph. Darker edges indicate higher transition probabilities. The transition probabilities are chosen to decay with Euclidean distance in the plane, consistent with, for example, a Gaussian transition kernel in the plane. The minorizing graph on the right retains only those edges exceeding the threshold c , which is chosen large enough to remove most distant edges, but small enough to result in a connected graph.

In Section 3, we considered the effect of the choice of τ on relative asymptotic variance. In that example, a larger choice of τ initially increased the probability of nearest neighbor transitions and would allow for a larger choice of c . As τ increases further however, the probabilities of distant transitions would become larger, potentially decreasing the maximum allowed choice of c and possibly requiring a smaller choice of β and/or a larger choice of C in Assumption 3. The quantitative interplay among c , β and C are determined by the specific Markov chain under study, but we expect similar behavior for common transition probabilities such as Gaussian transition kernels.

As the experiments in Section 3 clearly demonstrate, these assumptions allow both the typical escape time and the relative variance of the MC estimator to scale exponentially with n . The failure of MC in these scenarios does not contradict this paper's theoretical results because the MC estimator does not satisfy Assumption 2. Indeed, the MC estimator corresponds to τ = ∞ , for which S τ ( i, j ) can only be non-zero if i ∈ D , j ∈ D c .

With these assumptions, we can now state our upper bound for the rare event setting, whose proof is very simple but informative. The key idea is that, under these assumptions, Q τ ( k, ℓ ) in the denominator is indeed much larger than the transition probabilities S t ( k, ℓ ) in the numerator.

Theorem 2. Under Assumptions 1, 2 and 3, we have the following asymptotic variance bound:

Proof of Theorem 2. Because Q τ ( k, ℓ ) is the sum of the probabilities of all paths connecting states k and state ℓ that do not return to k and do not exit D (except through ℓ ), Assumption 2 implies that

Plugging this bound and the ones in Assumptions 1 and 3 into (6) we find that

̸

Optimizing the summand over d ( k, ℓ ) gives the bound in (12). The bound in (13) follows from the same argument using (7) instead of (6).

Remarkably, the total amount of data (measured in observed transitions of X t ) required to achieve a fixed relative accuracy has reduced from as bad as exponential in n for the MC estimator to no worse than n 3 for the short trajectory estimator.

5 Conclusions and future work

In this article we show that the LSTD method can produce relatively accurate estimators of rare event statistics with a data set of observed Markov chain transitions that is dramatically smaller than would be required by the Monte Carlo estimator. In particular, the LSTD estimator can achieve relative accuracy with a number of observed transitions much smaller than typical timescale of the rare event. This contrasts with the classical worst case perturbation bounds, which predict a large error when the typical timescale of the rare event is large.

Generalization of our basic conclusions beyond the tabular setting is a natural goal for future work. Our proof strategy can be used to establish general, entry-wise perturbation bounds for linear systems of the form in (2). A version of those bounds that applies to Markov processes in continuous spaces would have many interesting consequences, including in the analysis of both approximation and estimation error for TD approaches beyond the tabular setting.

We have not considered online TD approaches [Sutton and Barto, 2018]. In the tabular setting considered here, standard online TD corresponds to a variant of classical Richardson iteration in which the residual of (2) is replaced by an independent realization of the residual of (3) at each iteration. Both deterministic Richardson iteration and online TD will converge very slowly when the typical timescale of the rare event is large. This issue has been explored in Strahan et al. [2023b], where the authors suggest a batch version of subspace iteration for online policy evaluation and demonstrate improved convergence. The data requirements of that scheme should be studied theoretically.

When R ( i ) = 0 for all i ∈ D ,

References

Adam Antoszewski, Chatipat Lorpaiboon, John Strahan, and Aaron R. Dinner. Kinetics of phenol escape from the insulin r6 hexamer. The Journal of Physical Chemistry B , 125(42):11637-11649, 2021. doi: 10.1021/acs.jpcb.1c06544. URL https://doi.org/10.1021/acs.jpcb.1c06544 . PMID: 34648712.

Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Armand Prieditis and Stuart Russell, editors, Machine Learning Proceedings 1995 , pages 30-37. Morgan Kaufmann, San Francisco (CA), 1995. ISBN 978-1-55860-377-6. doi: https://doi.org/ 10.1016/B978-1-55860-377-6.50013-X. URL https://www.sciencedirect.com/science/ article/pii/B978155860377650013X .

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research , pages 1691-1692. PMLR, 06-09 Jul 2018. URL https://proceedings.mlr.press/v75/bhandari18a.html .

Steven J. Bradtke and Andrew G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning , 22(1):33-57, Jan 1996. ISSN 1573-0565. doi: 10.1023/A:1018056104778. URL https://doi.org/10.1023/A:1018056104778 .

James Antonio Bucklew. Introduction to Rare Event Simulation . Springer Series in Statistics. Springer New York, New York, New York, 1 edition, 2004. ISBN 978-0-387-20078-1. doi: 10.1007/978-1-4757-4078-3.

Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/ 2019/file/98baeb82b676b662e12a7af8ad9212f6-Paper.pdf .

David Cheikhi and Daniel Russo. On the statistical benefits of temporal difference learning. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Pierre Collett, Servet Martinez, and Jamie San Martin. Quasi-Stationary Distributions . Probability and its Applications. Springer Berlin Heidelberg, October 2012. ISBN 978-3-642-33130-5. URL https://link.springer.com/book/10.1007/978-3-642-33131-2 .

Gal Dalal, Balazs Szorenyi, Gugan Thoppe, and Shie Mannor. Finite sample analyses for td(0) with function approximation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.

Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research , 15(24):809-883, 2014. URL http://jmlr.org/papers/v15/dann14a.html .

Peter Dayan and Terrence J. Sejnowski. Td( λ ) converges with probability 1. Machine Learning , 14: 295-301, 1994. URL https://api.semanticscholar.org/CorpusID:476312 .

James W. Demmel. Applied Numerical Linear Algebra . Society for Industrial and Applied Mathematics, 1997. doi: 10.1137/1.9781611971446. URL https://epubs.siam.org/doi/abs/10. 1137/1.9781611971446 .

Justin Finkel, Robert J Webber, Edwin P Gerber, Dorian S Abbot, and Jonathan Weare. Learning forecasts of rare stratospheric transitions from short simulations. Monthly Weather Review , 149 (11):3647-3669, 2021.

Justin Finkel, Edwin P Gerber, Dorian S Abbot, and Jonathan Weare. Revealing the statistics of extreme events hidden in short weather forecast data. AGU Advances , 4(2):e2023AV000881, 2023a.

Justin Finkel, Robert J Webber, Edwin P Gerber, Dorian S Abbot, and Jonathan Weare. Data-driven transition path analysis yields a statistical understanding of sudden stratospheric warming events in an idealized model. Journal of the Atmospheric Sciences , 80(2):519-534, 2023b.

Steffen Grunewalder, Sepp Hochreiter, and Klaus Obermayer. Optimality of lstd and its relation to mc. In 2007 International Joint Conference on Neural Networks , pages 338-343, 2007. doi: 10.1109/IJCNN.2007.4370979.

Steffen Grünewälder and Klaus Obermayer. The optimal unbiased value estimator and its relation to lstd, td and mc. Machine Learning , 83, 08 2009. doi: 10.1007/s10994-010-5220-9.

Spencer C. Guo, Rong Shen, Benoît Roux, and Aaron R. Dinner. Dynamics of activation in the voltagesensing domain of ciona intestinalis phosphatase ci-vsp. Nature Communications , (15):1408, 2024. doi: 10.1038/s41467-024-45514-6. URL https://doi.org/10.1038/s41467-024-45514-6 .

Brooke E. Husic and Vijay S. Pande. Markov state models: From an art to a science. Journal of the American Chemical Society , 140(7):2386-2396, 2018. doi: 10.1021/jacs.7b12191. URL https://doi.org/10.1021/jacs.7b12191 . PMID: 29323881.

Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems , volume 6. MorganKaufmann, 1993. URL https://proceedings.neurips.cc/paper_files/paper/1993/ file/5807a685d1a9ab3b599035bc566ce2b9-Paper.pdf .

Valérian Jacques-Dumas, René M. van Westen, Freddy Bouchet, and Henk A. Dijkstra. Datadriven methods to estimate the committor function in conceptual ocean models. Nonlinear Processes in Geophysics , 30(2):195-216, 2023. doi: 10.5194/npg-30-195-2023. URL https: //npg.copernicus.org/articles/30/195/2023/ .

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.

Dario Lucente, Joran Rolland, Corentin Herbert, and Freddy Bouchet. Coupling rare event algorithms with data-based learned committor functions using the analogue markov chain, 2021. URL https://arxiv.org/abs/2110.05050 .

Peter J. Schmid. Dynamic mode decomposition and its variants. Annual Review of Fluid Mechanics , 54(Volume 54, 2022):225-254, 2022. ISSN 1545-4479. doi: https://doi.org/ 10.1146/annurev-fluid-030121-015835. URL https://www.annualreviews.org/content/ journals/10.1146/annurev-fluid-030121-015835 .

Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and td learning. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2803-2830. PMLR, 25-28 Jun 2019. URL https://proceedings.mlr.press/v99/srikant19a.html .

John Strahan, Adam Antoszewski, Chatipat Lorpaiboon, Bodhi P. Vani, Jonathan Weare, and Aaron R. Dinner. Long-time-scale predictions from short-trajectory data: A benchmark analysis of the trp-cage miniprotein. Journal of Chemical Theory and Computation , 17(5):2948-2963, May 2021. ISSN 1549-9618. doi: 10.1021/acs.jctc.0c00933. URL https://doi.org/10.1021/acs.jctc. 0c00933 .

John Strahan, Justin Finkel, Aaron R. Dinner, and Jonathan Weare. Predicting rare events using neural networks and short-trajectory data. Journal of Computational Physics , 488: 112152, 2023a. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2023.112152. URL https://www.sciencedirect.com/science/article/pii/S0021999123002474 .

John Strahan, Spencer C. Guo, Chatipat Lorpaiboon, Aaron R. Dinner, and Jonathan Weare. Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction. The Journal of Chemical Physics , 159(1):014110, 07 2023b. ISSN 0021-9606. doi: 10.1063/5.0151309. URL https://doi.org/10.1063/5.0151309 .

Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning , 3 (1):9-44, Aug 1988. ISSN 1573-0565. doi: 10.1007/BF00115009. URL https://doi.org/10. 1007/BF00115009 .

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6.

Erik Thiede, Brian Van Koten, and Jonathan Weare. Sharp entrywise perturbation bounds for markov chains. SIAM Journal on Matrix Analysis and Applications , 36(3):917-941, 2015. doi: 10.1137/140987900. URL https://doi.org/10.1137/140987900 .

Erik H. Thiede, Dimitrios Giannakis, Aaron R. Dinner, and Jonathan Weare. Galerkin approximation of dynamical quantities using trajectory data. The Journal of Chemical Physics , 150(24):244111, June 2019. ISSN 0021-9606, 1089-7690. doi: 10.1063/1.5063730.

John N. Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control , 42(5):674-690, 1997. doi: 10.1109/9. 580874.

A Proof of Lemma 1

Proof. Irreducibiliy and aperiodicity of S D implies that ν is the unique left eigenvector of S D with largest eigenvalue, λ max. Using the fact that the u ( i ) = E i [ T ] solves (2) with R ( i ) = 1 for i ∈ D and R ( i ) = 0 for i / ∈ D , a direct calculation yields E ν [ T ] = (1 -λ τ max ) -1 E ν [ T ∧ τ ] . Therefore

and

B Properties of the Markov chain studied in Section 3

Proof of lower bound of ∥ I -S D ∥ 2 . Let ˆ S D be the restriction of S D to row and column indices in D , i.e. the ( n -2) × ( n -2) submatrix in the middle of S D . ˆ S D is a substochastic matrix that satisfies detailed balance with respect to p in (9) for those entries in D . Therefore, ˆ S D is similar to a real symmetric matrix, and all eigenvalues of ˆ S D are real. Let λ min be the smallest eigenvalue of ˆ S D . We have

We will derive an upper bound of λ min to show that ∥ I -S D ∥ 2 is bounded away from 0 .

Define an ( n -2) × ( n -2) diagonal matrix D p as D p ( i, i ) = p ( i +1) . Note the index adjustment is due to the size difference of D p and p , and similar adjustment will persist in this proof. By the detailed balance property ˆ S D ( i, j ) p ( i +1) = ˆ S D ( j, i ) p ( j +1) for i, j = 1 , . . . , n -2 , the matrix A = D 1 2 p ˆ S D D -1 2 p is a symmetric matrix, and it has the same eigenvalues as ˆ S D .

Now we will give an upper bound of λ min, the smallest eigenvalue of A . By Rayleigh quotient characterization of λ min, any length n -2 nonzero vector v satisfies λ min ≤ v T Av v T v . Specifically, we choose v as

The motivation is that in the limit n →∞ , A will be related to a (weighted) finite difference matrix for the Laplace operator in one dimension, so we would like to use v which is a unit eigenvector corresponding to the smallest eigenvalue of the Laplace operator.

Our remaining work is to calculate v T Av and upper bound it. A is a tridiagonal matrix with A ( i, i ) = P ( i +1 , i +1) , A ( i, i +1) = 1 / [ 2 (√ p i +1 p i +2 + √ p i +2 p i +1 )] , A ( i -1 , i ) = 1 / [ 2 (√ p i p i +1 + √ p i +1 p i )] . By direct calculation, with α = π n -1 we have

According to (9), e -1 ≤ | p ( j ) /p ( i ) | ≤ e for | j -i | = 1 . Wealso have the inequality √ p j p i + √ p i p j ≥ 2 . Applying these bounds and separately summing terms of cosines with positive and negative signs, we

have an upper bound

Using exact formulas of sum of cosines, we have

with a 1 n = ⌊ 3( n -1) 4 ⌋ -⌈ n -1 4 ⌉ +1 , b 1 n = ⌊ 3( n -1) 4 ⌋ + ⌈ n -1 4 ⌉ , a 2 n = ⌊ n -1 4 ⌋ , b 2 n = ⌊ n -1 4 ⌋ +1 , a 3 n = n -1 -⌈ 3( n -1) 4 ⌉ , b 3 n = n -2 + ⌈ 3( n -1) 4 ⌉ , a 4 n = ⌊ 3 n -1 4 ⌋ -⌈ n +1 4 ⌉ +1 , b 4 n = ⌊ 3 n -1 4 ⌋ + ⌈ n +1 4 ⌉ -1 , a 5 n = ⌊ n +1 4 ⌋ -1 , b 5 n = ⌊ n +1 4 ⌋ +1 , a 6 n = n -1 -⌈ 3 n -1 4 ⌉ , b 6 n = n -3 + ⌈ 3 n -1 4 ⌉ . Note that αa 1 n , αa 4 n → π 2 and αa in → π 4 as n → ∞ for all other i 's. Similarly, αb in have corresponding limits. Also, ( n -1) sin α → π . Based on these limits, as n →∞ , the right hand side of the inequality above also approaches a limit. Specifically in the limit,

Therefore, lim inf n →∞ ∥ I -S D ∥ 2 ≥ 1 -lim sup n →∞ λ min ≥ 0 . 36 . This shows that for large n , ∥ I -S D ∥ 2 is bounded away from 0 .

Proof of (10) . We can find an explicit formula for u by solving the linear system

with u (1) = u ( n ) = 0 . This yields

By the symmetry of p ,

We can derive bounds based the last expression. A loose bound uses the fact that the sum is larger than the reciprocal of the smallest probability, and this probability is exponentially small.

More carefully, for any small ε > 0 , there are O ( n ) , cos( 4 π ( i -1) n -1 ) terms that are smaller than -1 + ε , and for these terms, p ( i ) ≤ 1 3 exp( n -1 4 π ( -2 + ε )) . Therefore,

Proof of (11) . This time u ( i ) solves

with u (1) = 0 , u ( n ) = 1 . We find that

A loose bound is

More carefully, for any small ε > 0 , there are O ( n ) , cos ( 4 π ( i -1) n -1 ) terms that are smaller than -1 + ε , resulting in the bound

C Proofs of main results

Proof of (5) in Theorem 1 . Each entry of ˜ S t -S t is a ratio of two random variables, as

We will prove all entries of ˜ S t -S t for all 1 ≤ t ≤ τ satisfy a central limit theorem by a central limit theorem for the (normalized) numerators, and then apply Slutsky's theorem to include the denominators. By an argument similar to the one used to establish Lemma 1 in Thiede et al. [2015], u is a differentiable function of entries of S t . The delta method, therefore, transforms a central limit theorem for these entries into a central limit theorem for u .

Let ˜ M t denote a matrix containing only the (normalized) numerators of ˜ S t -S t , that is, ˜ M t ( k, ℓ ) = 1 M ∑ M j =1 1 { k } ( X j 0 ) ( 1 { ℓ } ( X j t ∧ T j ) -S t ( k, ℓ ) ) . Let ˜ M 1: τ represent a long vector formed by flattening out each matrix ˜ M t with 1 ≤ t ≤ τ in row major order and gluing up in time order. Then, since trajectories are independent, ˜ M 1: τ satisfies a central limit theorem as

For convenience, let Σ 1: τ ˜ M (( t 1 , k 1 , ℓ 1 ) , ( t 2 , k 2 , ℓ 2 )) denote the entry of Σ 1: τ ˜ M that records the covariance

Since 1 { k } ( X 0 ) for different k can not be nonzero at the same time, we immediately obtain

̸

Now we deal with the denominators. Let ˜ µ be the vector containing all the (normalized) denominators, that is, ˜ µ ( k ) = 1 M M k . Then, again because of independence of trajectories, we have a weak law of large numbers as

where we recall that µ is the initial distribution supported on D , and P - → means convergence in probability.

Combining (23), (24), (26), and (27) , by Slutsky's theorem, we obtain a central limit theorem for ˜ S as

where ˜ S 1: τ and S 1: τ are vectors flattened out from ˜ S t and S t for 1 ≤ t ≤ τ with the same order as in ˜ M 1: τ , and

According to (25) and (26),

̸

To transform the central limit theorem (28) to a central limit theorem of ˜ u ( i ) , we point out that u ( i ) is a differentiable function of entries of S t and now we are going to calculate these derivatives. Entries in u outside of D are fixed under any perturbation in S t . To write down an equation satisfied by all entries of u , we rearrange (2) into

where R D c ( i ) = 1 i ∈ D c R ( i ) for i ∈ [ n ] . R D c is a vector whose nonzero entries are those fixed boundary values of u . Because S t is a stochastic matrix, we consider it to be determined by off diagonal entries as

̸

This implies that

̸

̸

For a specific pair ( k, ℓ ) with k ∈ D and ℓ = k , we define S t ( ε ) := S t + ε ( e k e T ℓ -e k e T k ) . Then we define

Differentiating (31) with respect to ε and taking care of (33), we find the derivative with respect to S t ( k, ℓ ) for different k, ℓ, t . The formulas are different for different t . For t = τ , k, ℓ ∈ D , we have

Instead, for k ∈ D,ℓ ∈ D c , taking derivatives leads to

which gives the same formula for ∂u ∂S τ ( k,ℓ ) as in (34) after rearranging using the fact that R D c ( ℓ ) = u ( ℓ ) for ℓ ∈ D c .

Therefore,

For t < τ , k ∈ D and ℓ ∈ [ n ] , taking derivatives in (31) gives

Now, we can use the delta method to transform (28) to a central limit theorem for ˜ u ( i ) for any i ∈ D .

̸

with

The second equality is obtained by plugging in (35), (34) and (30). The third equality uses the fact that

The fourth equality uses the Bellman equation (2).

Now we introduce the key lemma for Theorem 1. The lemma characterizes the continuity of the operator E ℓ [ ∑ T t =0 1 { j } ( X t ) ] in terms of the initial state ℓ . Note that a dot product of this operator with the corresponding reward R ( j ) generates the value function u ( ℓ ) .

We define additional hitting times counting from t = 0 . That is, ˆ T k = min { t ≥ 0 : X t = k } . This differs from T k because ˆ T k includes t = 0 . Similarly, we define ˆ T τ k and more generally ˆ T A and ˆ T τ A for any subset A ⊆ [ n ] as the counterparts of T τ k , T A , and T τ A but counted from t = 0 . But notice that we have defined T and T τ to be counted from t = 0 . In all hitting time definitions, when the set over which the minimum is taken is empty, the hitting time is defined to be + ∞ .

̸

Lemma 2. For i, k ∈ D , ℓ, j ∈ [ n ] , k = ℓ ,

̸

̸

Proof of Lemma 2. The proof will proceed by representing all quantities by probabilities, and then with the help of a complementary Markov chain, we can bound the difference of involved probabilities. We start with two basic identities that hold for any i, k ∈ D ,

and

Note that when j ∈ D c , P j [ T ≤ T j ] = 1 . With (39) and (40), the left hand side of (38) becomes

To make probabilities related to X t in (41) become probabilities related to τ -step transitions, we consider another Markov chain Y τ,j t that follows the transition probability matrix S τ j := ( S j ) τ , where S j is defined as

̸

S j is the same as S except it is stopped once j is hit. Note that when j ∈ D c , S j = S . Define hitting times T τ,j k , T τ,j and ˆ T τ,j k of Y τ,j t similarly as those of Y τ t . If Y τ,j t hits j before k , then T τ,j k = + ∞ , and this exception is similarly defined for other hitting times. Note that T τ,j counts from t = 0 as does T . One key relation between Y τ,j t and X t ∧ T is that

This relation is guaranteed by the fact that Y τ,j t will stay at j if a coupled X t ∧ T hits j at any time within each lengthτ time interval.

̸

Now consider the case where ℓ ∈ D , j = k and j = ℓ , and identify all of D c \ { j } with a single additional index ∆ . Note that whether j ∈ D or j ∈ D c ,

̸

Consider another Markov chain Z t ∈ { ℓ, k, j, ∆ } that records transitions only when Y τ,j t transitions between one of these states. We will use Q to denote the transition probabilities of Z t . For example,

and

In fact, P ℓ [ T τ,j j < T τ,j ∆ ] and P k [ T τ,j j < T τ,j ∆ ] satisfy the linear system

It is convenient to work with the normalized transition probabilities

With this notation, solving (45) reveals the identities

and

from which, after a little more algebra, we find that

This is the key quantity that characterizes the difference between probabilities with initial states k and ℓ . By bounding this, we can finally get relative variance bounds. We observe immediately that

Now we relate these probabilities to Y τ t . We have that P k [ T τ ℓ < T τ k ∧ T τ ] ≤ Q ( k, ℓ ) + Q ( k, j ) , so

This yields

Bringing these inequalities back into (46), with (43) and (44) we have

Now notice that P k [ T τ < T τ k ] ≥ Q ( k, ∆) + P k [ T τ ℓ < T τ k ∧ T τ ] Q ( ℓ, ∆) , and therefore,

Plugging (47) back into (41) we find that

̸

By (40) and the definition of Q τ ( k, ℓ ) in Theorem 1, this is the stated bound in the case where ℓ ∈ D,j = k and j = ℓ .

̸

Now we deal with the exceptional cases. When ℓ ∈ D but j = k or j = ℓ , the same conclusion holds with minor modifications of the intermediate steps. Specifically, take j = ℓ (so that j ∈ D and ∆ = D c ) as an example. Then Z t ∈ { ℓ, k, ∆ } records transitions of Y τ,j t among these three states. With Q denoting its transition probabilities.

and P k [ T τ ℓ < T τ k ∧ T τ ] = Q ( k, ℓ ) , P k [ T j < T ] = Q ( k, ℓ ) . Therefore,

We also note that P k [ T τ < T τ k ] ≥ Q ( k, ∆) . Then the following steps are the same as (47) and (48) in the general case.

Applying (40) to the right hand side, this gives the bound (38).

When ℓ ∈ D c and j = ℓ , P ℓ [ ˆ T j ≤ T ] = 1 . We again use Q to record transitions of Y τ,j t (= Y τ t ) among { k, j, ∆ } . Then,

The last inequality follows from the facts that

We thus prove that the bound (38) holds for any indices

Now we are well prepared to prove (6) in Theorem 1.

Proof of (6) in Theorem 1. We will interpret two main terms in (5) probabilistically. First, for i, k ∈ D ,

As for the expectation following e T i ( I -S τ D ) -1 e k in (5), we note from the Bellman equation (2) that

where the subscript k indicates conditioning on X 0 = k . Since this is a variance, we can decompose it into conditional variance of each step by the law of total variance, as

By the definition of u , E ( ∑ τ -1 t =1 R D ( X t ∧ T ) + u ( X τ ∧ T ) ∣ ∣ ∣ X 0 = k, X 1 ∧ T ) = u ( X 1 ∧ T ) , which helps to simplify the second term. Meanwhile, the first term can be simplified as

̸

We can further condition it on X 2 ∧ T and perform a similar decomposition as in (50), which can be done inductively for X t ∧ T up to t = τ . As a result of this inductive conditioning, we obtain

̸

For the inequality, we use the property of variance that

Now, we further decompose the difference u ( ℓ ) -u ( k ) into components that we have already bounded in Lemma 2, using the fact that R is nonnegative throughout our paper.

Then according to Lemma 2, we can bound the variation in u between states as

Combined with (49) and (51), this yields

As for Corollary 1, the strengthening comes from an intermediate step in the proof of (6).

Proof of Corollary 1. If R ( i ) = 0 for any i ∈ D (for example, when u is the committor function), then

̸

Compared with (51), we see that the same following steps will replace ∑ t S t ( k, ℓ ) in Theorem 1 by S τ for Corollary 1.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We made such statements in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have mentioned that we only deals with the simple case that is LSTD of the tabular setting of a finite state space in Section 1.1. We have also mentioned that the bound is not good for large lag times in Section 3. We discuss in Section 5 that we do not deal with the online setting which is more commonly used in practice for data efficiency.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide assumptions with each theoretical statement. The complete proofs are either in the main text or in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 1.1, we state the linear system for the estimator. In Section 3, we state the detailed setting of the experiments. The readers should be able to reproduce experimental results using the information.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The paper contains experiments, but the main part of codes is just solving some linear systems. We provide all details of experiment settings, and readers should be able to write their own code and reproduce our results. There is no dataset used by this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: For experiments in Section 3, the corresponding settings are stated there.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our theoretical result is asymptotic in nature, so such error bars are not directly related to the goal of this paper. In section 3, along with the theoretical relative asymptotic variance, we also provide plots of relative empirical variance with large sample sizes. This shows an agreement between the reality and the theory, but not to an extent of an actual error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: There are experiments involved in the paper in Section 3, but they are quite light in computational resources. The goal of experiments is to motivate the theoretical assumptions and bounds in Section 4, rather than argue computational feasibility or efficiency. Therefore, we think such information will be a departure from the key contributions of this paper and is better to omit.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have been aware of the NeurIPS Code of Ethics and the research conforms with thtat.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is on theoretical advantages of temporal difference learning. We expect it to have long-term impact on many different applications in machine learning, some of which might have positive effects on society, some negative. However, since the work is currently quite distant from concrete applications, it is difficult to speculate more specifically than that.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out

that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such models or data released in the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the estimation of rare event statistics in discrete-time, discrete-state Markov chains. It shows that when the transitions probabilities of the chain satisfy certain locality assumptions then temporal difference estimators are exponentially more efficient than Monte-Carlo estimators. Concretely, under these assumptions, the variance of the TD estimator scales as $n^3$ in the number $n$ of states of the Markov chain, whereas the variance for MC estimators scales as $e^n$.\n\nThe proofs are accompanied by two numerical examples that provide intuition and illustrate the applicability of the theory to mean first passage time and committor function computation. They also provide an additional validation of the theoretical results.\n\nSTRENGTHS:\nOverall great paper. The setting is sharp and precisely captures the phenomenon to be studied. The paper relies on a key original insight that links locality of transitions and efficiency of TD estimators for rare event estimation.\n\nThe related work is clearly covered, and the paper convincingly distills the key insights from previous empirical and theoretical studies into a set of assumptions and their consequences in the setting of an asymptotic central limit theorem.\n\nThe setting is consistent with standard notions of Markov chain literature. The key ideas are nicely illustrated using examples and pictures. Presentation is sharp overall.\n\nThe clear understanding of why and when TD performs well allow practitioners to make an informed choice on when to use TD for sampling problems.\n\nWEAKNESSES:\nThe problem of rare event estimation is essentially a statistics problem, and the results seem more typical of a probability or statistics journal paper than a machine learning conference.\n\nQUESTIONS:\nNo questions at this point.\n\nLIMITATIONS:\nIn general, the theoretical nature of the paper and clear exposition make most of the limitations appear clearly in the statements of the theorems.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of efficiency comparison between temporal difference (TD) and Monte Carlo (MC) methods for learning value functions (policy evaluation) especially in conditions involving rare events. The authors identify two challenges in policy evaluation with respect to rare events - long timescales of the rare event and need for relative accuracy in estimates of very small values. For their analysis, the authors choose the LSTD algorithm and upper bound its relative asymptotic variance. Using this bound they establish the better efficiency of TD versus MC methods in the presence of rare events. Specifically teh authros define rare events in terms of escape from a given set of states in a Markov reward process.\n\nSTRENGTHS:\nThe paper is mostly clear and the problem selected is an interesting and relevant one.\n\nWEAKNESSES:\nThe authors compare LSTD and MC methods for policy estimation in the presence of rare events, but do not present any numerical experiments to illustrate their claims. It is not clear how the results of this paper can be practically used.\n\nQUESTIONS:\n1. What do the authors mean by \"appropriate choice of reward function\" in the statement in line 49: \"Rare events can be embedded in such a formulation either as “escape from D,” or as “escape from D in an unusual way,” by appropriate choice of the reward function.\"\n2. Is $T$ a terminal time value for the process as given in equation (1)? Is it assumed that the process ends as soon as the trajectory leaves the set $D$?\n3. How does $u(i)$ reflect/determine a rare event? Can't two states resulting in two trajectories - one with a rare event and one without have the same value $u$?\n4. Can the authors elaborate this point in line 68: \"The MC estimator does not allow finite values of $\\tau$ (unless $T$ is bounded by a constant).\". Will this point change in discounted return settings?\n4. $S$ is used in footnote 1, but defined later. Can the authors fix this?\n5. How have the authors addressed the bias-variance tradeoff in TD vs MC methods?\n6. As per the expressions given after line 87, are all states $i \\notin D$ stationary states?\n7. I did not understand the experiments given in Section 3. Don't escape events need to be observed in both TD and MC cases for the algorithm to evaluate the value function accurately? Don't TD and MC only vary in their ability to generalize as TD makes a Markovian assumption, while MC does not? Do the examples mainly illustrate this point?\n8. What errors does the minorizing step in Section 4 introduce?\n\nLIMITATIONS:\nThe authors mention the limitations in their work as points in different sections but do not provide a separate section only discussing these.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper provides a rigorous sample complexity comparison between MC sampling and LSTD method, and shows that for rate event, LSTD can provide relatively accurate estimation with a much smaller dataset compared to MC sampling. This contradicts to the intution from the classical worst perturbation bounds.

STRENGTHS:
- An upper bound on the relative asymptotic variance is provided for LSTD

- This paper considers the policy evaluation for rare event, which is new to the best of my knowledge

WEAKNESSES:
- There is no simulation results for rate event scenario

- The writing can be more polished. Some explanation can be more formal. For example, it would be great if the authors can formalize their discussion in section 1.2.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16