Title: Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Low-rank RL; Entry-wise matrix estimation; Matrix completion
Abstract: We consider the problem of learning an $\varepsilon$-optimal policy in controlled dynamical systems with low-rank latent structure. 
For this problem, we present LoRa-PI (Low-Rank Policy Iteration), a model-free learning algorithm alternating between policy improvement and policy evaluation steps. In the latter, the algorithm estimates the low-rank matrix corresponding to the (state, action) value function of the current policy using the following two-phase procedure. The entries of the matrix are first sampled uniformly at random to estimate, via a spectral method, the *leverage scores* of its rows and columns. These scores are then used to extract a few important rows and columns whose entries are further sampled. The algorithm exploits these new samples to  complete the matrix estimation using a CUR-like method. For this leveraged matrix estimation procedure, we establish entry-wise guarantees that remarkably, do not depend on the coherence of the matrix but only on its spikiness. These guarantees imply that LoRa-PI learns an $\varepsilon$-optimal policy using $\tilde{\cal O}({(S+A)\over \mathrm{poly}(1-\gamma)\varepsilon^2})$ samples where $S$ (resp. $A$) denotes the number of states (resp. actions) and $\gamma$ the discount factor. Our algorithm achieves this order-optimal (in $S$, $A$ and $\varepsilon$) sample complexity under milder conditions than those assumed in previously proposed approaches.

Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation

Abstract

We consider the problem of learning an ε -optimal policy in controlled dynamical systems with low-rank latent structure. For this problem, we present LoRa-PI (Low-Rank Policy Iteration), a model-free learning algorithm alternating between policy improvement and policy evaluation steps. In the latter, the algorithm estimates the low-rank matrix corresponding to the (state, action) value function of the current policy using the following two-phase procedure. The entries of the matrix are first sampled uniformly at random to estimate, via a spectral method, the leverage scores of its rows and columns. These scores are then used to extract a few important rows and columns whose entries are further sampled. The algorithm exploits these new samples to complete the matrix estimation using a CUR-like method. For this leveraged matrix estimation procedure, we establish entry-wise guarantees that remarkably, do not depend on the coherence of the matrix but only on its spikiness. These guarantees imply that LoRa-PI learns an ε -optimal policy using ˜ O ( S + A poly(1 -γ ) ε 2 ) samples where S (resp. A ) denotes the number of states (resp. actions) and γ the discount factor. Our algorithm achieves this order-optimal (in S , A and ε ) sample complexity under milder conditions than those assumed in previously proposed approaches.

1 Introduction

Reinforcement Learning (RL) methods when applied to dynamical systems with large state and action spaces suffer from the curse of dimensionality. For example, learning an ε -optimal policy in tabular discounted Markov Decision Processes (MDPs) with S states and A actions requires a number of samples scaling at least as SA (1 -γ ) 3 ε 2 [17, 36]. Fortunately, many real-world systems exhibit a latent structure that if learnt and exploited could drastically improve the statistical efficiency of RL methods [25, 38]. In this paper, we are interested in developing methods to leverage low-rank latent structures. These structures have attracted a lot of attention recently, see e.g. [22, 11, 15, 28, 16, 45, 39, 2, 29, 40, 32, 35, 37, 34]. Here, we consider a structure where the (state, action) value functions of policies, viewed as S × A matrices, are low-rank. This structure has been empirically motivated and studied in [35, 34, 43, 33]. The hope is that when exploiting it optimally, learning an ε -optimal policy would only require O ( S + A (1 -γ ) 3 ε 2 ) samples. Such an improvement would also imply significant statistical gains in MDPs with continuous state and action spaces. If these spaces are of dimensions d 1 and d 2 , under natural smoothness conditions and using an appropriate discretization [35], the sample complexity would indeed be reduced from 1 ε d 1 + d 2 +2 (without structure) to 1 ε max( d 1 ,d 2 )+2 .

In this paper, we present LoRa-PI (Low Rank Policy Iteration), a model-free algorithm that learns and exploits an initially hidden low-rank structure in MDPs. Unlike existing algorithms, LoRa-PI does not require any prior information on the structure. Yet, the algorithm offers the promised statistical gains: its sample complexity essentially exhibits an order-optimal dependence in S , A and ε (i.e., S + A ε 2 ).

Contributions. Our algorithm LoRa-PI relies on approximate policy iteration [4]. As such, it alternates between policy evaluation and policy improvement steps. The design and performance analysis of these two steps constitute our main contributions.

1. Leveraged matrix estimation with entry-wise guarantees. LoRa-PI sequentially updates a candidate policy whose (state, action) value function has to be estimated. This function can be seen as an S × A matrix that we consider to be low rank. The policy evaluation step then boils down to a novel low-rank matrix estimation procedure. We have two main constraints for this procedure. (i) To be sample efficient, the matrix should be estimated from noisy observations of only a few of its entries. (ii) For RL purposes (when integrated to LoRa-PI ), the procedure should offer entry-wise performance guarantees. We present LME (Leveraged Matrix Estimation), a low-rank matrix estimation algorithm that meets these constraints. LME does not require knowledge of a priori unknown parameters of the matrix (such as its rank, condition number, spikiness, or coherence), and it is the first algorithm enjoying non-vacuous entry-wise guarantees even for coherent matrices.

More precisely, LME guarantees an entry-wise estimation error within ε using only ˜ O ( κ 4 α 2 ( S + A )+ α 2 (1 -γ ) 3 ε 2 ) samples, where α and κ denote the spikiness and the condition number of the matrix, respectively. Note that in particular, this sample complexity does not depend on the coherence of the matrix. Its dependence in S , A and ε cannot be improved. To reach this level of performance, LME relies on an adaptive sampling strategy. It first estimates, via a spectral method, the so-called leverage scores of the matrix. These scores quantify the amount of information about the matrix available in the different rows and columns. The algorithm then exploits the leverage scores to adapt its strategy and in turn, drive the sampling process towards more informative entries.

2. Design and sample complexity of LoRa-PI . Our RL algorithm LoRa-PI is a policy iteration algorithm that relies on LME to perform policy evaluation steps. The algorithm inherits the advantages of LME . In contrast to existing algorithms, it is parameter-free and its performance can be analyzed and guaranteed under mild assumptions on the (state, actions) value functions. In particular, the corresponding low-rank matrices do not need to be incoherent. We establish that LoRa-PI learns an ε -optimal policy using ˜ O ( κ 4 α 2 ( S + A )+ α 2 (1 -γ ) 8 ε 2 ) samples, where α and κ are upper bounds on the spikiness and the condition number of the (state, action) value functions.

3. Numerical experiments. We illustrate numerically the performance of our algorithms, LME and LoRa-PI , using synthetically generated low-rank MDPs. The experiments are presented in Appendix A due to space constraints.

Notation. We denote the Euclidean norm of a vector x by ∥ x ∥ 2 . Let M be an m × n matrix. We we denote its i -th row (resp. j -th column) by M i, : (resp. by M : ,j ). We denote its operator norm by ∥ M ∥ op, it Frobenius norm by ∥ M ∥ F, its infinity norm by ∥ M ∥ ∞ = max i ∈ [ m ] ,j ∈ [ n ] | M ij | , and its two-to-infinity norm by ∥ M ∥ 2 →∞ = max i ∈ m ∥ M i, : ∥ 2 . We denote by M † the Moore-Penrose inverse of M . For given subsets I ∈ [ m ] , J ∈ [ n ] , we denote by M I , J the sub-matrix whose entries are { M ij : ( i, j ) ∈ I × J } . Finally, we use a ∧ b = min( a, b ) and a ∨ b = max( a, b ) .

2 Related Work

Low-rank MDPs. MDPs with low-rank latent structure have been extensively studied recently. We may categorize these studies according to the type of the underlying low-rank structure and to the nature of the algorithms used to learn this structure.

The most studied low-rank structure concerns MDPs whose transition kernels and the expected reward functions are low-rank. For instance, it is assumed that the transition probabilities can be written as p ( s ′ | s, a ) = ϕ ( s, a ) ⊤ µ ( s ′ ) , where ϕ ( s, a ) and µ ( s ′ ) are d -dimensional feature maps [22, 11, 15, 28, 16, 45, 39, 2, 29, 40, 32]. These work additionally assume that the feature map ϕ (and similarly for µ ) belongs to a rich function class H . In this setting, the typical upper bounds derived for the sample complexity of identifying an ε -optimal policy scale as poly ( A, (1 -γ ) -1 ) log |H| ε 2 . When no restrictions are imposed on the class H , one can find a low-rank structure such that log |H| scales as the number S of states [20]. In this case, the aforementioned upper bounds are the same those for MDPs without structure. We also note that most algorithms using this framework rely on strong computational oracles (e.g., empirical risk minimizers, maximum likelihood estimators), see [23, 18, 44] for detailed discussions. In this paper, we do not limit our analysis to low-rank structures based on a given restricted class of functions, and our algorithms do not rely on any kind of oracle.

The low-rank structure we consider is similar to that in [35, 34] and just assumes that the (state, action) value functions are low-rank. Actually, [35] considers the case where only the optimal Q-function is low-rank, say of rank d . As shown in [35], such a structure naturally arises when discretizing smooth MDPs with continuous state and action spaces. In both papers [35, 34], the authors devise algorithms with a minimax-optimal sample complexity to identify an ε -optimal policy roughly scaling as ( S + A ) /ε 2 . But the analysis presented in [35] suffers from the following important limitations. 1. First, it is assumed that the learner is aware of a set I (resp. J ) of so-called anchors states (resp. actions), such that the rank of the matrix Q I , J := ( Q ( s, a )) ( s,a ) ∈I×J is the same as that of the entire matrix Q . Such anchors are however initially unknown (since Q is unknown). Importantly, the proposed RL algorithms rely on a low-rank matrix estimation procedure whose performance strongly depends on the smallest singular value σ d ( Q I , J ) of Q I , J . The authors circumvent this difficulty by actually parametrizing their algorithms using σ d ( Q I , J ) . But again, the latter is unknown, and it remains unclear how one can avoid this issue. 2. The second limitation is that the analysis is valid for small values of the discount factor γ (the authors need to impose an upper bound on γ/σ d ( Q I , J ) ). When σ d ( Q I , J ) is small, the analysis is limited to very short horizons. Note that in addition, [35] assumes that the collected rewards are deterministic, which together with the short horizon issue, greatly simplifies the learning problem.

To address the first limitation, the authors of [34] propose to sample rows and columns uniformly at random to get anchors. This solution requires to sample at least dµ 2 states and actions (Lemma 10 in [34]) where µ is the (unknown) coherence of the matrix to be estimated. Hence this essentially amounts to sampling almost the whole matrix for coherent matrices. The authors of [34] also propose a solution to the second limitation, but at the expense of imposing additional restrictive conditions. In this paper, we address both limitations and devise RL algorithms that rely on a new low-rank matrix estimation procedure that works without imposing the incoherence of the matrix and that does not require knowledge on a priori unknown parameters of this matrix.

Low-rank matrix estimation with entry-wise guarantees. Until recently, most results on low-rank matrix recovery concerned guarantees with respect to the spectral or Frobenius norms, see e.g. [12] and references therein. Over the past few years, methods to derive entry-wise guarantees have been developed. These include spectral approaches [1, 8, 37], nuclear-norm penalization and convex optimization techniques [9], CUR-based (or Nyström-like) methods [35, 3, 34].

The aforementioned literature provides guarantees not for all low-rank matrices, but for those typically enjoying additional structural properties such as incoherence. Relaxing the incoherence assumption is not easy, but can be achieved using adaptive sampling [24, 7, 41]. As far as we are aware, all results applicable to somewhat coherent matrices provide guarantees with respect to the spectral or Frobenius norms. In this paper, we develop a first adaptive matrix estimation method with provable entry-wise guarantees, valid for matrices with well-defined spikiness but not necessarily incoherent. Refer to [26, 30] and to §3.2 for a detailed discussion about the notion of spikiness.

3 Preliminaries

3.1 Low-rank Markov Decision Processes

We consider a discounted MDP with finite state and action spaces S and A . These spaces are of cardinality S and A , respectively. The dynamics are described by the transition kernel p where p ( s ′ | s, a ) denotes the probability to move to state s ′ given current state s and that the action a is selected. The collected rewards are random but bounded by r max , and r ( s, a ) is the expected reward collected when action a is selected in state s . A deterministic Markovian policy π is described by a mapping from S to A . We denote by V π the state value function of π : for all s ∈ S , V π ( s ) = E [ ∑ ∞ t =0 γ t r ( s π t , a π t ) | s π 0 = s ] , where s π t and a π t are, at time t , the state and the action selected under π . Similarly, the (state, action) value function of π is defined by: for all ( s, a ) ∈ S × A , Q π ( s, a ) = r ( s, a ) + γ ∑ s ′ p ( s ′ | s, a ) V π ( s ′ ) . Q π can be seen as a S × A matrix, referred to as the value matrix of π in the remainder of the paper. Let κ π denote the condition number of Q π . Finally, let V ⋆ be the value function of the MDP (the value function of the optimal policy).

The objective is to learn an ε -optimal policy by interacting with the MDP. Such a policy satisfies: for all s ∈ S , V π ( s ) ≥ V ⋆ ( s ) -ε . Without any assumption on the structure of the MDP, to identify such a policy, the learner needs to gather, even with a generative model, a number of samples 1 that scales as SA ε 2 (1 -γ ) 3 [17, 36]. The hope is that exploiting an a-priori known structure in the MDP may considerably accelerate the learning process. In this paper, we focus on a low-rank latent structure. Formally, we define:

Definition 1 (Rank of a policy, rank of the MDP) . The rank d π of a deterministic policy π is the rank of its value matrix Q π . The rank of an MDP is then defined as d = max π d π , where the maximum is over all deterministic policies.

Throughout the paper, we assume that the MDP is low-rank: its rank d satisfies d ≪ ( S + A ) . This assumption is merely made to simplify the exposition of our results and proof techniques. As we shall argue in Appendix E, our findings can naturally be extended to MDPs that are only low-rank in an approximate and well-precised sense.

3.2 Matrix estimation: coherence and spikiness

Our learning algorithm relies on the approximate policy iteration method, and in particular, in each iteration, it needs to estimate the low-rank value matrix of the current policy. To be sample efficient, the algorithm will estimate the matrix from the noisy observations of a few of its entries. Recovering a low-rank matrix from a few of its entries is not always possible (see e.g. [12] for a survey), and conditions on the degree to which information about a single entry is spread out across a matrix must be imposed. Examples of such conditions pertain to the coherence [6, 31] or the spikiness [30] of the matrix.

Matrix coherence. Let Q be a rankd S × A matrix with SVD U Σ W ⊤ . The coherence of Q is defined as µ ( Q ) = max { √ S/d ∥ U ∥ 2 →∞ , √ A/d ∥ W ∥ 2 →∞ } . Q is µ -coherent if µ ( Q ) ≤ µ . Low coherence means that the energy of U and W are not concentrated around a few rows and columns.

Matrix spikiness. The spikiness of Q is defined as α ( Q ) = √ SA ∥ Q ∥ ∞ / ∥ Q ∥ F ∈ [1 , √ SA ] . Q is α -spiky if α ( Q ) ≤ α . A matrix has low spikiness if the amplitude of its maximal entry is not much larger than the average amplitude of its entries, in which case, it is intuitively easier to estimate.

Most existing guarantees for low-rank matrix estimation are expressed through the spectral or Frobenius norm of the error matrix. For this type of guarantees, the estimation error scales polynomially either with the matrix coherence or with its spikiness [12, 30]. The matrix spikiness was introduced in the matrix completion literature [30] to obtain guarantees under less restrictive conditions than the incoherence conditions imposed in previous work. Indeed, there are matrices with bounded spikiness but high coherence (say close to √ S/d , in which case the aforementioned coherence-based guarantees are vacuous). In contrast, bounded incoherence provides an upper bound on spikiness since α ( Q ) = √ SA ∥ Q ∥ ∞ / ∥ Q ∥ F ≤ √ SA ∥ U ∥ 2 →∞ ∥ Q ∥ op ∥ W ∥ 2 →∞ / ∥ Q ∥ F ≤ µ ( Q ) 2 d .

For RL purposes, we need to derive entry-wise guarantees for the estimate of the value matrix of some policy as demonstrated in [35, 37, 21]. Existing upper bounds for the entry-wise estimation error exhibit a strong dependence in the matrix coherence and its condition number, see e.g. [9, 8, 37]. For instance, in [9], this dependence comes as a multiplicative factor µ ( Q ) 2 α ( Q ) 2 κ ( Q ) 2 in the number of samples required for a given level of estimation accuracy. As far as we are aware, our matrix estimation method is the first able to yield entry-wise guarantees that do not exhibit a dependence on the matrix coherence but only on its spikiness (see Table 1). Our algorithm is better by a factor of µ ( Q ) 2 than algorithms based on uniform sampling (studied in [34]), and requires the same sample complexity as the algorithm of [35], which has prior knowledge of anchor states. It remains however unclear whether the dependence of the entry-wise estimation error in the condition number can be avoided. This last observation guides the design of RL algorithms for low-rank MDPs as we discuss next.

3.3 Policy vs. Value Iteration: the condition number issue

We aim at devising an algorithm learning an efficient policy with provable guarantees while imposing conditions on the MDP that are as mild as possible. To this aim, one may think of applying either a policy iteration approach, as we do, or a value iteration approach.

Policy Iteration. Using this approach, in each iteration, we need to estimate the low-rank value matrix of the current candidate policy. As mentioned above, the entry-wise error of this estimation procedure depends on the condition number of the matrix. Note that this matrix belongs to the finite set of (state, action) value functions of deterministic policies. As shown in [10, 42], this set can be seen as the vertices of a simple polytope P . Hence to get performance guarantees when applying a PI approach, it is sufficient to impose an upper bound on the condition numbers κ π for all deterministic policies π , or equivalently, on the condition numbers of matrices corresponding to the vertices of P .

Value Iteration. Here, we would maintain, in iteration t , an estimate V ( t ) of the value function V ⋆ , and samples would be used to compute V ( t +1) , an estimate of T ⋆ ( V ( t ) ) , where T ⋆ denotes Bellman's operator. More precisely, starting from V ( t ) , we would estimate the low-rank matrix Q ( t +1) = F ( V ( t ) ) defined by for all ( s, a ) , F ( V ( t ) )( s, a ) = r ( s, a ) + γ ∑ s ′ p ( s ′ | s, a ) V ( t ) ( s ′ ) . Then we would define V ( t +1) as the value function of the greedy policy with respect to Q ( t +1) . Hence to get provable performance guarantees using a value iteration approach, we would need to impose an upper bound on the condition number of Q ( t ) in all iterations t . The main issue is that the set of matrices { Q ( t ) , t ≥ 1 } is stochastic and hard to predict. Indeed, we have no way of confining the iterates Q ( t +1) to the polytope P : as shown in [10], the polytope is not stable by Bellman's operator. As a consequence, if we wish to get performance guarantees for a value iteration approach, we would need to impose an upper bound on the condition number of all possible matrices of the form F ( V ) for some vector V .

In summary, policy iteration approaches offer a theoretical advantage compared to value iteration. It requires the control of the condition numbers of matrices in a set much smaller than that for value iteration. This advantage is illustrated in Figure 1 on a toy example of an MDP. Refer to Appendix A for additional numerical experiments (with larger MDPs).

4 Leveraged Matrix Estimation

In this section, we present Leveraged Matrix Estimation ( LME ), an algorithm that estimates the value matrix Q π of a policy π . The algorithm relies on an active strategy for sampling the entries of the matrix based on its estimated leverage scores as defined below. This active strategy accelerates the learning process and allows us to obtain entry-wise guarantees that do not depend on the coherence of the matrix but on its spikiness only.

Definition 2 (Leverage scores 2 ) . Let Q be a rankd S × A matrix with SVD U Σ W ⊤ . Its left and right leverage scores ℓ and ρ are defined as ℓ s = ∥ U s, : ∥ 2 2 /d for all s ∈ S , and ρ a = ∥ W a, : ∥ 2 2 /d for all a ∈ A .

LME only takes as inputs a policy π and a sampling budget T . It proceeds in two phases: first, it uses half of the sampling budget to estimate the leverage scores of Q π via singular subspace recovery.

Second, it selects a few anchor rows and columns sampled using the estimated leverage scores, and uses the remaining budget to sample the entries of these rows and columns. It finally completes the matrix estimation using a CUR-based method. The full pseudo-code of LME is presented in Appendix C. Observe that LME is parameter-free: it does not require knowledge of the policy rank d π , nor upper bounds on unknown parameters such as κ π or α ( Q π ) or µ ( Q π ) . Throughout this section, when presenting our guarantees, we will abuse notation and use d , κ and α , instead of d π , κ π and α ( Q π ) .

4.1 Preliminaries

LME exploits a natural empirical estimator of Q π entries at numerous stages. This empirical estimator is essentially based on Monte-Carlo rollouts with truncation as described next. Define the truncated value matrix at a horizon τ as follows: for all ( s, a ) ∈ S × A , Q π τ ( s, a ) = E [∑ ∞ t =0 γ t r t ( s π t , a π t ) 1 { t ≤ τ } ∣ ∣ s π 0 = s, a π 0 = a ] . By choosing τ appropriately, we may control the level of the approximation error Q π τ -Q π . We make this observation precise in the following lemma, proved in Appendix F.1.

In view of the above, to estimate an entry, say ( s, a ) , of Q π , we will use an empirical estimator based on trajectories of length τ +1 of the system under π and starting with (state, action) pair ( s, a ) . In our algorithms, this length is chosen to get an appropriate accuracy level. Specifically, we choose ϵ and τ as follows:

These choices will become apparent from our analysis.

4.2 Phase 1: Leverage scores estimation via spectral subspace recovery

The first phase of LME is devoted to the estimation of the leverage scores of Q π . To this aim, using half of the sampling budget T/ 2 , we estimate the singular subspaces of the matrix via a spectral method.

Phase 1a. Data collection and the empirical truncated value matrix. As suggested in §4.1, to estimate individual entries of Q π , we sample system trajectories of length τ +1 . More precisely, for each of the N := T/ (2( τ +1)) trajectories, we first sample the starting (state, action) pair uniformly at random, and then observe the trajectory obtained under the policy π and initiated at this pair. The data collected this way is D = { ( s π k, 0 , a π k, 0 , r π k, 0 , . . . , s π k,τ , a π k,τ , r π k,τ ) : k ∈ [ N ] } . Using this data, we construct an empirical estimate of the truncated value matrix as follows ∀ ( s, a ) ∈ S × A :

Phase 1b. Singular subspace recovery. We compute the SVD of the empirical truncated value matrix ˜ Q π τ . We obtain ˜ Q π τ = ∑ S ∧ A i =1 ˆ σ i ˆ u i ˆ w ⊤ i , where ˆ σ 1 , . . . , ˆ σ S ∧ A correspond, in decreasing order, to its singular values and ˆ u 1 , . . . , ˆ u S (resp. ˆ w 1 , . . . , ˆ w A ) to its left (resp. right) singular vectors. Using this decomposition, we construct our estimate of Q π as follows:

where β > 0 is a threshold that we will precise shortly. We view ̂ Q π as a biased estimate of Q π with controlled bias through τ . We also use β to estimate the rank of Q π : ̂ d = ∑ S ∧ A i =1 1 { ˆ σ i ≥ β } . Finally, the estimated left (resp. right) singular subspace is denoted ̂ U = [ˆ u 1 · · · ˆ u ˆ d ] ∈ R S × ˆ d (resp. ̂ W = [ ˆ w 1 · · · ˆ w ˆ d ] ∈ R A × ˆ d ). In the following proposition, we provide a choice for the threshold β that yields appropriate guarantees regarding our subspace recovery.

Proposition 1. Let δ ∈ (0 , 1) and choose the threshold β as

Then, provided that 3 :

An analogous result holds for ˆ ρ .

we have that events: ̂ d = d π , and for all s ∈ S ,

hold with probability at least 1 -δ . An analogous result holds for ̂ W .

The precise statement (Theorem 4) and the proof are presented in Appendix B.3 and B.4.

Phase 1c. Leverage Scores Estimation. To conclude, using the recovered subspaces ̂ U and ̂ W , we estimate the leverage scores as follows ˆ ℓ = ∥ ∥ ˜ ℓ ∥ ∥ -1 1 ˜ ℓ and ˆ ρ = ∥ ∥ ˜ ρ ∥ ∥ -1 1 ˜ ρ , where:

The performance of the estimation of the leverage scores is summarized in the following theorem, proved in Appendix B.2.

Theorem 1 (Leverage Scores Estimation) . Let δ ∈ (0 , 1) . Suppose the threshold β is chosen as in (4) . Then, we have that: P ( ∀ s ∈ S , ℓ s ≤ 4 ˆ ℓ s ) ≥ 1 -δ, provided that

4.3 Phase 2: Leveraged CUR-based Matrix Completion

Before we proceed with the description of the second phase, we briefly recall the so-called CUR decomposition [19, 27] for low-rank matrices. The decomposition says that for a given rankd S × A matrix Q , there always exists I ⊆ [ S ] , J ⊆ [ A ] , with |I| = |J | = d , such that the sub-matrix Q I , J is full rank and for all entries ( i, j ) , Q ij = Q i, J ( Q I , J ) † Q I ,j . As in [35, 34, 3], we leverage this decomposition in our matrix estimation procedure, but without any requirement such as knowledge of I , J for which σ d ( Q I , J ) bounded away from zero or upper bounds on parameters like the matrix coherence.

Phase 2a. Data collection to estimate the skeleton of the value matrix. We start by sampling K := 64 d log(64 d/δ ) rows (resp. columns) without replacement according to ˆ ℓ (resp. ˆ ρ ) to form a skeleton of the matrix. These rows and columns are referred to as anchors. We denote the set of selected rows (resp. columns) by I ⊆ S (resp. J ⊆ A ). We use the remaining sample budget T/ 2 to get samples of the entries of Q π in the skeleton. To this aim, we use the procedure described in §4.1, and sample trajectories of length τ +1 . For each entry ( s, a ) ∈ Ω □ := I × J , we use N 1 := T/ (4( τ +1) K 2 ) trajectories to compute ˜ Q π τ ( s, a ) , an empirical estimate of Q π ( s, a ) (see (2)). For each entry ( s, a ) ∈ Ω + := (( S\I ) ×J ) ∪ ( I× ( A\J )) , we use N 2 := T/ (4( τ +1)( K ( S + A ) -2 K 2 ) trajectories. Note that N 2 ≤ N 1 (this plays a role in the analysis).

Phase 2b. CUR-based completion with Inverse Leverage Scores Weighting. First, using the leverage scores, and the set of rows I and columns J , we define K × K diagonal matrices L and R as follows:

Next, starting from the values of ˜ Q π τ ( s, a ) for ( s, a ) in the skeleton, we perform a CUR matrix completion to obtain ̂ Q π : (i) for all ( s, a ) ∈ ( S × J ) ∪ ( I × A ) , we set ̂ Q π ( s, a ) = ˜ Q π τ ( s, a ) ; (ii) for all ( s, a ) ∈ ( S\I ) × ( A\J ) , we set

Note that the use of L and R in (8), referred to as Inverse Leverage Scores Weighting, corresponds to an importance sampling procedure. It allows us to account for the fact that the skeleton has been sampled using the (estimated) leverage scores.

The next theorem summarizes the performance guarantees under LME . Its proof is presented in Appendix C.1.

Theorem 2. Let ε > 0 , δ ∈ (0 , 1) . Given a deterministic policy π , and a sampling budget T , the algorithm LME ensures that P ( ∥ ̂ Q π -Q π ∥ ∞ ≤ ε ) ≥ 1 -δ , provided that ε ≲ ∥ Q π ∥ ∞ and

Theorem 2 states that the sample complexity of LME to obtain entry-wise guarantees does not depend on the coherence µ of Q π but rather on its spikiness α and condition number κ only. Hence LME provides entry-wise guarantees even for coherent matrices. In addition, its sample complexity scales with S , A , γ and ε optimally. Indeed if α, κ = Θ(1) and d ≪ S + A , it scales as ( S + A ) ε 2 (1 -γ ) 3 . We also wish to emphasize that LME is parameter-free, in the sense that it does not require knowledge of the so-called anchor rows and columns, nor does it require upper bounds on unknown parameters such as coherence, spikiness, rank or condition number. These properties are desirable for RL purposes.

5 Low-Rank Policy Iteration

In this section, we present and evaluate LoRa-PI (Low Rank Policy Iteration), a model-free variant of the approximate policy iteration algorithm [4]. It alternates between policy improvement and policy evaluation steps and uses LME , our low rank matrix estimation procedure for policy evaluation. Refer to Algorithm 1 for the pseudo-code.

Algorithm 1: Low-Rank Policy Iteration ( LoRa-PI )

The following theorem provides performance guarantees for LoRa-PI .

We state the results under the assumption that for any deterministic policy π , Q π is α -spiky and has a condition number upper bounded by κ .

The proof of Theorem 3 is presented in Appendix D. Having entry-wise guarantees (as stated in Theorem 2) at each iteration of the algorithm is critical to establish Theorem 3. In fact, the proof starts from the observation (see Lemma 9) that

LoRa-PI combines numerous advantages. (i) It is parameter-free: it does not require the knowledge of upper bounds on parameters such as the ranks, condition numbers, and spikiness of the value matrices of policies. This is thanks to LME , which is itself parameter-free. (ii) Its sample complexity does not depend on the coherence of the value matrices but only on their spikiness; which is an important improvement over existing algorithms [34]. (iii) LoRa-PI offers performance guarantees without having access to good anchor states and actions, without assuming that the rewards are deterministic and that the discount factor is (far too) small, as in [35] (refer to Section 2 for a detailed discussion). (iv) Its sample complexity has an order-optimal scaling in S , A and ε . (v) Finally, since LoRa-PI uses policy iteration, its theoretical guarantees can be established under milder assumptions than if value iteration was used instead (see §3.3).

The dependence of order (1 -γ ) -8 is far from the ideal minimal dependence of order (1 -γ ) -3 that one would typically obtain in RL without low-rank structure. This is an artifact of using a model-free approach, and more specifically the Monte-Carlo estimator of entries of the value matrices. Avoiding such high dependence requires further assumptions and a model-based approach.

Furthermore, it is worth mentioning that the guarantees enjoyed by LoRa-PI can be naturally extended to MDPs that are low-rank only in an approximate sense. We refer the reader to Appendix E for further details.

6 Conclusion

In this work, we considered a class of MDPs where the Q-function, viewed as a state-action matrix, admits a low-rank representation under any deterministic policy. We devised LoRa-PI , a model-free learning algorithm based on approximate policy iteration, that provably exploits such low-rank representation to output a near-optimal policy. Critical to the design and performance guarantee of LoRa-PI is a novel low-rank matrix estimation procedure referred to as LME . LME is shown to enjoy a tight entry-wise guarantee while being parameter-free, i.e., it does not require knowledge of the so-called anchor rows and columns, nor upper bounds on unknown parameters such as spikiness, coherence, rank, or condition number. More importantly, its sample complexity does not scale with

the coherence but instead with the spikiness of the matrix. This allows us to estimate a wider class of low-rank matrices with entry-wise guarantees than previous work. Such desirable properties are what make LME appealing for RL purposes, and in particular what allows us to show that LoRa-PI is sample-efficient under mild conditions. From a design perspective, LME and its analysis features many interesting tools and ideas. Notably, (i) we derived instance-dependent row-wise singular subspace recovery guarantees, and (ii) we combined the use of the so-called leverage scores with a CUR-based approximation for matrix estimation. We believe such tools and ideas to be of independent interest. Finally, we provided experimental results that suggest the superior performance of our proposed algorithms.

Acknowledgment

This research was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, the Swedish Research Council (VR), and Digital Futures. YJ is supported by the Knut and Alice Wallenberg Foundation Postdoctoral Scholarship Program under grant KAW 2022.0366.

References

Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Annals of statistics , 48(3):1452, 2020.

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs. In Advances in Neural Information Processing Systems , volume 33, pages 20095-20107. Curran Associates, Inc., 2020.

Anish Agarwal, Munther Dahleh, Devavrat Shah, and Dennis Shen. Causal matrix completion. In The Thirty Sixth Annual Conference on Learning Theory , pages 3821-3826. PMLR, 2023.

Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

Christos Boutsidis, Michael W Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms , pages 968-977. SIAM, 2009.

Emmanuel J. Candes and Yaniv Plan. Matrix completion with noise. Proceedings of the IEEE , 98(6):925-936, 2010.

Yudong Chen, Srinadh Bhojanapalli, Sujay Sanghavi, and Rachel Ward. Completing any low-rank matrix, provably. The Journal of Machine Learning Research , 16(1):2999-3034, 2015.

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning , 14(5):566-806, 2021.

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization , 30(4):3098-3121, 2020.

Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, and Marc G Bellemare. The value function polytope in reinforcement learning. In International Conference on Machine Learning , pages 1486-1495. PMLR, 2019.

Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On Oracle-Efficient PAC RL with Rich Observations. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Mark A. Davenport and Justin K. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE J. Sel. Top. Signal Process. , 10(4):608-622, 2016.

Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. J. Mach. Learn. Res. , 13(1):3475-3506, dec 2012.

Petros Drineas, Michael W Mahoney, and Shan Muthukrishnan. Relative-error cur matrix decompositions. SIAM Journal on Matrix Analysis and Applications , 30(2):844-881, 2008.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with Rich Observations via Latent State Decoding. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1665-1674. PMLR, 09-15 Jun 2019.

Dylan Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspective. In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 2059-2059. PMLR, 15-19 Aug 2021.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91:325-349, 2013.

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. Learning in Observable POMDPs, without Computationally Intractable Oracles. In Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022.

Sergei A Goreinov, Eugene E Tyrtyshnikov, and Nickolai L Zamarashkin. A theory of pseudoskeleton approximations. Linear algebra and its applications , 261(1-3):1-21, 1997.

Yassir Jedra, Junghyun Lee, Alexandre Proutiere, and Se-Young Yun. Nearly optimal latent state decoding in block mdps. In International Conference on Artificial Intelligence and Statistics , pages 2805-2904. PMLR, 2023.

Yassir Jedra, William Réveillard, Stefan Stojanovic, and Alexandre Proutiere. Low-rank bandits via tight two-to-infinity singular subspace recovery. In Forty-first International Conference on Machine Learning , 2024.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 06-11 Aug 2017.

Daniel Kane, Sihan Liu, Shachar Lovett, and Gaurav Mahajan. Computational-statistical gap in reinforcement learning. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1282-1302. PMLR, 02-05 Jul 2022.

Akshay Krishnamurthy and Aarti Singh. Low-rank matrix and tensor completion via adaptive sampling. Advances in neural information processing systems , 26, 2013.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5639-5650. PMLR, 13-18 Jul 2020.

Lester W Mackey, Ameet Talwalkar, and Michael I Jordan. Distributed matrix completion and robust factorization. J. Mach. Learn. Res. , 16(1):913-960, 2015.

Michael W Mahoney and Petros Drineas. Cur matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences , 106(3):697-702, 2009.

Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6961-6971. PMLR, 13-18 Jul 2020.

Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Modelfree representation learning and exploration in low-rank mdps. Journal of Machine Learning Research , 25(6):1-76, 2024.

Sahand Negahban and Martin J Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research , 13:16651697, 2012.

Benjamin Recht. A simpler approach to matrix completion. Journal of Machine Learning Research , 12(12), 2011.

Tongzheng Ren, Tianjun Zhang, Lisa Lee, Joseph E Gonzalez, Dale Schuurmans, and Bo Dai. Spectral decomposition representation for reinforcement learning. In Proc. of ICLR , 2023.

Sergio Rozada, Santiago Paternain, and Antonio G. Marques. Tensor and matrix low-rank valuefunction approximation in reinforcement learning. IEEE Transactions on Signal Processing , 72:1634-1649, 2024.

Tyler Sam, Yudong Chen, and Christina Lee Yu. Overcoming the long horizon barrier for sample-efficient reinforcement learning with latent low-rank structure. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(2):1-60, 2023.

Devavrat Shah, Dogyoon Song, Zhi Xu, and Yuzhe Yang. Sample efficient reinforcement learning via low-rank matrix estimation. Advances in Neural Information Processing Systems , 33:12092-12103, 2020.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Stefan Stojanovic, Yassir Jedra, and Alexandre Proutiere. Spectral entry-wise matrix estimation for low-rank reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling Representation Learning from Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9870-9879. PMLR, 18-24 Jul 2021.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Modelbased RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches. In Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2898-2933. PMLR, 25-28 Jun 2019.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation Learning for Online and Offline RL in Low-rank MDPs. In International Conference on Learning Representations , 2022.

Yining Wang and Aarti Singh. Provably correct algorithms for matrix column subset selection with selectively sampled data. Journal of Machine Learning Research , 18(156):1-42, 2018.

Yue Wu and Jesús A. De Loera. Geometric policy iteration for markov decision processes. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 2070-2078, New York, NY, USA, 2022. Association for Computing Machinery.

Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi. Harnessing structures for value-based planning and reinforcement learning. In International Conference on Learning Representations , 2020.

Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making Linear MDPs Practical via Contrastive Representation Learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26447-26466. PMLR, 17-23 Jul 2022.

Xuezhou Zhang, Yuda Song, Masatoshi Uehara, Mengdi Wang, Alekh Agarwal, and Wen Sun. Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26517-26547. PMLR, 17-23 Jul 2022.

A Numerical Experiments

All experiments in this section were performed on HP EliteBook 830 G8 with an Intel i7 core and 16 GB of RAM. Each experiment's runtime for individual realizations took at most 2-3 hours, and reproducing all results is feasible within a day.

A.1 Parameters of the toy example in Figure 1

We considered an MDP with S = A = 2 , γ = 0 . 87 , a reward matrix given by

and the following transition probabilities:

We initialized VI with V (0) = [2 . 86 2 . 98] ⊤ . Note that V max = r max 1 -γ = 3 . 69 and thus V (0) ∈ [ -V max , V max ] 2 .

For this example, the condition numbers of the Q-functions induced by policies are 16 . 08 , 4 . 38 , 15 . 29 , 12 . 07 , while the maximum condition number during value iteration is ≈ 2497 . 82 .

We stress here that this MDP is full-rank, and the purpose of this example is to demonstrate the potential instability of VI in the presence of large condition numbers. For low-rank MDPs, this corresponds to the matrix Q π having an effectively smaller rank than expected, and estimating all d singular vectors despite σ 1 ( Q π ) /σ d ( Q π ) →∞ .

A.2 Matrix completion with leveraged anchors

We consider matrix completion with a fixed matrix M ⋆ to be estimated, testing four different methods. First, we test a method based on CUR-approximation with anchors chosen uniformly at random. Next, we have a method based on the estimation of leverage scores, where, for a given budget of samples, we use half of them for estimating leverage scores as described in the main text. Then, we consider a method with oracle anchors, where the anchors are chosen with respect to the true leverage scores. Lastly, we consider standard SVD decomposition, where we keep only the first d largest singular values of the matrix.

25

20

15

10

0

0

0.3

uniform anchors

leveraged anchors

0.25

oracle anchors

SVD

8

0.2

0.15

M

0.1

0.05

0

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

number of samples

×106

0.8

1

1.2

number of samples uniform anchors

leveraged anchors oracle anchors

SVD

1.4

1.6

1.8

2

×106

As expected, CUR-based methods depend heavily on the quality of anchor selection. The gap between leverage-score-based anchors and oracle anchors is slight, even when half the samples are used to estimate the leverage scores. While SVD shows a smaller Frobenius error, it has higher entrywise error compared to CUR-based methods with good anchors.

M*IlF

一

0.2

0.4

0.6

A.3 Leverage scores for VI and PI

We demonstrate the importance of choosing anchors based on leverage scores for value iteration (VI) and policy iteration (PI). We postpone learning of the anchor states to the next subsections and assume that the true leverage scores of matrices ( Q ( t ) ) t ≥ 1 are given. For methods with leveraged anchors, anchors are chosen as those with the highest leverage scores (true leverage scores of Q ( t ) ). For uniform anchors, anchors are chosen uniformly without repetitions.

2.5

oracle anchors

uniform anchors

2.5

oracle anchors

uniform anchors

8

1.5

1.5

一

0.5

0.5

0

5

10

15

20

25

30

35

40

45

50

2

4

6

8

10

12

14

16

18

20

iteration number t

iteration number t

(a) Value iteration with anchors.

(b) Policy iteration with anchors.

These results highlight that leveraged anchors reduce entrywise error significantly for general matrices. In contrast, uniform anchors show significant randomness, although the error decreases in expectation over iterations.

A.4 Low-rank Value Iteration

We evaluate a VI-based variant of Algorithm 1, that we refer to as LoRa-VI . We do not assume prior knowledge of the matrices, and use samples to estimate leverage scores and matrices ( Q ( t ) ) t ≥ 1 .

11

0.16

VI with leveraged anchors

VI with leveraged anchors

10

VIwith uniform anchors

0.14

VIwith uniform anchors

6

full-matrix VI

full-matrix VI

0.12

7

8

0.1

6

一

0.08

4

0.06

3

0.04

2

0.02

1

0

0

0

2

4

6

8

10

12

14

0

2

4

6

8

10

12

14

total number of samples

×107

total number of samples

×107

Even though we did not theoretically analyze the VI-based method in this work, for the reasons mentioned in Section 3.3, we note that this method works well in practice for the settings considered in this study. We consider three methods: VI with leveraged anchors, where we use half of the experiences to estimate leverage scores and based on them sample the second half in a CUR-like fashion. Next, we consider VI with uniform anchors, where anchors are chosen uniformly at random without repetitions. And finally, we consider full-matrix VI, a standard VI approach without any matrix completion steps, where each entry of the matrix gets observed a certain number of times.

We see in Figure 4 that VI with leveraged anchors achieves the best performance measured in Frobenius and entrywise norm. On the other hand, VI with uniform-anchors does not recover specific entries with high values well (as seen in the right figure), but because there are not too many entries with high values, it achieves decent performance in the Frobenius norm. Finally, even though fullmatrix VI can observe all entries of the matrix, it still lags behind VI with leveraged anchors. We also want to remind the reader that VI with leveraged anchors uses only half of the available samples for matrix recovery, while the other half is used for learning the leverage scores.

The algorithm used in the experimental section of [35] closely resembles our LoRa-VI algorithm when uniform anchors are applied. As a result, the numerical results from [35] can be reproduced within our framework, which offers a more general and flexible setting.

A.5 Low-rank Policy Iteration

Finally, we experimentally study performance of the proposed algorithm LoRa-PI .

0.25

PIwith leveraged anchors

PI with leveraged anchors

10

PI with uniform anchors

PI with uniform anchors

6

full-matrix PI

0.2

full-matrix PI

8

7

0.15

0.1

3

2

0.05

0

1

2

3

4

5

6

0

1

2

3

4

5

6

total number of trajectories

×107

×107

total number of trajectories

Similarly as in the previous subsection we study performance of three different methods using PI instead of VI this time, and the observed performance is similar to the one of VI-based methods. In contrast to the other methods, using leverage scores seems to ensure that Frobenius error behaves similarly to entrywise error, up to a scaling factor. This might be caused by uniform dispersion of the estimation error over the entries with large values for PI/VI with leveraged anchors.

The choice of γ = 0 . 1 is governed by an observation that this value of parameter γ ensures that the largest singular values of Q ⋆ are scaling similarly. In other words, it is a heuristic for ensuring small κ needed for CUR-like methods. Furthermore, we believe that performance could be improved if a more tuned way of pseudoinversion is used. Namely, as Q ( t ) is effectively rank-deficient for many epochs, it is crucial to implement a stable way of calculating the pseudoinverse of L ˜ Q ( I , J ) R , and make it dependent on the current epoch and the level of the estimation error.

Lastly, we believe that the performance of the proposed methods can be significantly improved (compared to full-matrix methods) for larger state-action spaces, as well as by implementing a more advanced way of distributing samples across epochs.

B Leverage Scores Estimation Analysis

In this section we provide the proof of Theorem 1. The proof relies on a tight instance dependent row-wise guarantee on the singular subspace recovery which is provided in Theorem 4 together with a proof. Throughout this section and for brevity, we use the notation

in the entirety of this section to denote the number of sampled trajectories T τ , and spikiness-related parameter ¯ α (recall definition of spikiness α from Section 3.2). Furthermore, recall the truncated value matrix Q π τ defined in Section 4.1, and let us define its corresponding approximation error by ∆ = Q π τ -Q π .

B.1 Instance-dependent row-wise singular subspace recovery

Below, we present Theorem 4, which as highlighted before, is crucial in deriving Theorem 1.

Theorem 4. If T τ = ˜ Ω ( ¯ α 2 ( S + A ) ) and ∥ ∆ ∥ op ≤ σ d ( Q π ) / 32 , then we have that the event: for every i ∈ [ S ] , j ∈ [ A ]

holds with probability at least 1 -δ , where we define O ̂ U = ̂ U ⊤ U and O ̂ W = ̂ W ⊤ W .

Corollary 1. If ∥ ∆ ∥ ∞ ≤ min { r max 1 -γ √ d ( S ∧ A ) T τ , σ d ( Q π ) 32 √ SA } and T τ = ˜ Ω ( ¯ α 2 ( S + A ) ) , then w.h.p:

An analogous inequality holds for ∥ W i, : -̂ W i, : ( ̂ W ⊤ W ) ∥ 2 .

It is a simple algebraic exercise to show that ϵ = ∥ ∆ ∥ ∞ from (1) satisfies condition of the corollary above in given regime of T τ .

B.2 Proof of Theorem 1

Proof. First we consider those states with ∥ U s, : ∥ 2 2 > d 4 S . From Corollary 1 we obtain that for these states and large enough T τ :

w.h.p. and for some universal constant c 1 > 0 . Since ∥ U s, : ∥ 2 > √ d 4 S this implies that ∥ U s, : ∥ 2 > √ d 4( S + A ) κ 2 , we can simplify last inequality as follows:

Next, for T τ ≥ 50 c 2 1 ¯ α 2 ( S + A ) κ 2 log 3 ( T τ ( S + A ) δ ) , we have: ∥ U s, : -̂ U s, : ( ̂ U ⊤ U ) ∥ 2 ≤ (1 -1 √ 2 ) ∥ U s, : ∥ 2 . Finally, using reverse triangle inequality we obtain:

and thus:

Now we consider states with ∥ U s, : ∥ 2 2 ≤ d 4 S . Again, by means of Corollary 1 we get w.h.p:

for T τ ≥ 16 c 2 1 ¯ α 2 κ 2 ( S + A ) log 3 ( T τ ( S + A ) δ ) . Thus we obtain that for all s with ∥ U s, : ∥ 2 2 ≤ d 4 S , it also holds:

Since by definition ˜ ℓ s ≥ d S , we obtain that ˜ ℓ s ≥ ∥ U s, : ∥ 2 2 for states with ∥ U s, : ∥ 2 2 ≤ d 4 S .

Finally, we show similar inequalities hold for leverage scores ℓ and ˆ ℓ . Namely, we have:

where we used first part of the proof for the final inequality.

B.3 Proof of Theorem 4

Proof is based on leave-one-out method used for proving entry.wise guarantees for singular vectors of SVD estimates. We refer the interested reader to [8] for a comprehensive survey about the method. Here, we repeat the main arguments of the proof and improve the analysis in the following two ways:

(i) We keep track of approximation error during the whole proof in order to be able to apply it to approximately low rank matrix Q π τ ;

(ii) Instead of showing guarantees in ∥ · ∥ 2 →∞ norm, we prove row/column specific guarantees. Indeed, note that our guarantees do not depend explicitly on the incoherence parameter µ but instead the guarantee specific to the row vector U i, : of the singular subspace U , depends only on its own incoherence parameter, i.e., ∥ U i, : ∥ 2 . This enables us to do leverage score analysis and obtain Theorem 1.

Leave-one-out method is applied to symmetric matrices, so we first redefine our matrices in this context. For a matrix Q π ∈ R S × A with SVD Q π = U Σ W ⊤ define symmetrizated matrix M d as follows:

and similarly define symmetrized matrix M from Q π τ , ∆ from ∆ , ̂ M d from ̂ Q π , and E from E = ˜ Q π τ -Q π τ . Note that, using this notation, we have M = M d + ∆ , with rank d ′ = 2 d matrix M d and ∥ ∆ ∥ ∞ = ∥ ∆ ∥ ∞ , ∥ ∆ ∥ op = ∥ ∆ ∥ op. Assume observation matrix is given by ˜ M = M + E = M d + ∆ + E and note that ̂ M d = ̂ U ̂ Σ ̂ U ⊤ and ̂ M d = ˜ M ̂ U ̂ U ⊤ . Thus, in order to prove lemma, it is sufficient to show:

For now, let us assume that ∥ E ∥ op ≤ σ d ′ ( M ) / 32 , and prove that such inequality holds for T τ large enough (consequence of Proposition 2).

Define ˜ M d = M d + E . Fix any i ∈ [ S + A ] and for any matrix A ∈ R ( S + A ) × n define seminorm: ∥ A ∥ 2 ,i = ∥ A i, : ∥ 2 . Now using that U = M d UΣ -1 = ( ˜ M d -E ) UΣ -1 , we have:

To bound the numerator of the first term, note that ̂ U ( ̂ U ⊤ U ) Σ = ̂ U ̂ U ⊤ M d U = ̂ U ̂ U ⊤ ( ˜ M -∆ -E ) U . Since ̂ U ⊤ ˜ M = ̂ U ⊤ ̂ M d = ̂ Σ ̂ U ⊤ , we get

Consequently, for ( ⋆ ) := ∥ ˜ M d U -̂ U ( ̂ U ⊤ U ) Σ ∥ 2 ,i we have:

Throughout the proof we use Davis-Kahan's inequality (Corollary 2.8 in [8]): if ∥ E ∥ op < (1 -1 / √ 2) σ d ′ ( M ) , then:

Now, similarly to (39) in [37] and using that ∥ AB ∥ 2 ,i ≤ ∥ A ∥ 2 ,i ∥ B ∥ op we obtain:

Using these inequalities together with Lemma 3 and ∥ E ∥ op + ∥ ∆ ∥ op σ d ′ ( M ) ≤ 1 / 16 we obtain:

which in the end gives

Leave-one-out decomposition: Define matrix ˜ M ( i ) as follows:

̸

̸

and let ̂ U ( i ) denote matrix of d ′ dominant singular vectors of ˜ M ( i ) . Then, by triangle inequality we can write:

We bound the last term using Lemma 4 to obtain:

Substituting (10) into (9) we obtain:

Then we apply Proposition 2 on ∥ EU ∥ 2 ,i and ∥ E ( U -̂ U ( i ) ( ̂ U ( i ) ) ⊤ U ) ∥ 2 ,i . Weuse that ∥ U ∥ F ≤ √ d ′ and ∥ U -̂ U ( i ) ( ̂ U ( i ) ) ⊤ U ∥ F ≤ 2 √ d ′ . Finally, we use that ∥ ∆ ∥ 2 ,i ≤ √ S + A ∥ ∆ ∥ ∞ , the fact that σ d ′ ( M ) ≥ σ d ′ ( M d ) -∥ ∆ ∥ op ≥ σ d ′ ( M d )(1 -1 / 32)

and thus ∥ Σ ∥ op σ d ′ ( M ) ≤ 2 ∥ M d ∥ op σ d ′ ( M d ) ≤ 2 κ .

B.4 Rank estimation guarantee

Recall from Section 4.2 and Proposition 1 that, given the singular values (ˆ σ i ) i of our estimates ˜ Q π τ , we estimate effective rank as follows: ˆ d = ∑ S ∧ A i =1 1 { ˆ σ i ≥ β } with

Here we repeat first part of the Proposition 1 and prove it:

Lemma 2. If T satisfies (5) , then estimated rank ˆ d satisfies ˆ d = d π with probability at least 1 -δ .

Proof. By our assumptions σ i ( Q π ) > 0 only for i ∈ [ d π ] , and thus ∀ i > d π : ˆ σ i ≤ ∥ ˜ Q π τ -Q π ∥ op and ∀ i ≤ d π : ˆ σ i ≥ σ d π ( Q π ) -∥ ˜ Q π τ -Q π ∥ op. Recall that E = ˜ Q π τ -Q π τ and ∆ = Q π τ -Q π , and thus:

We bound the second term by ∥ ∆ ∥ op ≤ √ SA ∥ ∆ ∥ ∞ and use that ∥ ∆ ∥ ∞ ≤ r max T from Lemma 1. The first term is upper bounded by Proposition 2. Combining the two, we obtain that ∥ ˜ Q π τ -Q π ∥ op ≤ β with high probability. Thus, for 2 β ≤ σ d π ( Q π ) we are guaranteed to recover the true rank d π , since then ∀ i ∈ [ d π ] :

It is straightforward to check that, if

then first term in definition of β is ≤ σ d π ( Q π ) / 4 , and similar conclusion hold for the second term after noting that r max √ SA σ dπ ( Q π ) ( S + A ) ≥ 1 .

B.5 Technical lemmas from the proof of Theorem 4

In this section we shortly present concentration results used in the proof of Theorem 4. We refer reader to Section F.2 for discussion about equivalent noise model and Poisson approximation. Concentration inequalities proposed are fairly standard (see for example [8]), but as discussed in [37] because of the sampling pattern, entries of the matrix E are slightly dependent. A way to deal with these dependencies has been discussed in [37], and we refer to the results from that paper here directly.

Proposition 2. Let B be a ( S + A ) × 2 d matrix independent of E . Then, we have for all δ ∈ (0 , 1) , for all T τ ≳ ( S + A ) log 3 (( S + A ) /δ ) , both events:

hold with probability at least 1 -δ .

Proof. Follows straightforwardly from proofs of Propositions 26 and 27 in [37] and using noise equivalent model from Section F.2. Note that we keep the variance term upper bounded by ∥ A ∥ 2 F in the proof of Proposition 27 as in [37], and make use of inequality ∥ B ∥ 2 →∞ ≤ ∥ B ∥ F to obtain dependence on ∥ B ∥ F in the second inequality.

Lemma 3. If ∥ E ∥ ≤ σ d ′ ( M ) / 32 , then for every i :

Proof. Under condition ∥ E ∥ ≤ σ d ′ ( M ) / 32 we have σ d ′ ( ˜ M ) ≥ σ d ′ ( M ) -∥ E ∥ ≥ σ d ′ ( M ) / 2 . Thus, we have:

implying that ∥ ˜ M ̂ U ∥ 2 ,i ≤ 2 ∥ ˜ M ̂ U ̂ U ⊤ U ∥ 2 ,i . Furthermore, we have:

where we have used that M U = UΣ . Substituting this expression back into (11) finishes the proof.

Lemma 4. Under assumptions ∥ ∆ ∥ op ≤ σ d ′ ( M ) / 32 and ∥ E ∥ op ≤ σ d ′ ( M ) / 32 , we have with high probability for every i :

Proof. Proof follows similar steps as Step 2.2 in the proof of Theorem 4.2 in [8], but we repeat it here for the sake of completeness and focus on differences in the proof caused by ∆ matrix. First, we use that U is an orthogonal matrix ( ∥ U ∥ op = 1 ) and Davis-Kahan's inequality to obtain:

Note that under the assumption ∥ ∆ ∥ op ≤ σ d ′ ( M ) analysis of singular values stays the same as in [8], since, for example:

Thus, we can lower bound denominator in the inequality above by σ d ′ ( M ) / 2 . Now, term in the numerator can be written as:

and bounded in the same way as in [8]:

where we used that ∥ (( ̂ U ( i ) ) ⊤ U ) -1 ∥ 2 ≤ 2 under our assumptions. Finally, we obtain:

and under condition ∥ E ∥ op ≤ σ d ′ ( M ) / 32 we obtain result claimed in the lemma.

B.6 Nuclear norm minimization for leverage scores estimation

The authors of [34] leveraged the guarantees for nuclear norm minimization from [9] to learn Q matrices. However, several factors make the application of nuclear norm minimization theoretically challenging in our context:

Approximate low-rank structure. As our algorithm is based on policy iteration, our estimates Q π τ are only approximately low-rank. The result from [9] rely on non-convex optimization, leaving it unclear how this approximation error affects the final guarantees of the algorithm. In contrast, a more straightforward analysis using singular value decomposition allows us to explicitly express our bounds in terms of the approximation error.

Coherence-free subspace recovery. In our subspace recovery result (Theorem 4), we can bound the subspace error ∥ U i, : -̂ U i, : O ̂ U ∥ 2 in relation to ∥ U i, : ∥ 2 . It is uncertain whether current guarantees for nuclear norm minimization can achieve a similar outcome, which might instead depend on max i ∈ [ S ] ∥ U i, : ∥ 2 . We believe this would introduce dependency on the incoherence constant into the sample complexity of our algorithm.

C Leveraged Matrix Estimation Analysis

In this appendix, we provide the proof of Theorem 2 corresponding to sample complexity guarantee enjoyed by LME . First, we provide the pseudo-code of LME :

Algorithm 2: Leverage Matrix Estimation ( LME )

Input: Deterministic policy π , sampling budget T

(Phase 1). Leverage Scores Estimation:

(Phase 1a.) Data Collection & Emprirical Truncated Value matrix.

Sample uniformly at random T 1 / ( τ +1) trajectories of length τ +1 using policy π and gather them in D

Use the collected dataset D , to construct ˜ Q π τ as in (2)

(Phase 1b.) Singular Subspace Recovery

Set the threshold β as in (4)

Compute the SVD of ˜ Q π τ and threshold with β as described in (3) to obtain ̂ d , ̂ U , ̂ W and ̂ Q π (Phase 1c.) Leverage Scores.

Set the left (resp. right) leverage scores ˆ ℓ ( resp. ˆ ρ ) as described in (6).

(Phase 2.) CUR-based Matrix Estimation with Leverage.

(Phase 2a.) Data Collection with Leverage & Empirical Truncated Value Matrix:

Set K ← 64 ˆ d log(64 ˆ d/δ )

Sample K rows (resp. columns) I ⊂ S (resp. J ⊂ A ) without replacement according to the ˆ

leverage scores ℓ (resp. ˆ ρ )

For all ( s, a ) ∈ Ω □ (resp. ( s, a ) ∈ Ω + ) sample N 1 (resp. N 2 ) trajectories of length τ +1 using policy π and construct the empirical estimate ˜ Q π τ ( s, a ) based on these trajectories

(Phase 2b. CUR-based Matrix estimation)

Set the matrices L and R as in (7)

Construct ̂ Q π using a CUR-based approach as in (8)

Output: ̂ Q π .

C.1 Proof of Theorem 2

Before showing the proof of Theorem 2 we present two intermediate results used in the proof. As an immediate consequence of Hoeffding's inequality we have:

Lemma 5. With probability at least 1 -δ we have ∀ ( s, a ) ∈ ( I × A ) ∪ ( S × J ) :

where N = N 1 if ( s, a ) ∈ Ω □ or N = N 2 if ( s, a ) ∈ Ω + .

Theorem 5. Let I and J be such that |I| , |J | = K , and Q π ( I , J ) has rank d . Assume that for some ε □ , ε + > 0 :

If ε □ ≤ 1 8 c I c J σ d ( Q π ) √ SA log -2 ( S + A δ ) , ε + ≤ ∥ Q π ∥ ∞ and K ≥ 64 d log(64 d/δ ) , then with probability ≥ 1 -δ :

Proof of Theorem 5 is deferred to C.2.

Proof of Theorem 2. First, by Theorem 1 we require at least

number of samples to recover leverage scores of Q π . During the whole proof of Theorem 2 we condition on the event where Theorem 1 holds.

Recall that we use N 1 = T 4( τ +1) K 2 , N 2 = T 4( τ +1)( K ( S + A ) -2 K 2 ) and define the following quantities:

Note that by definitions of N 1 and N 2 we have that ε □ = ε + √ K S + A -2 K . Next, recall that ∥ Q π τ -Q π ∥ ∞ is upper-bounded by r max T from (1). Combining this with Lemma 5 and our definition of ε □ , ε + we can see that the conditions a ) and b ) of Theorem 5 are met.

Hence, by Theorem 5 we obtain that ∥ ̂ Q π -Q π ∥ ∞ ≤ ε if:

Setting ε + to be equal to this value we obtain that is sufficient to have:

Using definition of N 2 and rewriting inequality above in terms of T gives the following condition:

Combining this with condition (12) and using the fact that ∥ Q π ∥ 2 ∞ SA σ 2 d ( Q π ) ≤ κ 2 α 2 d gives the final condition:

Finally we verify that ε □ and ε + satisfy conditions from Theorem 5. Note that ∥ Q π ∥ ∞ √ SA σ d ( Q ) ≥ ∥ Q π ∥ ∞ √ SA ∥ Q π ∥ F ≥ 1 , as well as log 2 ( S + A δ ) ≥ 1 for any δ ∈ (0 , 1) . Thus we obtain that ε + ≲ ε and thus, in order to have ε + ≲ ∥ Q π ∥ ∞ it is sufficient to assume that ε ≲ ∥ Q π ∥ ∞ . Using the same reasoning we have:

whenever ε ≲ ∥ Q π ∥ ∞ and S + A ≳ K .

C.2 Proof of Theorem 5

Note that ( L ˜ Q π τ ( I , J ) R ) † = R † ( ˜ Q π τ ( I , J )) † L † in general, and thus our estimation is different from ˜ Q π τ ( s, J )( ˜ Q π τ ( I , J )) † ˜ Q π τ ( I , a ) used in [35]. However, weighting estimates by inverse leverage scores as proposed in Section 4.3 still provides unbiased estimates, in the following sense:

̸

Lemma 6. Assume that |I| , |J | = K and rank( Q π ) = d . Then:

Proof of Theorem 5. The proof follows from the proof of Proposition 13 of [35], to which we refer the reader for a more detailed exposition. Based on Lemma 6 and following the proof of Proposition 13 in [35] (see (22) and (23) in [35]), we have ∀ ( s, a ) ∈ S × A and ( ∗ ) := | ̂ Q π ( s, a ) -Q π ( s, a ) | :

We will repeatedly use result from Lemma 8 and condition on the event when given bounds on ∥ L ∥ op and ∥ R ∥ op hold. We begin by bounding the first term in (13). Using the assumption that ∀ ( s, a ) ∈ I × J : | ˜ Q π τ ( s, a ) -Q π ( s, a ) | ≤ ε □ we obtain:

Combining this inequality with our assumption on ε □ and Corollary 2 with η = 1 / 4 gives:

Second term in (13) can be bounded as follows:

and then use that:

Combining this result with Lemma 8 we get:

Similarly to the proof of Proposition 13 in [35], we bound the third term from 13 using inequality ∥ B † -A † ∥ op ≤ 1+ √ 5 2 min {∥ A † ∥ 2 op , ∥ B † ∥ 2 op }∥ B -A ∥ op as follows:

And the last term from (13) can be bounded as follows:

where we used that ∥ Q π ( I , a ) Q π ( s, J ) ∥ F ≤ K ∥ Q π ∥ 2 ∞ .

Combining all derived inequalities we obtain:

Proof of Lemma 6. First, define matrices D U , D W ∈ R K × d by D U = LU I , : and D W = RW J , : , and note that D U and D W are not orthogonal. However, we claim and prove in the end of this proof that:

Moreover, since D U and D W have full column rank, we have that D † U D U = I d × d and D ⊤ W ( D ⊤ W ) † = I d × d . Thus we have ∀ ( s, a ) ∈ S × A :

Now we proceed with proving (14) following similar argument as in Lemma 1 in [14]. Let SVD of D U and D W be given by D U = U D U Σ D U W ⊤ D U and D U = U D W Σ D W W ⊤ D W . First, we use that U D U and U D W are orthogonal matrices to get:

Since D U and D W are matrices with full column rank, all matrices inside of the pseudoinverse are of size d × d and full rank. Thus, their product is as well full rank and replacing pseudoinverse by inverse we obtain:

C.3 Concentration results for the proof of Theorem 5

Lemma 7. Assume that p is a probability measure on S such that p i ≥ η ∥ U i, : ∥ 2 2 d for all i ∈ [ S ] and some η ∈ [0 , 1] . Let I be a set obtained by sampling K entries of S according to p i.e. for any i ∈ [ S ] : i ∈ I with probability min { 1 , Kp i } . Define diagonal matrix L with entries 1 min { 1 , √ Kp i } for i ∈ I and matrix D U ∈ R K × d given by D U = LU I , : . Then, for any δ ∈ (0 , 1) :

holds with probability at least 1 -δ whenever K ≥ 4 d 9 η log(2 d/δ ) .

Proof. First we argue that case p i > 1 K is simple. Denote by S + states for which p i ≤ 1 K . Then, we have:

where Z ( i ) are obtained from U by zeroing all rows except i -th, and δ i 's are i.i.d. Bernoulli( Kp i ) for i ∈ S + . Now we can rewrite the first term:

and take expectation over δ i 's to get E [ ∑ i ∈S + δ 2 i ( Z ( i ) ) ⊤ Z ( i ) L 2 i,i ] = ∑ i ∈S + ( Z ( i ) ) ⊤ Z ( i ) .

Now, define X ( i ) = ( δ i -Kp i ) 1 Kp i U ⊤ i, : U i, : for i ∈ S + . Note that:

by our assumption on p . Moreover, using that Var( δ i ) = Kp i (1 -Kp i ) ≤ Kp i we have:

implying that ∥ E [ ∑ i ∈ [ S ] X ( i ) ( X ( i ) ) ⊤ ] ∥ op ≤ d Kη . Finally, noting that X ( i ) are symmetric matrices ∀ i , we apply matrix Bernstein inequality to obtain:

Setting right hand side equal to δ finishes the proof.

Corollary 2. If anchor states of size at least K ≥ 16 d η log(4 d/δ ) are chosen according to Lemma 7, we have with probability ≥ 1 -δ :

Note that we could use inequality above since D U and D W have full column rank.

Lemma 8. Consider setting of Lemma 7. Then there exist universal constants c I , c J > 0 such that with probability at least 1 -δ :

Proof. We note that if L i,i = 1 (i.e. Kp i ≥ 1 ), then obviously the inequality above holds for S ≥ K , and thus we consider only cases where L i,i = 1 √ Kp i . Now, note that ∥ L ∥ op = ∥ L ext ∥ op, where L ext = ∑ S i =1 δ i 1 √ Kp i e i e ⊤ i , and where δ i are i.i.d. Bernoulli( Kp i ) . Next, we have: E [ δ i 1 √ Kp i e i e ⊤ i ] = √ Kp i e i e ⊤ i , and thus:

Define Y ( i ) = ( δ i -Kp i ) 1 √ Kp i e i e ⊤ i . We have E [ Y ( i ) ( Y ( i ) ) ⊤ ] = (1 -Kp i ) e i e ⊤ i , and hence the variance term in matrix Bernstein is upper bounded by 1 . Lastly by our assumption on p we have for all i ∈ [ S ] :

By matrix Bernstein we obtain:

Equating last term with δ and using that S, A ≫ d we obtain statement of the lemma.

D Sample Complexity Analysis of LoRa-PI

In this appendix, we present the proof of Theorem 3. It is a direct consequence of the performance guarantee of LME (see Theorem 2) and an error bound on approximate policy iteration, which we provide in this appendix (see Lemma 9).

D.1 Proof of Theorem 3

Proof of Theorem 3. To start with, we first observe that, according to Lemma 9, LoRa-PI outputs ˆ π with ∥ V ⋆ -V ˆ π ∥ ∞ ≤ ε , if it holds that

where we introduce the notation Q ( t ) := Q π ( t ) as a shorthand. Now, we note that condition (i) is satisfied if

which is already as chosen in LoRa-PI . Now, in order for ( ii ) to hold we use Theorem 2. We define the events:

We show that ∩ t ∈ [ N epochs ] E t holds with high probability. To that end, it is sufficient to analyse for each t ∈ [ N epochs ] , the event E c t conditionally on the event that ( ∩ k ∈ [ t -1] E k ) holds. Indeed, by using the elementary inequality P ( E c ∪ B c ) ≤ P ( E c |B ) + P ( B c ) in a recursive manner, we can write

We will show that for all t ∈ [ N epochs ] , P ( E c t | ∩ k ∈ [ t -1] E k ) ≤ δ/N epochs, which would entail that P (( ∩ t ∈ [ N epochs ] E t ) c ) ≤ δ and ensure that ∥ V ⋆ -V ˆ π ∥ ∞ ≤ 1 -ε holds with probability at least 1 -δ . Let t ∈ [ N epochs ] . Note that by using Theorem 2, we can immediately show that P ( E c t | ∩ k ∈ [ t -1] E k ) ≤ δ/N epochs provided that

which entails, by definition of N epochs as chosen in LoRa-PI , an equivalent sample complexity to

where we emphasize that ˜ Ω( · ) may hide poly-log dependencies on S , A , (1 -γ ) -1 , d , κ , α , r max , log( e/ε ) , log( e/δ ) . This the desired sample complexity in Theorem 3.

Note that Theorem 2 also requires that

We show that this is satisfied by the condition ε ≲ ε . First, we show that under this condition, we have ∥ Q (1) ∥ ≤ 2 ∥ Q ( t ) ∥ ∞ . Using Lemma 10, and conditionally on the event ∩ k ∈ [ t ] E k holding, we have that for all k < t ,

where ϵ = (1 -γ ) 2 ε 4 , implying, in particular, that

Summing the above inequalities from 1 to t -1 , together with the fact that t -1 ≤ N epochs, gives

In view of this inequality, we note that 2 ∥ Q ( t ) ∥ ∞ ≥ ∥ Q (1) ∥ ∞ , if

We can verify that the above condition is implied by:

where we used the elementary fact x ≥ 2 a log(2 a ) + 2 b = ⇒ x ≥ a log( x ) + b for all a, b > 0 .

Thus, from (17) we conclude that the condition on ε , (16), is satisfied if the following condition holds:

This is the desired condition on ε in Theorem 3. With this we have concluded the proof.

D.2 Error Bound for Approximate Policy Iteration

The following result, a standard variant of Proposition 6.2 in [4], shows that the described approximate policy iteration is guaranteed to converge within an ϵ -accuracy.

Lemma 9. Let ( π ( t ) ) t ≥ 1 be a sequence of deterministic policies selected recursively as described in LoRa-PI , and denote V ( t ) = V π ( t ) for all t ≥ 1 . Let ϵ > 0 and suppose that for all t ≥ 1 , it holds that

Then, for all t ≥ 1 , we have

The proof of Lemma 9 follows standard arguments, but we provide it for completeness.

Lemma 10. Let π be a deterministic policy, and assume that ∥ ̂ Q π -Q π ∥ ∞ ≤ ϵ . Assume that policy π ′ is selected greedily with respect to ̂ Q π , i.e., for all s ∈ S , π ′ ( s ) = arg max a ∈A ̂ Q π ( s, a ) , then

Proof of Lemma 10. Before we proceed with the proof, let us define the composition of a deterministic policy π ′′ and a Q π function, π ′′ ◦ Q π ( s ) := Q π ( s, π ′′ ( s )) . We know that

where ≤ is applied component-wise. Next, we have

where T π is the Bellman policy evaluation operator. By monotonicity of the operator T π , we can re-iterate

Thus, we finally obtain

Taking k →∞ , we get

Proof of Lemma 9. We start by noting that, thanks to Lemma 10, we have: for all t ≥ 1 ,

where ≥ is applied component-wise. Thus, applying this inequality recursively we obtain

which gives at the end

Thus, we have

Thus, using the contraction property of T ⋆ , and that 1 -γ t ≤ 1 , we have ∥ V ⋆ -V ( t +1) ∥ ∞ ≤ ∥ ( T ⋆ ) t ( V ⋆ ) -( T ⋆ ) t ( V (1) ) ∥ ∞ + 2 ϵ (1 -γ ) 2 ≤ γ t ∥ V ⋆ -V (1) ∥ ∞ + 2 ϵ (1 -γ ) 2 .

E Extension of Guarantees to Approximately-Low Rank MDPs

We consider the setting where the matrix Q π is approximately low rank. Specifically, we define a constant ζ d such that ζ d = ∥ Q π ( s, a ) -Q π d ( s, a ) ∥ ∞ , where Q π d is the best d -rank approximation of Q π in the operator norm. Note that ζ d ≤ σ d +1 ( Q π ) ≤ √ SAζ d . In contrast to Theorem 4, where the additional perturbation term ∆ arises from a controllable quantity (through roll-out length τ ), here we assume that ζ d is fixed in advance and unknown. For simplicity, we omit terms stemming from the ∆ perturbation, but the results still hold in that setting. Here, we show that if:

we can obtain similar guarantees for ∥ V ⋆ -V ˆ π ∥ ∞ as in Theorem 3 even in the approximate low rank setting, with an additive error scaling with ˜ O ( 1 1 -γ ζ d dκ 2 α 2 ) . Next, we show that our three main theorems still hold in this setting.

Theorem 1: Leverage scores estimation. We can repeat the arguments from the proof of Theorem 4 to obtain, with high probability, ∀ s ∈ [ S ] :

if T τ = ˜ Ω ( ¯ α 2 ( S + A ) ) , and σ d +1 ( Q π ) ≤ σ d ( Q π ) / 64 . New terms are highlighted in blue in the inequality above. A similar inequality holds for the rows of the matrix of right singular vectors W .

Under Assumption A + and using that σ d +1 ( Q π ) ≤ √ SAζ d , we have:

indicating that the contributions of the two newly added terms are negligible for leverage score estimation and that Theorem 1 still holds in this setting.

Theorem 2: Complete matrix estimation. Theorem 5 holds with the same arguments. Instead of Lemma 5, we have that with high probability: ∀ ( s, a ) ∈ ( I × A ) ∪ ( S × J ) :

Note that our conditions on ζ d and σ d +1 ( Q π ) ensure that the conditions on ε □ and ε + in Theorem 5 ( ε □ ≲ σ d ( Q π ) √ SA log -2 ( S + A δ ) , ε + ≲ ∥ Q π ∥ ∞ ) still hold, as:

Then, the upper bound on ∥ ̂ Q π -Q π ∥ ∞ from Theorem 5 will include an additive term: ζ d SA ∥ Q π ∥ 2 ∞ σ 2 d ( Q π ) log 4 ( S + A δ ) = ˜ O ( ζ d dκ 2 α 2 ) . Finally, under approximate low-rank structure, Theorem

This aligns with Theorem 14 in [34], where the approximation error scales by terms corresponding to SA ∥ Q π ∥ 2 ∞ σ 2 d ( Q π ) in our setting, as both methods use CUR-like matrix recovery.

Theorem 3: Guarantee for LoRa-PI . Based on the approximate policy iteration theorem, which claims:

we observe that the error from approximate low rank propagates through the second term, yielding an additive error of magnitude 1 1 -γ ζ d dκ 2 α 2 to the error of Theorem 3.

F Miscellaneous Results

In this section, we provide some of the observations and results about the truncated value matrix. More specifically, we present the proof to Lemma 1, and a discussion on the variance proxy of the truncated discounted sum of rewards.

F.1 Truncated Value Matrix

Proof of Lemma 1. We know that Q π satisfies the following identity: for all ( s, a ) ∈ S × A ,

Furthermore, note that

and thus

where we used that γ ≤ exp( γ -1) for γ ∈ (0 , 1) . Setting the right hand side of the last inequality equal to ϵ we obtain statement of the lemma.

F.2 Equivalent Noise Model

Recall definition of ˜ Q π τ from (2):

Consider one of N sampled trajectories with index k starting from ( s π k, 0 , a π k, 0 ) = ( s, a ) , and note that

Moreover, since Q π τ is given by:

we have

In other words, each term inside of the outer loop in definition of ˜ Q π τ is uniformly bounded and equal to Q π τ ( s, a ) 1 { ( s π k, 0 ,a π k, 0 )=( s,a ) } in expectation. Thus we can view estimate ˜ Q π τ ( s, a ) equivalently as:

where ξ s,a,k are i.i.d. across k , and | Q π τ ( s, a ) + ξ s,a,k | ≤ r max 1 -γ , implying that ξ s,a,k are 2 r max 1 -γ -subgaussian random variables.

Next note that the number of times N ( s, a ) = ∑ N k =1 1 { ( s π k, 0 , a π k, 0 ) = ( s, a ) } that we sample entries are random variables with multinomial distribution, since P (( s π k, 0 , a π k, 0 ) = ( s, a )) = 1 SA and ∑ ( s,a ) N ( s, a ) = N . This weak dependence between the entries can be dealt with using the Poisson approximation argument (see Section C.2 in [37]). Essentially, this enables us to rewrite matrix ˜ Q π τ as a matrix with i.i.d. entries. Namely, we have for all ( s, a ) :

where Y ( s, a ) are i.i.d. Poisson random variables with parameter E [ Y ( s, a )] = N SA . The fact that the two noise models are equivalent is depicted in Lemma 20 in [37] claiming that probability of an event under the multinomial model can be upper bounded by √ T times probability of the same event under the Poisson model. Practically, this adds a multiplicative factor of T in our probabilistic claims. For more thorough exposition of this issue check Section C.2 in [37].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: To the best of our ability, we believe this to be the case as can be seen in our manuscript. Please refer to our theoretical results throughout our manuscript and Appendix A for the numerical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Our work is primarily theoretical, and we have discussed our results and assumptions whenever presented in our manuscript.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the statements of our theoretical result and corresponding proofs in the appendix. To the best of our ability, we believe the proofs to be correct.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Appendix A and provided code in the supplementary material. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to Appendix A and provided code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Appendix A and provided code in the supplementary material. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to Appendix A and see provided code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our ability, we believe this to be the case.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work that is primarily theoretical and whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a leveraged matrix estimation method for low-rank policy evaluation and extends it to policy iteration as a model-free learning algorithm. The main idea is to separate the Q matrix estimation into two phases. The first step is to use half of the sample budget to estimate the leverage scores of the Q matrix. Based on the leverage scores, a skeleton of the value matrix is sampled. Then roughly one-fourth of the sample budget is used to sample the entries in the skeleton. The left sample budget is used to sample the entries outside the skeleton, and the final Q-value matrix is constructed by CUR decomposition with inverse leverage scores weighting. It's proved that the leveraged matrix estimation method relaxed eliminates the commonly used assumption of incoherence., and guarantees an entry-wise estimation error. The effectiveness of the proposed method is demonstrated via simulation in the Appendix.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The theoretical investigation comes with rigorous proof. By adopting leveraged matrix estimation, the authors can relax the incoherence assumption of the low-rank Q matrix, which serves as a major advantage of the proposed method.\n\nWEAKNESSES:\nIt seems there is a lack of discussion about leveraged matrix estimation methods in matrix completion/low-rank matrix estimation literature. How does the proposed method perform when compared with other regularized policy evaluation approaches (for example, nuclear-norm/max-norm regularization)? It is crucial in assessing the significance and novelty of this work.\n\nIs that possible to include experiments in real-world datasets to enhance the significance of this work?\n\nQUESTIONS:\nIs the low-rank assumption of the Q function under any deterministic policy very restrictive? How does that compare to related works?\n\nIt would be better to replace shadows in the figures (experiments) with error bars, as shallows overlay each other and make it hard to read.\n\nLIMITATIONS:\nThe paper is more in a theoretical flavor and lacks real-world experiments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the reinforcement learning problem with low-rank latent structure. The objective of this problem is of learning an $\\epsilon$-optimal policy in a tabular setting. For this problem, they devised LoRa-PI (Low-Rank Policy Iteration), a model-free learning algorithm alternating between policy improvement and policy evaluation steps. \n\nTheir main innovation lies in the latter part where the algorithm estimates the low-rank matrix corresponding to the (state, action) value function of the current policy using the following two-phase procedure. The entries of the matrix are first sampled uniformly at random to estimate, via a spectral method, the leverage scores of its rows and columns. These scores are then used to extract a few important rows and columns whose entries are further sampled. The algorithm exploits these new samples to complete the matrix estimation using a CUR-like method. For this leveraged matrix estimation procedure, they used spikiness instead of coherence (two-to-infinity norm assumption).\n\nSTRENGTHS:\n1) According to the authors, their matrix estimation method is the first able to yield entry-wise guarantees that do not depend on the matrix coherence but on its spikiness instead. \n\n2) The authors used the tabular setting where entrywise estimation is important and applied an estimation method that fits well with entry-wise estimation. As they mentioned in Lemma 1, the singular value difference could be very large, but the tabular setting lets authors focus more on entrywise estimation. I believe this is a nice strategy for writing. \n\n3) The authors clearly state why policy iteration is much more useful than value iteration to alleviate condition number assumptions.\n\nWEAKNESSES:\n1) Readers might think this is another application of low-rank estimation literature. Of course, this type of 'adapting results of other fields appropriately' studies is also important in the research, but I'd say this is not ground-breaking. \n\n2) (Important) I am not sure how meaningful the transition from 'coherence' to 'spikiness' is. Maybe it is directly related to Question 1. If I understand correctly, this is the main reason for using the CUR-based matrix completion instead of others, but I believe the authors should provide a more convincing reason (such as theorems?) to make authors study CUR-based matrix completion.\n\n3) (Important) It is not clear how much improvement they made. It would be great if there's a table for low-rank RL that compares the performance/assumptions of previous works concisely. Could you add a table of reference that includes all low-rank and many recent tabular RL results to see how much improvement you made?\n\n4) (Important) Dependence of the threshold parameter $\\beta$ with parameter $d$ or $\\sigma_d(Q^\\pi)$. See the questions section for details. Also, this $\\beta$ seems very important, but it is written in the appendix. It feels like the authors are trying to deceive readers.\n\nQUESTIONS:\n- (Important) Is spikiness always better than coherence? The authors state that in a specific case, spikiness could be finite but coherence could be very large. Is there a case where coherence is finite but spikiness blows up? To use the word 'less restrictive' I think this proof is also necessary. \n\n- (Important) Table (mentioned in the weakness section)\n\n- (Important) Could you explain how your threshold of the singular value could be free from the rank $d$? In many sparse linear or low-rank estimations, rank works as a parameter for the threshold. To alleviate the rank condition from the threshold, one must assume the minimum signal condition, such as $\\sigma_{min}>\\epsilon$ for some $\\epsilon>0$. Seems like in your paper it was possible since you assumes 'minimum signal condition' on Theorem 2, $\\epsilon < \\sqrt{\\frac{S+A}{SAd} \\sigma_d (Q^\\pi)}$. You don't know $d$ or $\\sigma_d$, how could you be sure about this? One main selling point of this paper is that the learner doesn't need to know $d$ or minimum eigenvalue conditions before the game starts. (line 224-225)\n\n- Is there any great idea to extend your estimation idea to the model-based method? \n\n- This idea works well in the tabular setting. What about the non-tabular settings, such as linear RL? Let me know the difficulty of extending your result to linear RL. \n\n\n*** Minor points\n\n1) Maybe using $\\sigma_{min}$ instead of $\\sigma_d$? Researchers frequently use $\\sigma_d (Q^\\pi)$ as $d$-th largest eigenvalue and $d=\\max_\\pi rank(Q^\\pi)$. This means in the traditional notation $\\sigma_d(Q^\\pi)=0$. Though they've mentioned it in their related works, it would be better to change it as $\\sigma_{min}$ or define $\\sigma_d$ somewhere near the Notation paragraph.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
The works present a policy iteration algorithm that relies on supposedly low rank structure of value function (Q-function)
The proposed work first estimates the low-rank matrix of a given policy before performing policy iteration step. The Leveraged Matrix Estimation (LME) algorithm learns the Q-matrix for a given policy. LME for a given sampling budget estimates Q-matrix in a 3 step process.
The policy iteration algorithm also uses LME at the policy evaluation stage.

STRENGTHS:
The proposed Q-matrix estimation and policy iteration algorithm are interesting for sparse MDPs.
Sample complexity bounds for the algorithm
Analysis is easy to follow

WEAKNESSES:
It is not clear where such low-rank Q-matrix appear in real-world. Is coherence and spikiness of Q generally appear in control problems?
Can authors show any real-world application like gym environments?
There is significant body of work that leverage rank deficit structure of Q, it is not immediately what are the additional
insights the current draft has over the past work.

Can you compare with other algorithms proposed.

QUESTIONS:
See weakness

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider the problem of learning an $\\epsilon$-optimal policy in systems with low-rank latent structures and is order optimal under weaker conditions. The proposed algorithm iterates between exploitation (policy evaluation) and exploration (policy improvement). For policy evaluation, the entries are sampled in two stages to complete matrix estimation. The adopted approach provides entry-wise guarantees with respect to the spikiness of the matrix.\n\nSTRENGTHS:\n- The authors have done a great job of clearly defining the problem they are trying to solve and conducting a thorough literature survey showcasing the existing work as well as how their results improve upon the existing work.\n- The proposed algorithm is parameter-free which implies that it does not rely on spikiness/rank/coherence bounds to work. Similarly, the lack of dependence of sample complexity bounds also shows the strengths of these results.\n\nWEAKNESSES:\n- The notation can be hard to read especially as l and L and r and R imply completely different things, while in most cases they imply smaller letters imply matrix entries for the matrix indicated by the larger letter.\n\nQUESTIONS:\n- Theorem 1 shows that the Leverage Scores Estimation depends on the $d^{th}$ singular value, Wouldn't that impose constraints indirectly on the rank/spikiness of the original matrix?\n\n- In Line 285, is the subscript of $\\Omega$ a typo.\n\nLIMITATIONS:\n- Minor Comment: Please elaborate on acronyms before using them. The goal is to make the paper self-contained for a first-time reader as well even though these acronyms are widely known in practice. Line 10 - CUR, Line 30 - MDP, Line 88 - ERM, MLE\n- Can the authors compare the simulation results in Appendix A with existing approaches akin to [35]  to showcase if their algorithm indeed works in a practical setting?\n- Can the authors also showcase how their algorithm holds for different values of rank, spikiness, and condition number?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}