Title: Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Low-rank matrix estimation; low rank bandits; low rank MDP; spectral methods
Abstract: We study matrix estimation problems arising in reinforcement learning with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it characterizes the transition kernel of the MDP. In both cases, each entry of the matrix carries important information, and we seek estimation methods with low entry-wise prediction error. Importantly, these methods further need to accommodate for inherent correlations in the available data (e.g. for MDPs, the data consists of system trajectories). We investigate the performance of  simple spectral-based matrix estimation approaches: we show that they efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise prediction error. These new results on low-rank matrix estimation make it possible to devise reinforcement learning algorithms that fully exploit the underlying low-rank structure. We provide two examples of such algorithms: a regret minimization algorithm for low-rank bandit problems, and a best policy identification algorithm for low-rank MDPs. Both algorithms yield state-of-the-art performance guarantees.

Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learning

Abstract

We study matrix estimation problems arising in reinforcement learning (RL) with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it may for example characterize the transition kernel of the MDP. In both cases, each entry of the matrix carries important information, and we seek estimation methods with low entry-wise error. Importantly, these methods further need to accommodate for inherent correlations in the available data (e.g. for MDPs, the data consists of system trajectories). We investigate the performance of simple spectral-based matrix estimation approaches: we show that they efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error. These new results on low-rank matrix estimation make it possible to devise reinforcement learning algorithms that fully exploit the underlying low-rank structure. We provide two examples of such algorithms: a regret minimization algorithm for low-rank bandit problems, and a best policy identification algorithm for reward-free RL in low-rank MDPs. Both algorithms yield state-of-the-art performance guarantees.

1 Introduction

Learning succinct representations of the reward function or of the system state dynamics in bandit and RL problems is empirically known to significantly accelerate the search for efficient policies [38, 55, 13]. It also comes with interesting theoretical challenges. The design of algorithms learning and leveraging such representations and with provable performance guarantees has attracted considerable attention recently, but remains largely open. In particular, significant efforts have been made towards such design when the representation relies on a low-rank structure. In bandits, assuming such a structure means that the arm-to-reward function can be characterized by a low-rank matrix [37, 32, 7, 27]. In MDPs, it implies that the reward function, the Q -function or the transition kernels are represented by low-rank matrices [56, 4, 46, 60, 49]. In turn, the performance of algorithms exploiting low-rank structures is mainly determined by the accuracy with which we are able to estimate these matrices.

In this paper, we study matrix estimation problems arising in low-rank bandit and RL problems. Two major challenges are associated with these problems. (i) The individual entries of the matrix carry important operational meanings (e.g. in bandits, an entry could correspond to the average reward of an arm), and we seek estimation methods with low entry-wise error. Such requirement calls for a

fine-grained analysis, typically much more involved than that needed to only upper bound the spectral or Frobenius norm of the estimation error [22, 21, 12, 2, 54, 15, 53, 47]. (ii) Our estimation methods should further accommodate for inherent correlations in the available data (e.g., in MDPs, we have access to system trajectories, and the data is hence Markovian). We show that, essentially, spectral methods successfully deal with these challenges.

Contributions. 1) We introduce three matrix estimation problems. The first arises in low-rank bandits. The second corresponds to scenarios in RL where the learner wishes to estimate the (lowrank) transition kernel of a Markov chain and to this aim, has access to a generative model. The last problem is similar but assumes that the learner has access to system trajectories only, a setting referred to as the forward model in the RL literature. For all problems, we establish strong performance guarantees for simple spectral-based estimation approaches: these efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error. To prove these results, we develop and combine involved leave-one-out arguments and Poisson approximation techniques (to handle the correlations in the data).

2) We apply the results obtained for our first matrix estimation problem to devise an efficient regret-minimization algorithm for low-rank bandits. We prove that the algorithm enjoys finite-time performance guarantees, with a regret at most roughly scaling as ( m + n ) log 3 ( T ) ¯ ∆ / ∆ 2 min where ( m,n ) are the reward matrix dimensions, T is the time horizon, ¯ ∆ is the average of the reward gaps between the best arm and all other arms, and ∆ min is the minimum of these gaps.

3) Finally, we present an algorithm for best policy identification in low-rank MDPs in the rewardfree setting. The results obtained for the second and last matrix estimation problems imply that our algorithm learns an /epsilon1 -optimal policy for any reward function using only a number of samples scaling as O ( nA/ /epsilon1 2 ) up to logarithmic factors, where n and A denote the number of states and actions, respectively. This sample complexity is mini-max optimal [28], and illustrates the gain achieved by leveraging the low-rank structure (without this structure, the sample complexity would be Ω ( n 2 A/ /epsilon1 2 ) ).

Notation. For any matrix A ∈ R m × n , A i, : (resp. A : ,j ) denotes its i -th row (resp. its j -th column), A min = min ( i,j ) A i,j and A max = max ( i,j ) A i,j . We consider the following norms for matrices: ‖ A ‖ denotes the spectral norm, ‖ A ‖ 1 →∞ = max i ∈ [ m ] ‖ A i, : ‖ 1 , ‖ A ‖ 2 →∞ = max i ∈ [ m ] ‖ A i, : ‖ 2 , and finally ‖ A ‖ ∞ = max ( i,j ) ∈ [ m ] × [ n ] | A i,j | . If the SVD of A is U Σ V /latticetop , we denote by sgn( A ) = UV /latticetop the matrix sign function of A (see Definition 4.1 in [14]). O r × r denotes the set of ( r × r ) real orthogonal matrices. For any finite set S , let P ( S ) be the set of distributions over S . The notation a ( n, m, T ) /lessorsimilar b ( n, m, T ) (resp. a ( n, m, T ) = Θ ( b ( n, m, T )) ) means that there exists a universal constant C > 0 (resp. c, C > 0 ) such that a ( n, m, T ) ≤ Cb ( n, m, T ) (resp. cb ( n, m, T ) ≤ a ( n, m, T ) ≤ Cb ( n, m, T ) ) for all n, m, T . Finally, we use a ∧ b = min( a, b ) and a ∨ b = max( a, b ) .

2 Models and Objectives

Let M ∈ R m × n be an unknown rank r matrix that we wish to estimate from T noisy observations of its entries. We consider matrices arising in two types of learning problems with low-rank structure, namely low-rank bandits and RL. The SVD of M is U Σ V /latticetop where the matrices U ∈ R m × r and V ∈ R n × r contain the left and right singular vectors of M , respectively, and Σ = diag ( σ 1 , . . . , σ r ) . We assume without loss of generality that the singular values have been ordered, i.e., σ 1 ≥ . . . ≥ σ r . The accuracy of our estimate ̂ M of M will be assessed using the following criteria:

(ii) Matrix estimation. To assess the accuracy of ̂ M , we will upper bound the row-wise error ‖ ̂ M -M ‖ 1 →∞ or ‖ ̂ M -M ‖ 2 →∞ , as well as the entry-wise error ‖ ̂ M -M ‖ ∞ (the spectral error ‖ ̂ M -M ‖ is easier to deal with and is presented in appendix only).

(i) Singular subspace recovery. Let the SVD of ̂ M be ̂ U ̂ Σ ̂ V /latticetop . To understand how well the singular subspaces of M are recovered, we will upper bound min O ∈ O r × r ‖ U -̂ UO ‖ 2 →∞ and min O ∈ O r × r ‖ V -̂ V O ‖ 2 →∞ (the min O ∈ O r × r problem corresponds to the orthogonal Procrustes problem and its solution aligns ̂ U and U as closely as possible, see Remark 4.1 in [14]).

We introduce two classical quantities characterizing the heterogeneity and incoherence of the matrix M [11, 48]. Let κ = σ 1 / σ r , and let µ ( U ) = √ m/r ‖ U ‖ 2 →∞ (resp. µ ( V ) = √ n/r ‖ V ‖ 2 →∞ ) denote the row-incoherence (resp. column-incoherence) parameter of M . Let µ = max { µ ( U ) , µ ( V ) } . Next, we specify the matrices M of interest in low-rank bandits and RL, and the way the data used for their estimation is generated.

Model I: Reward matrices in low-rank bandits. For bandit problems, M corresponds to the average rewards of various arms. To estimate M , the learner has access to data sequentially generated as follows. In each round t = 1 , . . . , T , an arm ( i t , j t ) ∈ [ m ] × [ n ] is randomly selected (say uniformly at random for simplicity) and the learner observes M i t ,j t + ξ t , an unbiased sample of the corresponding entry of M . ( ξ t ) t ≥ 1 is a sequence of zero-mean and bounded random variables. Specifically, we assume that for all t ≥ 1 , | ξ t | ≤ c 1 ‖ M ‖ ∞ a.s., for some constant c 1 > 0 .

Model II: Transition matrices in low-rank MDPs. In low-rank MDPs, we encounter Markov chains whose transition matrices have low rank r (refer to Section 5 for details). Let P ∈ R n × n be such a transition matrix. We assume that the corresponding Markov chain is irreducible with stationary distribution ν . The objective is to estimate P from the data consisting of samples of transitions of the chain. More precisely, from the data, we will estimate the long-term frequency matrix M = diag( ν ) P ( M ij is the limiting proportion of transitions from state i to state j as the trajectory grows large). Observe that M is of rank r , and that P i, : = M i, : / ‖ M i, : ‖ 1 . To estimate M , the learner has access to the data ( x 1 , . . . , x T ) ∈ [ n ] T generated according to one of the following two models.

(a) In the generative model, for any t ∈ [ T ] , if t is odd, x t is selected at random according to some distribution ν 0 , and x t +1 is sampled from P x t , : .

(b) In the forward model, the learner has access to a trajectory ( x 1 , . . . , x T ) of length T of the Markov chain, where x 1 ∼ ν 0 and for any t ≥ 1 , x t +1 ∼ P x t , : .

3 Matrix Estimation via Spectral Decomposition

In the three models (Models I, II(a) and II(b)), we first construct a matrix ˜ M directly from the data, and from there, we build our estimate ̂ M , typically obtained via spectral decomposition, i.e., by taking the best rankr approximation of ˜ M . In the remaining of this section, we let ̂ U ̂ Σ ̂ V /latticetop denote the SVD of ̂ M . Next, we describe in more details how ̂ M is constructed in the three models, and analyze the corresponding estimation error.

3.1 Reward matrices

For Model I, for t = 1 , . . . , T , we define ˜ M t = ( ( M i t ,j t + ξ t ) 1 { ( i,j )=( i t ,j t ) } ) i,j ∈ [ m ] × [ n ] and ˜ M = nm T ∑ T t =1 ˜ M t . Let ̂ M denote the best rankr approximation of ˜ M . Theorem 1. Let δ > 0 . We introduce:

Assume that T ≥ cµ 4 κ 2 r 2 ( n + m ) log 3 ( e ( m + n ) T/ δ ) for some universal constant c > 0 . Then there exists a universal constant C > 0 such that the following inequalities hold with probability at least 1 -δ :

Corollary 2. (Homogeneous reward matrix) When m = Θ ( n ) , κ = Θ (1) , µ = Θ (1) , ‖ M ‖ ∞ = Θ (1) , r = Θ (1) , we say that the reward matrix M is homogeneous. In this case, for any δ > 0 , when T ≥ c ( n + m ) log 3 ( e ( m + n ) T/ δ ) for some universal constant c > 0 , we have with probability at least 1 -δ :

For a homogeneous reward matrix, ‖ U ‖ 2 →∞ = Θ (1 / √ m ) and ‖ M ‖ ∞ = Θ (1) , and hence, from the above corollary, we obtain estimates whose relative errors (e.g., ‖ ̂ M -M ‖ ∞ / ‖ M ‖ ∞ ) scale at most as √ m/T up to the logarithmic factor.

3.2 Transition matrices under the generative model

We may also compare the results of the above corollary to those of Theorem 4.4 presented in [14]. There, the data consists for each pair ( i, j ) of a noisy observation M i,j + E i,j . The E i,j 's are independent across ( i, j ) . This model is simpler than ours and does not include any correlation in the data. But it roughly corresponds to the case where T = nm in our Model I. Despite having to deal with correlations, we obtain similar results as those of Theorem 4.4: for example, ‖ ̂ M -M ‖ ∞ /lessorsimilar √ 1 / ( n + m ) (up to logarithmic terms) with high probability.

For Model II(a), the matrix ˜ M records the empirical frequencies of the transitions: for any pair of states ( i, j ) , ˜ M i,j = 1 /floorleft T/ 2 /floorright ∑ /floorleft T/ 2 /floorright k =1 1 { ( x 2 k -1 ,x 2 k )=( i,j ) } . ̂ M is the best rankr approximation of ˜ M and the estimate ̂ P of the transition matrix P is obtained normalizing the rows of M : for all i ∈ [ n ] , where ( · ) + is the function applying max(0 , · ) component-wise and 1 n is the n -dimensional vector of ones. The next theorem is a simplified version and a consequence of a more general and tighter theorem presented in App. B.2. To simplify the presentation of our results, we define

Theorem 3. Let δ > 0 . Introduce B = µ κ √ ( r ‖ M ‖ ∞ /T ) log( n √ T/ δ ) . Assume that we have ( ν 0 ) min = min i ∈ [ n ] ( ν 0 ) i > 0 . If (a) n ≥ c log 2 ( nT 3 / 2 / δ ) and (b) T ≥ cg ( M,T, δ ) for some universal constant c > 0 , then there exists a universal constant C > 0 such that the following inequalities hold with probability at least 1 -δ :

where (iv) holds if in addition T ≥ cn ‖ M ‖ ∞ ( ν 0 ) -2 min rµ 2 κ 4 log( n √ T/ δ )

Note that in theorem, the condition (a) on n has been introduced just to simplify the expression of B (refer to App. B.2 for a full statement of the theorem without this condition).

Corollary 4. (Homogeneous transition matrix) When κ = Θ (1) , µ = Θ (1) , r = Θ (1) , M max = Θ ( M min ) , we say that the frequency matrix M is homogeneous. If T ≥ cn log( nT ) for some universal constant c > 0 , then we have with probability at least 1 -min { n -2 , T -1 } :

For a homogeneous frequency matrix, ‖ U ‖ 2 →∞ = Θ (1 / √ n ) , ‖ M ‖ 2 →∞ = Θ (1 /n √ n ) , ‖ M ‖ ∞ = Θ (1 /n 2 ) , ‖ P ‖ 1 →∞ = 1 , ‖ P ‖ ∞ = Θ (1 /n ) . Thus for all these metrics, our estimates achieve a relative error scaling at most as √ n/T up to the logarithmic factor.

3.3 Transition matrices under the forward model

For Model II(b), we first split the data into τ subsets of transitions: for k = 1 , . . . , τ , the k -th subset is (( x k , x k +1 ) , ( x k + τ , x k +1+ τ ) , . . . , ( x k +( T τ -1) τ , x k +1+( T τ -1) τ )) where T τ = /floorleft T/ τ /floorright . By separating two transitions in the same subset, we break the inherent correlations in the data if τ is large enough. Now we let ˜ M ( k ) be the matrix recording the empirical frequencies of the transitions in the k -th subset: ˜ M ( k ) i,j = 1 T τ ∑ T τ -1 l =0 1 { ( x k + l τ ,x k +1+ l τ )=( i,j ) } for any pair of states ( i, j ) . Let ̂ M ( k ) be the best r -rank approximation of ˜ M ( k ) . As in (1), we define the corresponding ̂ P ( k ) . Finally we may aggregate these estimates ̂ M = 1 τ ∑ τ k =1 ̂ M ( k ) and ̂ P = 1 τ ∑ τ k =1 ̂ P ( k ) . We present below the performance analysis for the estimates coming from a single subset; the analysis of the aggregate estimates easily follows.

For any ε > 0 , we define the ε -mixing time of the Markov chain with transition matrix P as τ ( ε ) = min { t ≥ 1 : max 1 ≤ i ≤ n 1 2 ‖ P t i, : -ν /latticetop ‖ 1 ≤ ε } , and its mixing time as τ /star = τ (1 / 4) . The next theorem is a simplified version and a consequence of a more general and tighter theorem presented in App. B.3. To simplify the presentation, we define:

Theorem 5. Let δ > 0 . Assume that ν min = min i ∈ [ n ] ν i > 0 and that τ / ( τ /star log( T ν -1 min )) ∈ [ c 1 , c 2 ] for some universal constants c 2 > c 1 ≥ 2 . Introduce:

If (a) n ≥ c τ /star log 3 / 2 ( nT 3 / 2 / δ ) log 1 / 2 ( T ν -1 min ) and (b) T ≥ ch ( M,T, δ ) for some universal constant c > 0 , then there exists a universal constant C > 0 such that the following inequalities hold with probability at least 1 -δ :

where (iv) holds if in addition T ≥ cn ‖ M ‖ ∞ ν -2 min τ /star rµ 2 κ 4 log( n √ T/ δ ) log( T ν -1 min ) .

Note that our guarantees hold when τ roughly scales as τ /star log( T ν -1 min ) . Hence to select τ , one would need an idea of the latter quantity. It can be estimated typically using τ /star ν -1 min samples [61] (which

is small when compared to the constraint T ≥ ch ( M,T, δ ) as soon as ν min = Ω (1 /n ) ). Further observe that in the theorem, the condition (a) can be removed (refer to App. B.3 for a full statement of the theorem without this condition).

Corollary 6. (Homogeneous transition matrices) Assume that M is homogeneous (as defined in Corollary 4). Let τ = log( Tn ) . If T ≥ cn log 2 ( nT ) for some universal constant c > 0 , then we have with probability at least 1 -min { n -2 , T -1 } :

As for the generative model, for a homogeneous frequency matrix, our estimates achieve a relative error scaling at most as √ n/T up to the logarithmic factor for all metrics. Note that up to a logarithmic factor, the upper bound for ‖ ̂ P -P ‖ 1 →∞ (and similarly for ̂ M ) matches the minimax lower bound derived in [63].

3.4 Elements of the proofs

The proofs of the three above theorems share similar arguments. We only describe elements of the proof of Theorem 5, corresponding to the most challenging model. The most difficult result concerns the singular subspace recovery (the upper bounds (i) in our theorems), and it can be decomposed into the following three steps. The first two steps are meant to deal with the Markovian nature of the data. The third step consists in applying a leave-one-out analysis to recover the singular subspaces.

Step 1: Multinomial approximation of Markovian data. We treat the matrix ˜ M ( k ) arising from one subset of data, and for simplicity, we remove the superscript ( k ) , i.e., ˜ M = ˜ M ( k ) . Note that T τ ˜ M is a matrix recording the numbers of transitions observed in the data for any pair of states: denote by N i,j this number for ( i, j ) . We approximate the joint distribution of N = ( N i,j ) ( i,j ) by a multinomial distribution with n 2 components and parameter T τ M i,j for component ( i, j ) . Denote by Z = ( Z i,j ) ( i,j ) the corresponding multinomial random variable. Using the mixing property of the Markov chain and the choice of τ , we establish (see Lemma 21 in App. C) that for any subset Z of { z ∈ N n 2 : ∑ ( i,j ) z i,j = T τ } , we have P [ N ∈ Z ] ≤ 3 P [ Z ∈ Z ] .

Step 2: Towards Poisson random matrices with independent entries. The random matrix Z does not have independent entries. Independence is however a requirement if we wish to apply the leaveone-out argument. Consider the random matrix Y whose entries are independent Poisson random variables with mean T τ M i,j for the ( i, j ) -th entry. We establish the following connection between the distribution of Z and that of Y : for any Z ⊂ N n 2 , we have P [ Z ∈ Z ] ≤ e √ T τ P [ Y ∈ Z ] . Refer to Lemma 22 in App. C for details.

Step 3: The leave-one-out argument for Poisson matrices. Combining the two first steps provides a connection between the observation matrix ˜ M and a Poisson matrix Y with independent entries. This allows us to apply a leave-one-out analysis to ˜ M as if it had independent entries (replacing ˜ M by Y ). The analysis starts by applying the standard dilation trick (see Section 4.10 in [14]) so as to make ˜ M symmetric. Then, we can decompose the error ‖ U -̂ U ( ̂ U /latticetop U ) ‖ 2 →∞ (see Lemma 32 in App. E) into several terms. The most challenging of these terms is ‖ ( M -˜ M )( U -̂ U ( ̂ U /latticetop U )) ‖ 2 →∞ = max l ∈ [ n ] ‖ ( M l, : -˜ M l, : )( U -̂ U ( ̂ U /latticetop U )) ‖ 2 because of inherent dependence between M -˜ M and U -̂ U ( ̂ U /latticetop U ) . The leave-one-out analysis allows us to decouple this statistical dependency. It consists in exploiting the row and column independence of matrix ˜ M to approximate ‖ ( M l, : -˜ M l, : )( U -̂ U ( ̂ U /latticetop U )) ‖ 2 by ‖ ( M l, : -˜ M l, : )( U -̂ U ( l ) (( ̂ U ( l ) ) /latticetop U ) ‖ 2 where ̂ U ( l ) is the matrix of eigenvectors of matrix ˜ M ( l ) obtained by zeroing the l -th row and column of ˜ M . By construction, ( M l, : -˜ M l, : ) and U -̂ U ( l ) (( ̂ U ( l ) ) /latticetop U ) are independent, which simplifies the analysis. The proof is completed by a

further appropriate decomposition of this term, combined with concentration inequalities for random Poisson matrices (see App. D).

4 Regret Minimization in Low-Rank Bandits

Consider a low-rank bandit problem with a homogeneous rankr reward matrix M . We wish to devise an algorithm π with low regret. π selects in round t an entry ( i π t , j π t ) based on previous observations, and receives as a feedback the noisy reward M i π t ,j π t + ξ t . The regret up to round T is defined by R π ( T ) = TM i /star ,j /star -E [ ∑ T t =1 M i π t ,j π t ] , where ( i /star , j /star ) is an optimal entry. One could think of a simple Explore-Then-Commit (ETC) algorithm, where in the first phase entries are sampled uniformly at random, and where in a second phase, the algorithm always selects the highest entry of ̂ M built using the samples gathered in the first phase and obtained by spectral decomposition. When the length of the first phase is T 2 / 3 ( n + m ) 1 / 3 , the ETC algorithm would yield a regret upper bounded by O ( T 2 / 3 ( n + m ) 1 / 3 ) for T = Ω (( n + m ) log 3 ( n + m )) .

To get better regret guarantees, we present SME-AE (Successive Matrix Estimation and Arm Elimination), an algorithm meant to identify the best entry as quickly as possible with a prescribed level of certainty. After the SME-AE has returned the estimated best entry, we commit and play this entry for the remaining rounds. The pseudo-code of SME-AE is presented in Algorithm 1. The algorithm runs in epochs: in epoch /lscript , it samples T /lscript entries uniformly at random among all entries (in T /lscript , the constant C just depends on upper bounds of the parameters µ , κ , and ‖ M ‖ ∞ , refer to App. G); from these samples, a matrix ̂ M ( /lscript ) is estimated and A /lscript , the set of candidate arms, is pruned. The pruning procedure is based on the estimated gaps: ̂ ∆ ( /lscript ) i,j = ̂ M ( /lscript ) /star -̂ M ( /lscript ) i,j where ̂ M ( /lscript ) /star = max i,j ̂ M ( /lscript ) i,j .

Algorithm 1: S uccesive M atrix E stimation and A rm E limination ( SME-AE )

The following theorem characterizes the performance of SME-AE and the resulting regret. To simplify the notation, we introduce the gaps: for any entry ( i, j ) , ∆ i,j = ( M i /star ,j /star -M i,j ) , ∆ min = min ( i,j ): ∆ i,j > 0 ∆ i,j , ∆ max = max ( i,j ) ∆ i,j , and ¯ ∆ = ∑ ( i,j ) ∆ i,j / ( mn ) . We define the function ψ ( n, m, δ ) = c ( m + n ) log( e/ ∆ min ) ∆ 2 min log 3 ( e ( m + n ) log( e/ ∆ min ) ∆ min δ ) for some universal constant c > 0 .

E [ ∧ ] ≤ ( ) + . 2

Theorem 7. (Best entry identification) For any δ ∈ (0 , 1) , SME-AE( δ ) stops at time τ and recommends arm (ˆ ı τ , ˆ  τ ) with the guarantee P ( (ˆ ı τ , ˆ  τ ) = ( i /star , j /star ) , τ ≤ ψ ( n, m, δ ) ) ≥ 1 -δ . Moreover, for any T ≥ 1 and α > 0 , the sample complexity τ of SME-AE( 1 /T α ) satisfies τ T ψ n, m, T -α T 1 -α

(Regret) Let T ≥ 1 . Consider the algorithm π that first runs SME-AE( 1 /T ) and then commits to its output (ˆ ı τ , ˆ  τ ) after τ . We have: R π ( T ) ≤ ¯ ∆ ( ψ ( n, m, T -2 ) + 1 ) + ∆ max T .

The proof of Theorem 7 is given in App. G. Note that the regret upper bounds hold for any time horizon T ≥ 1 , and that it scales as O (( m + n ) log 3 ( T ) ¯ ∆ / ∆ 2 min ) (up to logarithmic factors in m,n and 1 / ∆ min ). The cubic dependence in log 3 ( T ) is an artifact of our proof techniques. More precisely, it is due to the Poisson approximation used to obtain entry-wise guarantees. Importantly, for any time horizon, the regret upper bound only depends on ( m + n ) rather than mn (the number

of arms / entries), and hence, the low-rank structure is efficiently exploited. If we further restrict our attention to problems with gap ratio ∆ max / ∆ min upper bounded by ζ , our regret upper bound becomes O ( ζ ( m + n ) log 3 ( T ) / ∆ min ) , and can be transformed into the minimax gap-independent upper bound O ( ζ (( m + n ) T ) 1 / 2 log 2 ( T )) , see App. G. Finally note that Ω ((( m + n ) T ) 1 / 2 ) is an obvious minimax regret lower bound for our low-rank bandit problem.

Avery similar low-rank bandit problem has been investigated in [7]. There, under similar assumptions (see Assumption 1 and Definition 1), the authors devise an algorithm with both gap-dependent and gap-independent regret guarantees. The latter are difficult to compare with ours. Their guarantees exhibit a better dependence in T and ∆ min , but worse in the matrix dimensions n and m . Indeed in our model, b /star in [7] corresponds to ‖ M ‖ 2 →∞ and scales as √ n . As a consequence, the upper bounds in [7] have a dependence in n and m scaling as √ n ( n + m ) in the worst case for gap-dependent guarantees and even nm (through the constant C 2 in [7]) for gap-independent guarantees.

5 Representation Learning in Low-Rank MDPs

The results derived for Models II(a) and II(b) are instrumental towards representation learning and hence towards model-based or reward-free RL in low-rank MDPs. In this section, we provide an example of application of these results, and mention other examples in Section 7. A low-rank MDP is defined by ( S , A , { P a } a ∈ A , R, γ ) where S , A denote state and action spaces of cardinalities n and A , respectively, P a denotes the rankr transition matrix when taking action a , R is the reward function, and γ is the discount factor. We assume that all rewards are in [0 , 1] . The value function of a policy π : S → A is defined as V π R ( x ) = E [ ∑ ∞ t =1 γ t -1 R ( x π t , π t ( x π t )) | x π 1 = x ] where x π t is the state visited under π in round t . We denote by π /star ( R ) an optimal policy (i.e., with the highest value function).

Reward-free RL. In the reward-free RL setting (see e.g. [36, 30, 66]), the learner does not receive any reward signal during the exploration process. The latter is only used to construct estimates { ̂ P a } a ∈ A of { P a } a ∈ A . The reward function R is revealed at the end, and the learner may compute ˆ π ( R ) an optimal policy for the MDP ( S , A , { ̂ P a } a ∈ A , R, γ ) . The performance of this model-based approach is often assessed through Γ = sup R ‖ V π /star ( R ) R -V ̂ π ( R ) R ‖ ∞ . In tabular MDP, to identify an /epsilon1 -optimal policy for all reward functions, i.e., to ensure that Γ ≤ /epsilon1 , we believe that the number of samples that have to be collected should be Ω ( poly ( 1 1 -γ ) n 2 A /epsilon1 2 ) (the exact degree of the polynomial in 1 / (1 -γ ) has to be determined). This conjecture is based on the sample complexity lower bounds derived for reward-free RL in episodic tabular MDP [30, 43]. Now for low-rank MDPs, the equivalent lower bound would be Ω ( poly ( 1 1 -γ ) nA /epsilon1 2 ) [28] (this minimax lower bound is valid for Block MDPs, a particular case of low-rank MDPs).

Leveraging our low-rank matrix estimation guarantees, we propose an algorithm matching the aforementioned sample complexity lower bound (up to logarithmic factors) at least when the frequency matrices { M a } a ∈ A are homogeneous. The algorithm consists of two phases: (1) in the model estimation phase, it collects A trajectories, each of length T/A , corresponding to the Markov chains with transition matrices { P a } a ∈ A . From this data, it uses the spectral decomposition method described in §3 to build estimates { ̂ P a } a ∈ A . (2) In the planning phase, based on the reward function R , it computes the best policy ˆ π ( R ) for the MDP ( S , A , { ̂ P a } a ∈ A , R, γ ) . The following theorem summarizes the performance of this algorithm. To simplify the presentation, we only provide the performance guarantees of the algorithm for homogeneous transition matrices (guarantees for more general matrices can be derived plugging in the results from Theorem 5).

Theorem 8. Assume that for any a ∈ A , M a is homogeneous (as defined in Corollary 4). If T ≥ cnA log 2 ( nAT ) for some universal constant c > 0 , then we have with probability at least 1 -min { n -2 , T -1 } : Γ = sup R ‖ V π /star ( R ) R -V ˆ π ( R ) R ‖ ∞ /lessorsimilar 1 (1 -γ ) 2 √ nA T log( nAT ) .

Theorem 8 is a direct consequence of Corollary 6 and of the fact that for any reward function R : ‖ V π /star ( R ) R -V ˆ π ( R ) R ‖ ∞ ≤ 2 γ (1 -γ ) 2 max a ∈ A ‖ P a -̂ P a ‖ 1 →∞ , see App. A. The theorem implies that if we wish to guarantee Γ ≤ /epsilon1 , we just need to collect O ( nA /epsilon1 2 (1 -γ ) 4 ) samples up to a logarithmic factor.

This sample complexity is minimax optimal in n , A , and /epsilon1 in view of the lower bound presented in [28].

6 Related Work

Low-rank matrix estimation. Until recently, the main efforts on low-rank matrix recovery were focused on guarantees w.r.t. the spectral or Frobenius norms, see e.g. [19] and references therein. The first matrix estimation and subspace recovery guarantees in /lscript 2 →∞ and /lscript ∞ were established in [22], [21] via a more involved perturbation analysis than the classical Davis-Kahan bound. An alternative approach based on a leave-one-out analysis was proposed in [2], and further refined in [10, 12, 17], see [14] for a survey. Some work have also adapted the techniques beyond the independent noise assumption [39, 1, 5], but for very specific structural dependence. We deal with a stronger dependence, and in particular with Markovian data (an important scenario in RL).

The estimation of low-rank transition matrices of Markov chains has been studied in [63, 9] using spectral methods and in [40, 67] using maximum-likelihood approaches. [63] does not conduct any fine-grained subspace recovery analysis (such as the leave-one-out), and hence the results pertaining to the ‖ · ‖ 1 →∞ -guarantees are questionable; refer to App. H for a detailed justification. All these papers do not present entry-wise guarantees.

It is worth mentioning that there exist other methods for matrix estimation that do not rely on spectral decompositions like ours, yet enjoy entry-wise matrix estimation guarantees [51, 3, 50]. However, these methods require different assumptions than ours that may be too strong for our purposes, notably having access to the so-called anchor rows and columns. Moreover, we do not know if these methods also lead to guarantees for subspace recovery in the norm ‖ · ‖ 2 →∞ , nor how to extend those results to settings with dependent noise.

Low-rank bandits. Low-rank structure in bandits has received a lot of attention recently [35, 37, 32, 57, 41, 7, 34, 27]. Different set-ups have been proposed (refer to App. H for a detailed exposition, in particular, we discuss how the settings proposed in [32, 7] are equivalent), and regret guarantees in an instance dependent and minimax sense have been both established.

Typically minimax regret guarantees in bandits scale as √ T , but the scaling in dimension may defer when dealing with a low rank structure [32, 34, 7]. In [32], the authors also leverage spectral methods. They reduce the problem to a linear bandit of dimension nm but where only roughly n + m dimensions are relevant. This entails that a regret lower bound of order ( n + m ) √ T is inevitable. Actually, in their reduction to linear bandits, they only use a subspace recovery in Frobenius norm, which perhaps explains the scaling ( n + m ) 3 / 2 in their regret guarantees. It is worth noting that in [34], the authors manage to improve upon the work [32] and obtain a scaling order ( m + n ) in the regret. Our algorithm leverages entry-wise guarantees which rely on a stronger subspace recovery guarantee. This allows us to obtain a scaling √ n + m in the regret. The work of [27] is yet another closely related work to ours. There, the authors propose an algorithm achieving a regret of order polylog ( n + m ) √ T for a contextual bandit problem with low rank structure. However, their result only holds for rank 1 and their observation setup is different than ours because in their setting, the learner observes m entries per round while in ours the learner only observes one entry per round. In [7], the authors use matrix estimation with nuclear norm penalization to estimate the matrix M . Their regret guarantees are already discussed in §4.

Some instance-dependent guarantees with logarithmic regret for low rank bandits have been established in [35, 37, 57]. However, these results suffer what may be qualified as serious limitations. Indeed, [35, 57] provide instance dependent regret guarantees but only consider low-rank bandits with rank 1 , and the regret bounds of [35] are expressed in terms of the so-called column and row gaps (see their Theorem 1) which are distinct from the standard gap notions. [37] extend the results in [35] to rank r with the limitation that they require stronger assumptions than ours. Moreover, the computational complexity of their algorithm depends exponentially on the rank r ; they require a search over spaces of size ( m r ) and ( n r ) . Our proposed algorithm does not suffer from such limitations.

We wish to highlight that our entry-wise guarantees for matrix estimation are the key enabling tool that led us to the design and analysis of our proposed algorithm. In fact, the need for such guarantees arises naturally in the analysis of gap-dependent regret bounds (see Appendix G.1). Therefore, we

believe that such guarantees can pave the way towards better, faster, and efficient algorithms for bandits with low-rank structure.

Low-rank Reinforcement Learning. RL with low rank structure has been recently extensively studied but always in the function approximation framework [29, 18, 20, 44, 24, 65, 56, 4, 46, 59, 60, 49]. There, the transition probabilities can be written as φ ( x, a ) /latticetop µ ( x ′ ) where the unknown feature functions φ ( x, a ) , µ ( x ′ ) ∈ R r belong to some specific class F of functions. The major issue with algorithms proposed in this literature is that they rely on strong computational oracles (e.g., ERM, MLE), see [33, 25, 64] for detailed discussions. In contrast, we do not assume that the transition matrices are constructed based on a given restricted class of functions, and our algorithms do not rely on any oracle and are computationally efficient. In [51, 50], the authors also depart from the function approximation framework. There, they consider a low rank structure different than ours. Their matrix estimation method enjoys an entry-wise guarantee, but requires to identify a subset of rows and columns spanning the range of the full matrix. Moreover, their results are only limited the generative models, which allows to actually rely on independent data samples.

7 Conclusion and Perspectives

In this paper, we have established that spectral methods efficiently recover low-rank matrices even in correlated noise. We have investigated noise correlations that naturally arise in RL, and have managed to prove that spectral methods yield nearly-minimal entry-wise error. Our results for low-rank matrix estimation have been applied to design efficient algorithms in low-rank RL problems and to analyze their performance. We believe that these results may find many more applications in low-rank RL. They can be applied (i) to reward-free RL in episodic MDPs (this setting is easier than that presented in §5 since successive episodes are independent); (ii) to scenarios corresponding to offline RL [62] where the data consists of a single trajectory generated under a given behavior policy (from this data, we can extract the transitions ( x, a, x ′ ) where a given action a is involved and apply the spectral method to learn ̂ P a ); (iii) to traditional RL where the reward function R has to be learnt (learning R is a problem that lies in some sense between the inference problems in our Models I and II); (iv) to model-free RL where we would directly learn the Q function as done in [52] under a generative model; (v) to low-rank RL problems with continuous state spaces (this can be done if the transition probabilities are smooth in the states, and by combining our methods to an appropriate discretization of the state space).

Acknowledgment

This research was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

References

Emmanuel Abbe, Jianqing Fan, and Kaizheng Wang. An /lscript p theory of pca and spectral clustering. The Annals of Statistics , 50(4):2359-2385, 2022. (Cited on page 9.)

Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Annals of statistics , 48(3):1452, 2020. (Cited on pages 2, 9, and 34.)

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019. (Cited on pages 9 and 18.)

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs. In Advances in Neural Information Processing Systems , volume 33, pages 20095-20107. Curran Associates, Inc., 2020. (Cited on pages 1 and 10.)

Joshua Agterberg, Zachary Lubberts, and Carey E Priebe. Entrywise estimation of singular vectors of low-rank matrices with heteroskedasticity and dependence. IEEE Transactions on Information Theory , 68(7):4618-4650, 2022. (Cited on page 9.)

Afonso S Bandeira and Ramon Van Handel. Sharp nonasymptotic bounds on the norm of random matrices with independent entries. The Annals of Probability , 44(4):2479-2506, 2016. (Cited on page 31.)

Mohsen Bayati, Junyu Cao, and Wanning Chen. Speed up the cold-start learning in two-sided bandits with many arms. arXiv preprint arXiv:2210.00340 , 2022. (Cited on pages 1, 8, 9, and 50.)

George Bennett. Probability inequalities for the sum of independent random variables. Journal of the American Statistical Association , 57(297):33-45, 1962. (Cited on page 26.)

Shujun Bi, Zhen Yin, and Yihong Weng. A low-rank spectral method for learning markov models. Optimization Letters , 17(1):143-162, 2023. (Cited on page 9.)

T Tony Cai and Anru Zhang. Rate-optimal perturbation bounds for singular subspaces with applications to high-dimensional statistics. The Annals of Statistics , 46(1):60-89, 2018. (Cited on page 9.)

Emmanuel J Candès and Terence Tao. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory , 56(5):2053-2080, 2010. (Cited on page 3.)

Joshua Cape, Minh Tang, and Carey E Priebe. The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics. The Annals of Statistics , 47(5):24052439, 2019. (Cited on pages 2, 9, 18, 35, and 43.)

Yash Chandak, Shantanu Thakoor, Zhaohan Daniel Guo, Yunhao Tang, Remi Munos, Will Dabney, and Diana L Borsa. Representations and exploration for deep reinforcement learning using singular value decomposition. In Proc. of ICML , 2023. (Cited on page 1.)

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning , 14(5):566-806, 2021. (Cited on pages 2, 4, 6, 9, 34, 39, 40, 41, and 43.)

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization , 30(4):3098-3121, 2020. (Cited on page 2.)

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM Journal on Optimization , 30(4):3098-3121, 2020. (Cited on page 50.)

Yuxin Chen, Jianqing Fan, Cong Ma, and Kaizheng Wang. Spectral method and regularized mle are both optimal for top-k ranking. Annals of statistics , 47(4):2204, 2019. (Cited on page 9.)

Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On Oracle-Efficient PAC RL with Rich Observations. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. (Cited on page 10.)

Mark A. Davenport and Justin K. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE J. Sel. Top. Signal Process. , 10(4):608-622, 2016. (Cited on page 9.)

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with Rich Observations via Latent State Decoding. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1665-1674. PMLR, 09-15 Jun 2019. (Cited on page 10.)

Justin Eldridge, Mikhail Belkin, and Yusu Wang. Unperturbed: spectral analysis beyond davis-kahan. In Algorithmic Learning Theory , pages 321-358. PMLR, 2018. (Cited on pages 2 and 9.)

Jianqing Fan, Weichen Wang, and Yiqiao Zhong. An /lscript ∞ eigenvector perturbation bound and its application to robust covariance estimation. Journal of Machine Learning Research , 18(207):1-42, 2018. (Cited on pages 2 and 9.)

Vivek Farias, Andrew A Li, and Tianyi Peng. Near-optimal entrywise anomaly detection for low-rank matrices with sub-exponential noise. In International Conference on Machine Learning , pages 3154-3163. PMLR, 2021. (Cited on page 34.)

Dylan Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspective. In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 2059-2059. PMLR, 15-19 Aug 2021. (Cited on page 10.)

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. Learning in Observable POMDPs, without Computationally Intractable Oracles. In Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022. (Cited on page 10.)

Samuel B Hopkins, Tselil Schramm, Jonathan Shi, and David Steurer. Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 178-191, 2016. (Cited on page 26.)

Prateek Jain and Soumyabrata Pal. Online low rank matrix completion. In Proc. of ICLR , 2023. (Cited on pages 1, 9, and 50.)

Yassir Jedra, Junghyun Lee, Alexandre Proutiere, and Se-Young Yun. Nearly optimal latent state decoding in block mdps. In International Conference on Artificial Intelligence and Statistics , pages 2805-2904. PMLR, 2023. (Cited on pages 2, 8, and 9.)

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 06-11 Aug 2017. (Cited on page 10.)

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-Free Exploration for Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4870-4879. PMLR, 13-18 Jul 2020. (Cited on page 8.)

Kwang-Sung Jun, Rebecca Willett, Stephen Wright, and Robert Nowak. Bilinear bandits with low-rank structure. In International Conference on Machine Learning , pages 3163-3172. PMLR, 2019. (Cited on page 50.)

Kwang-Sung Jun, Rebecca Willett, Stephen J. Wright, and Robert D. Nowak. Bilinear bandits with low-rank structure. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 3163-3172. PMLR, 2019. (Cited on pages 1 and 9.)

Daniel Kane, Sihan Liu, Shachar Lovett, and Gaurav Mahajan. Computational-statistical gap in reinforcement learning. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1282-1302. PMLR, 02-05 Jul 2022. (Cited on page 10.)

Yue Kang, Cho-Jui Hsieh, and Thomas Chun Man Lee. Efficient frameworks for generalized lowrank matrix bandit problems. Advances in Neural Information Processing Systems , 35:1997119983, 2022. (Cited on pages 9 and 50.)

Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Claire Vernade, and Zheng Wen. Stochastic Rank-1 Bandits. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 392-401. PMLR, 20-22 Apr 2017. (Cited on page 9.)

Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive Reward-Free Exploration. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132 of Proceedings of Machine Learning Research , pages 865-891. PMLR, 16-19 Mar 2021. (Cited on page 8.)

Branislav Kveton, Csaba Szepesvari, Anup Rao, Zheng Wen, Yasin Abbasi-Yadkori, and S. Muthukrishnan. Stochastic low-rank bandits, 2017. (Cited on pages 1 and 9.)

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5639-5650. PMLR, 13-18 Jul 2020. (Cited on page 1.)

Lihua Lei. Unified /lscript 2 →∞ eigenspace perturbation theory for symmetric random matrices. arXiv preprint arXiv:1909.04798 , 2019. (Cited on page 9.)

Xudong Li, Mengdi Wang, and Anru Zhang. Estimation of markov chain via rank-constrained likelihood. In International Conference on Machine Learning , pages 3033-3042. PMLR, 2018. (Cited on page 9.)

Yangyi Lu, Amirhossein Meisami, and Ambuj Tewari. Low-rank generalized linear bandit problems. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 460-468. PMLR, 2021. (Cited on page 9.)

Andrew D McRae and Mark A Davenport. Low-rank matrix completion and denoising under poisson noise. Information and Inference: A Journal of the IMA , 10(2):697-720, 2021. (Cited on page 31.)

Pierre Menard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7599-7608. PMLR, 18-24 Jul 2021. (Cited on page 8.)

Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6961-6971. PMLR, 13-18 Jul 2020. (Cited on page 10.)

Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis . Cambridge university press, 2017. (Cited on page 22.)

Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration in low-rank mdps. To appear in Journal of Machine Learning Research (JMLR) , 2023. (Cited on pages 1 and 10.)

Sahand Negahban and Martin J Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research , 13(1):1665-1697, 2012. (Cited on page 2.)

Benjamin Recht. A simpler approach to matrix completion. Journal of Machine Learning Research , 12(12), 2011. (Cited on page 3.)

Tongzheng Ren, Tianjun Zhang, Lisa Lee, Joseph E Gonzalez, Dale Schuurmans, and Bo Dai. Spectral decomposition representation for reinforcement learning. In Proc. of ICLR , 2023. (Cited on pages 1 and 10.)

Tyler Sam, Yudong Chen, and Christina Lee Yu. Overcoming the long horizon barrier for sample-efficient reinforcement learning with latent low-rank structure. ACM SIGMETRICS Performance Evaluation Review , 50(4):41-43, 2023. (Cited on pages 9 and 10.)

Devavrat Shah, Dogyoon Song, Zhi Xu, and Yuzhe Yang. Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation. In Advances in Neural Information Processing Systems , volume 33, pages 12092-12103. Curran Associates, Inc., 2020. (Cited on pages 9 and 10.)

Devavrat Shah, Dogyoon Song, Zhi Xu, and Yuzhe Yang. Sample efficient reinforcement learning via low-rank matrix estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20. Curran Associates Inc., 2020. (Cited on page 10.)

Ohad Shamir and Shai Shalev-Shwartz. Matrix completion with the trace norm: Learning, bounding, and transducing. The Journal of Machine Learning Research , 15(1):3401-3423, 2014. (Cited on page 2.)

Nathan Srebro and Adi Shraibman. Rank, trace-norm and max-norm. In International conference on computational learning theory , pages 545-560. Springer, 2005. (Cited on page 2.)

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling Representation Learning from Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9870-9879. PMLR, 18-24 Jul 2021. (Cited on page 1.)

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Modelbased RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches. In Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2898-2933. PMLR, 25-28 Jun 2019. (Cited on pages 1 and 10.)

Cindy Trinh, Emilie Kaufmann, Claire Vernade, and Richard Combes. Solving bernoulli rankone bandits with unimodal thompson sampling. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117 of Proceedings of Machine Learning Research , pages 862-889. PMLR, 08 Feb-11 Feb 2020. (Cited on page 9.)

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015. (Cited on page 34.)

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652 , 2021. (Cited on page 10.)

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation Learning for Online and Offline RL in Low-rank MDPs. In International Conference on Learning Representations , 2022. (Cited on pages 1 and 10.)

Geoffrey Wolfer and Aryeh Kontorovich. Estimating the mixing time of ergodic markov chains. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 3120-3159. PMLR, 25-28 Jun 2019. (Cited on page 5.)

Ming Yin and Yu-Xiang Wang. Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings. In Advances in Neural Information Processing Systems , volume 34, pages 12890-12903. Curran Associates, Inc., 2021. (Cited on page 10.)

Anru Zhang and Mengdi Wang. Spectral State Compression of Markov Processes. IEEE Transactions on Information Theory , 66(5):3202-3231, 2020. (Cited on pages 6, 9, 18, 21, 44, 49, and 50.)

Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making Linear MDPs Practical via Contrastive Representation Learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26447-26466. PMLR, 17-23 Jul 2022. (Cited on page 10.)

Xuezhou Zhang, Yuda Song, Masatoshi Uehara, Mengdi Wang, Alekh Agarwal, and Wen Sun. Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26517-26547. PMLR, 17-23 Jul 2022. (Cited on page 10.)

Zihan Zhang, Simon Du, and Xiangyang Ji. Near Optimal Reward-Free Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 12402-12412. PMLR, 18-24 Jul 2021. (Cited on page 8.)

Ziwei Zhu, Xudong Li, Mengdi Wang, and Anru Zhang. Learning markov models via low-rank optimization. Operations Research , 70(4):2384-2398, 2022. (Cited on page 9.)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a detailed analysis of matrix estimation problems in low-rank bandit and low-rank RL scenarios. The authors investigate the effectiveness of spectral-based matrix estimation methods and demonstrate their ability to accurately recover the singular subspaces of the matrix with minimal entry-wise error. Building upon these findings, the paper introduces a regret minimization algorithm for low-rank bandit problems and a best policy identification algorithm for reward-free RL in low-rank MDPs.\n\nSTRENGTHS:\n1. The paper tackles the challenging task of deriving the entry-wise error of low-rank matrix estimation, specifically focusing on the correlated noise case. This analysis extends existing results and enables the study of bandit and RL settings. The techniques employed, such as leave-one-out arguments and Poisson approximation, demonstrate technical prowess.\n\n2. Despite being a theoretical paper, the writing is clear and easy to comprehend, enhancing its accessibility.\n\nWEAKNESSES:\n1. The paper fails to address the necessity of deriving entry-wise error in low-rank matrix estimation for bandit and RL settings. In the context of low-rank bandit problems, the primary objective is to identify the best entry, such as the largest value, in the matrix as quickly as possible. Consequently, the emphasis should be on selecting the arm with the highest reward, rather than achieving a small entry-wise error for the entire matrix. Entries with very small values do not necessitate extensive exploration. Thus, caring about entry-wise error may lead to unnecessary over-exploration, which is not desirable in low-rank bandit or low-rank MDP settings.\n\n\n2. In low-rank bandit and MDP scenarios, the focus should primarily be on minimizing regret. Several existing approaches have achieved minimax regret bounds in these settings. For instance, Kang et al. (2022) addressed a more general low-rank matrix bandit problem and proved a minimax regret of $O((n+m)\\sqrt{T})$. Jain and Pal (2023) also considered a similar low-rank matrix bandit problem and used a similar entry-wise matrix estimation result to achieve a regret bound of $polylog(n+m)\\sqrt{T}$ for the rank-1 case. Unfortunately, the paper only discusses sub-optimal approaches in the main body, while the superior results of these references are mentioned in the last section of the supplementary materials.\n\n\n3. The lack of numerical experiments to evaluate the proposed regret minimization algorithm and compare it against benchmark methods in the literature is a notable drawback. Merely comparing regret upper bounds does not provide a comprehensive understanding of the algorithms' actual performance. It would be more convincing to include extensive numerical experiments that directly compare the regret achieved by different algorithms, thereby justifying the usefulness of deriving entry-wise error of low-rank matrix estimation in bandit and RL settings.\n\nQUESTIONS:\n1. Could you elaborate on how deriving the entry-wise error of low-rank matrix estimation is justified in bandit and RL settings, providing detailed explanations and extensive numerical experiments?\n\n2. In the discussion of related work, it is crucial to compare the paper's results with state-of-the-art approaches and provide a fair and objective analysis.\n\nLIMITATIONS:\nNo numerical experiments were provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate matrix estimation in reinforcement learning and bandit settings with low-rank structures. They demonstrate the effectiveness of spectral-based methods in recovering matrix subspaces and minimizing entry-wise error. This enables the development of efficient RL algorithms tailored for low-rank bandits and Markov Decision Processes (MDPs).\n\nThis paper is solid, supported by robust results and technically intriguing proofs. It builds upon the recent trend of examining entry-wise error bounds for matrix completion problems and extends it to dynamic settings like RL and bandits. While the guarantees may not be surprising and the algorithms largely follow an explore-and-commit approach, the authors introduce an interesting technique using Compound Poisson Noise to transform a non-i.i.d. problem into an i.i.d. problem, which may have a larger implication for other applications.\n\nSTRENGTHS:\n- Theoretical Soundness\n- Compound Poisson Noise Reduction for MDP is interesting\n- Exploring the potential of entry-wise guarantees of matrix completion in a dynamic setting is in general interesting\n\nWEAKNESSES:\nWhile acknowledging the efforts invested in developing the theory, the novelty of the research appears limited given the well-established techniques of entry-wise recovery and leave-one-out. Moreover, the current algorithms are constrained to explore-and-commit and (pseudo) i.i.d. sampling schemas, compared to adaptive algorithms such as UCB, due to the limitations of the theoretical guarantees.\n\nQUESTIONS:\nHow is the result for matrix completion with Poisson noise in this paper related to \"Near-optimal entrywise anomaly detection for low-rank matrices with sub-exponential noise.\" In International Conference on Machine Learning, pp. 3154-3163. PMLR, 2021?\n\nLIMITATIONS:\nI do not perceive any immediate potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical study of low-rank matrix estimation in three contexts: \n\n**Context 1**: in the case of standard matrix estimation with uniformly sampled entries, where the authors prove a sample complexity comparable to approximate recovery results of $\\tilde{O}(m+n)$ (when the rank is $O(1)$), with the following two main differences compared to existing methods: \n\na (makes the result easier). It is assumed the rank is known, and the function class restriction is based directly on an explicit low-rank assumption. \nb (makes the result much harder and stronger). the error is measured with the maximum norm rather than Frobenius norm or expected loss.  \n\nBecause of the general context of the paper which concerns mostly reinforcement learning (see other results below), these results are formulated as a reinforcement learning/bandit problem where the arms are arranged as pairs of integers and the matrix of rewards is assumed to be low-rank. However, note that independence of the noise components is required (cf. lines 689-693 in the supplementary). \n\n**Context 2.1** \n\nIn this model, recovery guarantees are proved for estimating a low-rank transition matrix from observations of independent transitions. Here, the matrix must be square, with the dimension being equal to the number of states. \n\n\n**Context 2.2** \nIn this more natural setting, we estimate the transition matrix from context 2.2 based only on full trajectories. Regret bounds proved in this case rely on the ergodicity of the Markov chain by splitting the expression of the average error over time into sums where the time steps are separated by a large number $\\tau$, so that the distributions of the corresponding observations are approximately independent and distributed as the limiting distribution. \n\n\n\nFinally, the authors introduce a novel algorithm for performing reinforcement learning in a low-rank bandit setting. The algorithm works in epochs: at a given epoch, with each iteration providing a more and more refined estimate of a set of candidate arms. Whilst the set of candidate arms has more than 2 elements, the algorithm recommends a uniformly random arm, after the set of candidate arms degenerates to a single element, the algorithm keeps recommending that one. A bound is shown on the expected amount of time it takes to identify the best arm. This culminates in a gap-independent regret bound of $\\tilde{O}(T^{1/2})$ for this case, which to the best of my knowledge, solves a highly significant open problem in the nascent theory of low-rank bandits. \n\n\n\nThe proofs rely on an ingenious combination of existing ideas, with the most salient one being that of poison approximation: the matrix of observed rewards is approximated by a matrix with entries equal to $M_{i,j}T_i$ where $T_i$s are independent Poisson random variables. (this general technique has previously been introduced in [7)]. Then, concentration inequalities for such matrices are used in combination with bounds on the error in approximating the true matrix of observed rewards by this Poisson approximation. \n\n==========Post-rebuttal=======\n\nMy doubts have been adequately addressed in the rebuttal and I will keep my score of **strong accept**. I remain convinced this is an excellent paper. However, I do hope the authors will take the trouble to add more detailed derivations and also tone down the statements and claims of novelty over the lack of independence of the $E_{i,j}$. Indeed, since this is an impressive contribution that many readers will be interested in studying, it would be a great service to the community to make the paper as approachable as possible. \n\n\n\n========\n\n\n\n**References**\n[1] Nathan Srebro, \"Rank, Trace-Norm and Max-Norm\", COLT 2005.\n\n[2] Ohad Shamir, Shai Shalev-Shwartz. Matrix Completion with the Trace Norm: Learning, Bounding, and Transducing. JMLR 2014.\n\n[3] Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, Yuling Yan. Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization. SIAM journal of Optimization, 2020.\n\n[4] Prateek Jain, Soumyabrata Pal Online Low Rank Matrix Completion. ICLR 2023\n\n[5] Sahand Negahban, Martin J. Wainwright. Restricted Strong Convexity and Weighted Matrix Completion: Optimal Bounds with Noise. JMLR 2012. \n\n[6] Mitzenmacher, Michael and Upfal, Eli. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge UNiversity Press 2017. \n\n[7] Yuxin Chen, Yujie Chi, Jianqing Fan, Cong Ma et al. Spectral Methods for data science: a statistical perspective.\n\nSTRENGTHS:\n**1** This is an excellent contribution to the field: although I am not completely sure about the originality (especially that of theorem  1 as compared to other results in [5]) since I am not fully up to speed on the reinforcement learning theory, I think there is a good chance this is groundbreaking work. The algorithm of successive arm elimination is simple and elegant, with a nearly optimal regret bound. Whilst it is true that the fact that explicit rank restriction is used (as opposed to, for instance, nuclear norm regularization) makes the matrix completion results seem a little trivial, the regret bound in the RL setting seems to be new and is a natural starting point to kick-start this branch of research. \n\n\n**2** Aside from very minor things (mostly related to the contextualization of the work within the related works), this paper is excellently written. The proofs are very well organized, to an extent which is demonstrably superior to the majority of accepted papers I have reviewed.    The authors clearly organize the proof into steps (Poisson approximation, dilation trick, general concentration inequalities for Poisson matrices, specific calculations for the Poisson approximation to this problem, bounds on the error introduced by the approximation). Each parts of the proofs contain references to existing literature where relevant.\n\nWEAKNESSES:\n**1** One thing that threw me off a bit was the lack of contextualization and explanation of the first results (Theorem 1). In particular, on lines 90 to 92 (see also lines 125 to 130), it seems like the authors want to distance their results from standard matrix completion results by hinting that noise components are not independent (they are not stated to be independent at that point), when in fact it also seems this setting is exactly matrix completion with an explicit rank restriction. Indeed,  lines 689-693 in the supplementary appear to suggest independence **is** required. It would be quite appropriate to compare to existing results in this case as well. For a bounded loss, a similar sample complexity goes all the way back to [1] for approximate recovery with a bounded loss. For connected problems with the trace norm, we also have the result of [2] and the extremely impressive result of [3], which is much stronger than the results presented here due to improved dependence on the variance of the noise. \n\nSimilarly, I am also a bit confused by the statement on line 334 that \"we do not assume the transition matrices are constructed based on a given restricted class of functions\" since it appears there is a low rank assumption in the work. \n\n**2**  Similarly, I feel like the related works on low-rank bandits missed some of the recent literature [4]. \n\n**3** In the low rank bandit result, the algorithm eventually recommends the same arm, that is, the same entry, all the time. This is highly restrictive in a recommender systems context: the algorithm isn't even able to identify the best item for each user, only the best user-item combination...\n\n\n\n\n\n\n\n**References**\n[1] Nathan Srebro, \"Rank, Trace-Norm and Max-Norm\", COLT 2005.\n\n[2] Ohad Shamir, Shai Shalev-Shwartz. Matrix Completion with the Trace Norm: Learning, Bounding, and Transducing. JMLR 2014.\n\n[3] Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, Yuling Yan. Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization. SIAM journal of Optimization, 2020.\n\n[4] Prateek Jain, Soumyabrata Pal Online Low Rank Matrix Completion. ICLR 2023\n\n[5] Sahand Negahban, Martin J. Wainwright. Restricted Strong Convexity and Weighted Matrix Completion: Optimal Bounds with Noise. JMLR 2012. \n\n[6] Mitzenmacher, Michael and Upfal, Eli. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge UNiversity Press 2017. \n\n[7] Yuxin Chen, Yujie Chi, Jianqing Fan, Cong Ma et al. Spectral Methods for data science: a statistical perspective.\n\nQUESTIONS:\n**1** Could you clarify what you mean in lines 90 to 92 and lines 125 to 130 regarding the independence of the noise? \nAlthough I can find such an example in the literature, it is seems unlikely to me that this result  (corollary 1) is new. Is there any similar result for matrix completion with the maximum absolute value of the error as the performance measure? Perhaps in [5]? \n\n**2** Could you clarify to what extent theorem 1 and its proof differs from the existing results and proofs from [7] (what is the most similar result to Theorem 1 which was proved in [7])?\n\n**3** ( I would really like to have the answer to this!) In lines 957 to 958, could you explain clearly ho you arrive at the equation on the bottom of page 33? I am a little stuck there, I need far more details than what is written as it seems you are using another equation from the same page without stating it. Also, did you really mean $\\sigma_{2r}(M)$ in the denominator or did you mean $\\sigma_r(M)$ instead (which is the quantity mentioned in the textual explanation on the line above)?\n\n**4** Could you give a few more details in the page 32? It feels the reference to Lemma 4.16  in other work breaks the reader's momentum, it would be nice to write down the lemma again in the paper. Indeed, I think it is a key step in the argument that allows you to obtain this fascinating result with the max norm (theorem 1). \n\n**5** Line 1098, second equation, do you mean $\\ell+2$ instead of $\\ell-2$ in the exponent\n\n\n**6** In the second to last equation on line 1114 (page 42) should it not be $\\lceil  \\log_2(1/\\Delta_{min})\\rceil$ or $\\log_2(e/\\Delta_{min})$ instead of $\\log_2(1/\\Delta_{min})$? Again, I see that you are not being too careful about constants in general but there is no $O$ notation in this particular inequality. \n\n\n**7** In line 90, you say that the entries are sampled \"uniformly at random for simplicity\", do you in fact require this assumption in the proof of the theorem?  (based on existing results for matrix completion with explicit low-rank assumptions, it would seem that this is not required. In general, it would be nice to add a bit more details to each of the theorem statements to clarify the assumptions. \n\n\n**8** Is Lemma 25 known? It seems very general and standard, I feel like it is probably in some well known book. \n\n\n\n\n\n\n\n\n\n\n\n\n**Minor comments/typos**\n\nA citation would be nice on lines 226-227\n\nIt could be nice to reorganize sections C1 and C2.2 a little bit. For instance, in line 707, you are using a lemma which comes from a subsection that concerns the Markov problem  when proving a result about Model 1, it would be nice to make it clear that Lemma 22 doesn't, in fact, require any proof elements from the context of Model 2. \n\nIn the same vein, I think it would be nice, for completeness, to write down the proof of Lemma 22, even if it is very similar to the proof of Theorem 5.7 in [6]. It is a bit funny that the proof of Lemma 21 is there but not that of Lemma 22 since Lemma 21 is pretty much trivial whilst Lemma 22 is a bit more advanced. Making the paper fully self contained would be a nice way to get even more citations by further improving the encylopaedic quality of the appendix. \n\n\n\nLine 1098 (page 41), second equation, I think you mean $4e$ not $2\\sqrt{e}$, though I understand you are not being too bothered about constants given the final results are in $O$ notation\n\n\nIn the last inequality of the sequence of inequalities on page 43, and in line 1124 I think it would be nice to remind the reader of the definition of $\\bar{\\Delta}$ from line 239, since it is instrumental in deriving the equality on line 1124, which unfortunately relies on the fact that the recommendations are uniformly at random in $[m]\\times [n]$ instead of the more natural $\\mathcal{A}_{\\ell}$.\n\n\n\nTypos: \n\nLine 951: missing \"the\" \n\nLine 706: extra square bracket  inside the equation.\n\nIn line 727, I recommend adding more details for ho you prove that $\\tau(\\epsilon)\\leq \\tau$, it takes quite a bit of massaging of previous equations to get this with the correct constant!\n\nLine 741: be joint probability==> be the joint probability \n\nline 742 \"for Poisson and multinomial model ==> \"for the Poisson and multinomial models\"\n\nline 793 (proof of Lemma 25): \"variables\" ==> \"variable\" \n\nline 957 add \"following\" before \"condition\n\nLine 980, Model\\ref{10} should probably be Model\\eqref{10}\n\nline 1097  (page 41) You say \" in order for the above guarantee to hold we require the following two conditions to hold. I think that is not technically true: the conditions below are what is needed to apply the specific previously established results, so technically those are sufficient rather than necessary conditions. \n\n\nLine 1114 (page 42):  I think there was an error with copy and paste: \"rounds\" should be after $\\tau$ and before the first, not the second comma of the line. \n\nIn the first equality after line 1122, using \\left and \\right might be nicer. \n\n\n[1] Nathan Srebro, \"Rank, Trace-Norm and Max-Norm\", COLT 2005.\n\n[2] Ohad Shamir, Shai Shalev-Shwartz. Matrix Completion with the Trace Norm: Learning, Bounding, and Transducing. JMLR 2014.\n\n[3] Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, Yuling Yan. Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization. SIAM journal of Optimization, 2020.\n\n[4] Prateek Jain, Soumyabrata Pal Online Low Rank Matrix Completion. ICLR 2023\n\n[5] Sahand Negahban, Martin J. Wainwright. Restricted Strong Convexity and Weighted Matrix Completion: Optimal Bounds with Noise. JMLR 2012. \n\n[6] Mitzenmacher, Michael and Upfal, Eli. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge UNiversity Press 2017. \n\n[7] Yuxin Chen, Yujie Chi, Jianqing Fan, Cong Ma et al. Spectral Methods for data science: a statistical perspective.\n\nLIMITATIONS:\nSee \"weaknesses\", especially Weakness 3. In addition, it seems that the algorithm is a little unnatural in that it doesn't use the successive estimates of the set of candidate arms and merely recommends uniformly random entries amongst all arms until it is confident it has identified the best arm overall. It would be interesting to see if it is possible to prove similar guarantees for the algorithm which recommends a random arm in the set $\\mathcal{A}_{\\ell}$ instead of ($\\mathcal{A}_1=[m]\\times [n]$) at each time step $\\ell$.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies two problems in RL involving low-rank matrix estimation. It provides entry-wise estimation error bounds for simple spectral methods in low-rank bandits and low-rank MDP under different sampling mechanisms. Based on these, it provides performance guarantees for two algorithms designed for low-rank bandit and low-rank MDP respectively.\n\nSTRENGTHS:\nThe paper makes contribution towards the estimation of low-rank matrices in the setting of bandits and MDP respectively, as well as how these results can be useful in devising efficient algorithms to make full use of the low-rank structure. \n\nSpecifically, the paper considers the challenging case of low-rank matrix estimation in the presence of dependent noise and provides sharp entry-wise estimation error bounds using involved techniques. This is likely to help the development of algorithms which are better fitted for low-rank settings.\n\nWEAKNESSES:\n1. Compared with standard matrix completion problem, Model I in this paper extends to the case of dependent noise, but it restricts the magnitude of noise terms $\\xi_t$ to be upper bounded by $c_1\\Vert M\\Vert_{\\infty}$ (see Page 3 Line 91). Therefore, the results here can’t reduce to the state-of-the-art results in matrix completion with independent noise, where the noise level can be orderwise larger than $\\Vert M\\Vert_{\\infty}$ [1]. We suggest the authors to discuss why their condition becomes more stringent when the noise are independent.\n\n2. There are a few typos in the paper. For example:\n\nOn Page 2 Line 71, it would be better to avoid using acronyms list “wlog”. \n\nOn Page 7 Line 224, “where in a first phase” —> “where in the first phase” \n\nOn Page 5 Line 175, “is a small when“\n\n[1] Chen, Yuxin, Yuejie Chi, Jianqing Fan, and Cong Ma. \"Spectral methods for data science: A statistical perspective.\" Foundations and Trends® in Machine Learning 14, no. 5 (2021): 566-806.\n\nQUESTIONS:\nOn Page 7 Line 220, it is assumed that the low-rank bandit has an homogeneous rank-$r$ reward matrix M, while this is not mentioned in the corresponding result Theorem 7. This makes the setting kind of confusing. Can you provide more explanations on this?\n\nLIMITATIONS:\nThis paper does not have potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}