Provably Efficient Reinforcement Learning with Linear Function Approximation under Adaptivity Constraints

Abstract: Tianhao Wang, Dongruo Zhou, Quanquan Gu

Reviews and author responses:

The paper considers RL with limited number of batches. They show that they can achieve the same regret as full adaptivity with lower number of batches.   The result seems well explored. The first result reduces the number of batches to achieve the same regret as with T batches. The second result reduces the number of policy switches. As I see, the result is interesting.

The related work is well explained, where the related work to this problem exist in different setups, and thus the overall result is not that surprising. Ofcourse, the detailed proof changes with the problem setup. The detailed technical novelty compared to the related works would have improved the paper.  Yes

 **Q1**: ‘However, the main term $\tilde O(\sqrt{d^3H^3T})$ does not match the lower bound of $dH\sqrt{T}$’.

**A1**: We want to clarify that in the regret bound of LSVI-UCB-Batch, the dominating term should be $dHT/B$ since the budget $B$ is pre-determined and $T=KH$ can go to infinity. In addition, our lower bound suggests that the dependence on $B$ in our upper bound is tight. Furthermore, when choosing $B = \sqrt{T/(dH)}$, our regret upper bound matches the lower bound. 

---

**Q2**: ‘... provide a more general analysis of the batch learning model under other RL algorithms’, ‘the novelty is limited if only the LSVI-UCB algorithm is considered’.

**A2**: In this paper, we only focus on LSVI-UCB-type algorithms under the assumption of linear MDPs which we think is important since this setting is a generalization of the tabular MDP. We believe that the key idea in the algorithm design and analysis developed in this paper is generic and can be applied to other UCB-type algorithms in RL and other kinds of MDPs (e.g., linear mixture MDPs [Ayoub et al. 2020]). Our results take an initial step on adaptivity constraint problems in RL beyond the tabular setting and shed light on the algorithmic design and analysis under more general assumptions, e.g., RL with general function approximation [Wang et al. 2020].

 **Q1**: ‘It would be good to have some extra experiments showing how the proposed algorithms perform under different levels of model misspecification.’

**A1**: Thank you for the suggestion! We have conducted the following experiments under different levels of model misspecification. In particular, based on the linear MDP constructed in Example 6.1, we follow the definition of $\zeta$-approximate linear MDPs in [Jin et al. 2020], and consider a corrupted transition given by $\mathbb{P}_h(s’|0,a) = (1-\zeta(a))\phi(0,a)^\top \mu_h(s’) + \zeta(a)\cdot \mathbf{1}(s’=g(a))$, where $\zeta(\cdot): \mathcal{A}\to[0,\zeta]$ and $g:\mathcal{A}\to\mathcal{S}$. These two additional functions can be constructed by random sampling before running the algorithms, and the magnitude of $\zeta\in(0,1)$ characterizes the level of model misspecification. All the other components of the model remain the same. 
Under this misspecified model with different levels $\zeta=0.05,0.1,0.2,0.4$, we have run LSVI-UCB-Batch with $B=50$ and LSVI-UCB-RareSwitch with $\eta=8$ respectively. Our preliminary experiments are presented in the following two tables where we report the average regret at episodes $K=500,1000,1500,2000,2500$ (aggregated over 50 independent rounds):

| LSVI-UCB-Batch, $B=50$ | K=500   | K=1000  | K=1500  | K=2000  | K=2500  |
| ---------------------- | ----- | ----- | ----- | ----- | ----- |
| $\zeta=0$              | 1.186 | 0.801 | 0.598 | 0.481 | 0.406 |
| $\zeta=0.05$           | 1.230 | 0.851 | 0.643 | 0.522 | 0.441 |
| $\zeta=0.1$            | 1.258 | 0.869 | 0.658 | 0.540 | 0.459 |
| $\zeta=0.2$            | 1.362 | 1.008 | 0.784 | 0.648 | 0.557 |
| $\zeta=0.4$            | 1.565 | 1.255 | 1.016 | 0.865 | 0.766 |

and

| LSVI-UCB-RareSwitch, $\eta=8$ | K=500   | K=1000  | K=1500  | K=2000  | K=2500  |
| ----------------------------- | ----- | ----- | ----- | ----- | ----- |
| $\zeta=0$                     | 1.004 | 0.750 | 0.613 | 0.527 | 0.471 |
| $\zeta=0.05$                  | 1.068 | 0.828 | 0.684 | 0.592 | 0.531 |
| $\zeta=0.1$                   | 1.128 | 0.889 | 0.749 | 0.650 | 0.578 |
| $\zeta=0.2$                   | 1.238 | 1.011 | 0.874 | 0.780 | 0.703 |
| $\zeta=0.4$                   | 1.447 | 1.258 | 1.143 | 1.064 | 1.002 |

Note that we also include the well-specified model ($\zeta=0$) which corresponds to the plots we have already shown in the main context. These preliminary results verify the effectiveness of our algorithms under considerable levels of model misspecification. We will conduct more experiments and include corresponding plots in the revision.

---

**Q2**: ‘technique novelty and contribution’.

**A2**: First, for the batch learning model, when controlling the bonus sum, our proof is simpler than that in Section 3.3 in [Han et al. 2020]. Also, the corresponding analysis for the rare policy switch model shares the same strategy, thus we have proposed a unified framework to conquer adaptivity constraints for UCB-type algorithms in RL. Our analysis is succinct yet generic, and we believe that such ideas can be applied to more general RL settings, e.g., RL with general function approximation [Wang et al. 2020].

Moreover, compared with the bandits problem studied in [Han et al. 2020] and [Abbasi-Yadkori et al. 2011], which is a special case of finite-horizon MDPs with $H=1$, it requires non-trivial effort in terms of the algorithmic design since each stage $h$ can trigger the policy update. One needs to carefully check the dependence structure when applying the martingale concentration inequality, otherwise the argument could easily fail. We believe it is still valuable to generalize the previous works on adaptivity constraints from traditional bandits setting to RL. We take an initial step in this direction for RL with linear function approximation, and we hope this will help advance the community’s knowledge for more complex models.

---

**Q3**: ’... provide high-level ideas regarding how to get a lower bound for the rare policy switch model (or some reasonable conjectures of how it should look like)’

**A3**: For the rare policy switch model, we conjecture the regret lower bound to be $\tilde\Omega(\sqrt{d^3H^3T [1+T/(dH)]^{dH/B}})$. We would like to point out that one possible way to establish this kind of lower bound is by adapting the construction of the hard-to-learn instance for stochastic linear bandits proposed in [Ruan et al., 2020]. Yet the problem of this construction is that it requires the action set to be pre-fixed and change from episode to episode, which is not true in the standard linear MDP setting that we consider here. Therefore, we did not include this part due to the mismatch between the assumptions. We will add these remarks in the revision. If the reviewer thinks the lower bound result in a slightly different setting is still helpful, we will include the complete statement and proof of this lower bound in the revision.


 Thank you for your positive comments!

 **Q1**: ‘For the LSVI-UVB-RareSwitch, it would be better if you could explain the bound on the adaptivity in terms of $B$ and the role of $B$.’

**A1**: Thank you for your suggestion. We have discussed this in Remarks 5.3 and 5.4.  We will add more discussion on the regret bound of LSVI-UCB-RareSwitch in terms of $B$ in the revision.

---

**Q2**: ‘When specialized to the tabular setting, the provided regret bounds both scale poorly in $|\mathcal{S}|,|\mathcal{A}|$, ..., but it is worth remaking this issue’

**A2**: We agree that if we specialize the linear MDP to tabular MDP using the one-hot feature mapping (see Example 2.1 in [Jin et al. 2020]), then the dependence on $|\mathcal{S}|$ and $|\mathcal{A}|$ would be unsatisfactory, yet we think this is caused by the fact that the one-hot feature mapping is not compact enough. In fact, even if specializing the regret bound of (fully adaptive) LSVI-UCB to tabular MDPs, its regret will suffer from the same loose dependence on $|\mathcal{S}|$ and $|\mathcal{A}|$. We will comment on this issue in the revision.

---

**Q3**: ‘In practice, $B$ is often the constraint that is fixed and we are free to choose algorithm parameters, e.g. $T$. It would be useful if you could add comments on how to select algorithm parameters like $T$. What does the PAC bounds look like?’

**A3**: Recall that our regret bound is given by $\tilde O(\sqrt{d^3H^3T} + dHT/B)$. If we consider the corresponding PAC bounds, then for any $\epsilon>0$, it requires that $\tilde O(\sqrt{d^3H^3/T} + dH/B) \leq \epsilon$, which implies that $T\geq \tilde\Omega(d^3H^3/\epsilon^2)$ and $B\geq \tilde\Omega(dH/\epsilon)$. Therefore, to get a meaningful PAC bound, we require both $T$ and $B$ to be sufficiently large. This is reasonable since for a fixed $B$, when $T$ is too large, the dominant term in the regret bounds (both the upper and lower bound) is $dHT/B$ which scales linearly in $T$. We will add these comments in the revision.

---

**Q4**: ‘Did you try other schemes adjusting $B$ like diminishing $B$ as the learned policy goes to be near-optimal?’

**A4**: This may be a misunderstanding. Note that $B$ is the number of batches rather than the size of each batch.  So diminishing $B$ does not make sense. We guess what you mean is diminishing batch size. This is actually related to our second algorithm LSVI-UCB-RareSwitch, where the number of policy switches is stochastic and the size of each batch is adaptive (depending on the data). Also, by the design of the algorithm, indeed the agent would switch policies less often as the learned policy goes to near-optimal (in this case, the batch size is actually not diminishing, but increasing).

---

**Q5**: ‘The authors also remark on a similar work [3] that could be a case of their results. It would be helpful for readers to differentiate them if you could add more comments on the technical side.’

**A5**: To compare our work with [Gao et al., 2021], we would like to remark that their main algorithm is a special case of our LSVI-UCB-RareSwitch by setting $\eta$ to be roughly a constant. Our analysis is more general yet succinct. Moreover, they did not study the batch learning model which we think is also of important practical interest. We will add more detailed comments in the revision as you suggested.


The paper studies learning episodic finite-horizon MDPs featured by the policy adaptivity, in the linear function approximation setting. The authors consider two models of adapting policies: batch learning case and rare policy switch, propose two LSVI-UCB algorithms with policy adaptivity by generalizing the well-known LSVI-UCB algorithm and prove their regret bounds. The proposed algorithms and regret bounds generalize the well-known LSVI-UCB algorithm to have policy adaptivity. The authors also verify the effectiveness via computational experiments. 

The main contribution relies on the proposed two LSVI-UCB algorithms with added policy adaptivity and their regret guarantees.   The paper is well written. The main results are well explained and it is easy to follow proof steps. Regarding quality and clarity, I only have a few remarks. 

(1). It is nice to match/improve the regret bound for the LSVI-UCB algorithms constrained by the policy adaptivity. For the LSVI-UVB-RareSwitch, it would be better if you could explain the bound on the adaptivity in terms of $B$, and the role of $B$.  

(2). When specialized to the tabular setting, the provided regret bounds both scale poorly in $S$, $A$, if we compare them with [1, 2]. This seems to be the state of the art, but it is worth remaking this issue. 

(3). As shown in Theorem 4.1 and Theorem 5.1, the regret bounds match the LSVI-UCB's regret with full adaptivity when $B=O(\sqrt{ T })$. In practice, $B$ is often the constraint that is fixed and we are free to choose algorithm parameters, e.g. $T$. It would be useful if you could add comments on how to select algorithm parameters like $T$. What does the PAC bounds look like?

(4). It is nice to see the effect of batch size $B$ on the regret in the computational experiment. Did you try other schemes adjusting $B$ like diminishing $B$ as the learned policy goes to be near-optimal? This would add more flexibility of algorithms so that practitioners like.     

Originality and Significance. The proposed LSVI-UCB algorithms are featured by adaptively switching policies. The method is new in the function approximation setting and the proofs offer a way to deal with samples from adaptively switched policies. The authors also remark a similar work [3] that could be a case of their results. It would be helpful for readers to differentiate them if you could add more comments on the technical side. 


References:

[1] Provably Efficient Q-Learning with Low Switching Cost. NeurIPS. 2019.

[2] Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decomposition. NeurIPS. 2020.

[3] A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Cost. arXiv:2101.00494. Yes.

This paper examines two variants of the LSVI-UCB algorithm in limited adaptiveness, which is the case where policies cannot be updated continuously. To deal with the limited adaptivity, the authors consider two models and proposes two algorithms, respectively. The LSVI-UCB-Batch algorithm corresponds to the batch learning model where we update the policy every $K/B$ epochs. The authors prove a $\tilde{O}(\sqrt{d^3 H^3 T} + d H T/B)$ regret and a $\Omega(dH \sqrt{T} + d H T/B)$ regret lower bound under this scenario. The LSVI-UCB-RareSwitch algorithm corresponds to the rare policy switch model where we update the policy when $det(\Lambda_h^k) > \eta det (\Lambda_h)$. The authors prove a $\tilde{O}(\sqrt{d^3 H^3 T[1 + T/(dH)]^{dH/B}})$ regret for this case. Experimental results in this paper show that the performance of both algorithms is comparable to that of the LSVI-UCB algorithm, with a substantially reduced number of policy switches.

  
In Theorem 4.2, by comparing the regret lower bound with the obtained bound of LSVI-UCB-Batch in Theorem 4.1, the authors claim that the dependency of the total regret on the number of batches for the batch learning model is optimal. However, the main term $\sqrt{d^3 H^3 T}$ does not match the lower bound of $d H \sqrt{T}$. It would be helpful if the authors could provide a more general analysis of the batch learning model under other RL algorithms, especially those that match the lower bound.

The focus of this paper is the adaptivity constrained model. However, all the two algorithms proposed are based on the LSVI-UCB algorithm proposed in Jin et al. (2020). Theorem 4.1 of this work extends Theorem 3.1 in Jin et al. (2020) by comparing the difference between $\Lambda_h^{b_k}$ and $\Lambda_h^{k}$ in Lemma A.3. Theorem 5.1 of this work further combines the idea of Lemma B.2 (Lemma 12, Abbasi-Yadkori et al. 2011).

This work provides a new aspect of the LSVI-UCB algorithm, but the novelty is limited if only the LSVI-UCB algorithm is considered.

 The authors mentioned in their paper that the LSVI-UCB-RareSwitch algorithm does not have a lower bound result yet.

This paper considers online episodic RL with linear function approximation under adaptivity constraints. In particular, two types of limited adaptivity models are considered:
- Batch learning model, where the agent is allowed to update the policy at $B$ pre-specifed batch grids. Using a batched version of LSVI-UCB we can achieve $\tilde{O}(\sqrt{d^3 H^3 T} + dHT/B)$ regret with $B$ uniform batch grids, so $\sqrt{T/dH}$ batches are sufficient. A lower bound is provided which shows the tight dependency on $B$.
- Rare policy switch model, where the agent is allowed to switch policy ($i.e., \pi^k \ne \pi^{k-1}$) for at most $B$ times. Using a modified version of the rare switching OFUL we can achieve $\tilde{O}(\sqrt{d^3 H^3 T[1+ T/(dH)]^{dH/B}} )$ regret with $B$ switches, so $dH \log T$ switches are sufficient. No lower bound is provided in this setting.

The authors test out their algorithm with the hard-to-learn linear MDP (Zhou et al. 2021b) and the numerical results match theoretical gurantees.


  All references here can be found in the original paper.

### Originality
Online learning with adaptivity constraints has been popular recently, but ideas in this paper does not seem too novel to me. 
-  Under the batch learning model, the LSVI-UCB-Batch algorithm is literally LSVI-UCB with Q-function updates only at pre-specified batch grids. The analysis for the upper bound mostly combines techniques from Jin et al. 2020 (to control the LSVI-UCB estimation quality) and Abbasi-Yadkori et al. 2011 (to control the sum of bonus terms). The message that we can use uniform batch grids is somewhat interesting, but it's quite predictive from Han et al. 2020 that uses uniform batches in SBUCB.
- Under the rare policy switch model, the LSVI-UCB-RareSwitch uses the determinant based criterion from Abbasi-Yadkori et al. 2011 to determine when to switch the policy. My understanding is that they basically changed the OFUL updates in the rarely switching OFUL algorithm (Abbasi-Yadkori et al. 2011) to LSVI-UCB.  Also, in my opinion, providing a reasonable lower bound in this setting could be the most novel contribution, but unfortunately the authors defer that to future work.

### Quality
Overall this paper tells a clean and complete story (except missing the lower bound for the rare policy switch model) with all arguments well supported.
- Theory. I briefly checked the proofs in the appendix (which is generally well-orginized), and the high-level ideas look correct to me.
- Experiments. It is good to see that the numerical experiments matches what the theory predicts (when the linear MDP assumption holds). It would be good to have some extra experiments showing how the proposed algorithms perform under different levels of model misspecification.

### Clarity
The paper is clearly written and well organized. It's very easy to follow, and provides enough information to reproduce experimental results.

### Significance
RL under adaptivity constraints is a relevant problem in many practices, so I agree that the problem studied here can be valuable. However, as a theoretical paper, its technical contributions look somewhat limited. Also, since the paper assumes linear MDP throughout and it often does not hold in practice, the current experiment results may not be strong enough to convince practitioners that the batch schedules / policy update criterion's work well in practice. To reiterate some concerns I listed in Main Review:
- I'm mainly concerned with the theoretical novelty of this paper. It would be good if the authors could 1) explicitly explain the novelties in their proof techniques compared to existing work, and 2) provide high-level ideas regarding how to get a lower bound for the rare policy switch model (or some reasonable conjectures of how it should look like).
- Since the paper assumes linear MDP throughout and it often does not hold in practice, it should be good to either 1) discuss how misspecification can affect the regret rate (Like Theorem 3.2 in Jin et al. 2020), or 2) provide some extra numerical experiments to show how the algorithms perform under misspecification.

Meta-review: The paper studies reinforcement learning in linear MDPs when the agent is limited in how frequently it can change its policy (e.g., the rarely-switching setting). This setting has been well-developed in simpler problems like online learning and linear bandits, and the theoretical results are what one would expect. The main concern is that the technical novelty is fairly low here, the paper seems to combine known techniques for linear MDPs, with techniques from the bandit literature on limited adaptivity. As such, we recommend rejecting the paper. 

Decision: accept