Title: The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition
Authors: Tiancheng Jin
Longbo Huang
Haipeng Luo
Venue: NeurIPS 2021
Keywords: reinforcement learning
online learning
best of both worlds

Abstract: We consider the best-of-both-worlds problem for learning an episodic Markov Decision Process through $T$ episodes, with the goal of achieving $\widetilde{\mathcal{O}}(\sqrt{T})$ regret when the losses are adversarial and simultaneously $\mathcal{O}(\log T)$ regret when the losses are (almost) stochastic. Recent work by [Jin and Luo, 2020]  achieves this goal when the fixed transition is known, and leaves the case of unknown transition as a major open question. In this work, we resolve this open problem by using the same Follow-the-Regularized-Leader (FTRL) framework together with a set of new techniques. Specifically, we first propose a loss-shifting trick in the FTRL analysis, which greatly simplifies the approach of [Jin and Luo, 2020] and already improves their results for the known transition case. Then, we extend this idea to the unknown transition case and develop a novel analysis which upper bounds the transition estimation error by the regret itself in the stochastic setting, a key property to ensure $\mathcal{O}(\log T)$ regret.

TL;DR: We propose algorithms to achieve best-of-both-worlds results of learning episodic MDPs. 

Review discussion:
In some sense, the contribution may be seen as a minor improvement from (Jin and Luo 2020) for known transition case.
In fact, maybe one can first run reward-free exploration schemes to learn transition kernels and then execute any known algorithms for best-of-both-worlds. It would result in O(T^{2/3})-regret nevertheless.
Hence getting $O(\sqrt{T})$ and $O(polylog(T))$ regrets in adversarial and stochastic cases respectively does not seem to be so trivial.
On the other hand, analysis seems not particularly novel but a variation of FTRL method (e.g., Jin et al. 2020) and logarithmic-regret for MDPs (e.g., Simchowitz and Jamieson 2019).
Nevertheless, they are well-adapted to a challenging problem of achieving best of both worlds with unknown transition kernels.
Overall the paper is well-written and easy to follow. Comparisons to previous work are well discussed whenever needed. Main results and analysis are well-presented, hence I recommend the acceptance for this paper.
There are some points I want to be clarified.
- In Section 3, the main idea seems to add additional term (loss-shifting?) $g_\tau$.
However, I do not see what conceptually plays the role of $g_\tau$ in Algorithm 1. (Eq (11) is, in my interpretation, simply an adjusted loss with standard exploration bonus). Could you be more specific about the connection?
- Line 231-234: I do not understand what it does mean by "This framework, however, does not allow the loss-shifting trick discussed in Section 3" (or why it has to be).
- Line 293-294 "In fact, ... complicated": at least, could you explain which improvement you have or can be made?
I also raise some other questions on the analysis
- Could you be more specific on your technical novelty? In which part of the analysis is different from (Jin et al. 2020) for adversarial case?
- How is the regret decomposition for the stochastic case different from (Simchowitz and Jamieson 2019)? (or are they the same?)
- Line 345-346 "In fact, the reason ... terms":  I don't understand what you mean by $Reg_T(\pi^o)$ cannot be directly bounded - isn't your theorem bounds it by $O(\sqrt{T})$?
Minor Comments
- Line 259: "in the bandit case" -> do you mean in the bandit-feedback case?
- Line 137: $\Delta(s,a)$ is not defined before used here.
- Line 177: "this natural and simple candidate" - do you mean the one used in (Zimmert and Seldin 2019)?
No
We thank the reviewer for the valuable feedback.
1\. *maybe one can first run reward-free exploration schemes to learn transition kernels and then execute any known algorithms for best-of-both-worlds.*
We want to emphasize that this completely defeats the purpose of obtaining best-of-both-world guarantees, as it results in $T^{2/3}$ regret for both the adversarial setting and the stochastic setting.
2\. *In Section 3, the main idea seems to add additional term (loss-shifting?) However, I do not see what conceptually plays the role of in Algorithm 1. (Eq (11) is, in my interpretation, simply an adjusted loss with standard exploration bonus). Could you be more specific about the connection?*
The loss shifting technique does not require actually adding the loss-shifting function explicitly in the algorithm (as shown in Eq.(5)). This is only an analytic tool that enables us to apply several useful properties (e.g., Lemma A.1.3 for the known transition setting), which is critical to achieve the best-of-both-worlds guarantee.
3\. *Line 231-234: I do not understand what it does mean by "This framework, however, does not allow the loss-shifting trick discussed in Section 3" (or why it has to be).*
As we explain immediately after that sentence, our loss-shifting function is defined with respect to a fixed transition, and the important invariant Eq.(5) (or Corollary A.1.2) would not hold if we use the standard framework which performs FTRL over the larger set $\Omega(\mathcal{P}_i)$.
In other words, the standard method does not allow analyzing the regret using the imaginary additional loss-shifting function, forcing us to come up with a different framework.
4\. *Line 293-294 "In fact, ... complicated": at least, could you explain which improvement you have or can be made?*
We are able to reduce the dependency of L for the stochastic world by carefully separating the state-action pairs into those that are visited often enough and those that are not.
5\. *Could you be more specific on your technical novelty? In which part of the analysis is different from (Jin et al. 2020) for adversarial case? How is the regret decomposition for the stochastic case different from (Simchowitz and Jamieson 2019)? (or are they the same?)*
We reiterate our technical novelty stated in Lines 40-71 (and explained further in other sections): First, we propose a novel loss-shifting trick for the FTRL analysis for MDPs with fixed transition, which greatly simplifies the approach of Jin and Luo (2020) for the known transition case and lays the foundation for the analysis of unknown transition.
Second, we design a framework for the unknown transition under adversarial losses that enables the application of the same loss-shifting trick.
Third, we develop an analysis to show that in the stochastic world the transition estimation error of our algorithm is only polylogarithmic in T with the help of a new regret decomposition (Lemma D.1.1 and Corollary D.1.2) and several self-bounding terms (Appendix D.2).
For the adversarial world, although our algorithm is different from Jin et al. (2020), the analysis is very similar to theirs, especially for bounding the ERR_1 term as we point out in Lines 310-311.
However, the key challenge is on handling to the stochastic world.
Although our regret decomposition is inspired by Simchowitz and Jamieson (2019), they are not the same. Our decomposition (Lemma D.1.1) holds generally and is agnostic to specific algorithms, while theirs is very specific to the so-called "strongly optimistic" algorithms (see their Appendix D).
In fact, they do not really have an equality decomposition but only an upper bound after applying a certain clipping trick. Our decomposition does not rely on the clipping trick, and instead makes use of a novel quantity $q_\pi^\star(s,a)$, the probability of visiting a trajectory of the form $(s_0, \pi^\star(s_0)), (s_1, \pi^\star(s_1)), \ldots, (s_{k(s)-1}, \pi^\star(s_{k(s)-1})), (s,a)$ when executing policy $\pi_t$.
6\.*Line 345-346 "In fact, the reason ... terms": I don't understand what you mean by $\text{Reg}_T(\mathring{\pi})$ cannot be directly bounded - isn't your theorem bounds it by $\mathcal{O}(\sqrt{T})$?*
Sorry for the confusion. Here, we mean that $\text{Reg}_T(\mathring{\pi})$ cannot be bounded  by $\text{polylog}(T)$ using our proposed techniques.
7\. *Line 259: "in the bandit case" -> do you mean in the bandit-feedback case?*
Yes.
8\. *Line 137: $\Delta(s,a)$ is not defined before used here.*
Actually, $\Delta(s,a)$ is defined in Condition (1) already.
9\. *Line 177: "this natural and simple candidate" - do you mean the one used in (Zimmert and Seldin 2019)?*
Yes.
10\. *Do you think the regret bound can be improved to $O(\log T)$ in the stochastic cases in (Simchowitz and Jamieson2019)?*
We believe that this is possible, but we do not know how to achieve this at the moment. The extra $\log(T)$ factor of our bound comes from the fact that we run $\log(T)$ independent instances of FTRL with a fixed transition, which is currently crucial for our loss-shifting technique.
11\. *Can you improve the regret bound by a factor of $S$ in the adversarial case?*
Note that our bound is only $\sqrt{S}$ (not $S$) away from the best lower bound. Please see more discussions on this in our second response to reviewer o6kk.
This work is an improvement compared to previous works on several fronts.
First, in the setting with fixed transitions, this new algorithm and its analysis are both more efficient and simpler to study.
In the setting with unknown transition, the results are novel.
Overall, the framework provided is very complete, as they study episodic MDPs in a Best of Both world setting, both with known and unknown transitions and both with bandit and full information feedback.
The algorithm proposed is part of the FTRL framework, where they use some  existing tools, such as the self-bounding technique as well as new tools such as the Loss-Shifting Technique, which is efficient and easier to analyse than the previously existing algorithm from Jin and Luo (2020).
The paper is quite well written, and the proofs are extensively detailled in the appendix. The amount of material presented is important for the length of the paper.
A conclusion that discuss possible extensions would be a useful addition.
No
We thank the reviewer for the valuable feedback.  We will incorporate your minor comments on writing in the next version.
Significance: Best of both worlds results are desirable and have received a good amount of attention in the RL / bandit theory community. This paper significantly advances the state of the art in this area by showing the first such results in MDPs with unknown transitions.
Besides the actual regret bound, I think the proof techniques in this paper will be useful in other settings as well. I particularly liked the simpler analysis using shifted losses compared to prior works.
Clarity: The paper is very well written, illustrating the main technical challenges and clearly explaining how they were overcome. I think the paper does a particularly good job at sketching the intuition in the main paper while providing the verbose arguments with all details in the appendix.
Originality: Many of the technologies used in this paper have been used elsewhere before, including loss shifting, using phases where the empirical model is kept fixed and adding a reward bias to account for uncertainty in transition estimates (e.g. in UCBVI algorithms). I still think that the combination and adaptation of these tools to the setting addressed here is technically highly nontrivial and sufficiently novel.
Quality: I checked several technical lemmas in the proofs and did not find any issues. The main arguments also make sense to me, but I did not go through all lemmas in the appendix carefully.
Minor comments and typos:
- One log(T) factor seems to come from the fact that the algorithm runs log(T) independent instances of FTRL. If one could avoid that, would that allow us to achieve a log(T) instead of log^2(T) bound or are there other places that contribute the additional log-factor?
- I understand that it is not the main focus of the paper to push the best results for either case, adversarial or stochastic, but I wonder whether the techniques developed in this paper can shed light on the additional sqrt(S) / S factor in the presented upper-bounds compared to the lower bounds when transitions are stochastic & unknown?
- Line 342: challenging -> challenges
No
We thank the reviewer for the valuable feedback.
1\. *One $\log(T)$ factor seems to come from the fact that the algorithm runs $\log(T)$ independent instances of FTRL. If one could avoid that, would that allow us to achieve a $\log(T)$ instead of $\log^2(T)$ bound or are there other places that contribute the additional log-factor?*
Indeed, the fact that we run $\log(T)$ independent instances of FTRL is the only source of the extra $\log(T)$ factor in our regret bound. It is unclear to us how to get rid of it though, since having a fixed transition in each epoch is critical to our loss shifting technique.
2\. *I understand that it is not the main focus of the paper to push the best results for either case, adversarial or stochastic, but I wonder whether the techniques developed in this paper can shed light on the additional $\sqrt{S}$ / $S$ factor in the presented upper-bounds compared to the lower bounds when transitions are stochastic & unknown?*
Unfortunately, no (or at least not that we are aware of). Our algorithm uses the same Bernstein-style confidence interval as UOB-REPS (Jin et al., 2020), and we at best achieve the same bound as theirs for the adversarial environment, which is $\sqrt{S}$ away from the best lower bound unfortunately. We in fact conjecture that the upper bound is tight already, given the lower bound of the reward-free exploration setting (see Jin et al. "Reward-Free Exploration for Reinforcement Learning"), which shows unavoidable extra $S$ dependence when the learner has little knowledge on the future reward function of interest.
Thank you for responding to my questions. This is very interesting work!
The paper is very well-written and studies an interesting problem. It provides new techniques:
- The choice and use of a shifting function. It is a very nice solution.
- The idea to move the use of exploration bonus from the search of the occupancy measure for the confidence set of transitions
to the alternative solution of substructing the exploration bonus from the loss. This trick looks potentially useful in other applications.
Remarks:
- The statement of lines 167 - 170 and equation (7), and later reasoning on the same topic at lines 295-300 suffer from lack of clarity. Reference to a more detailed explanation in the appendix is needed and also reformulate a thought more clearly.
- (1) comes initially from best of both worlds analysis for bandits.
- The message of lines 231-234 is not very clear. The shifting function as well can be computed using \mathcal{P}_i.
- It is not very clear why the best of both world problem in the full information setting requires special attention. The stochastic regime with full information is not very challenging.
No
We thank the reviewer for the valuable feedback
1\. *The statement of lines 167 - 170 and equation (7), and later reasoning on the same topic at lines 295-300 suffer from lack of clarity. Reference to a more detailed explanation in the appendix is needed and also reformulate a thought more clearly.*
Thanks for the suggestion. We will clarify these in the next version.
2\. *Eq.(1) comes initially from best of both worlds analysis for bandits.*
Indeed. We will add the reference.
3\. *The message of lines 231-234 is not very clear. The shifting function as well can be computed using $\mathcal{P}_i$.*
Our shifting function is defined with respect to one fixed transition, so we assume what the reviewer means is that we perform FTRL over $\mathcal{P}_i$, whose solution corresponds to one transition in $\mathcal{P}_i$, and then we define the shifting function using this particular transition. This does not work as it does not ensure the invariant condition Eq. (5) (when $\bar{P}$ is changed to $\mathcal{P}_i$). Please also see Corollary A.1.2. for more details on this invariant.
4\. *It is not very clear why the best of both world problem in the full information setting requires special attention. The stochastic regime with full information is not very challenging.*
We agree that the stochastic regime with full information is not very challenging.
However, for the corrupted stochastic setting (which we cover), it is harder to identify the nature of world, and achieving similar guarantees as our Theorem 4.1.1. does not appear to be easy.

Meta-review:
role: PC
data: comment: This paper has been very well-received by the reviewers, who all appreciated the significance of the contribution and the high quality of both the writing and the technical content. This is an excellent paper that clearly deserves to be published at the conference.

Decision: Accept (Oral)