Title: Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithms
Authors: Fan Chen
Yu Bai
Song Mei
Venue: ICLR 2023
Keywords: reinforcement learning theory
POMDPs
predictive state representations
partially observable reinforcement learning

Abstract: Partial Observability---where agents can only observe partial information about the true underlying state of the system---is ubiquitous in real-world applications of Reinforcement Learning (RL). Theoretically, learning a near-optimal policy under partial observability is known to be hard in the worst case due to an exponential sample complexity lower bound. Recent work has identified several tractable subclasses that are learnable with polynomial samples, such as Partially Observable Markov Decision Processes (POMDPs) with certain revealing or decodability conditions. However, this line of research is still in its infancy, where (1) unified structural conditions enabling sample-efficient learning are lacking; (2) existing sample complexities for known tractable subclasses are far from sharp; and (3) fewer sample-efficient algorithms are available than in fully observable RL.

This paper advances all three aspects above for Partially Observable RL in the general setting of Predictive State Representations (PSRs). First, we propose a natural and unified structural condition for PSRs called \emph{B-stability}. B-stable PSRs encompasses the vast majority of known tractable subclasses such as weakly revealing POMDPs, low-rank future-sufficient POMDPs, decodable POMDPs, and regular PSRs. Next, we show that any B-stable PSR can be learned with polynomial samples in relevant problem parameters. When instantiated in the aforementioned subclasses, our sample complexities improve substantially over the current best ones. Finally, our results are achieved by three algorithms simultaneously: Optimistic Maximum Likelihood Estimation, Estimation-to-Decisions, and Model-Based Optimistic Posterior Sampling. The latter two algorithms are new for sample-efficient learning of POMDPs/PSRs.
We additionally design a variant of the Estimation-to-Decisions algorithm to perform sample-efficient \emph{all-policy model estimation} for B-stable PSRs, which also yields guarantees for reward-free learning as an implication.

TL;DR: We propose a unified structural condition for sample-efficient partially observable RL (POMDPs/PSRs), and establish substantially sharper learning results than existing ones.

Review discussion:
Strengths: the paper is clearly written, gives a detailed account of related work and necessary background, and introduces a new property, B-stability, which is intuitive and possessed by several important classes of partially observable RL problems. This opens up directions for research related to details of proposed algorithms, learning of B-stable PSRs, and computational issues.
Weaknesses: the paper is of theoretical nature, and as such relies on definitions and proofs of theorems related to B-stability and algorithms. The proofs are not a part of the body of the paper but rather of the appendices, and the full length to be read to understand and check correctness is in excess of 40 pages. I tried to dive into the appendices but didn't have enough time and cannot confirm that the proofs are correct.  My humble opinion is that the paper, with all the contributions which it claims, is perhaps to long for a conference, either the reviewing process or presentation later on.
The paper is clear and well organized. The contribution is incremental  --- which by no means a negative comment: building on the body of previous work is, in my opinion, an advantage of this research.
However, since the paper is theoretical and most of the results can only be evaluated by careful verification of proofs in the appendices, 'reproducibility' is questionable (is reproducibility of a theoretical paper means being able to prove the results from the paper alone?)
NO.
We thank the reviewer for the valuable feedback to our paper. We respond to the main concerns as follows.
*--- “The proofs are not a part of the body of the paper but rather of the appendices, and the full length to be read to understand and check correctness is in excess of 40 pages”*
We agree our results are theoretical in nature and require all the detailed proofs in the appendix. We have included a one-page “Overview of Techniques” in Appendix B, which outlines the main steps of the proofs (including their improvements over existing techniques) and gives pointers to the key lemmas. (We would have loved to include that in the main text if there was one more page for the space limit :) A shorter version of this overview is presented at the bottom of Page 7, after Theorem 9.
We hope those could at least help the readers get a gist of the proof steps as well as their relationship with existing work, and further help save some time for reading the actual proofs.
This paper provides a unified theoretical framework for online learning in PSRs. The proposed framework, called B-stability, subsumes several known tractable subclasses of models in the literature. I have not read through all details of the proof.
However, the PAC results in Theorem 9 seem reasonable. Given the importance of latent representation learning and the relevance of partially observable RL, the unified theoretical framework in this paper could have an impact across different fields of machine learning.
As for the weakness, this paper is certainly notation heavy.
While some of them might be inevitable given the technical nature of this paper, some of the notations are not well defined.
For instance, it would be appreciated if the authors could further elaborate on Step 6 of OMLE. It is unclear how the exploratory policy is being set.
This paper is mainly theoretical. The main results seem technically sound. Detailed proofs are provided in Appendix. The idea of developing a unified theoretical framework for online learning in PSRs is certainly novel, and could have impact across disciplines in machine learning.
NO.
We thank the reviewer for the valuable feedback to our paper. We respond to the main concerns as follows.
*--- “For instance, it would be appreciated if the authors could further elaborate on Step 6 of OMLE. It is unclear how the exploratory policy is being set..”*
The exploration policy $\pi_{h,{\rm exp}}^k$ appearing in Step 6 is defined via the $\circ_h$ notation as follows: Take $\pi^k$ for the first $h-1$ steps, take a uniform action ${\rm Unif}(\mathcal{A})$ at step $h$, take an action sequence sampled from ${\rm Unif}(\mathcal{U}_{A,h+1})$ at step $h+1$, and behave arbitrarily afterwards. This definition was presented in the text “2. (Data collection)” after Algorithm 1.
To highlight this definition and avoid confusions, we have slightly changed the text to highlight where we define $\pi_{h,{\rm exp}}^k$ (together with the $\circ_h$ notation) in our revision.
*Strength*
- The paper is very comprehensive -- the framework is quite general and can imply many previous results with improved upper bounds.
- It is interesting to see the connections with DEC.
*Weaknesses*
- In a current form, the paper sounds in nature too similar to (Zhan et al., 2022). It would have been better to position the paper by, for example, focusing more on the technical innovations to get better sample-complexity.
- The term "sharp" is used in several places to emphasize that the upper bound analysis is improved from previous work, but I think this term is somewhat conventionally used when the upper bound matches the lower bound which is not discussed in the paper.
- It is somewhat hard to appreciate the proposed B-stability condition (maybe it is too abstract? or only algebraically explained?). This is a personal opinion but impacts my overall impression of the paper. If there is a better and more intuitive explanation for this, it would be great.
The writing is in general very clear.
But the paper could be hard for broader audience to read.
In terms of originality, I feel that the paper could have emphasized more on technical innovations.
NO.
We thank the reviewer for the valuable feedback to our paper. We respond to the main concerns as follows.
*--- “In a current form, the paper sounds in nature too similar to (Zhan et al., 2022). It would have been better to position the paper by, for example, focusing more on the technical innovations to get better sample-complexity.”*
We agree that the technical innovations to obtain the improved sample complexity is an important part of our contribution. We have a brief overview of techniques at the bottom of Page 7 (after Theorem 9), with a more elaborated version in Appendix B, which presents the main steps (Performance decomposition, Bounding the squared-B errors, and generalized L2 Eluder argument) and discusses how they improve over existing work. In the rest of the main text, we chose to focus on other ingredients (the B-stability condition and algorithms) due to their necessity and the space limit.
We also briefly remark that, apart from the sample complexity improvement, even just as a condition for learnability with polynomial samples, our B-stability condition is also more expressive than Zhan et al.'s “regular PSR” condition (see the paragraph on Page 6 after Example 8 for detailed discussions).
*--- “The term "sharp" is used in several places to emphasize that the upper bound analysis is improved from previous work, but I think this term is somewhat conventionally used when the upper bound matches the lower bound which is not discussed in the paper.”*
We indeed mainly used "sharp" to indicate our substantial sample complexity improvement over existing results. We agree that obtaining a lower bound for our problem (learning POMDPs / PSRs), as well as understanding its gap with the current upper bounds, is an important question. Currently, a (non-trivial) lower bound for learning tractable subclasses of POMDPs / PSRs (such as revealing / decodable POMDPs) is still largely open, which we believe is beyond the scope of the current paper but an important question for future work.
*--- “ [...] somewhat hard to appreciate the proposed B-stability condition (maybe it is too abstract? or only algebraically explained?). If there is a better and more intuitive explanation for this, it would be great.”*
Intuitively, the B-operator $\mathcal{B}_ {H:h}$ has the following probabilistic meaning: It maps any predictive state $\mathbf{q}=\mathbf{q}(\tau_{h-1})$
to the vector
$\mathcal{B}_ {H:h}\mathbf{q} = (\mathbb{P}(\tau_ {h:H}|\tau_ {h-1}))_ {\tau_ {h:H}}$
, which governs the probability of transitioning to all possible futures (i.e. $\mathbb{P}(\tau_ {h:H}|\tau_ {h-1}) \times \pi(\tau_ {h:H}|\tau_{h-1}) = \mathbb{P}^\pi(\tau_ {h:H}|\tau_ {h-1})$ for any policy $\pi$).
The B-stability condition in turn just requires the above B-operator to have a bounded operator norm, so that small errors in the input $\mathbf{q}_ 1-\mathbf{q}_ 2$ will result in the small error in the output $\mathcal{B}_ {H:h}(\mathbf{q}_1-\mathbf{q}_2)$, in suitable norms.
To better illustrate this, we have added a short explanation of the above intuition for the B-operators at the beginning of Page 5 (Section 3.1) in our revision.
We thank all reviewers again for their valuable feedback to our paper.
We have incorporated the reviewers’ suggestions and made several edits on the presentation in our revision. All changes (except for typo fixes) are marked in red, for clarity.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors investigated the unified structural condition, i.e., B-stability, under which the PSR can be learned sample-efficiently. The authors also introduced the connection between the B-stable PSR and the learnable subclasses of POMDP and provided tighter bounds. Finally, the authors proved that these bounds can be achieved by three different algorithms. 


The paper is well-organized and clear. As a theoretical paper, this paper is comprehensive in the sense that it covers a variety of models and plenty of algorithms. All of the reviewers agree this paper should be accepted. 


Minor:

- The current version is relatively similar to (Zhan et al., 2022). It will be better if the paper can be re-organized to emphasize the novelty and significance. 

justification_for_why_not_higher_score: 
The paper is a theoretical paper to provide a unified condition, under which the RL is sample efficient in the partial observable environment.

The paper is rigorous and comprehensive to cover several known sample-efficient subclasses of POMDP, and improves the known results with three different algorithms.
justification_for_why_not_lower_score: 
The presentation of the paper can be further improved to emphasize the significance of the contribution, comparing to the existing literature (Zhan et al., 2022). 

Decision: Accept: notable-top-25%