Variational oracle guiding for reinforcement learning

Abstract: How to make intelligent decisions is a central problem in machine learning and artificial intelligence. Despite recent successes of deep reinforcement learning (RL) in various decision making problems, an important but under-explored aspect is how to leverage oracle observation (the information that is invisible during online decision making, but is available during offline training) to facilitate learning. For example, human experts will look at the replay after a Poker game, in which they can check the opponents' hands to improve their estimation of the opponents' hands from the visible information during playing. In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods. Our key contribution is to propose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL. VLOG is featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value-based DRL algorithm. We empirically demonstrate the effectiveness of VLOG in online and offline RL domains with tasks ranging from video games to a challenging tile-based game Mahjong. Furthermore, we publish the Mahjong environment and an offline RL dataset as a benchmark to facilitate future research on oracle guiding (https://github.com/Agony5757/mahjong).

Reviews and author responses:

 Dear Reviewer 8vyx and AC/SAC/PC,

\
In case you haven't see our reply (posted 22:02 Nov. 29 AOE Time) to this comment (posted 19:23 Nov. 29 AOE Time) due to nested replies, we attach [the link to our reply](https://openreview.net/forum?id=pjqqxepwoMy&noteId=sH68-l8YU1X) . Thanks! 

\
Best regards,

Paper 135 authors

 ======== The follows are the reply to [reviewer 8vyx's lastest comment](https://openreview.net/forum?id=pjqqxepwoMy&noteId=9thKi4V687m) =========


Thanks for the prompt response and the clarifications of concern!

### Core concern

>  Instead, my concern is on whether the problem of formulating the sequence model to generate the v_st is solvable and whether the proposed way of modelling the posterior/prior distribution is valid. With this regard, I do not see convincing response. Also, I think it is not that leading to the good final RL performance is the key. Instead, I look forward to see more results on the quality generative model, i.e., whether the prior distribution learned in such a way is good enough for modelling the latent distribution.

If our understanding of "valid" is correct, the heart of this concern is that "*is the prior representation learned well by our method in practice?*". 

Although we agree that the RL performance is not the only "key" to unbox the learned model representation, the RL performance should be the most important and the most fair criteria of the "validity" of the learn prior and posterior representations (given that our framework is mathematically justified) since we are considering RL problems, after all. 

In additional to the promising RL performance of our framework, we also visualized an example of the learned prior distribution in Fig.2 C,D and provided detailed discussion about it (Sec. 5.1) as a piece of evidence to support our claim that the prior distribution is learned well using the proposed Bayesian formulation. 

Another possible way to help validating the learned prior distribution could be plotting the KLD between posterior and prior during training (it will be very close to $D_{KL}^{tar}$ (Sec. 4.1) after training). We are happy to provide the corresponding plots in the paper if the reviewer thinks this helps to support our claims. Meanwhile, we would appreciate if the reviewer could suggest more concrete metrics to "validate the way of modelling the posterior/prior distribution".

### About other concerns 

Thanks again for your time and feedbacks! 

Please kindly inform us if there is any other remaining concern that we have not resolved in the rebuttals (e.g., real-life scenarios, in "Reply to the feedback from reviewer 8vyx (Part 2)" below). 




 I fully understand what VAE, ELBO and prior/posterior mean so I do not need explanation on this. My point is also not about the mathematical correctness of ELBO. Instead, my concern is on whether the problem of formulating the sequence model to generate the v_st  is solvable and whether the proposed way of modelling the posterior/prior distribution is valid. With this regard, I do not see convincing response. Also, I think it is not that leading to the  good final RL *performance* is the key. Instead, I look forward to see more results on the quality generative model, i.e., whether the *prior* distribution learned in such a way is good enough for modelling the latent distribution. 



 
# References

- **[Kingma2014ICLR]** Kingma, Diederik P., and Max Welling. “Auto-Encoding Variational Bayes.” ICLR 2014 : International Conference on Learning Representations (ICLR) 2014, 2014.

- **[Sohn2015NeurIPS]** Sohn, Kihyuk, Honglak Lee, and Xinchen Yan. "Learning structured output representation using deep conditional generative models." Advances in neural information processing systems 28 (2015): 3483-3491.

- **[Goyal2017NeurIPS]** Goyal, Anirudh, et al. “Z-forcing: training stochastic recurrent networks.” Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.

- **[Kaspar2020IROS]** Kaspar, Manuel, Juan D. Muñoz Osorio, and Jürgen Bock. "Sim2real transfer for reinforcement learning without dynamics randomization." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

- **[Li2020arXiv]** Li J, Koyamada S, Ye Q, et al. Suphx: Mastering mahjong with deep reinforcement learning[J]. arXiv preprint arXiv:2003.13590, 2020.

- **[Guez2020NeurIPS]** Arthur Guez, Fabio Viola, et al. Value-driven Hindsight Modelling. In Advances in Neural Information Processing Systems, 2020.

- **[Chen2020CoRL]** Chen, Dian, et al. "Learning by cheating." Conference on Robot Learning. PMLR, 2020.

- **[Fang2021AAAI]** Fang, Yuchen, et al. “Universal Trading for Order Execution with Oracle Policy Distillation.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

- **[Lee2020ScienceRobotics]** Lee, Joonho, et al. "Learning quadrupedal locomotion over challenging terrain." Science robotics 5.47 (2020).

 
> I think the authors also do not validate that the trained prior distribution could properly encode the distribution or they could really be generative to yield valid output when fed to the decoder that it never do so during training.

We validated "the trained prior distribution could properly encode the distribution" by visualizing the prior distribution of $z$ and by demonstrating the empirical performance using prior distribution fed to the decoder (prior encoder + decoder = executor model).

In Fig. 2C,D, we visualize the prior distribution in the interpretable Maze task to show how it can be better shaped by leveraging oracle observation, after training (as described in the caption of Fig.2C, the $z$ shown was computed during execution using the prior encoder and executor observations.

We also demonstrated the empirical performance of VLOG using the prior distribution of $z$ for execution in Fig. 2B, Fig. 3 and Table 1, 2, for various tasks. These results clearly indicate the prior encoding is effectiveness for RL performance.


## Novelty 

We would like to clarify that the general oracle guiding scheme is not originally proposed by us. The oracle guiding problems have already been studies in previous researches including imitation learning [Chen2020CoRL] and RL [Guez2020NeurIPS, Li2020arXiv, Fang2021AAAI]. **The recent trend of publications indicates the significance of oracle guiding problems**.

While there are increasing research interests on oracle guiding problems, existing studies on oracle guiding RL [Guez2020NeurIPS, Li2020arXiv, Fang2021AAAI] all used heuristic algorithms. **Our novelty is established on the fact that VLOG is the first to mathematically derive a lower bound objective from a Bayesian perspective for oracle guiding RL. The proposed theoretically-guaranteed optimization for executor model leveraging oracle observations has also been demonstrated to outpeform the baseline and previous heuristic methods on various RL tasks.**


## Empirical Evaluation

The reviewer mentioned remaining concern about empirical evaluations. Could the reviewer inform us which part of the our experiments is not justified?


## Motivation and real-life examples

> I still think this work is not very general because none of the given example is a real-life scenario and I'm extremelly not happy with the "a robot on the Moon" case.

We disagree that "none of the given example is a real-life scenario". The examples in our paper include playing Poker/Mahjong (opponents' hand as oracle observations available in training) and taking exams (solutions as oracle observations available in training). These examples are clearly describing real-life scenarios.

Sorry for the cofusion of the "robot on the Moon" example, we mean a ["Sim2Real"](https://sim2real.github.io/) scenario where a robot is pretrained in simulators (full environmental information can be given as oracle observation) and is then assigned to perform real-world tasks in partially observable environments such as the Moon, this is of great research interests and has been studies by many recent works, e.g., [Kaspar2020IROS, Lee2020ScienceRobotics]. We will amend this example to be more clear and add the references in the paper.


## Mahjong Benchmark Releasing

> To my knowledge, many important RL environments are open-sourced before they got published so that researchers in the community could use them and help verify how they contribute to the community.

Our Mahjong envrionment and dataset had been already published on [github.com/pymahjong/pymahjong](https://github.com/pymahjong/pymahjong) and we welcome everyone to use it and to provide feedbacks to us for improvement. Whilst it remains anonymous as now due to the double-blind review policy.  Mahjong is played by more than a billion people over the world. We believe our environment will not only contribute to the RL community, but also will contribute to the Mahjong community.

### (The references are in the next part)



 > Though some of the confusing points have been clarified the authors, I still have remaining conerns on the Bayesian formulation, novelty of this work and the empirical evaluation part.

Thanks very much for your feedback and we are glad that our rebuttal resolved your concerns except **Bayesian formulation**, **novelty**, and **empirical evaluation**.

#### We hope the follows help to resolve the remaining concerns.

## VLOG Bayesian formulation

The evidence lower bound (ELBO) (or variational lower bound) is a family of objective functions which is often optimized in variational Bayesian methods. **What the prior distribution and posterior distribution condition on varies depending on what kind of task one considers**.

For example, 

- The **ELBO of the original VAE [Kingma2014ICLR] is a special case** in which the prior $z$ is independent Gaussian variable and the posterior $z$ conditions on the data. The prior $z_t$ can be used for generating data, similar to that in VLOG we using the prior $z_t$ to estimate value function.

- For the conditional VAE [Sohn2015NeurIPS], the prior distribution of $z$ conditions on the raw data (e.g., Images of MNIST), and the posterior $z$ conditions on raw data and extra information (e.g., Images of MNIST and corresponding one-hot labels). Both the prior distribution and posterior distribution of $z$ can be used for generating outputs.

- The ELBO of a bi-directional variational RNN [Goyal2017NeurIPS], which performs sequence modeling, has another formulation. Suppose we have a sequential data $X_{1:T}$. The prior distribution of latent variable $z_t$ is computed using $X_{1:t}$ with a forward-directional RNN where $1\le t \le T$, and the posterior $z_t$ is computed using $X_{t:T}$ (which is available during learning, like oracle information) with a backward-directional RNN. And the prior $z_t$ can be used for predicting $X_{t+1}$, similar to that in VLOG we are using the prior $z_t$ to estimate value function.


> So I do not think such a design is similar to the conventional variational autoencoders, because for me I think if follow the conventional design, the executor observation part should be considered as the posterior. 

The reviewer is right that the objective of VLOG (Eq. 1) is a different type of ELBO from the ELBO of the original VAE [Kingma2014ICLR], since VAE considers low-dimensional embbeding of high-dimensional data, while we consider oracle guiding RL, which are two totally different problems.

> My another concern is that the variational formulation in this part is not very valid.

While our variational formulation is different from that of VAE, this does not mean our formulation is wrong. VLOG is the first work applies variational Bayes to oracle guiding tasks, this is exactly where the novelty and significance of our work is established. 

#### We hope the following explanations further help to resolve your concerns about validity of our Beysian formulation.

**Mathematically**, we derived the VLOG obective function in Eq. 1, which is the lower bound of our original objective -- the log-likelihood objective of the estimation of return based on executor observation $x_t$ (i.e., for the executor model). We would appreciate if the reviewer could point out any mistakes in the derivations (one notable issue is that the latent distributions should satisfy $q(z_t|\hat{x}_t) > 0$ and $p(z_t|x_t) > 0$ because they could be on the denominator, however, there is no problem since we employed Gaussian distributions for $q(z_t|\hat{x}_t)$ and $p(z_t|x_t)$). 

**Intuitively**: "Posterior" means something comes later (i.e., hindsight). In VLOG, oracle observation is available to the agent during experience replay after decision making, so it is natural to use oracle observation for posterior. Also, posterior generally contains more information than the prior as in variatianal Beysian formulations [Kingma2014ICLR, Sohn2015NeurIPS], which also matches the characteristic of oracle observation (which contains executor observations (Sec. 3)).

**Empirically**, our experiments demonstrated the effectiveness on various kinds of tasks by outperforming previous oracle guiding methods that were not theoretically justified.

> The training of the encoder for the raw/executor observation in the proposed model seems to be really challenging task, because the encoder's output latent is only updated through the KL loss, as the authors model the oracle as the posterior to feed to the decoder. 

While such a training scheme may seem to be challenging at the first glance, it indeed works quite well in practice. **Not only our experiments showed the effectiveness of this training scheme (prior only updated through the KL divergence), but it is also demonstrated by also many existing variational Bayesian studies such as [Sohn2015NeurIPS, Goyal2017NeurIPS]**.


 Thanks the authors to revising the paper according to the review comments and providing detailed response. 

Though some of the confusing points have been clarified the authors, I still have remaining conerns on the Bayesian formulation, novelty of this work and the empirical evaluation part so that I decide not to increase the recommendation score at the moment. 

**Variational Formulation**: I  wish to highlight that the encode-decoder design in this work is not the same as the conventional VAE, because it trains the decoder to output a v_st which is a value function and this could be really hard as v_st is a changing target being updated during RL.  

My  another concern is that the variational formulation in this part is not very valid. The training of  the encoder for the raw/executor observation in the proposed model seems to be really challenging task, because the encoder's output latent is only updated through the KL loss, as the authors model the oracle as the posterior to feed to the decoder. So I do not think such a design is similar to the conventional variational autoencoders, because for me I think if follow the conventional design, the executor observation part should be considered as the posterior. I think the authors also do not validate that the trained prior distribution could properly encode the distribution or they could really be  `generative`  to yield valid output when fed to the decoder that it never do so during training.


**Motivation**: I still think this work is not very general because none of the given example is a real-life scenario and I'm extremelly not happy with the "a robot on the Moon" case. 

**The introduced Mahjoong environment: I appreciate the authors for releasing another RL environment but for me it is not sure how significant this new released environment could contributed to the community. To my knowledge, many important RL environments are open-sourced before they got published so that researchers in the community could use  them and help verify how they contribute to the community. So I think Mahjoong also need some time to be testified by some works before we could confirm it is a valid contribution worth being considered for ICLR. 

 Thanks for the reponse! We are working on making our Mahjong environment ([Anonymous Git Repo](https://github.com/pymahjong/pymahjong)) easier to install (using pip) and use (more elegant wrapping). We would appreciate very much if you have any suggestions to improve the Benchmark.

In this paper, reinforcement learning (RL) with training-time privileged information is studied. This scenario is the relaxation of traditional POMDP by allowing additional observations to be available during training time. A variational Bayes approach is proposed under this scenario. The main idea lies in learning a joint latent space for both testing-stage partial observations and the training-stage full observations.  Experiments over several benchmark tasks, including Mahjong, a large-scale multi-player game with imperfect information are conducted to verify the performance of the approach.

 Strengths of the paper:
- Adopting variational Bayes is an interesting and reasonable approach to make use of the training-time priviledged information. 

Weakness of the paper:
- In fact, the studied problem is not absolutely new. There have been similar settings studied under imitation learning (IL),  such as [1]. Furthermore, applying variational Bayes for latent space learning is also not a very novel idea in both RL and IL. 

- According to the paper, one of the major drawbacks for previous approaches in this problem is lacking theoretical guarantees. However, in my view, the variational Bayes is also not possible to work in all situations. Why and when the proposed approach would indeed work? The theoretical analysis is missing in the paper. Even though the experimental results show some advatange of the proposed approach, after reading the paper, I still don't get solid justifications on why variational Bayes could outperform previous approaches in principle. 

- Some descriptions could be clearer:
>- In Equation (1), how $v^{tar}_t$ is obtained? The paper just says that it could be calculated by any policy evaluation method. But policy iteration relies on knowing the transition model and the reward function. This is not assumed in the paper.
>- In the Mahjong experiment, I wonder why VLOG-no oracle could improve over the comparison methods. According to Equation (1), if no oracle observation is available, VLOG reduces to minimizing the maximum-likehood loss of fitting the value function. I thought it would have no difference to the baseline method, i.e. vallina policy learning. 

Reference:
[1]  Learning by cheating. https://arxiv.org/pdf/1912.12294.pdf

 To summarize, I think the paper proposes a reasonable approach for RL with training-time privileged information. But I think further efforts should be made to better justify the proposed approach both theoretically and empirically, as discussed above.

 Dear authors,

I think the feedback has made several points clarified, including:
- The relationship between other related works
- How $v^{tar}$ is obtained
- The performance of VLOG-no oracle in Mahjong

I think the most attractive part of the paper for me is the Mahjong experiment, thus I am looking forward to the release of the code and environment. While my biggest concern is about the novelty of the proposed approach: The Bayesian-based approach is not significantly novel, thus the technical contribution is somehow limited. After all, I decide to raise my score to six.





 Thanks for the response! 

 Thanks for the response! 

 Thanks for replying to my concerns. I think the clarification solves my questions. I have scanned through other reviewers' concerns and it seems the author(s) has(ve) well resolved most of them. I prefer voting for acceptance. 

 Thank you for all your clarifications!

I am still in favour of accepting this paper, although I still find the Mahjong experiments a little contrived. I can appreciate the value of the environment (in part thanks to the clarifications provided), but it still seems to me like it distracts a bit from the main contributions of the paper (it takes over a page just to explain the game/environment!).

As such, I'll keep my score where it is (which is in favour of acceptance).

 We are greatly thankful to the reviewer for the comments. We hope the following replies resolve the concerns 
___

>  the oracle and raw observations are processed by two individual encoders and then the two latent spaces are fused and fed to a shared decoder. 

We would like to clarify that the two latent spaces are not fused. As shown in Fig.1

- During learning, the posterior z (input is oracle observation) is fed to the shared decoder. 
- During execution, the prior z (input is executor observation) is fed to the shared decoder. 

The derivation of Eq. 1 guarantees the correctness. Such scheme was also widely used in other variational Beysian models, e.g., [1]
___

> The assumption that there are always oracle observations available during policy learning does not hold for many applications, such as Atari 2600 and noisy robotics navigation tasks. And in the more challenging problems with partial observability, it is unclear whether this method could be directly applied and how to effectively regularize the posterior distribution which meant to represent partially observed states.

This misunderstanding may be due to that we introduced POMDP in the preliminaries. The reviewer might think that we were trying to solve general POMDP problem by assuming oracle observation exists. However, this is not our original intention. The scope of our study is oracle guiding RL problem in which oracle observation exists. **We have revised the paper to include a new section (Sec. 3) to formally define the oracle guiding RL problem we consider.**

**Oracle observations RL problems are actually quite common in real life**. Not only in card games, but also, for example, when taking a final-term examination (oracle observation is answers to similar questions, which are available only during preparing); Or training a robot to perform some tasks on the Moon (when training the robot, we can provide it with the information of the whole territory, which are not available during execution).

Nonetheless, VLOG can be directly applied to any POMDP task by considering oracle observation equal to the executor observation (this is the case of the "VLOG - no oracle" model). In most cases, VLOG - no oracle performs similar to the baseline because oracle observation is not used. However, as we discussed in the last paragraph of Sec. 5.2, the stochasticity of $\boldsymbol{z}_t$ may facilitate exploration, and thus contribute to a better performance than the deterministic baseline model. This is an unintentional surprise thanks to the variational Bayesian network model. 
___
> I feel the formulation of ELBO is not that sound. I do not know why the latent of oracle observations are set as the posterior instead of that from executor observation.

We hope the following two explanations resolve questions of "why the latent of oracle observations are set as the posterior".

First, our final goal is to get a model that generates policy using executor observation. It is normal in variational Bayesian models that the generative model uses prior distribution of $z$ [1]. So executor observation is used for prior and thus oracle observation is used for posterior.

Second, according to the definition in Bayesian statistics, posterior is the probability calculated when some outcomes/evidences after the event are given and the posterior is calculated condition on them. In VLOG, oracle observation is available to the agent during experience replay AFTER decision making, so it is natural to use oracle observation for posterior. Besides, posterior generally contains more information than the prior, which also matches the characteristic of oracle observation.

____

>  Also, it seems that there are two execution paths for the ELBO, i.e., there should be one for the oracle and another for the executor observation. But in Sec 4.2, this is not properly specified as Eq (2) only tackles one of the path.

If our understanding is correct, the reviewer meant "how to compute Q-function during execution" by "execution path". If so, we would like to clarify that **VLOG has only one "execution path" as shown in Fig.1 B. The oracle observations are not involved in execution, but only in learning (Fig.1 A)**. Our derivation of Eq. (1) shows that VLOG can correctly optimize the executor model (input is executor observation) using oracle observation data.

Eq. (2) defines the loss function (negative ELBO) used in learning of VLOG, which tackles with both executor and oracle observations.
_____

> It is unclear how the variational model is related to the policy model, e.g., is the input to the policy $x_t$ or $z_t$?

$z_t$ is the input to the policy model (MLP for estimating the Q function) (Fig.1 (A) and (B)). We thank the reviewer for pointing out this potential unclear point and have **revised the paper to clarify this in the end of Sec. 4.1**.

### **Continue in the next part**


 
We greatly thank all the reviewers for the insightful comments and suggestions, which are very helpful for us to further improve this work. Especially, we are encouraged by the recognitions from the reviewers, e.g.,
- *The paper is easy to follow and well-written* (Reviewer Wjja & 8vyx)
- *I really like the motivation and idea behind this paper* (reviewer 5XWD)
- *The proposed method is intuitive and effective* (Reviewer Wjja)
- *Adopting variational Bayes is an interesting and reasonable approach* (Reviewer 7M2j)

There were some unclear writings in the paper that caused confusion. **For the the unclear points, we made thorough revisions to address them.  The parts with major revisions are shown in blue fonts in the updated version of the manuscript**. The siginificant revisions are summarized as follows:

1. We complemented the discussion of related works according to the suggestions of reviewer 7M2j and 8vyx in Sec. 2

2. We formally defined oracle guiding POMDP in Sec. 3, as suggested by reivewer Wjja.

3. We clarified how to obtain $v^{tar}$ in Sec. 4 and Appendix B.1, as asked by reviewer Wjja and 7M2j. 

4. We justified our claim of VLOG's theoretical guarantee in Sec. 4, as commented by reviewer 7M2j and Wjja.

4. We made more clear explanation for the Mahjong experiments (Sec. 5.3), as suggested by reviewer 5XWD and 7M2j.

 
We are greatly thankful to the reviewer for the comments. We hope the following replies resolve the concerns raised in the review.

 -----------------------------
> The problem is not absolutely new ... such as [1] ... 


The oracle-guiding setting in RL is not new. As discussed in the Related Works, it has been studied by several papers. However, our contribution is to derive an objective function for oracle guiding in a principled manner using variational Bayes and propose a practical algorithm with neural networks based on it. Experimental results demonstrated the effectiveness of our algorithm. 

We thank the reviewer for referring us to the related work. Although their setting differs from ours (IL vs. RL and deterministic vs Bayesian), we agree that it is highly relevant. Therefore, we have added a discussion to Related Works. 

---------------------------------

> ... lacking theoretical guarantee ... Variational Bayes is also not possible to work in all situations. ... why variational Bayes could outperform previous approaches in principle.



We agree that variational Bayes may fail if the function class for the approximate posterior is not flexible enough, which may result in a considerable gap between the original objective and variational lower bound. However, if a function class for the approximate posterior contains the true (or nearly true) posterior, maximizing the variational lower bound leads to maximising the original objective. **In other words, given that the approximate posterior model is flexible enough, VLOG maximizes the objective of the executor model (whose input is executor observation) using oracle observation data.**

**In contrast, previous oracle-guiding algorithms are heuristic and have no theoretical consideration** (dropout in Suphx [1] and knowledge distillation in OPD [2]). For example, in OPD-style oracle guiding (Appendix A), the loss function is:

$$ J_{OPD} = \mbox{MSE} \left(Q_t^{tar} , Q(x_t, a_t) \right) + \mu \mbox{MSE} \left(Q_{teacher}(\hat{x}_t, a_t) , Q(x_t, a_t) \right), $$
where the second term is the distillation loss ($\mu$ is a scalar hyper-parameter), which may introduce bias to the original learning objective (the first term).


----------------------------

> how $v^{tar}$ is obtained?


We apologize for the confusion. $ v_t^{tar} $ is a general notation for the target value (V or Q) obtained by any value learning algorithm (e.g., using any Bellman operation such as TD(0), TD($\lambda$), Q-learning(( Retrace($\lambda$), etc.). For example, $v^{tar}(z_t) = r_t + \gamma  max_{a}Q(z_{t+1}, a)$ for Q-learning. We have revised Sec. 4 to clarify this.

We included the full description of RL models and loss functions for all our experiments such as how to compute $v^{tar}$ in Appendix B.1. 

----------------------------

> In the Mahjong experiment, I wonder why VLOG-no oracle could improve over the comparison methods.

Mahjong is a game with highly stochastic state transitions (for example, when drawing a tile, it can be any of the total 34 types of tiles. And there are around 17 times of drawing for each player in one game). While there is no oracle information available, we consider that the **Bayesian property of the model is beneficial for RL in stochastic environments**, as we know abundant empirical results in SL or UL in which variational Bayesian models showed promising performance on stochastic datasets, e.g., [3], [4]. 

## References

[1] Li, Junjie, et al. "Suphx: Mastering mahjong with deep reinforcement learning." arXiv preprint arXiv:2003.13590 (2020).

[2] Fang, Yuchen, et al. "Universal Trading for Order Execution with Oracle Policy Distillation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

[3] Babaeizadeh, Mohammad, et al. "Stochastic Variational Video Prediction." International Conference on Learning Representations. 2018.

[4] Goyal, Anirudh, et al. "Z-forcing: training stochastic recurrent networks." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.



 
### Replies to Minors
---------------------------
> 1. Many sentences are repeated in the abstract and introduction, but readers with a reasonable RL/ML background can follow this message with one scan. Maybe use another example for better elaboration.

Thanks for the suggestion, we have removed some redundant parts and added more examples in the introduction. 

---------------------------
> 2. In remark one, the paper says "they can choose q(v_t|\hat_{x}_t) to be p(v_t|x_t)u(v_t; [l,u])". Are you multplying two distrbutions? In this sense, q will not define any distribution.

Thanks for pointing out this issue. We rewrote it to the following sentence: For example, if one knows the range of a state-value at $x_t$ is a closed interval $[l, u]$, the approximate posterior $q(v_t|\hat{x}_t)$ can be restricted to a family of probability distributions supported on $[l, u]$.

---------------------------
> 3. Poker games involve two players competing with each other. In this sense, maybe a (PO)Markov Game model can better describe this environment than a (PO)MDP.

We agree that multi-agent games cannot be appropriately described by POMDP. We would like to clarify that our work focused on the single-agent setting during training. For Mahjong, because the dataset used for offline learning is fixed, the opponents can be considered to have fixed policies. Therefore, our experiments are still within the domain of single-agent RL. We also have revised the Sec. 5.3 to clarify the single-agent setting.

However, we think that applying VLOG to multi-agent games will be interesting since the players can confuse other players by unusual actions. In the future, we would like to explore how VLOG can help multi-agent RL described by the PO Markov game model.


---------------------------

### typos

We also fixed the typo as the reviewer mentioned. 


---------------------------


## References

[1] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

[2] Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018).

[3] Burgess, Christopher P., et al. "Understanding disentangling in $\beta $-VAE." NIPS Workshop on Learning Disentangled Representations (2017).




 
We are greatly thankful to the reviewer for the comments. We hope the following reply resolve the concerns raised in the review.

### Replies to Concerns
---------------------------
> 1. .... summarize the different between offline RL and oracle guiding RL ... Please formally define the oracle-guiding RL problems anyway.


Thanks for the suggestion! Offline RL and oracle-guiding RL are orthogonal to each other. Offline RL considers the case that the agent cannot interact with the environment. Oracle guiding RL considers the case that the agent has access to oracle observations, which are available only during the training but not during action generation. Oracle observations may exist in either online or offline RL, although it is more likely that they are available in offline RL cases.

**We have complemented the manuscript with a section to formally define oracle guiding problems (Sec. 3)**. This should clarify the oracle guiding RL problem. 

---------------------------
> 2. The author claims "VLOG is theoretically guaranteed to ...", ... where are the new contributions?

We would also like to to clarify the "theoretical guarantee" mentioned in the paper is that **"VLOG is correctly optimizing the lower bound of the objective of the executor model (whose input is executor observation) using oracle observation data.** This is achieved by deriving the lower bound of the objective function of the executor model." We have revised the paper to clarify this (Sec.4). Previous oracle-guiding algorithms did not have theoretical guarantees, and it is unclear whether they are theoretically sound or not. Therefore, we think our results are an important first step towards understanding the oracle-guiding scheme.


---------------------------
> 3. How do you get the label $v^{tar}_{t}$? in (1) ...

We apologize for the confusion. $v_t^{tar}$ is a general notation for the target value (V or Q) obtained by any value learning algorithm (e.g., using any Bellman operation such as TD(0), TD($\lambda$), Q-learning, Retrace($\lambda$), etc.). For example, $v^{tar}(z_t) = r_t + \gamma  max_{a}Q(z_{t+1}, a)$ for Q-learning. We have revised Sec. 4 to clarify this.

We included the full description of RL models and loss functions for all our experiments such as how to compute $v^{tar}$ in Appendix B.1. 


----------------------------


> 4. Will it make things easier by replacing a hyper-parameter $\beta$ with another hyper-parameter $D^{tar}_{KL}$? ... linear or exponential scheduler ... works quite well in practice.



We thank the reviewer for raising the question. The following are our considerations which we now add as Appendix D into the revised paper.

In principle, replacing a hyper-parameter with another does not always make training easier. However, in practice (especially deep RL), the performance will be highly sensitive to some hyper-parameters (e.g., the entropy coefficient $\alpha$ in the original soft-actor critic algorithm [1] need to be tuned in each robotic task. This is because the reward magnitude is different among tasks.). As a result, it will be beneficial to replace a sensitive hyper-parameter with another one that does not need fine tuning. For example, in the follow-up paper of soft-actor critic, they let the entropy coefficient $\alpha$ to be adaptive by introducing another hyper-parameter, the entropy target [2]. They empirically found that it is good to set the entropy target equal to the negative of the degree of freedom of the agent, thus to avoid tuning $\alpha$. 

Our idea of replacing $\beta$ with $D_{KL}^{tar}$ is due to similar reasons. One obvious problem is that the magnitude of the "oracle prediction error" term (the first term in the RHS of Equation 2) relies on the reward magnitude of the task. Therefore $\beta$ should also be adjusted to match with the magnitude of task reward. However, $D_{KL}^{tar}$ is only relevant to the magnitude of prior $z_t$ and posterior $z_t$, which does not differ too much among tasks (usually at the order of 1). In practical we found that a single value for the target KLD, $D_{KL}^{tar} = 50$, worked well for all tasks including Maze, MinAtar and Mahjong. 

Introducing a linear or exponential scheduler, as the reviewer suggested, works quite well in practice. For example, in the paper "Understanding disentangling in $\beta$-VAE" [3], they used a linear scheduler for the target KL-divergence and got good results. However, using a scheduler introducing more hyper-parameters (at least two: initial $\beta$ and final $\beta$), which is opposite to our willing to reduce the impact of hyper-parameters.

### Continue in the next Part

 [1] Goyal, Anirudh, et al. "Z-forcing: training stochastic recurrent networks." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.

[2] Li, Junjie, et al. "Suphx: Mastering mahjong with deep reinforcement learning." arXiv preprint arXiv:2003.13590 (2020).

[3] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." arXiv preprint arXiv:2006.04779 (2020).

[4] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.


 
----------------------------

> For instance, in the first task of maze navigation, in such a simple case I'm very sure using something like PPO will successfully solve the problem. I don't understand why the problem could be unsolvable, i.e., the performance of baseline is such poor. I'm also concerned if the experimental settings are properly configured.


We used the Maze Task to **provide intuitive understandings of VLOG because of the simplicity and interpretability**, as we explained in the first paragraph of Sec. 5.1, **we demonstrate how VLOG helps to shape the latent representation by leveraging oracle observation**. We do not intent to use the Maze task to demonstrate that how VLOG can improve empirical performance in hard tasks. The MinAtar and Mahjong tasks are the challenging tasks we used to prove the effectiveness of VLOG.

Since we do not want VLOG to be a black box inside which the latent $\boldsymbol{z}$ is mystery, we visualized the latent representation (Fig.2 (C) and (D)) to explain that VLOG contributed to a more effective latent representation as the main results for the Maze task. 

The reason accounting unsatisfying performance of baseline is that the input to the baseline model is only the (noisy) x, y coordinate of the agent. The latent representation of the baseline model may mess up the adjacent grids blocked by walls (these grids have similar x, y), as shown in Fig.2 (C) and (D). While in VLOG, the oracle observations help shaping the latent representation to resolve this issue because the oracle observation clearly reflects the current state of the agent.

-----------------------

> For instance, in the maze task, the input is very shallow, i.e., with only $\sim$3 dimensions, but the authors specify the latent to have a dimension of 128. I believe such setting is not very reasonable. 


For the Maze and MinAtar tasks, we used hidden layer width 256 for all the layers except the latent layer $\boldsymbol{z}$ of VLOG (Hidden layer width 256 is a reasonable number which is very common in deep RL [4]) for consistency and simplicity. For the latent layer of VLOG, because both the mean and variance of $\boldsymbol{z}$ need to be computed, using dimension of 128 for z (half of 256) should be a reasonable choice for fairness.

----------------------------

> In many cases, even though the oracle features might be available, they might have very different format compared to the raw observations, e.g., (x, y, z) vs high-dimensional images. In such scenarios, would the two-encoder-one-decoder architecture still work?


Since our derivation for the VLOG objective function have NO assumptions for the prior and posterior encoders $p(\boldsymbol{z}_t|\boldsymbol{x}_t)$ and $q(\boldsymbol{z}_t|\boldsymbol{x}_t)$,  can use totally different encoder network architectures for processing executor and oracle observation, respectively.

We would like to highlight that in all our tasks, oracle observation and executor observation have different formats, since oracle observation is the executor observation concatenated with some additional useful information. For example, in Mahjong, oracle observation has the shape of (93+18) $\times$ 34, while executor observation is a part of oracle observation, with shape 93 $\times$ 34 (described in Appendix E.2). Thus, the prior encoder and posterior encoder in Fig. 1 need to process different formats of observations in all tasks, and experimental results demonstrate that this will not be the obstacle for applying VLOG.

___________________________


> Overall the novelty of this paper is relatively limited

We would like to highlight that the novelty of our algorithm is that **we are the first to derive an variational lower bound (Eq.1) for oracle-guiding RL.** By maximizing the lower bound, the executor model is optimized using oracle observation data. Furthermore, we **proposed the methodology to implement it with any value-based deep RL algorithm** (Fig.1 (A) and (B)). Our experiments incorporated VLOG with dueling double-DQN (Maze and MinAtar), conservative Q-learning (Mahjong) and Behavior cloning (Mahjong), respectively.

----------------------------

> there lacks discussion with a number of related works that combine variational inference with policy learning, such as: [1] Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model [2] Variational methods for Reinforcement Learning [3] Sequential Generative Exploration Model for Partially Observable Reinforcement Learning [4] Reinforced Variational Inference

We thank the review for the references. **We revised the Related Work section to include the discussion for the mentioned works.**

### Continue in the next part (References Only)


 
--------------------------------

> The method is developed upon a very simple RL algorithm. The authors employ a very simple RL algorithm to deal with very simple tasks.

The proposed framework, VLOG, is not developed upon a very simple RL algorithm. We actually demonstrated 3 different RL algorithms in this paper (Dueling double DQN, Conservative Q-learning, behavior cloning). Also, we examined VLOG using a very challenging imperfect-information, stochastic game Mahjong (game complexity $> 10^{166}$, Appendix E.1) and noisy MinAtar environments.

For easier understanding, we summarized the tasks, the RL algorithms and the motivations of each task set in the table below, followed by our detailed rebuttal to the comments.


| Task                   | Maze (Sec. 5.1)                                                                               | Noisy MinAtar (Sec. 5.2)                                              | Mahjong (Sec. 5.3)                                                                    | Mahjong (Sec. 5.3)                                                                           |
|------------------------|-----------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|---------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| Motivation             | To visualize the internal representation learned by VLOG since Maze is simple and explainable | Atari games are standard environments for deep RL                     | Mahjong is highly challenging and is very suitable for testing oracle guiding methods | Same as left                                                                                 |
| Source of experiences  | online                                                                                        | online                                                                | offline                                                                               | offline                                                                                      |
| RL algorithm           | Dueling Double DQN                                                                            | Dueling Double DQN                                                    | Dueling Double DQN + Conservative Q-learning                                          | Behavior Cloning                                                                             |
| Implication of results | VLOG can leverage oracle observation to improve internal representations                      | VLOG works well for noisy, high-dimensional observations in online RL | VLOG works well in this very challenging task using offline RL                       | The Bayesian property of the model is beneficial. |

### Tasks
 
We examined VLOG in the tasks of Maze, pixel-based MinAtar, and Mahjong (Game complexity $>10^{166}$). The first task Maze is relatively simple, but **we just use its simplicity and interpretability to visualize how VLOG can help with representation learning (Fig.2 C,D)**. We use the second task MinAtar to demonstrate the scalability of VLOG to tasks with more high-dimensional state space. Most importantly, as we discussed at the beginning of Sec. 5.3, **the last task Mahjong is a highly challenging game** for its (1) extremely high game complexity $>10^{166}$, (2) imperfect information due to huge invisible state space $\sim 10^{48}$ [2] and (3) highly stochastic state transitions as in many card games. Therefore, we believe that VLOG has been well examined on challenging tasks, and its effectiveness has been proved via superior performance.

### RL algorithms

As for RL algorithms, first we would like to clarify that VLOG is not a single algorithm, but a general Bayesian framework for any value-based deep RL method. Besides, in the most challenging task Mahjong, VLOG is examined upon conservative Q-learning (CQL) [3], which is currently the state-of-the-art offline RL algorithm.


### **Continue in the next part**

 
--------------------

> 4-6. What is "suphx-style oracle guiding"? What is "OPD-style oracle guiding"? What is the "(trained) baseline model" used in Table 1?


Thanks for raising these questions. We have revised the paper in Sec 6.2 (5.2 before revision) to clarify these.

Suphx is the name of the Mahjong AI (Li et. al. 2020). In the implementation of Suphx, oracle guiding was done by the dropout (Appendix A and Fig.4 Left).

OPD is the abbreviation for "oracle policy distillation", an algorithm proposed by (Fang et. al. 2021) for oracle RL problems (Appendix A and Fig.4 Right).

We implemented the oracle guiding algorithms of OPD and Suphx, but other parts (For example, Suphx also used supervised learning to pre-train the network) may be different from their original implementation. So we refer to them as "xxx-style".

The "(trained) baseline model used in Table 1" was trained using the same RL algorithm as VLOG, Suphx-style, etc., but there was no oracle guiding (Fig.1C) for the baseline model. We now clarify this in the caption of Table 1.


## References

[1] Li, Junjie, et al. "Suphx: Mastering mahjong with deep reinforcement learning." arXiv preprint arXiv:2003.13590 (2020).

[2] Goyal, Anirudh, et al. "Z-forcing: training stochastic recurrent networks." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.

[3] Babaeizadeh, Mohammad, et al. "Stochastic Variational Video Prediction." International Conference on Learning Representations. 2018.


 
---------------------------

> Why isn't the top-left value of table 2 bolded as well? The VLOG line in Table 2 has overlapping CIs, so should not be bolded.

Thanks for the comment, we bolded the top-left value of table 2 and unbolded The VLOG line in Table 2 to make it more rigorous.



---------------------------


> Finally, the experiments are combining too many things: offline RL, multi-agent RL, etc. This makes it harder to reliably ascertain what the benefits of VLOG are. While interesting, I'd recommend clarifying the exposition of the experiments, or finding another (better known) benchmark, and leave the Mahjong experiments as an extra set of appendix experiments.

Thanks for the comments. We would like to clarify more about the Mahjong experiments. First, although Mahjong is a 4-players game, using offline RL data to train an agent does not involve multi-agent RL because the offline dataset is fixed: the opponents have fixed policies and thus can be considered as part of the environment. **Accordingly, our experiments still focus on single-agent RL to avoid the complexity caused by considering multi-agent RL.** We have revised Sec. 5.3 to include these discussions to make it clear. Nonetheless, how VLOG facilitates multi-agent RL remains an interesting future direction. 

Then, we would like to discuss the necessities for including the offline RL setting (the third paragraph of Sec. 5.3). Since the number of possible states in Mahjong is enormous, it is extremely costly to explore with random actions in an online RL manner, and no previous work could train a strong Mahjong AI with purely online RL. Also,  we would like to examine the effectiveness of VLOG in offline RL settings because VLOG is not only designed for the online RL setting. For these reasons, we transferred to offline RL for the Mahjong task using expert demonstrations.

Finally, as we explained in the first response, Mahjong is a promising test platform for oracle guiding methods. Thus, we think the experiments on Mahjong provide strong empirical evidences of VLOG's advantage.

--------------------

## Questions 

> 1. In footnote 2 in page 3, is there any relationship between $X$ (set of executors) and $\hat{X}$ (set of oracles)? I assume the latter is a subset of the full ?

We use $\boldsymbol{X}$ to denote the executor observation space
(executor observation $\boldsymbol{x} \in \boldsymbol{X}$) and use $\hat{\boldsymbol{X}}$ to denote the oracle observation space (oracle observation $\hat{\boldsymbol{x}} \in \hat{\boldsymbol{X}}$). The inclusion relationship exists between $\boldsymbol{x}$ and $\hat{\boldsymbol{x}}$, but not $\boldsymbol{X}$ and $\hat{\boldsymbol{X}}$. Specifically, oracle observation $\hat{\boldsymbol{x}}$ is the executor observation $\boldsymbol{x}$ concatenated with some additional useful information. Although they both partially represent state $\boldsymbol{s}$, $\hat{\boldsymbol{x}}$ should always contain no less information about $\boldsymbol{s}$ than $\boldsymbol{x}$. In the POMDP setting, state $\boldsymbol{s}$ contains perfect information the agent needs for making good decisions. Thus, oracle observation can be used to facilitate policy learning.  **We have revised the paper accordingly in Sec. 3 to make it clear.**

--------------------

> 2. In remark 1, is the $v_t$ in $p(v_t| \boldsymbol{x}_t)$ equal to $V(\boldsymbol{x}_t)$?

The reviewer is right. It is a typo, and $v_t$ should be read as $v(\boldsymbol{x}_t)$

--------------------

> 3. In the last paragraph of Sec. 5.2 it says "VLOG-no oracle performed surprisingly well", but this is not the case in the figures shown.

Sorry for the confusion. The full sentence is that "It is notable that VLOG and **VLOG-no oracle performed surprisingly well in seaquest.**"  We did not mean that VLOG-no oracle performed well in all tasks, but only for seaquest (the second panel of Fig.3). In seaquest, VLOG-no oracle significantly outperformed the baseline (we said "surprising" since VLOG-no oracle was not expected to outperform the baseline) probably because the stochastic network encouraged exploration (the last paragraph of Sec. 5.2 and footnote 4.)


 
We are greatly thankful to the reviewer for the comments. We hope the following replies resolve the concerns raised in the review. 

> My main concern with the paper is that the evaluation on Mahjong:

Thanks for raising the questions about Mahjong. To provide more information and intuition of the Mahjong experiments, we revised the manuscript in Sec. 5.3 to more clearly explain the Mahjong task, and added a picture of an actual Mahjong game in Fig. 5 in the Appendix. And the follows are the replies to detailed concerns.

--------------------------

> It is somewhat confusing to follow (for readers not already familiar with Mahjong), and it is not clear what the takeaway is (other than VLOG seems to work).



Sorry for the confusion. We chose Mahjong for good  reasons. The main goal of this paper is to better leverage the oracle observation to facilitate RL. To this end, we propose a general learning framework referred to as VLOG for deep RL. To demonstrate its effectiveness, Mahjong is a suitable and challenging task.

**Mahjong is a very suitable test bed for oracle guiding problem due to its game mechanism.** Specifically, in Mahjong, it is hard to make optimal decisions based on executor observation (player's own tiles and discarded tiles) because the outcomes heavily depend on invisible oracle information (the private tiles of the opponent), and the complexity of invisible state space is as high as $10^{48}$ on average [1]. The professional human players heavily leverage oracle observation to improve their skills, by checking and performing reasoning between executor observations and oracle observations at match replays. Besides, oracle guiding has already been shown to improve RL for Mahjong in the previous work [1].

In addition, Mahjong is a highly challenging game for the following factors and complex enough to test if the method is effective: (1) extremely high game complexity $>10^{166}$, (2) imperfect information due to huge invisible state space $\sim 10^{48}$ and (3) highly stochastic state transitions as in many card games.

For these reasons, the superior experimental results of VLOG well demonstrate that VLOG can successfully utilize oracle information to make better decisions in a highly complex task.

--------------------

> The results are somewhat suspect, since the oracle does the worst in Table 2, which is not what one would expect.


Table 2 has two panels corresponding to the conservative Q-learning (CQL) and behavior cloning (BC), respectively.  For CQL (upper panel of Table 2), oracle performed quite well as expected. So we guess that the reviewer means the BC results (lower panel of Table 2). We would like to clarify that **such results comes from the characteristic of BC as it is more like supervised learning.**

In the behavior cloning (BC) setting, the agents did not learn a value function, but tried to predict human experts' actions. Therefore, the training procedure did not involve the reasoning of the relationship between playing outcome and oracle observation, but just imitating human behaviors. The human behaviors were also based on executor observations. In contrast, doing reinforcement learning (learning a value of current environmental observation) involves such reasoning process. Therefore, oracle does not outperform others in this setting. But oracle is not the worst since it has overlapping confidence intervals with Suphx-style.

As for VLOG and VLOG-no oracle, they slightly outperform oracle and Suphx-style in the BC setting. This can be explained by that Mahjong has stochastic state transitions like many other card games, and variational Bayes is known to well handle the stochasticity [2][3]. Also, VLOG and VLOG-no oracle performs similarly well in the BC setting, which is consistent with that oracle observations are not helpful in BC.


---------------------------

> The method for scoring described at the bottom of page 8 ("we simply added up the payoff of the two being tested agents") seems quite strange and not really a fair evaluation. Is this really how Mahjong is played/evaluated?

Sorry for the confusion caused by our writings. We amended the top left cell and the caption of Table 1 to clarify our evaluation method. The following explains why this is a reasonable and fair evaluation.

Mahjong is a 4-player game, where each player aims at maximizing its own payoff and ranking. We tested the performance of trained models by letting them play against baseline models. Among 4 independent players, two of them were using the model being tested, and the rest were using the baseline model (no communication or team cooperation between agents). To decrease the statistical variance, we averaged (summed up) the results of two players with the same model and use it as the performance of the corresponding model.


This paper presents an approach to dealing with problems where agents could benefit from learning from state features that are only available during training, but not during evaluation. Their approach leverages Bayesian theory to propose a variational learning method and is evaluated on a variety of tasks, including a Mahjong simulation (which the authors are providing as part of their contribution). I really like the motivation and idea behind this paper, and I think it is something that future researchers could make use of. In particular, I like the simplicity of the approach.

My main concern with the paper is that the evaluation on Mahjong:
*  It is somewhat confusing to follow (for readers not already familiar with Mahjong), and it is not clear what the takeaway is (other than VLOG seems to work). 
*  The results are somewhat suspect, since the oracle does the worst in Table 2, which is not what one would expect.
*  The method for scoring described at the bottom of page 8 ("we simply added up the payoff of the two being tested agents") seems quite strange and not really a fair evaluation. Is this really how Mahjong is played/evaluated?
*  Why isn't the top-left value of table 2 bolded as well? The VLOG line in Table 2 has overlapping CIs, so should not be bolded.
*  Finally, the experiments are combining too many things: offline RL, multi-agent RL, etc. This makes it harder to reliably ascertain what the benefits of VLOG are. While interesting, I'd recommend clarifying the exposition of the experiments, or finding another (better known) benchmark, and leave the Mahjong experiments as an extra set of appendix experiments.

Questions for the authors:
1. In footnote 2 in page 3, is there any relationship between $X$ (set of executors) and $\hat{X}$ (set of oracles)? I assume the latter is a subset of the full $X$?
1. In remark 1, is the $v_t$ in $p(v_t | {\bf x}_t)$ equal to $V({\bf x}_t)$?
1. In the last paragraph of section 5.2 it says "VLOG-no oracle performed surprisingly well", but this is not the case in the figures shown.
1. What is "suphx-style oracle guiding"?
1. What is "OPD-style oracle guiding"
1. What is the "(trained) baseline model" used in Table 1?

---
Minor comments:
1. The abstract starts with "How to make intelligent decisions is a central problem in machine learning and cognitive science". Minor nit, but I'd say cognitive science is more concerned with _understanding_ how decisions are made.
1. In the abstract should say "RL using variational method***s***."
1. In the abstract should say "decision-making tasks rang***ing*** from video games..."
1. At the end of the second paragraph of the introduction, it's not necessary to include "etc" since you started the sentence with "including".
1. In section 2, remove "different" after "Another" in the second sentence.
1. In section 2 should say "make the student policy behave more simliar to ***the*** teacher policy. Both methods"
1. Title of section 3 should be "PRELIMINAR***IES***"
1. In the first sentence of section 4 should say "variable representing the environmental state"
1. In the first sentence of section 5.1, should say "by leveraging oracle observation***s*** in learning."
1. In the end of the first paragraph of 5.1, should say "another grid if not ***blocked by a wall***."
1. In the first sentence of 5.2 should say "we tested it ***on*** a set of MinAtar video games"
1. When citing the ALE, you should cite [Bellemare et al, 2013](https://arxiv.org/abs/1207.4708) instead of Machado et al.
1. In the third paragraph of page 8, should say "are sophisticated ways to encode the state and action space of Mahjong (Li et al., 2020), we attempt to make simplifications with reasonable amount***s*** of approximations since our goal is not to create strong Mahjong AI"
 See details above.

This paper tackles reinforcement learning scenarios when the raw observation obtained from the executor is not very descriptive and there are high quality oracle observations available for the learning process to guide the policy training. The proposed method aims to derive a representative latent state via incorporating a variational inference perspective. Specifically, the oracle and raw observations are processed by two individual encoders and then the two latent spaces are fused and fed to a shared decoder. The latent distribution obtained from the oracle observations serve as the posterior distribution and that from the raw observation serves as the prior distribution. The distance two distributions is minimized via minimizing a KL divergence term.  $\textbf{Strength}$:
- This paper is well written and easy to follow.
- Leveraging Bayesian theory to tackle the representation learning problem in reinforcement learning is an important research direction.

$\textbf{Limitations}$:
- The proposed method is not very general as it could only be used in the tasks where oracle observations are available.
- The  formulation of ELBO might not be very valid (refer to the detailed comments)
- The empirical evaluation domains are too simple and some of the experimental configurations are not properly specified.
- The method is developed upon a very simple RL algorithm. 

$\textbf{Detailed comments}$:
1. The assumption that there are always oracle observations available during policy learning does not hold for many applications, such as Atari 2600 and noisy robotics navigation tasks. And in the more challenging problems with partial observability, it is unclear whether this method could be directly applied and how to effectively regularize the posterior distribution which meant to represent partially observed states. 

2. I feel the formulation of ELBO is not that sound. I do not know why the latent of oracle observations are set as the posterior instead of that from executor observation. Also, it seems that there are two execution paths for the ELBO, i.e., there should be one for the oracle and another for the executor observation. But in Sec 4.2, this is not properly specified as Eq (2) only tackles one of the path. 

3. It is unclear how the variational model is related to the policy model, e.g., is the input to the policy $x_t$ or $z_t$? 

4. The authors employ a very simple RL algorithm to deal with very simple tasks. For instance, in the first task of maze navigation, in such a simple case I'm very sure using something like PPO will successfully solve the problem. I don't understand why the problem could be unsolvable, i.e., the performance of baseline is such poor. I'm also concerned if the experimental settings are properly configured. For instance, in the maze task, the input is very shallow, i.e., with only ~3 dimensions, but the authors specify the latent to have a dimension of 128. I believe such setting is not very reasonable.  

5. In many  cases, even though the oracle features might be available, they might   have very different  format  compared to the raw observations, e.g., (x, y, z) vs  high-dimensional images. In such scenarios,  would   the two-encoder-one-decoder architecture still work?

6. Overall the novelty of this paper is relatively limited and there lacks discussion with a number of related works that combine variational inference with policy learning, such as:
[1] Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
[2] Variational methods for Reinforcement Learning
[3] Sequential Generative Exploration Model for Partially Observable Reinforcement Learning
[4] Reinforced Variational Inference This paper proposes a method to leverage oracle observations with a variational inference framework. But the applicability of the method is rather limited as it  poses a very strong assumption that oracle observations need to be accessed for training the policy. The empirical evaluation domains are too trivial, e.g., very simple decision making tasks and there is no challenging POMDP task employed to demonstrate the KL regularization works. There are also some problems regarding to how the methods are implemented. 

This paper proposed a variational-based model VLOG for solving the oracle-guiding RL problem. By minimizing the KL-divergence between the latent features from the oracle observations and the executor observations, the model can apply the oracle signals to boost the execution performance. The author(s) derived a lower bound objective as the training loss and empirically demonstrated the performance of their method under three different types of environments. The paper is easy to follow and generally well-written. The proposed method is intuitive and effective, as an application of variational models. The oracle-guiding problem is novel and intriguing. It is similar to the offline RL, but there are some key differences that can distinguish them. I urge the author to formally define the problem to help the readers who are less familiar with the oracle-guiding framework. I summarize some of my concerns in the following:

concerns:
1. Can you summarize the difference between offline RL and oracle-guide RL. I think the key difference is the training observations (or oracle observations) contain a richer signal than the observations in the testing environment (or executor observation), but there could be more. Please formally define the oracle-guiding RL problems anyway.
2. The author claims "VLOG is theoretically guaranteed to ...", where are the theoretical results? I think we can show by maximizing this variational lower bound, the KL divergence between the true posterior and the approximate posterior will be reduced, but this is a well-known property. Where are the new contributions?
3. How do you get the label v^{tar}_{t}? (in (1)). I assume a DQN or a critic network must be implemented. If it is the case, please define the model and its loss.
4. Will it make things easier by replacing a hyper-parameter \beta with another hyper-parameter D^{tar}_{KL}? I guess both hyper-parameters are difficult to determine. In fact, other works might design a linear or exponential scheduler for \beta that allows it smoothly grow from a min (in most cases, 0) to a max value during training. As far as I know, this method works quite well in practice.

Minors:
1. Many sentences are repeated in the abstract and introduction, but readers with a reasonable RL/ML background can follow this message with one scan. Maybe use another example for better elaboration.
2. In remark one, the paper says "they can choose q(v_t|\hat_{x}_t) to be p(v_t|x_t)u(v_t; [l,u])". Are you multplying two distrbutions? In this sense, q will not define any distribution.
3. Poker games involve two players competing with each other. In this sense, maybe a (PO)Markov Game model can better describe this environment than a (PO)MDP.

typos:
1) the oracle-estimated one as posterior distribution -> the oracle-estimated one as the posterior distribution. This paper tackles an intriguing oracle-guiding problem. The proposed method is intuitive and effective, as an application of variational models. The paper is generally well written. The concerns I proposed are mostly minor and can be easily resolved within a reasonable time. I did not capture major bugs in this paper. The experiment is complete and promising.  I am happy to vote for an accept given other reviewers do not find major issues.

Meta-review: This paper studies the problem of using oracle information that's only available during training in RL. The key contributions are 1) a variational Bayesian approach that models the oracle observation as latent variables; and 2) a Mahjong environment for benchmarking RL with oracle guiding. The novelty of the proposed approach is limited, but reviewers find the problem intriguing and agreed that it's a reasonable application of Bayesian approach to RL with latent oracle information. In addition, the Mahjong environment could benefit the community and spur new work in this direction. Therefore, I recommend this paper to be accepted as a poster.

Decision: Accept (Poster)