Variational oracle guiding for reinforcement learning
Authors: Dongqi Han
Tadashi Kozuno
Xufang Luo
Zhao-Yun Chen
Kenji Doya
Yuqing Yang
Dongsheng Li
Venue: ICLR 2022 Poster (2022)
Keywords: variational Bayes
oracle guiding
reinforcement learning
decision making
probabilistic modeling
game
Mahjong

Abstract: How to make intelligent decisions is a central problem in machine learning and artificial intelligence. Despite recent successes of deep reinforcement learning (RL) in various decision making problems, an important but under-explored aspect is how to leverage oracle observation (the information that is invisible during online decision making, but is available during offline training) to facilitate learning. For example, human experts will look at the replay after a Poker game, in which they can check the opponents' hands to improve their estimation of the opponents' hands from the visible information during playing. In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods. Our key contribution is to propose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL. VLOG is featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value-based DRL algorithm. We empirically demonstrate the effectiveness of VLOG in online and offline RL domains with tasks ranging from video games to a challenging tile-based game Mahjong. Furthermore, we publish the Mahjong environment and an offline RL dataset as a benchmark to facilitate future research on oracle guiding (https://github.com/Agony5757/mahjong).

Published as a conference paper at ICLR 2022
VARIATIONAL ORACLE GUIDING FOR REINFORCE- MENT LEARNING
Dongqi Han"!, Tadashi Kozuno", Xufang Luo", Zhaoyun Chen', Kenji Doya', Yuqing Yang". and Dongsheng L.i²
Okinawa Institute of Science and Technology University of Alberta
Microsoft Research Asia
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
ABSTRACT
How make intelligent decisions is a central problem in machine learning and artificial intelligence. Despite recent successes of deep reinforcement learning (RL) various decision making problems, an important but under-explored as- pect is how to leverage oracle observation (the information that is invisible during online decision making, but available during offline training) to facilitate learn- ing. For example, human experts will look at the replay after Poker game, in which they can check the opponents' hands to improve their estimation of the opponents' hands from the visible information during playing. In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods. Our key contribution pose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL. VLOG featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value- DRL algorithm. We empirically demonstrate the effectiveness of VLOG in online and offline RL domains with tasks ranging from video games to challenging tile- based game Mahjong. Furthermore, we publish the Mahjong environment and an offline RL dataset as a benchmark to facilitate future research oracle guiding
pro-
INTRODUCTION
Deep reinforcement learning (DRL) has undergone rapid development in recent years (utton & Barto 2018 Mnih et al.) 2015, Vinyals et al. 2019). However, there is a common and important but exper-explored aspect in RL: imagine that after playing Poker game, human player may look at the replay to check opponents' hands and analyze this information to improve his/her play- ing strategy (or policy) for the next time. We refer the information like opponents' hands as oracle observation, defined as the information invisible to the agent during online task execution but available during offline training. By contrast, the information available during task execution is called executor observation. Such a scenario has been referred to as oracle guiding for RL (Li al. 2020 Fang et al. 2021) (see Sec. [ for formal definition). Oracle guiding common in real life. For example, when taking an examination (the oracle observation is the answers to similar questions, which are available only during preparing); and training a robot to perform some tasks on the Moon (when training the robot, we can provide it with the information of the territory. which are not available during execution). The type of oracle observation can be diverse, including hindsight information (Harutyunyan al. [019]] Guez et al. (2020). human feedback (Knox & Stone, 2209); Lofún et al., 2016: MacGlashan et al. 2017), re-calibrated data with post-processing, and hidden states in partially observed setting (Li et al. 2020).
While humans naturally perform oracle guiding when learning to make decisions, it remains chal- lenging in RL. The difficulties include: (1) how to guarantee that learning with oracle observation
Work done during an internship in Microsoft Research Asia. Email: donggi.han@oist.j. http://hhwh..cm/////oo/////////m

---

Published as a conference paper at ICLR 2022
improves the main decision model using executor observation only, and (2) if introducing iliary loss leveraging oracle observation, how to tradeoff between the main loss and auxiliary loss. While recent studies attempted to model oracle guiding in RL (Guez et al. 2020: Li et al. 2020 Fang et al. 2021). none of them addressed these difficulties (refer to the Related Work section for more details). In particular, all these proposed methods are heuristic: although empirical results showed performance gain with oracle guiding, it is not theoretically guaranteed that the usage of oracle observation improves execution performance.
In this paper, we propose fundamentally new idea for oracle guiding based on Bayesian theory. Taking Poker as an example, we know that the learning of optimal strategy is tractable if knowing the global, true state of the environment (or simply state 2 including all visible or invisible cards, the opponents' playing style, etc. (Azar et al. 2017; Jin et al. 2018 2020). A key part of skill improvement is learning to estimate the probabilistic distribution of environmental state from ex- ecutor observation. The common way to do this by human experts is to watch match replays where the oracle observation (e.g. opponents' hands) is available, and then use the oracle-estimated state to correct the executor-estimated state. We interpret this to Bayesian language: executor-estimated state the prior distribution, and the oracle-estimated one the posterior distribution. Thus, the training objective can be considered two-fold: learning to make decisions based on posterior esti- mation of state, and learning prior distribution of state closer to the posterior one.
We formulate this idea by proposing a novel learning framework for general oracle guiding prob- lems based variational Bayes (VB) (Kingma & Welling 2014), referred to as variational latent oracle guiding (VLOG). VLOG owns several preferable properties. First, VLOG is theoretically guaranteed to leverage oracle observation for improving the decision model using executor obser- vation. Second, VLOG is a versatile DRL framework that can be integrated into any value-based RL algorithm and agnostic to the type of oracle observation. Third, VLOG does not suffer from the necessity of tuning additional prpee-pprameters. Finally, we empirically show that VLOG con- tributes to better performance
variety of Decciiionmmking tasks in both online and offline RI, domains. The tasks include a simple maze navigation, video games playing, and a particularly chal- lenging tile-based game Mahjong in which humans heavily leverage oracle observation in learning (Li et al. 2020). We also contribute to the community by taking Mahjong as benchmarking task for oracle guiding and publishing the RL environment and dataset to facilitate future research.
2 RELATED WORK
In the past few years, research interests have grown on DRL and imitating learning (Chen et al.
2020 with leveraging oracle or hindsight information. For DRL, Guez al. (2020); Fang et al. 2021 considered hindsight observation (executor observation at future steps) as the oracle obser- vation during training. Guez et al. (2020) used hindsight observation facilitate learning rep- resentation of current state. Another method (Fang et al.) (021)) was used for stock trading: the authors trained a teacher (oracle) policy with hindsight information, and employed network distilla- tion to make the student policy behaves more similar to the teacher policy. Both the methods (Guez al., 2020) Fang et al., (02T) are heuristic and focused on making use of future observation for a better sequential modeling, while VLOG is theoretically guaranteed for any kind of oracle observa- tion. For applications in imperfeettinfooraaion games, Suphx (Li et al. 2020). a DRL-based Al for Mahjong, also introduced method to leverage oracle observation (opponents' hand) for stronger performance. They concatenate oracle observation with executor observation as the input of policy network, where the oracle observation is timed by scalar variable which is annealed from 0 during the training course. However, the method used in Li et al. (2020) is also heuristic and has only been tested in one task.
Variational Bayes (VB) is a well-established method and has been taken advantage of in RL. For example, control as probabilistic inference uses VB connect the objective function of RL and variational lower bound of a probabilistic inference problem (Furmston & Barber 2010; Weber et al.) 2015 Levine 2018). Our idea differs since it frames value regression problem by the maximum- likelihood problem, and then, it applies VB to solve it (see Sec. Also, the usage of VBiinn network models for DRL has captured attention from researchers recently. For example, Ha & Schmidhuber (2018) proposed to employ a VAE to reduce the high dimensionality of image observation; Igl et al.
Generally, oracle observation does not necessarily contain all the information of environmental state.
2

---

Published as a conference paper at ICLR 2022
2018 : Han et al. (2020); Lee al. (2020) proposed variational RNNs the state- transstranitiin models to encode the belief states of the agent; Yin et al. (202I) utilized a variational sequential generative model for predicting future observation and used the prediction error to infer intrinsic reward to encourage exploration; and [Okada et al. (2020) demonstrated performance gain by using a deep Bayesian planning model in continuous control tasks. Our study differs from the mentioned works
by focusing on oracle guiding, and VLOG does not involve learning state transition model.
ORACLE GUIDING POMDP
Here we define the problem based on the Partially Observable Markov decision process (POMDP) [Sondik 1978). An oracle guiding POMDP distinguishes from the original POMDP by having two types of observations: executor and oracle. The executor observation is always available to the agent, on which the agent's decision making (execution) relies. The oracle observation á is not accessible during execution, but can be obtained afterward. is included in & since the former is always available. Thus & contains no less information of the underlying environment state than æ.
A formal definition of oracle guiding POMDP is a tuple and A are the state and action spaces, respectively. Po specifies the initial state distribution such that (s((s) is the probability of state & E S being an initial state. T specifies the state transition probability such that T(s', r(ss,,)s,,)) a) is the probability of reaching to new state s' E S with an immediate reward E R after taking an action a E AA at state s E S. X denotes the executor observation space and X denotes the oracle observation space. O specifies the executor observation probability such that (x||s) 3) is the probability of an executor observation æ E at a state s e S. Similarly, Ô specifies the oracle observation probability such that Ö 8) is the probability of an oracle observation & E X at a state & E S. 7 E [0. 1) is the discount factor. Value functions are the expected value of return from a particular state or state-action pair. For policy n, its Q-function is defined by q² (s, a) = E, 1>== 7"
T, X, 7), where S
8, ar = a). where Ex indicates the expectation when the policy æ is followed. The state value function T defined by u (s) == [a)())) (9" (s, (s.a)] The agent aims maximize (e)," (T (s) with respect to æ, and the value-functions play an important
role to this end (Sutton & Barto 2018)
VLOG: VARATIONAL LATENT ORACLE GUIDING
Let us introduce a latent vector 2t, which is probabilistic variable representing the environmental state 8t. From a Bayesian perspective, we consider the prior distribution p(z,| (2+)) as the agent's estimated probability density function (PDF) for based on executor observation æ,, Meanwhile, the posterior distribution PDF ((z (.:)) is modeled based on oracle observation i;.
In RL, the most basic requirement is to make a good estimation of return by a value function approx- imator vxx;)) == ((,,,,))) )dz, (we denote by v, but it can also be a Q function by simply replacing æ, with (x,; a:)) based on available information, i.c. exccutor observation œ;. The target of return, denoted by of"T, can be estimated by any value learning algorithm such as TD(0) (Sutton & Barto 2018). Peng's Q()) (Kozuno al. 2021), etc. (generally, pfar can always be given by Bell- man equations. However, one can also use Monte-Card return as ytar available). In particular, we employed double Q-learning with dueling architecture (Wang et al. 2016) compute yfar due to its effectiveness and simplicity (Sec. and B. We want to maximize the log-likelihood objective of the estimation of return based on executor observation æ, (i.e., for the executor model)
/ P (u(z)) = ((aT)z)) (2)) 21
L = log P (v(#)) = (20)) = log
{
q(z, (e)) p (u(2+) = ofar |2x) (((z1 xx))
=log
3

---

Published as a conference paper at ICLR 2022
By Jensen's inequality, we have
c 2z g(z (ix) log (w(z,) = ofar| (ze) p(zz (())) dz
((2, ((x)
= S1 ((2)) log P (u(z+) = v/2 (1))))))=))) - ((2e((0)) log
g(2e (2t) dzc
pl Ze xx4)
( [log P (u(z+) 1t"0" 12)]] DrL (a (ze (z)((z)) lpp (ze (xx)) == LvLOG.
(1)
oracle prediction error
regularization term
Thus can maximize initial objective L via maximizing L which is also known as the vari- ational lower bound (Kingma & Welling, 2014), but in our oracle-guiding scheme. Since p(zz (x)) and g( (z)) represents the PDF of the latent vector obtained from executor observation and oracle observation, respectively, the meanings of the two terms in LyLog appear clear now: the first term, i.e., oracle prediction error, helps to improve value estimation from posterior latent state distribution (z) computed with oracle observation); and the second term, i.c., regularization term, helps to shape the prior representation of 2, closer to the posterior one as latent oracle guiding. We would like to highlight that the VLOG objective is the lower bound of the objective for the prior executor model (x2e) (the estimation of return using executor observation 2;²) with the usage of oracle observation Ct. This lower bound guarantees that the usage of oracle observation facilitates the learning of the executor model, which our original motivation.
One may use any shape of the approximate posterior q, depending on which different instance of VLOG is possible. Furthermore, one may directly use v(x;) instead of p(zz)æ)). These design choices allow users to incorporate any prior knowledge on oracle observation. For example, if one knows the range of a state-value at I closed interval [t. u), the approximate posterior q(v) (t) can be restricted to family of probability distributions supported on (1. u).
Remark 1.
(A)
VLOG: learning
(B)
VLDG: execution
(C)
Baseline
Qtanget
Quast
Orade prediction error
Regularization
prror
post
2 PITTO
Posterior enpodet
Price cocodel
ensoler
Figure Implementation of VLOG for deep Q-learning during (A) learning and (B) execution. Exccutor observation I; and oracle observation ã; use two different encoders but share one decoder (MIP for estimating Q function). (C) Baseline model without using oracle observation. rectangle means deterministic layer/variable, and an ellipse means a stochastic one. Here the outputs are multi-head Q-values corresponding to discrete actions.
4.1 IMPLEMENTATION WITH NEURAL NETWORKS
Inspired by the implementation of variational auto-encoder (VAE, Kingma & Welling (0014), propose the neural network architecture of VLOG [T.. The executor observation D; and oracle observation d; are processed by two distinct encoder networks to compute the prior and posterior distribution of the latent vector, respectively. During training, both æ, and é; are available, and all the network parameters are updated by maximizing the VLOG objective in an end-to-end manner (Fig. [A). During execution, the agent computes prior distribution p( p(z(æ) (xx) for decision making (Fig. BB) without using oracle observation. z; is computed by parameterized normal distribution p(z1(x;) = N (Hf, exp (log of)), (n,, log o = prior encoder(x)))
(22,(,,) = Au, exp (log of)), (#L) log of) posterior encoder((.).
For computing P(o(zz) ofar|2 lz.) in Eq. I we simply assume it follows normal distribution, and estimate it with mean square error between ((2)) and vaar in practice. The reparameterization trick is used to perform end-to-end training as in VAE (Kingma & Welling, 2014). Then the output of the decoder (value function) can be obtained by v(2,) = decoder(2), (zz). Note that 2, is obtained using the posterior encoder during training, and using the prior encoder during execution (Fig. (A, B).
4

---

Published as a conference paper at ICLR 2022
4.2
LATENT BOTTLENECK CONTROL
To learn a better representation, we borrow the idea from 3-VAE (Higgins et al. 2016) to multiply coefficient 3 to the regularization term. Thus we have the loss function (negative lower bound)
Rooa - By( (a,) [log P (v(z+)
[=)) + 3Dx1. (1 (éx) lpp (ze
(2)
The hyper- parameter 3 controls the capacity of the latent information bottleneck (Tishby & Za-
slavsky 2015 Alemi al. 2017). We found the choice of 3 is important for the performance of are not desired. Inspired by the method used in Burgess et al. (2017) for controlling the scale of KL divergence in B-VAE, we propose a task-agnostic method to automatically adjust /3 by setting target of KL divergence Dier. particular, we minimize the auxiliary loss function (3 the being optimized parameter)
VLOG in RL (see Appendix B33 However, having extra
Tg = (log10 DKL 1og DKL (4 (z, (+)2))llp (z log(())
The intuition here to strengthen the regularization by increasing 3 when the divergence between prior and posterior is too large, and vice versa. This method is similar to that used in soft actor-critic for automatically adjusting the entropy coefficient (Haarnoja et al. 2019). while we used this for KL divergence coefficient. Importantly, we found well-performing value Dier 50 agnostic to other design choices. It worked well in a range of different tasks and networks (Sec. 5). Therefore, we do not need tune 3. We provide more discussion about this method in Appendix
EXPERIMENTS
How does VLOG perform in practice? We investigated the empirical performance of VLOG in three types of tasks using online or offline RL, from simple to difficult. In the following experiments, we used double DQN with ducling network architecture (van Hasselt al., 2016 Wang et al.) 2016) as the base RL algorithm, the model and loss functions for RL are defined in Appendix. B.1 As DRL is susceptible to the choice of hyper-paramette the effect of oracle guiding. Double DQN and dueling architecture are preferable for the base algo- contrast to other DQN variants (Hessel et al. 2018), such prioritized experience replay (Schaul et al.) 2016), noisy network (Fortunato let al. 2018), categorical DQN (Bellemare et al. (017). and distributed RL (Kapturowski et al. 2018 Importantly, used the same hyper-parameter setting for all methods and environments as much as possible (see Appendix B.2).
introducing any new hyper-parameters might obscure
rithm since they require no additional hyper-parameterts
5.1 MAZE
We first demonstrate how VLOG helps to shape the latent representation by leveraging oracle obser- vation in learning. The testbed is maze navigation (Fig. with 10x 10 grids. The exccutor observation is the (x, y) position, where x, y are continuous values randomly sampled within each grid of the maze (thus the observations in two adjacent but wall-separated grids may be very close). At each step, the agent selects an action (going up, down, right, or left) and moves to another grid if not blocked by wall. We provided the VLOG agent with oracle observation (Ie yo. dg) during training, where Te. De are coordinates of the center of the current grid, and d, is the (shortest) path distance to goal from the current grid. It is intuitive that although the raw observation is (æ. y). dg matters more in a maze navigation task. We empirically investigated how much such oracle observa- tion used in learning of VIOOG could help to shape the latent representation of z w.r.t. dy rather than position. The encoder and decoder were both 2-layers multi-layer perceptrons (MIP) with width 256 and ReLU activation. The size of latent vector 2, for VLOG was 128 since we computed both and o (Appendix C
Experiments show that the baseline agent struggled to reach the goal (Fig. 2),, while VLOG agents stably solved the task after learning. To check how the usage of VLOG affected the learned la- tent representation, we visualize the latent representation of both VLOG and baseline models with principal component analysis (PCA. Pearson F.R.S. (190I). In Fig, D map path distance to goal dy to color and plot the scores of the first 2 PCS of z (computed using executor observation)
5

---

Published as a conference paper at ICLR 2022
(1)
VLOG
Baselline
agent
3
grod
Latert state PC1
Long stoee
VLDG
Baseline
oo
nillon 0895
Figure 2: Learning latent representations by oracle guiding a simple maze. (A) Illustration of the task. (B) Success rate of using VLOG and not using VLOG (baseline), each using 8 random seeds. (C) PCA of in VLOG and the corresponding hidden state baseline model (z collected during execution, not using oracle observation). Color indicates normalized path distance to goal. The latent state of VL.OG, trained with oracle observation, showed more sequential and interpretable representation w.r.t path distance to goal. We circled the latent regions corresponding to some places in the maze that the (z, 3) positions are close but distances to goal distinguish.
(D) The learned latent representation w.r. position in the maze, where the green and black component of the color corresponds to the score of the first 2 PCs of the latent state, respectively, as shown by the colormap (similar colors indicate similar representations).
for VLOG and the corresponding latent state for baseline using successful trials ('latent layer" in Fig. TC). The latent state of VLOG showed a relatively smoother and more interpretable represen- tation of distance-to-pn to-gooll to-goal comparing to that of baseline. We then plot the latent representations for different positions in the maze in Fig. PD. The latent state of VLOG more clearly represented dg. looked into rectangle region (denoted by rect- angles in Fig. 2D) inside which the left 2 grids and right 2 grids are segregated by a wall. We found the corresponding areas in the latent PC space and circled them in Fig. Dc While these grids are close in (z. y) (executor observation), their distances-to- tooccss-t-ggoall (oracle observation) are highly distin- guished. By leveraging oracle guiding using VLOG, the agents can clearly differentiate the left grids and the right 2 grids in latent space as shown in Fig. 2 D, left (note that the latent state here of VLOG was computed using executor observation only). By contrast, the latent representations of these grids were overlapped for the baseline model, which did not utilize the oracle observation Fig.2c.. D. right). In sum, we demonstrated with a toy example that VLOG effectively helped the latent space to couple with oracle state useful for the task. The following sections will transfer to experiments on more complicated tasks and discuss how VLOG can improve practical performance.
consistent with the result (Fig. 2C). In particular,
5.2 NOISY MINATAR
To evaluate how VLOG scales with more high-dimensional state space, we tested it a set of MinAtar videos games. MinAtar (Young & Tian 2019) is a test platform for AI agents, which implements 5 miniaturized Atari 2600 games with discrete actions (Seaquest, Breakout, Space In- vaders, Freeway and Asterix). MinAtar is inspired by Arcade Learning Environment (Bellemare al. 2013) but simplifies the environments for efficiency. The observation is 10x 10 pixels with multiple channels indicating different objects. In real-world, the observation usually contains some noise. Thus it is natural consider the noisy observation as the partially-observable executor obser- vation, and the original, non-noisy observation the oracle one. Suppose that at each frame, each pixel may "break" randomly with an independent probability of 1/8 (Fig. 3A). The original obser- vation broken pixel is erased and is replaced by different value in all channels. We consider such noisy MinAtar environments with the noisy pixels as the executor observation and the original pixels the oracle observation.
The network structure was the same as that for Maze, but the encoder was replaced by a CNN (Appendix C). We ran experiments all the 5 environments of MinAtar with VLOG well as

---

Published as a conference paper at ICLR 2022
(A)
(B)
breakkou
Invaders
Sestest
freeway
random
VILGG modd
broken
pixels
famos
Figure 3: (A) Illustration of the MinAtar environments with randomly broken pixels. (B) Learning curves showing normalized average return (divided by the average return of trained oracle model) of VLOG and alternative methods each using 8 random seeds.
baseline, oracle and alternative oracle guiding methods (see Appendix A for details). Baseline model always uses executor observation as network input (Fig. [C) Oracle is the same as baseline except that it always receives oracle observation (i.e., cheating, we ran the experiments of oracle for reference). VLOG-no oracle is an ablation of VLOG where we use the executor observation as the input to posterior encoder in VLOG (oracle observation is not used). Suphx-style oracle guiding is the oracle guiding method used in the Mahjong AI Suphx (Li et al. 2020). in which the exccutor observation and the dropout-ed oracle observation (with dropout probability p pdropout) are concatenated the input to the network. With training proceeds, proopout is gradually increased from 0 to 1, thus the trained network does not need oracle observation for input (Appendix (A]. OPD-style oracle guiding is the oracle guiding method used in oracle policy distillation (OPD) (Fang et al.) 2021 OPD-style oracle guiding first trains a teacher model using oracle observation as input, and then trains the executor model using an auxiliary loss, which is the error between the executor and the teacher models' estimation for value function (Appendix AA.
The results show that oracle performed usually the best, as expected. We normalized the perfor- mance of non-oracle models using oracle model as reference, for more clear comparison (Fig. BB). Among all the oracle guiding methods (VLOG, OPD-style and Suphx-style), VLOG consistently performed the best. It is notable that VLOG and VLOG-no oracle performed surprisingly well in Seaquest. This can be explained by that Seaquest is a task with a local optimum (see Appendix [E) while the stochasticity in hidden states of VLOG helped exploration in latent space to escape from the local optimum (a similar idea is Fortunaao et al. (2018), but their noise was added to the network weights). Except in Seaquest, VLOG-no oracle did not show significant performance different with baseline, showing that the performance gain of VLOG in this task set mainly came from leveraging oracle observation for shaping latent distribution; and the usage of variational Bayesian model was, at least not harming the performance when there is no helpful oracle information.
5.3 OFFLINE LEARNING ON MAHIONG
Mahjong is a popular tile-based game with hundreds of millions of players worldwide (here we consider the Japanese variant). The game like many other card games (but using tiles instead of cards), in which multiple (usually four) players draw and discard tiles (totally 136 tiles) alternatively satisfy winning conditions. It is highly challenging game characterized with (I) imperfect information in executor observation (a player cannot see opponents' private tiles and the remaining tiles to be drawn), (2) stochastic state transitions as in many card games, and (3) extremely high game complexity (i.e.. the number of distinguished, legal game states). Complexity of Mahjong is much larger than 10 (Appendix R.I. For reference, complexity of Go 10 (Silver et al., 2016) and complexity of no- no-limit Poker is 102 (Johanson 2013).
In Mahjong, it is hard to make optimal decisions based on executor observation because the out- comes heavily depend on invisible information, and complexity of invisible state space is as high as 10 on average (Li et al. 2020). In response to this challenge, Li et al. (2020) introduced suphx-style oracle guiding and demonstrated a performance gain. Thus, consider Mahjong
promising test platform for oracle guiding methods. Since the number of possible states in Mahjong extremely large. It is costly to explore with random actions in an online RL manner, and no pre-
7

---

Published as a conference paper at ICLR 2022
vious work could train a strong Mahjong Al with purely online RL. Also, we would like examine the effectiveness of VLOG in offline RL settings. For these reasons, we transferred to offiie RL (Levine et al. 2020 for the Mahjong task using expert demonstrations.
We processed about 23M steps of human experts' plays from the online Mahjong game platform Tenhou https////ennoounn///mooo../ htmll to dataset for offiin RL (data were augmented using the symmetry in Mahjong, see Appendix F). Also, we created a simulator of Mahjong as the testing environment. Though there are sophisticated ways to encodes the state and action space of Mahjong (Li et al. 2020), we attempts to make simplifications with reasonable amounts of approximations since our goal is to not create an strong Mahjong AI, but to use Mahjong as a platform to study oracle guiding problems. In our case, the action space is composed of 47 discrete actions covering all decisions in Mahjong. An executor observation is a matrix encoding public information and the current player's private hand; An oracle observation concatenates the executor observation with the information of the opponents' private hands. (see Appendix F). We used an 1-D CNN as the encoder as done in Mahjong AIs commonly (Li et al. 2020), and the size of 2, and the decoder network width was increased to 512 and 1024, respectively (Appendix C
Note that although Mahjong is a 4-player game, using offline RL data to train agent does not involve multi-agent RL (Zhang et al. 202I) because the offline dataset is fixed: the opponents have fixed policies and thus be considered as parts of the environment. Our experiments focused on single-agent RL to avoid the complexity caused by considering multi-agent RI. We investigated two kinds of offline RL settings. The first is conservative Q-learning (CQL) (Kumar et al.] (2020). Our CQL setting differed from the online RL setting in previous sections by adding auxiliary CQL loss (ummar al. 2020) the Q-learning loss function (Appendix B.I.2). The other is behavior cloning (BC). Although VLOG was designed for value-based RL, we could straightforwardly incorporate VLOG with by letting the network predict action instead function. The learning was conducted by minimizing the cross entropy between the output and the target action (demonstration) as in classification problem. Note that we did not test OPD-style oracle guiding in BC setting because it was equal to the baseline since we can directly use demonstration actions as oracle policy for distillation.
being evaluated model
CQL
BC
vs. baseline model
avg. payoff
match win rate(%)
avg. payoff
match win rate(%)
Oracle (cheating) Suphx style
224 + 13
53.6 + 0.4
15 + 12
50.5 + 0.4
59 + 13
51.2 + 0.4
37 + 12
50.9 + 0.4
OPD style
-9 + 13
50.0 + 0.4
VLOG VLOG-no oracle
233 +
55.7 0.4
+ 12 67 12
51.4 + 0.4
61 + 13
52.0 + 0.4
52.2 1 0.4
Table 1: Performance of each model after training vs. the trained baseline model on Mahjong, each using 4 random seeds. Each match was played by four independent players (agents) for a series of 8 games. A player wins match if it has the highest points among all players when finishing 8 games. Each player played for itself and only consider to maximize its own payoff and ranking. Among the 4 players, two were being tested agents and the other two were the baseline models (no communication or team cooperation between agents). The bascline model was trained with the same offline RL process, but without oracle guiding (Fig, [[)). To decrease the variance, we sum up the results of two players with the same model, and use it as the performance of the corresponding method. Average payoff is the mean of points change per game. Each type of match was repeated 12,500 times (100,000 games). Data are Mean + S.E.M..
Because Mahjong is a zero-sum, four-player game, we tested the performance of trained models in two scenarios: playing with the (trained) baseline model (Table I} and playing with each other (fight each other, 2). In the first scenario, four agents were playing the same matches on the game table, where two of them were being tested agents and the other two were the baseline models. communication between players, we simply
Although each agent played for itself and there added up the payoff of the two being tested agents, and consider they won a match if one of them ranked top (so the match win rate will be 50% if equally strong as baseline). for statisties (Table T).
8

---

Published as a conference paper at ICLR 2022
Fight each other (CQL)
avg. payoff
match win ratc(%)
game win rate(
deal-in rate(
Oracle (cheating)
168 + 11
28.0 + 0.4
18.45
4.48
Suphx style
-75 12
23.2 0.4
18.32
8.84
OPD style
-209 = 12
20.1 = 0.4
14.39
8. 8.06
VLOG
116 = 12
28.7 = 0.4
19.36
8.28
Fight each other (BC)
avg. payoff
match win rato(%)
game win rate(%)
deal-in ratc(%)
Oracle (cheating)
-27 + 11
23.9 1: 0.4
20.69
7.96
Suphx style
-10 11
24.6 0.4
20,51
8.03
VLOG
26 = 11
25.6 = 0.4
20.82
8.00
VLOG-no oracle
11 11
25.9 + 0.4
20.76
8.15
Table 2: Trained models playing with each other the same table. Deal-in means the player discards tile and another player wins the game by picking up this tile to compose winning hand. Dealing-in results a punishment the player, thus is better be avoided. Each type of match was repeated for 12,500 times (i.e. 100,000 games). Data are Mean + S.E.M..
For CQL, the results (Table 1 left and 2 upper) show that VLOG substantially outperformed the baseline and alternative methods (because Mahjong is a highly random game, 55.7% match win rate indicates large skill gap). Interestingly, VLOG was even comparable to oracle. This can be explained by that VLOG also benefited from its Bayesian property, which consistent with that VLOG-no oracle showed a significant performance gain over the baseline model [Table[]]eft). Still, the oracle model learned to reduce deal-ins (i.e., player discards tile and another player wins the game by picking up this tile to compose a winning hand) since it could explicitly see the opponents private tiles, showing a much lower deal-in rate than other non-cheating models (Table 2 upper).
In BC setting, the agents did not learn value function, but tried to predict human experts actions. Therefore, the training procedure did not involve reasoning the relationship between playing out- come and oracle observation, but just imitating human behaviors. This can be seen from the results that oracle did not substantially outperform baseline in BC (Table I right and lower). However, VLOG and VLOG-no oracle still showed performance gain, thanks to the stochastic modeling.
6 SUMMARY
We have proposed VLOG variational Bayesian learning framework for leveraging oracle obser- vation to facilitate DRL especially in partially observable environments. VLOG is available for any RL problem in which there is oracle observation that may help the executor make decisions.
We first introduced a latent vector z to represent the environmental state. The prior and posterior distribution of is modeled using executor and oracle observation, respectively. Then, we derived a variational lower bound (Eq. 2 by maximizing which we can optimize the executor model us- ing the oracle observation. We developed the corresponding methodology for DRL, which can be incorporated with most RL algorithms that need to estimate value function.
If oracle observation contains more information to retrieval the environmental state (or, it the true environmental state), VLOG's oracle guiding in latent space helps to shape latent representa- tion neural networks closer to the one. We demonstrated this advantage of VLOG using the maze task. Then, we scaled VLOG up to solve image-based videos games, and compared it with alternative oracle-guidee gooidig methods. Though all goneelgggiiingg methods showed performance gain over the baseline model, VLOG performed consistently the best. Finally, we transferred to offline RI. domain using a challenging tile-based game Mahjong in which an executor plays with hidden information and random state transitions, and observed VLOG achieved best overall performance.
We also conducted an ablation study of VLOG (VLOG-no oracle) which posterior model did not receive oracle observation, but executor one. VLOG-no oracle demonstrated performance gain in the tasks that may benefit from the stochasticity; otherwise. it performs similar to the determinis- tic baseline. This clarified that the source of VLOG's promising performance is two-fold: oracle guiding and stochastic modeling. Finally, we publish the dataset of Mahjong for offline RL and the corresponding RI environment so to facilitate future research oracle guiding.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported by Microsoft Research Asia. Kenji Doya was supported by Japan So- ciety for the Promotion of Science KAKENHI Grant Numbers JP16K21738, JP16H06561 and JP16H06563, as well as by Okinawa Institute of Science and Technology.
REPRODUCIBILITY STATEMENT
The source code of VLOG can be found in Supplementary Material.
ETHICS STATEMENT
We declare no conflict of interests. We tried to use friendly colors to people with color recognition disabilities (Fig. cC, D) and distinguishable markers for performance curves of different models (Fig. 2B, Fig. and Fig Our Mahjong dataset was generated using downloadable, public game replay data from Tenhou.nel with post-processing. The dataset contains no private information about players. Since VLOG is general framework for leveraging the oracle information, we cannot foresee any direct application of VI.OG to malicious purposes. However, any new RI. algorithm might confer increased autonomy an agent, and eventually lead to completely autonomous agent, which can used for malicious purposes, e.g., fully autonomous soldiers.
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep Variational Information Bottleneck. In International Conference on Learning Representations, 2017.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In International Conference on Machine Learning, 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A Distributional Perspective on Reinforcement Learning. International Conference on Machine Learning, 2017.
Christopher Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in -3-AF. 2017.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Con- ference Robot Learning, pp. 66-75. PMLR, 2020.
Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal Trading for Order Execution with Oracle Policy Distillation. In AAAI Conference on Artificial Intelligence, 2021.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy Networks for Exploration. In International Conference on Learning Representations, 2018.
Thomas Furmston and David Barber. Variational methods for reinforcement learning. In Pro- ceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp- 241-248. JMLR Workshop and Conference Proceedings, 2010.
Arthur Guez, Fabio Viola, Theophane Weber, Lars Buesing, Steven Kapturowski, Doina Precup, David Silver, and Nicolas Heess. Value-driven Hindsight Modelling. In Advances in Neural Information Processing Systems, 2020.
10

---

Published as a conference paper at ICLR 2022
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In Ad- vances Neural Information Processing Systems, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1856-1865, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algo- rithms and Applications. arXiv, abs/1812.05999 2019.
Donggi Han, Kenji Doya, and Jun Tani. Variational Recurrent Models for Solving Partially Observ- able Control Tasks, In International Conference on Learning Representations, 2020.
Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nico- las Heess, Hado van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, and Remi Munos. Hindsight Credit Assignment. In Advances in Neural Information Processing Systems, 2019.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. In AAAI Conference Artificial Intelligence, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. 3-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In International Conference on Learning Representations, 2016.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep Variational Reinforcement Learning for POMDPs. In International Conference Machine Learning, 2018.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is Q-learning provably effi- cient? In Advances in Neural Information Processing Systems, 2018.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, 2020.
Michael Johanson. Measuring the size of large no-limit poker a302v 7008, 2013.
games.
arXiv preprint
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrett Expe- rience Replay in Distributed Reinforcement Learning. In International Conference on Learning Representations, 2018
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. International Confer- on Learning Representations, 2014.
W. Bradley Knox and Peter Stone. Interactively Shaping Agents via Human Reinforcement: The TAMER Framework. In International Conference on Knowledge Capture, 2009.
Tadashi Kozuno, Yunhao Tang, Mark Rowland, Remi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting Peng" Q()) for Modern Reinforcement Learning. In International Conference on Machine Learning, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Of- fline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2020.
Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXivv2005.01643, 2020.
11

---

Published as a conference paper at ICLR 2022
Junjie Li. Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang. Ruihan Yang, Li Zhao, Tao Qin. Tie- Yan Liu, and Hsiao- Hon. Suphx: Mastering Mahjong with Deep Reinforcement Learning. arXiv, abs/2003.13599) 2020.
Robert Loftin, Bei Peng, James MacGlashan, Michael L. Littman, Matthew E. Taylor, Jeff Huang, and David L. Roberts. Learning behaviors via human- deelveede discrete feedback: modeling implicit feedback strategies speed up learning. Autonomous agents and multi multi-agent systems, 30(1))30-59, 2016.
James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L Littman. Interactive Learning from Policy- Seiiey-Dependett Hu- man Feedback. In International Conference Machine Learning. 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning.
Nature, 518(7540):529, 2015.
Masashi Okada, Norio Kosaka, and Tadahiro Taniguchi. PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference. In International Conference on Intelligent Robots and Systems, 2020.
Karl Pearson F.R.S. LIll. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. In International Conference on Learning Representations, 2016.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Edward J. Sondik. The optimal control of partially observable markov processes over the infinite horizon: Discounted costs. Oper. Res., 26(2):22-3304 1978.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2nd edition, 2018
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop. IEEE, 2015.
Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-learning. In AAAI conference on artificial intelligence, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang. To- bias Pfaff, Yuhuai Wu, Roman Ring. Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando Freitas. Du- eling Network Architectures for Deep Reinforcement Learning. In International conference on machine learning, 2016.
Theophane Weber, Nicolas Heess, Ali Eslami, John Schulman, David Wingate, and David Silver. Reinforced variational inference. In Advances in Neural Information Processing Systems (NIPS) Workshops, 2015.
12

---

Published as a conference paper at ICLR 2022
Haiyan Yin, Jianda Chen, Sinno Jialin Pan, and Sebastian Tschiatschek. Sequential generative exploration model for partially observable reinforcement learning. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10700-10708, 2021.
Kenny Young and Tian Tian. MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. arXiv preprint
2019.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp- 321-384, 2021.
13

---

Published as a conference paper at ICLR 2022
A
IMPLEMENTATION OF ALTERNATIVE ORACLE GUIDING METHODS
Suphx-style a Koyamedoo B, at 2020)
OPD-style (Fang Y. al MN 2021)
Student model
Teacher mocel
Qlarget
@target
teacher
Error from target
Q
Qeacher
Teacher
phocee gradkollyy
feature
Dropout
enooder
Figure 4: Diagram of our implementation for Suphx-style and OPD-style oracle guiding. For OPD- style oracle guiding, the teacher model is trained and then fixed in prior to training the student model (executor model).
In the original works of Suphx (Li ct al. 2020 and OPD (Fang et al. 2021). the authors were per- forming oracle guiding different RL objectives to ours. For comparison, we re-implemented Suphx-style and OPD-style oracle guiding our settings using similar network structures (see Fig. 4 and used the same hyper-paramette wherever possible. However, the key methodology of Suphx-style and OPD-style oracle guiding remains the same as those in their original works.
As shown in Fig. 4 suppose executor observation is æ; and oracle observation is i.,, Suphx-style model receives input
Supex
concatenate(z,, 6g O tt),
where 6 is the dropout matrix whose elements Bernoulli variables with (i
pdropout, and @ denotes element-wise multiplication. During training, dropout was linearly increased from 0 to 1 in the first 2/3 training epochs, and kept to be in the remaining 1/3 epochs. The other parts of training were the same as the baseline model.
For OPD-style oracle guiding, directly used the trained oracle model as the teacher model (Fig right). The loss function (in Q-learning) was
TorD MSE (Qter a(), + μMSE (Queschrr (zt), (()).a)))
where the first term MSE (Qter Q(+,,a)) was the same the loss function for the baseline model. And the second term is the distillation loss with coefficient (note that the teacher model is fixed during training). We selected the sestt-rrffrminngge μ for each task using grid search (see Table 3)
B
RL ALGORITHMS AND
B.
RL ALGORITHMS
B.1.1
DUELING DOUBLE DQN FOR MAZE AND MINATAR TASKS
As discussed in Sec. used double DQN with dueling network architecture (van Hasselt
et al.
2016 Wang et al., 2016) as the base RL algorithm, because work relatively well (Hessel without introducing additional hyper- paramerr.
al.
2018
The Dueling architecture of DQN (Wang et al. 2016) is defined as follows (see Appendix C for hidden layer size):
14

---

Published as a conference paper at ICLR 2022
class DuelingQNetwork (nn (an.uulle
def .-init.- (self, input_size action_num layden.
super((DulinggNeetoon
self inputtsiiee = input.size
celf.action_nnu = action_num
self. Self.hiddennaaa layesn
hidden layers
self nework-moddla = mn ModuleList()
last last lare-llyerssai = input size
for layer-size in hidden_layers:
self network.modules.. ( Linaa((aalllayer.ssiz layer-size)) self network.moduless... secwork.mddules.appeddd (m... last last_layer size = sazer_size
(an.
self. value layer = mn (aastaaaaallllyyr..... size action num)
1)
self advantage layer mn. Linear (last sllast_laall (last_layer layer_sizeeeeeeeeeeeeeeaeealllll
self. main.
(e (+ network.modules)
def forwar x):
h self. main.networr(x*
v = self. value_layer (h). repeat.interlest (self action.num. dim=-1) qp self advantage _layer (h)
q0 torch. mean (q0, dim=-l, keepdim=True repeat,
action__mmn dim=-1)
= v +
return
Double deep Q-learning (van Hasselt et al. 2016) was used to compute Quie in Fig. A (one can any other algorithm compute Quiger without changing other parts). In particular, as in Wang et al. (0166), we have
Qutrze 2 72( arg max =a
where Ft the reward at step t, y is the discount factor (Table 3) denotes the parameters of the Q network (MLP decoder) for computing the Q-function (Appendix C). Note that is given by the posterior decoder with oracle observation æ as input, since oracle prediction error term Ba(2,/(e:) [log (u(2,) utar| [2,]] in Eq. iis the expectation over the posterior distribution g( ()). Following deep RL normals (Mnih et al., 2015) Wang et al. 2016 van Hasselt et al.) (016), we used the original Q network, of which the parameters denoted as f- (Table 3). Every 1,000 steps, the target Q network copies the parameters from the original network (Table 3). Then the first term of the VLOG loss function (Eq. (2) is simply given by the mean square error between Quipeer and the output of Q network (MLP decoder) for the Maze and MinAtar tasks.
target Q network with the same structure
1.2 DUELING DOUBLE DQN WITH CONSERVATIVE Q LEARNING FOR MAHJONG
In Mahjong, as we transfer offline RL domain (Sec. 5.3 directly using off-policy RL algorithm usually results to very unsatisfying performance (Levine et al. 2020).
Therefore, we complement the loss function of VLOG (Eq. with an auxiliary conservative Q- learning (CQL) loss (Kumar et al. 2020).
Jegu. a.E log Σ
d'EA
where D is the offline dataset we used for Mahjong and 1. The combined loss function used for Mahjong (CQL) is I JeoL-
B.2 HYPER-PARAMER PARAMETER SELECTION
We summarize the dyper-paramett in Table
15

---

Published as a conference paper at ICLR 2022
Explanation
Symbol
Value
Comment
Common
Discount factor Learning rate Batch size coefficient of c-greedy target network update interval environment steps per gradient step Optimizer
(Mahjong) 0.995 (others)
0.0001
For all loss functions
1024 (Mahjong) 128 (others)
0.1 (learning) 0 (evaluation)
For online RL
1000
4
For online RL For all loss functions
Adam
VLOG
KI. divergence target value Initial value of B
Dlar
50 0.00001
Fixed in all experiments
OPD style
coefficient of policy distillation loss
0.01 (Mahjong) 10 (others)
By grid search
Table 3: Hyper-parameters used in this paper. For OPD-style, we did grid search for the additional hyper-parameter js to select the best-performing value among [0.001, 0.01, 0.1, 1, 10].
B.3 SENSITIVITY ANALYSIS FOR /
As we mentioned in Sec. the coefficient 3 is important to the learning of VLOG agents. If we fix the value of 3 throughout training, too large or too small 33 will result worse performance. The corresponding results are shown in Fig. 6
C NETWORK STRUCTURES
For simplicity, we use the same Jayersn_ayersizzee size for all the fully connected layers, where the hid- den deyerryrssiz is 256 for maze and MinAtar, and 1024 for Mahjong.
C.I ENCODER
Since we targeted at various tasks, we used different encoder network each type environment. The prior and posterior encoder has the same structure except different sizes of input features/chan- nels.
For the maze task, the encoder was 2-layers MLP with ReLU activation. The output size is also equal to hidden layer size.
For the MinAtar tasks, used 2-D CNN encoder defined as follows:
import torch nn as nn enn_module_list = nn. ModuleLList ()
cnn-modulc.list. append (nn. Conv2d n_channels. 16. 3. 1, 0))
enn.modulc.list append (nn.ReLU
enn.module.list
(nn. 32, 3, 1, 0))
cnn.module.list appen
enn mommodule._ll list. (on 128, 4. 2, 0))
cnn enn_modulc .ist (nn. on. cnn_module_ htt.otlcc.iisappppedmooooooooooo (nn. cnn cnn.module.liis append (m). ReLU()
256, 2, 1, 0))
enn module list append Flatten ())
enn minatar mn. Sequential cn (eteennouuuleeeeeeeeeemeeeeoo module
where n channels the number of channels of executor or oracle observation. The output size is equal hidden layer size.
16

---

Published as a conference paper at ICLR 2022
For Mahjong, because the second dimension of observation (tile ID) has local contextual relationship
(Li et al., 2020), we used 1-D CNN (convolution along the tile ID dimension) as the encoders, defined as follows:
cnn cnn-modulc.list = mn ModuleLLiLit ()
cnn.modulc.list. append Convid channelss 64, 3, 1, 1)) enn cnn-module.list append (nn.Convdd 64, 3, 1, 1))
enn mon_module.o list. (nn.
enn mon_module.o list htt.mooduee...t..ppepmmmcmommmm (on. 64, 3, 1. 1))
cnn_modulc.list http::/iiiis..pppeeooooeeee.e cnn_module.list: http:oddiie..is..ppppeeneeeee Ctt.iiieeeeeeemmoooooocooeeee
32, 3, 1, 1))
cnn.modulc.liss htpp:lllllss..pppdddddcc (tn.
cnn mnn-module_lin append Flattttt ())
enn min.mahoong = mn. Sequential (* enn module_list)
The output size is 1088, close to hidden layer size.
C.2 LATENT LAYER
For VLOG and VLOG (no-oracle), the size of layer is half of hidden layer size because need to estimate both the mean and the variance of z. For all the other models (baseline, oracle, OPD-style, Suphx-style), the latent layer is fully connected layer with size hidden Jayer size and ReL.U activation.
C.3 DECODER
The decoders for all models were 2-layers MLPs with size hidden,Jayer_siz Except BC on Mahjong, the input of decoder was the output of the latent layer concatenated with the action, and we used the dueling Q-network structure (Wang et al. 2016) to output scalar Q value. ReLU activation was used except for the output.
For BC on Mahjong, the input of decoder was the output of the latent layer. The outputs of decoder was logit of actions, and the actions could be obtained using softmax.
D TASK-AGNOSTIC LATENT BOTTLENECK CONTROL
As we discussed in Sec. 4.2 the coefficient 33 the regularization term in the VLOG loss function (Eq. 2 is adaptively regularized by Eq. provided with Dir, which is another hyper-parameter. While our experiments demonstrated the effectiveness of this approach, the follows discuss more thoughts behind this design choice.
In principle, replacing a hyper-parameter with another does not always make training easier. How- ever, in practice (especially deep RL), the performance will be highly sensitive some hyper- parameters (c.g., the entropy coefficient a in the original soft-actor critic algorithm (Haarnoja et al.) 2018 need be tuned in each robotic task. This is because the reward magnitude is different among lasks.). As result, it will be beneficial to replace a sensitive hyper- parameter with another one that does not need fine tuning. For example, in the follow-up paper of soft-actor critic, the authors used adaptive entropy coefficient a by introducing another hyper-pand parameter, the entropy target (Haarnojal jet al. 2019). They empirically found that is good to set the entropy target equal to the negative of the degree of freedom of the agent, thus to avoid tuning a.
Our idea of replacing with Dirr is due to similar reasons. One obvious problem is that the magnitude of the "oracle prediction error" term (Eq. [ relics the reward magnitude of the task. Therefore / should also be adjusted to match with the magnitude only relevant the magnitude of prior and posterior z, which does not differ too much among tasks (usually at the order of 1). In practice, we found that Diar = 50 works well for all the tasks including Maze, MinAtar and Mahjong.
task reward. However, Dar
Another way of regularizing is to employ a linear or exponential scheduler. For example, in Burgess a (2017), the authors used linear scheduler for the target Kl-divergence and got good
17

---

Published as a conference paper at ICLR 2022
results. However, using scheduler introducing more hyper-parameters (at least two: initial | and final 3), which is against our intention to reduce the impact of eyper-parametent
E SEAQUEST LOCAL OPTIMUM
In Seaquest, the agent drives submarine diving into the sea to shoot enemies and rescue divers earn scores. However, the submarine has limited oxygen. must surface replenish the oxygen before running out for surviving, by doing which it temporarily cannot earn scores. A typical local optimum is to use the last remaining oxygen for diving instead of surfacing.
F MAHJONG
F.1 ESTIMATION OF GAME COMPLEXITY OF MAHJONG
We consider 4-players Japanese Mahjong Although there are minor variants rules, the following estimation applies to general cases.
For easier computation, we make two major simplifications (i.e., we estimate a lower bound of the game complexity): (I) melding from discard is not considered and (2) the information other than tiles, such as results of contextual games, points of players, is not considered.
There are 34 types of tiles, each with 4 duplicates (so totally 136 tiles). We further restrict our estimation to the last turn (i.e., the last tile drawn). Among 136 tiles, 53 tiles are in someone's hand (the players have 14, 13, 13. 13 tiles in hands, respectively). Permutation of tiles in one's hand does not make a different, while the permutation matters if not in one's hand. The number of distinguishable configurations of 136 tiles thus be computed (
136!
10145
Mcanwhile, for each discarded tile, important know whether it was discarded immediately after being drawn or not. For 70 discarded tiles, the number of possibilities is simply 2 ~ 1021
Therefore, the lower bound of game complexity of Mahjong is estimated as
1 x 10 16.
If considering the other being simplified information, the state space could be much larger. For example, let's consider the current points of each player. The most common rule is that each player starts with 25,000 points, with 100 points the minimal unit (totally 1,000 units), and the game terminates if someone got negative points. So the number of possibilities can be converted to the
answer of "how many ways to distribute 1000 candies to 4 kids", which (1000+1)(4-)
(1-1)! 10a.
F.2 DETAILS OF OBSERVATION SPACE AND ACTION SPACE ENCODING
In our Mahjong environmen the action space is composed of 47 discrete actions (Table Because not all actions are available at a certain step, we also provide the set of valid actions among all 47 actions each step according to the rules, which can be used during playing and learning.
The oracle observation has the shape of 111 x 34 (executor observation is a part of oracle obser- vation, with shape 93 x 34) (Fig. []. The first dimension corresponds to 111 features (channels). The second dimension of observation (with size 34) corresponds to 34 Mahjong tiles (the order is Character 1-9, Dot 1-9, Bamboo 1-9, East, South, West, North, White, Green, Red), We used 1-D CNNs with convolution along the second dimension for the encoders. Suppose the current player is player and other players are numbered 1, 2, counter-clockwise The value of any element in an observation is 1 or O, explained in Table
'htts://en.wwikeee wwwipedia..rg/wwweeeee www..//e/w/w////aee.......m MaaaapnnneeeeMaaonggggn Mahjong
' meld is specific pattern of three or four tiles. A player pick up discarded tile from others to form meld by displaying the meld public, if certain condition satisfied.
'The Mahjong environment we used in this papers is available on https//ggthb..ommmyymiooo//////t for reproducibility. However, recommend to use the newer version htnp://gthna.cmm//ggg5555///m// which is better-supported by the authors and much faster.
18

---

Published as a conference paper at ICLR 2022
Rough meaning
Feature index
Meaning (L is the corresponding tile)
If player has 21 in hand
If player has 22 in hand
2
If player has 23 in hand
Hand player 0
If player has t in hand player has dicarded in this game If player has Red Dora in hand
If player has 21 1 in callings
If player has 2: in callings
Callings of player (melding from discarded tiles)
player has 23 callings
If player has in callings
10
If player has 21 in callings and is from other's 's discarded tiles If player has Red Dora in callings
11
Callings of player
Same to but for player
Callings of player 2 Callings of player 3
18 to 23
Same as 5 to 11, but for player 2
24 to 29
Same 5 to 11, but for player 3
30
If player has discarded 21
31
player has discarded 22 If player has discarded 23
32
33
If player has 4 in calling
34
If player 0's first dicarded is Tegiri (if applicable) player second dicarded is Tegiri applicable) If player O's third dicarded Tegiri (if applicable)
Discarded tiles from player 0
37
If player O's fourth dicarded t is Tegiri (if applicable) If player has discarded Red Dora
38
39
If player has discarded to annouce Riichi Same as 30-39, but for player
Discarded tiles from player
40 to 49
Discarded tiles from player 2 Discarded tiles from player 3
50 to 59
Same as 30-39, but for player 2
60 69
Same 30-39, but for player
If Dora indicator (> repeats)
71
If Dora indicator (2 repeats)
72
If t is Dora indicator (2 3 repeats)
73
If is Dora indicator (4 repeats)
74
If is Dora (2 repeats)
Other public information
is Dora (2 repeats)
76
If Dora (2 3 repeats)
77
If Dora (4 repeats)
78
If wind of the table
79
Is wind of self
The tile the latest action
80
is the tile corresponding to the latest action at is in player 01's hand
81
82
If at be Chi, and is the smallest in the meld at can be Chi, and is middle
meld
84
If at can be Chi, and is the largest in the meld
85
If at can be Pong
86
If at can be An-Kan
Information for available actions
If at can be Kan
If at can be Ka-Kan
89
If Ricch is possible by discarding
90
If t is the latest discarded tile from others enabling RonAgari
91
If is the latest drawn tile enabling Tsumo
If Kyuhai and is in player 0's hand
Hand of player (only for oracle)
93 to 98
Same as O to 5, hut for player
Hand of player 2 (only for oracle) Hand of player 3 (only for oracle)
99 to 104
Same 0 to 5, hut for player
105 110
Same 0 to 5, but for player
Table 4: Explanation of the 111 features of oracle observation encoding (the first 93 features are available to the executor) of our Mahjong environment. Player 0 the current player who is making decision and player 2, 3 are opponents counterclockwise. Tegiri ("discard from hand"" means the tile not discarded immediately after drawing
19

---

Published as a conference paper at ICLR 2022
Action index
Explanation
0
Discard Character
1
Discard Character 2
2
Discard Character 3
3
Discard Character 4
4
Discard Character 5 (non-Red Dora with higher priority)
5
Discard Character
6
Discard Character 7
7
Discard Character 8
8
Discard Character 9
9
Diseard Dot
10
Discard Dot
11
Discard Dot
12
Discard Dot
13
Discard Dot (non-Red Dora with higher priority)
14
Discard Dot 6
15
Discard Dot
16
Discard Dot
17
Discard Dot
18
Discard Bamboo
19
Discard Bamboo 2
20
Discard Bamboo
21
Discard Bamboo
22
Discard Bamboo (non-Red Dora with higher priority)
23
Discard Bamboo 6
24
Discard Bamboo
25
Discard Bamboo 8
26
Discard Bamboo
27
Discard East Wind
28
Discard South Wind
29
Discard West Wind
30
Discard North Wind
31
Discard White Dragon Tile (Haku)
32
Discard Green Dragon Tile (Hatsu)
33
Discard Red Dragon Tile (Chu)
34
Chi (the picked up tile is the smallest in the meld)
35
Chi (the picked up tile is the middle the meld)
36
Chi (the picked up tile is the largest in the meld)
37
Pon
38
An-Kan
39
Kan
40
Ka-Kan
41
Riichi
42
Ron
43
Tsumo
44
Restart the game with Kyushukyuhai
45
Not to response (when Chi, Pon, Kan, Ron, etc. is possible)
46
Not to Riichi (When Riichi is possible)
Table 5: Action encoding of our Mahjong environment.
20

---

Published as a conference paper at ICLR 2022
additional oracle observation
executor observation
Figure 5: Picture of Mahjong game. The executor observation includes the publicly visible infor- mation and the player's private hand tiles. The oracle observation includes the executor observation and additional information about the opponents' private tiles.
F.3 DATA AUGMENTATION
In (Japanese) Mahjong, there are 3 suit sets of (characters, dots, bamboos) tiles, 4 wind tiles and dragon tiles. The 3 suit sets are symmetric and thus exchangeable with each other" So are the wind tiles and the dragon tiles. According such symmetry, we augmented the offline RL dataset by randomly exchanging the 3 suit sets, the 4 wind tiles and the 3 dragon tiles, respectively.
Thhere is one exemption of winning hand pattern "all green". Since it is extremely rare case, we simply
ignore it.
21

---

Published as a conference paper at ICLR 2022
breakout
seaquest
space invaders
average return
average return
average return
20
15
40
15
30
10
20
5
2
million frames
million frames
million frames
freeway
asterix
average return
50
average return
lngoo
40
8
logas -7
loga -6
30
6
logsa -3
logas 0 -1
Ingsa
10
2
3
3
million frames
million frames
Figure 6: Sensitivity analysis of VLOG in noisy MinAtar environments w.r.. the selection of 3 if B
fixed.
22


Reviews:
[{"review_id": "oL3cLvKrsl", "reviewer": "Reviewer_8vyx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strength}$:\n- This paper is well written and easy to follow.\n- Leveraging Bayesian theory to tackle the representation learning problem in reinforcement learning is an important research direction.\n\n$\\textbf{Limitations}$:\n- The proposed method is not very general as it could only be used in the tasks where oracle observations are available.\n- The  formulation of ELBO might not be very valid (refer to the detailed comments)\n- The empirical evaluation domains are too simple and some of the experimental configurations are not properly specified.\n- The method is developed upon a very simple RL algorithm. \n\n$\\textbf{Detailed comments}$:\n1. The assumption that there are always oracle observations available during policy learning does not hold for many applications, such as Atari 2600 and noisy robotics navigation tasks. And in the more challenging problems with partial observability, it is unclear whether this method could be directly applied and how to effectively regularize the posterior distribution which meant to represent partially observed states. \n\n2. I feel the formulation of ELBO is not that sound. I do not know why the latent of oracle observations are set as the posterior instead of that from executor observation. Also, it seems that there are two execution paths for the ELBO, i.e., there should be one for the oracle and another for the executor observation. But in Sec 4.2, this is not properly specified as Eq (2) only tackles one of the path. \n\n3. It is unclear how the variational model is related to the policy model, e.g., is the input to the policy $x_t$ or $z_t$? \n\n4. The authors employ a very simple RL algorithm to deal with very simple tasks. For instance, in the first task of maze navigation, in such a simple case I'm very sure using something like PPO will successfully solve the problem. I don't understand why the problem could be unsolvable, i.e., the performance of baseline is such poor. I'm also concerned if the experimental settings are properly configured. For instance, in the maze task, the input is very shallow, i.e., with only ~3 dimensions, but the authors specify the latent to have a dimension of 128. I believe such setting is not very reasonable.  \n\n5. In many  cases, even though the oracle features might be available, they might   have very different  format  compared to the raw observations, e.g., (x, y, z) vs  high-dimensional images. In such scenarios,  would   the two-encoder-one-decoder architecture still work?\n\n6. Overall the novelty of this paper is relatively limited and there lacks discussion with a number of related works that combine variational inference with policy learning, such as:\n[1] Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model\n[2] Variational methods for Reinforcement Learning\n[3] Sequential Generative Exploration Model for Partially Observable Reinforcement Learning\n[4] Reinforced Variational Inference", "summary_of_the_paper": "This paper tackles reinforcement learning scenarios when the raw observation obtained from the executor is not very descriptive and there are high quality oracle observations available for the learning process to guide the policy training. The proposed method aims to derive a representative latent state via incorporating a variational inference perspective. Specifically, the oracle and raw observations are processed by two individual encoders and then the two latent spaces are fused and fed to a shared decoder. The latent distribution obtained from the oracle observations serve as the posterior distribution and that from the raw observation serves as the prior distribution. The distance two distributions is minimized via minimizing a KL divergence term. ", "summary_of_the_review": "This paper proposes a method to leverage oracle observations with a variational inference framework. But the applicability of the method is rather limited as it  poses a very strong assumption that oracle observations need to be accessed for training the policy. The empirical evaluation domains are too trivial, e.g., very simple decision making tasks and there is no challenging POMDP task employed to demonstrate the KL regularization works. There are also some problems regarding to how the methods are implemented. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mE0RTF6KQK", "reviewer": "Reviewer_5XWD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the motivation and idea behind this paper, and I think it is something that future researchers could make use of. In particular, I like the simplicity of the approach.\n\nMy main concern with the paper is that the evaluation on Mahjong:\n*  It is somewhat confusing to follow (for readers not already familiar with Mahjong), and it is not clear what the takeaway is (other than VLOG seems to work). \n*  The results are somewhat suspect, since the oracle does the worst in Table 2, which is not what one would expect.\n*  The method for scoring described at the bottom of page 8 (\"we simply added up the payoff of the two being tested agents\") seems quite strange and not really a fair evaluation. Is this really how Mahjong is played/evaluated?\n*  Why isn't the top-left value of table 2 bolded as well? The VLOG line in Table 2 has overlapping CIs, so should not be bolded.\n*  Finally, the experiments are combining too many things: offline RL, multi-agent RL, etc. This makes it harder to reliably ascertain what the benefits of VLOG are. While interesting, I'd recommend clarifying the exposition of the experiments, or finding another (better known) benchmark, and leave the Mahjong experiments as an extra set of appendix experiments.\n\nQuestions for the authors:\n1. In footnote 2 in page 3, is there any relationship between $X$ (set of executors) and $\\hat{X}$ (set of oracles)? I assume the latter is a subset of the full $X$?\n1. In remark 1, is the $v_t$ in $p(v_t | {\\bf x}_t)$ equal to $V({\\bf x}_t)$?\n1. In the last paragraph of section 5.2 it says \"VLOG-no oracle performed surprisingly well\", but this is not the case in the figures shown.\n1. What is \"suphx-style oracle guiding\"?\n1. What is \"OPD-style oracle guiding\"\n1. What is the \"(trained) baseline model\" used in Table 1?\n\n---\nMinor comments:\n1. The abstract starts with \"How to make intelligent decisions is a central problem in machine learning and cognitive science\". Minor nit, but I'd say cognitive science is more concerned with _understanding_ how decisions are made.\n1. In the abstract should say \"RL using variational method***s***.\"\n1. In the abstract should say \"decision-making tasks rang***ing*** from video games...\"\n1. At the end of the second paragraph of the introduction, it's not necessary to include \"etc\" since you started the sentence with \"including\".\n1. In section 2, remove \"different\" after \"Another\" in the second sentence.\n1. In section 2 should say \"make the student policy behave more simliar to ***the*** teacher policy. Both methods\"\n1. Title of section 3 should be \"PRELIMINAR***IES***\"\n1. In the first sentence of section 4 should say \"variable representing the environmental state\"\n1. In the first sentence of section 5.1, should say \"by leveraging oracle observation***s*** in learning.\"\n1. In the end of the first paragraph of 5.1, should say \"another grid if not ***blocked by a wall***.\"\n1. In the first sentence of 5.2 should say \"we tested it ***on*** a set of MinAtar video games\"\n1. When citing the ALE, you should cite [Bellemare et al, 2013](https://arxiv.org/abs/1207.4708) instead of Machado et al.\n1. In the third paragraph of page 8, should say \"are sophisticated ways to encode the state and action space of Mahjong (Li et al., 2020), we attempt to make simplifications with reasonable amount***s*** of approximations since our goal is not to create strong Mahjong AI\"\n", "summary_of_the_paper": "This paper presents an approach to dealing with problems where agents could benefit from learning from state features that are only available during training, but not during evaluation. Their approach leverages Bayesian theory to propose a variational learning method and is evaluated on a variety of tasks, including a Mahjong simulation (which the authors are providing as part of their contribution).", "summary_of_the_review": "See details above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l2w-lfspd5M", "reviewer": "Reviewer_7M2j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n- Adopting variational Bayes is an interesting and reasonable approach to make use of the training-time priviledged information. \n\nWeakness of the paper:\n- In fact, the studied problem is not absolutely new. There have been similar settings studied under imitation learning (IL),  such as [1]. Furthermore, applying variational Bayes for latent space learning is also not a very novel idea in both RL and IL. \n\n- According to the paper, one of the major drawbacks for previous approaches in this problem is lacking theoretical guarantees. However, in my view, the variational Bayes is also not possible to work in all situations. Why and when the proposed approach would indeed work? The theoretical analysis is missing in the paper. Even though the experimental results show some advatange of the proposed approach, after reading the paper, I still don't get solid justifications on why variational Bayes could outperform previous approaches in principle. \n\n- Some descriptions could be clearer:\n>- In Equation (1), how $v^{tar}_t$ is obtained? The paper just says that it could be calculated by any policy evaluation method. But policy iteration relies on knowing the transition model and the reward function. This is not assumed in the paper.\n>- In the Mahjong experiment, I wonder why VLOG-no oracle could improve over the comparison methods. According to Equation (1), if no oracle observation is available, VLOG reduces to minimizing the maximum-likehood loss of fitting the value function. I thought it would have no difference to the baseline method, i.e. vallina policy learning. \n\nReference:\n[1]  Learning by cheating. https://arxiv.org/pdf/1912.12294.pdf\n\n", "summary_of_the_paper": "In this paper, reinforcement learning (RL) with training-time privileged information is studied. This scenario is the relaxation of traditional POMDP by allowing additional observations to be available during training time. A variational Bayes approach is proposed under this scenario. The main idea lies in learning a joint latent space for both testing-stage partial observations and the training-stage full observations.  Experiments over several benchmark tasks, including Mahjong, a large-scale multi-player game with imperfect information are conducted to verify the performance of the approach.\n\n", "summary_of_the_review": "To summarize, I think the paper proposes a reasonable approach for RL with training-time privileged information. But I think further efforts should be made to better justify the proposed approach both theoretically and empirically, as discussed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_ZCCVxTjz9q", "reviewer": "Reviewer_Wjja", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to follow and generally well-written. The proposed method is intuitive and effective, as an application of variational models. The oracle-guiding problem is novel and intriguing. It is similar to the offline RL, but there are some key differences that can distinguish them. I urge the author to formally define the problem to help the readers who are less familiar with the oracle-guiding framework. I summarize some of my concerns in the following:\n\nconcerns:\n1. Can you summarize the difference between offline RL and oracle-guide RL. I think the key difference is the training observations (or oracle observations) contain a richer signal than the observations in the testing environment (or executor observation), but there could be more. Please formally define the oracle-guiding RL problems anyway.\n2. The author claims \"VLOG is theoretically guaranteed to ...\", where are the theoretical results? I think we can show by maximizing this variational lower bound, the KL divergence between the true posterior and the approximate posterior will be reduced, but this is a well-known property. Where are the new contributions?\n3. How do you get the label v^{tar}_{t}? (in (1)). I assume a DQN or a critic network must be implemented. If it is the case, please define the model and its loss.\n4. Will it make things easier by replacing a hyper-parameter \\beta with another hyper-parameter D^{tar}_{KL}? I guess both hyper-parameters are difficult to determine. In fact, other works might design a linear or exponential scheduler for \\beta that allows it smoothly grow from a min (in most cases, 0) to a max value during training. As far as I know, this method works quite well in practice.\n\nMinors:\n1. Many sentences are repeated in the abstract and introduction, but readers with a reasonable RL/ML background can follow this message with one scan. Maybe use another example for better elaboration.\n2. In remark one, the paper says \"they can choose q(v_t|\\hat_{x}_t) to be p(v_t|x_t)u(v_t; [l,u])\". Are you multplying two distrbutions? In this sense, q will not define any distribution.\n3. Poker games involve two players competing with each other. In this sense, maybe a (PO)Markov Game model can better describe this environment than a (PO)MDP.\n\ntypos:\n1) the oracle-estimated one as posterior distribution -> the oracle-estimated one as the posterior distribution.", "summary_of_the_paper": "This paper proposed a variational-based model VLOG for solving the oracle-guiding RL problem. By minimizing the KL-divergence between the latent features from the oracle observations and the executor observations, the model can apply the oracle signals to boost the execution performance. The author(s) derived a lower bound objective as the training loss and empirically demonstrated the performance of their method under three different types of environments.", "summary_of_the_review": "This paper tackles an intriguing oracle-guiding problem. The proposed method is intuitive and effective, as an application of variational models. The paper is generally well written. The concerns I proposed are mostly minor and can be easily resolved within a reasonable time. I did not capture major bugs in this paper. The experiment is complete and promising.  I am happy to vote for an accept given other reviewers do not find major issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of using oracle information that's only available during training in RL. The key contributions are 1) a variational Bayesian approach that models the oracle observation as latent variables; and 2) a Mahjong environment for benchmarking RL with oracle guiding. The novelty of the proposed approach is limited, but reviewers find the problem intriguing and agreed that it's a reasonable application of Bayesian approach to RL with latent oracle information. In addition, the Mahjong environment could benefit the community and spur new work in this direction. Therefore, I recommend this paper to be accepted as a poster.
Decision: Accept (Poster)