Year: 2021

Abstract:
How to make intelligent decisions is a central problem in machine learning and artificial intelligence. Despite recent successes of deep reinforcement learning (RL) in various decision making problems, an important but under-explored aspect is how to leverage oracle observation (the information that is invisible during online decision making, but is available during offline training) to facilitate learning. For example, human experts will look at the replay after a Poker game, in which they can check the opponents’ hands to improve their estimation of the opponents’ hands from the visible information during playing. In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods. Our key contribution is to propose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL. VLOG is featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value-based DRL algorithm. We empirically demonstrate the effectiveness of VLOG in online and offline RL domains using decision-making tasks ranging from video games to a challenging tile-based game Mahjong. Furthermore, we publish the Mahjong environment and an offline RL dataset as a benchmark to facilitate future research on oracle guiding (https://github.com/pymahjong/pymahjong).

heading: 1 INTRODUCTION
text: Deep reinforcement learning (DRL) has undergone rapid development in recent year (Sutton & Barto, 2018; Mnih et al., 2015; Vinyals et al., 2019). However, there is an common and important but under-explored aspect in RL: imagine that after playing a Poker game, a human player may look at the replay to check opponents’ hands and analyze this information to improve his/her playing strategy (or policy) for the next time. We refer to the information like opponents’ hands as oracle observation, defined as the information invisible to the agent during online task execution but available during offline training. By contrast, the information available during task execution is called executor observation. Such a scenario has been refered to as oracle guiding for RL (Li et al., 2020; Fang et al., 2021) (see Sec. 3 for a formal definition). Oracle guiding is common in real life. For example, when taking an examination (the oracle observations are the answers to similar questions, which are available only during preparing); and training a robot to perform some tasks on the Moon (when training the robot, we can provide it with the information of the territory, which are not available during execution). The type of oracle observations can be diverse, including hindsight information (Harutyunyan et al., 2019; Guez et al., 2020), human feedback (Knox & Stone, 2009; Loftin et al., 2016; MacGlashan et al., 2017), re-calibrated data with post-processing, and hidden states in a partially observed setting (Li et al., 2020).
While humans naturally perform oracle guiding when learning to make decisions, it remains challenging in RL. The difficulties include: (1) how to guarantee that learning with oracle observation improves the main decision model using executor observation only, and (2) if introducing an auxiliary loss leveraging oracle observation, how to tradeoff between the main loss and auxiliary loss. While recent studies attempted to model oracle guiding in RL (Guez et al., 2020; Li et al., 2020; Fang et al., 2021), none of them addressed these difficulties (refer to the Related Work section for more details). In particular, all these proposed methods are heuristic: although empirical results showed performance gain with oracle guiding, it is not theoretically guaranteed that the usage of oracle observation improves execution performance.
In this paper, we propose a fundamentally new idea for oracle guiding based on Bayesian theory. Taking Poker as an example, we know that the learning of optimal strategy is tractable if knowing the global, true state of the environment (or simply state1), including all visible or invisible cards, the opponents’ playing style, etc. (Azar et al., 2017; Jin et al., 2018; 2020). A key part of skill improvement is learning to estimate the probabilistic distribution of environmental state from executor observation. The common way to do this by human experts is to watch match replays where the oracle observation (e.g. opponents’ hands) is available, and then use the oracle-estimated state to correct the executor-estimated state. We interpret this to Bayesian language: executor-estimated state as the prior distribution, and the oracle-estimated one as the posterior distribution. Thus, the training objective can be considered two-fold: learning to make decisions based on posterior estimation of state, and learning a prior distribution of state closer to the posterior one.
We formulate this idea by proposing a novel learning framework for general oracle guiding problems based on variational Bayes (VB) (Kingma & Welling, 2014), referred to as variational latent oracle guiding (VLOG). VLOG owns several preferable properties. First, VLOG is theoretically guaranteed to leverage oracle observation for improving the decision model using executor observation. Second, VLOG is a versatile DRL framework that can be integrated into any value-based RL algorithm and is agnostic to the type of oracle observation. Third, VLOG does not suffer from the necessity of tuning additional hyper-parameters. Finally, we empirically show that VLOG contributes to better performance in a variety of decision-making tasks in both online and offline RL domains. The tasks include a simple maze navigation, video games playing, and a particularly challenging tile-based game Mahjong in which humans heavily leverage oracle observations in learning (Li et al., 2020). We also contribute to the community by taking Mahjong as a benchmarking task for oracle guiding and publish the corresponding simulation environment and dataset to facilitate future research.

heading: 2 RELATED WORK
text: In the past few years, research interests have grown on DRL and imitating learning (Chen et al., 2020) with leveraging oracle or hindsight information. For DRL, Guez et al. (2020); Fang et al. (2021) considered hindsight observation (executor observation at future steps) as the oracle observation during training. Guez et al. (2020) used hindsight observation to facilitate learning a representation of current state. Another method (Fang et al., 2021) was used for stock trading: the authors trained a teacher (oracle) policy with hindsight information, and employed network distillation to make the student policy behaves more similar to the teacher policy. Both the methods (Guez et al., 2020; Fang et al., 2021) are heuristic and focused on making use of future observation for a better sequential modeling, while VLOG is theoretically guaranteed for any kind of oracle observation. For applications in imperfect-information games, Suphx (Li et al., 2020), a DRL-based AI for Mahjong, also introduced a method to leverage oracle observation (opponents’ hand) for stronger performance. They concatenate oracle observation with executor observation as the input of policy network, where the oracle observation is timed by a scalar variable which is annealed from 1 to 0 during the training course. However, the method used in Li et al. (2020) is also heuristic and has only been tested in one task.
Variational Bayes (VB) is a well-established method and has been taken advantage of in RL. For example, control as probabilistic inference uses VB to connect the objective function of RL and variational lower bound of a probabilistic inference problem (Furmston & Barber, 2010; Weber et al., 2015; Levine, 2018). Our idea differs since it frames a value regression problem by the maximumlikelihood problem, and then, it applies VB to solve it (see Sec. 4). Also, the usage of VBian network models for DRL has captured attention from researchers recently. For example, Ha & Schmidhuber (2018) proposed to employ a VAE to reduce the high dimensionality of image observation; Igl et al. (2018); Han et al. (2020); Lee et al. (2020) proposed variational RNNs as the state-transition models to encode the belief states of the agent; Yin et al. (2021) utilized a variational sequential generative model for predicting future observation and used the prediction error to infer intrinsic reward to encourage exploration; and Okada et al. (2020) demonstrated performance gain by using a deep Bayesian planning model in continuous control tasks. Our study differs from the mentioned works by focusing on oracle guiding, and VLOG does not involve learning a state transition model.
1Generally, oracle observation does not necessarily contain all the information of environmental state.

heading: 3 ORACLE GUIDING POMDP
text: Here we define the problem based on the Partially Observable Markov decision process (POMDP) (Sondik, 1978). An oracle guiding POMDP distinguishes from the original POMDP by having two types of observations: executor and oracle. The executor observation x is always available to the agent, on which the agent’s decision making (execution) relies. The oracle observation x̂ is not accessible during execution, but can be obtained afterward. x is included in x̂ since the former is always available. Thus x̂ contain no less information of the underlying environment state than x.
A formal definition of oracle guiding POMDP is a tuple 〈S,A,P0, T ,X, X̂,O, Ô, γ〉, where S and A are the state and action spaces, respectively. P0 specifies the initial state distribution such that P0(s) is the probability of a state s ∈ S being an initial state. T specifies the state transition probability such that T (s′, r|s, a) is the probability of reaching to a new state s′ ∈ S with an immediate reward r ∈ R after taking an action a ∈ A at a state s ∈ S. X denotes the executor observation space and X̂ denotes the oracle observation space. O specifies the executor observation probability such that O(x|s) is the probability of an executor observation x ∈ X at a state s ∈ S. Similarly, Ô specifies the oracle observation probability such that Ô(x|s) is the probability of an oracle observation x̂ ∈ X̂ at a state s ∈ S. γ ∈ [0, 1) is the discount factor. Value functions are the expected value of return from a particular state or state-action pair. For a policy π, its Q-function is defined by qπ(s, a) := Eπ[ ∑∞ n=0 γ
nrt+n|st = s, at = a], where Eπ indicates the expectation when the policy π is followed. The state value function vπ is defined by vπ(s) := Eπ(a|s)[qπ(s, a)]. The agent aims to maximize EP0(s)vπ(s) with respect to π, and the value-functions play an important role to this end (Sutton & Barto, 2018).

heading: 4 VLOG: VARATIONAL LATENT ORACLE GUIDING
text: Let us introduce a latent vector zt, which is a probabilistic variable representing the environmental state st. From a Bayesian perspective, we consider the prior distribution p(zt|xt) as the agent’s estimated probability density function (PDF) for zt based on executor observation xt. Meanwhile, the posterior distribution PDF q(zt|x̂t) is modeled based on oracle observation x̂t. In RL, the most basic requirement is to make a good estimation of return by a value function approximator v(xt) := ∫ v(zt)p(zt|xt)dzt (we denote it by v, but it can also be a Q function by simply replacing xt with (xt,at)) based on available information, i.e. executor observation xt. The target of return, denoted by vtart , can be estimated by any value learning algorithm such as TD(0) (Sutton & Barto, 2018), Peng’s Q(λ) (Kozuno et al., 2021), etc. (generally, vtar can always be given by Bellman equations. However, one can also use Monte-Carlo return as vtar if available). In particular, we employed double Q-learning with dueling architecture (Wang et al., 2016) to compute vtar due to its effectiveness and simplicity (Sec. 5 and B.1). We want to maximize the log-likelihood objective of the estimation of return based on executor observation xt (i.e., for the executor model)
L := logP ( v(xt) = v tar t |xt ) = log ∫ zt P ( v(zt) = v tar t |zt ) p(zt|xt)dzt
= log ∫ zt q(zt|x̂t) q(zt|x̂t) P ( v(zt) = v tar t |zt ) p(zt|xt)dzt.
By Jensen’s inequality, we have L ≥ ∫ zt q(zt|x̂t) log P (v(zt) = vtart |zt) p(zt|xt) q(zt|x̂t) dzt
= ∫ zt [ q(zt|x̂t) logP ( v(zt) = v tar t |zt ) − q(zt|x̂t) log q(zt|x̂t) p(zt|xt) ] dzt
= Eq(zt|x̂t) [ logP ( v(zt) = v tar t |zt )]︸ ︷︷ ︸ oracle prediction error −DKL (q (zt|x̂t)‖p (zt|xt))︸ ︷︷ ︸ regularization term := LVLOG. (1)
Thus we can maximize our initial objectiveL via maximizingLVLOG, which is also known as the variational lower bound (Kingma & Welling, 2014), but in our oracle-guiding scheme. Since p(zt|xt)
and q(zt|x̂t) represents the PDF of latent vector obtained from executor observation and oracle observation, respectively, the meanings of the two terms in LVLOG appear clear now: the first term, i.e., oracle prediction error, helps to improve value estimation from posterior latent state distribution (zt computed with oracle observation); and the second term, i.e., regularization term, helps to shape the prior representation of zt closer to the posterior one as latent oracle guiding. We would like to highlight that the VLOG objective is the lower bound of the objective for the prior executor model v(xt) (the estimation of return using executor observation xt) with the usage of oracle observation x̂t. This lower bound guarantees that the usage of oracle observation is positively affecting the learning of executor model, which is what we finally want. Now we have derived the theoretical objective of VLOG, the next section will explain its implementation in practice. Remark 1. One may use any shape of the approximate posterior q, depending on which different instance of VLOG is possible. Furthermore, one may directly use v(xt) instead of p(zt|xt). These design choices allow users to incorporate any prior knowledge on oracle observation. For example, if one knows the range of a state-value at xt is a closed interval [l, u], the approximate posterior q(vt|x̂t) can be restricted to a family of probability distributions supported on [l, u].

heading: 4.1 IMPLEMENTATION WITH NEURAL NETWORKS
text: Inspired by the implementation of variational auto-encoder (VAE, Kingma & Welling (2014)), we propose the neural network architecture of VLOG (Fig. 1). The executor observation xt and oracle observation x̂t are processed by two distinct encoder networks and to compute the prior and posterior distribution of the latent vector, respectively. During training, both xt and x̂t are available, and all the network parameters are updated by maximizing the VLOG objective in an end-to-end manner (Fig. 1A). During execution, the agent computes prior distribution p(z|xt) for decision making (Fig. 1B) without using oracle observation. zt is computed by parameterized normal distribution
p(zt|xt) = N (µpt , exp (logσ p t )), (µ p t , logσ p t ) = prior encoder(xt), q(zt|x̂t) = N (µqt , exp (logσ q t )), (µ q t , logσ q t ) = posterior encoder(x̂t).
For computing P(v(zt) = vtart |zt) in Eq. 1, we simply assume it follows normal distribution, and estimate it with mean square error between v(zt) and vtart in practice. The reparameterization trick is used to perform end-to-end training as in VAE (Kingma & Welling, 2014). Then the output of the decoder (value function) can be obtained by v(zt) = decoder(zt). Note that zt is obtained using the posterior encoder during training, and using the prior encoder during execution.

heading: 4.2 TASK-AGNOSTIC LATENT BOTTLENECK CONTROL
text: To learn a better representation, we borrow the idea from β-VAE (Higgins et al., 2016) to apply a coefficient β to the regularization term. Thus we have the loss function (negative lower bound) as
J βVLOG = −Eq(zt|x̂t) [ logP ( v(zt) = v tar t |zt )] + βDKL (q (zt|x̂t)‖p (zt|xt)) . (2)
The hyper-parameter β controls the capacity of the latent information bottleneck (Tishby & Zaslavsky, 2015; Alemi et al., 2017). We found the choice of β is important for the performance of
VLOG in RL (see Appendix B.3). However, having extra hyper-parameters are not desired. Inspired by the method used in Burgess et al. (2017) for controlling the scale of KL divergence in β-VAE, we propose a task-agnostic method to automatically adjust β by setting a target of KL divergence DtarKL. In particular, we minimize the auxiliary loss function (β as the being optimized parameter)
Jβ = (DtarKL −DKL (q (zt|x̂t)‖p (zt|xt))) log(β). (3)
The intuition here is to strengthen the regularization by increasing β when the divergence between prior and posterior is too large, and vice versa. This method is similar to that used in soft actorcritic for automatically adjusting the entropy coefficient (Haarnoja et al., 2019), but we used for KL divergence coefficient. Importantly, we found a well-performing value DtarKL = 50 agnostic to other design choices. It worked well in a range of different tasks and networks (Sec. 5). Therefore, we do not need to tune β. We provide more discussion about this method in Appendix D.

heading: 5 EXPERIMENTS
text: How does VLOG perform in practice? We investigated the empirical performance of VLOG in three types of tasks using online or offline RL, from simple to difficult. In the following experiments, we used double DQN with dueling network architecture (van Hasselt et al., 2016; Wang et al., 2016) as the base RL algorithm, the model and loss functions for RL are defined in Appendix. B.1. As DRL is susceptible to the choice of hyper-parameters, introducing any new hyper-parameters might obscure the effect of oracle guiding. Double DQN and dueling architecture are preferable for the base algorithm since they require no additional hyper-parameters, in contrast to other DQN variants (Hessel et al., 2018), such as prioritized experience replay (Schaul et al., 2016), noisy network (Fortunato et al., 2018), categorical DQN (Bellemare et al., 2017), and distributed RL (Kapturowski et al., 2018). Importantly, we used the same hyper-parameter setting for all methods and environments as much as possible (see Appendix B.2).

heading: 5.1 MAZE
text: We first demonstrate how VLOG helps to shape the latent representation by leveraging oracle observations in learning. The testbed is a maze navigation task2 (Fig. 2A) with 10×10 grids. The executor observation is the (x, y) position, where x, y are continuous values randomly sampled within each grid of the maze (thus the observations in two adjacent but wall-separated grids may be very close). At each step, the agent selects an action (going up, down, right, or left) and moves to another grid if not blocked by wall.
We provided the VLOG agent with oracle observation (xc, yc, dg) during training, where xc, yc are coordinates of the center of the current grid, and dg is the (shortest) path distance to goal from the current grid. It is intuitive that although the raw observation is (x, y), dg matters more in a maze navigation task. We empirically investigated how much such oracle observation used in learning of VLOG could help to shape the latent representation of z w.r.t. dg rather than position. The encoder and decoder were both 2-layers multi-layer perceptrons (MLP) with width 256 and ReLU activation. The size of latent vector zt for VLOG was 128 since we computed both µ and σ (Appendix C).
Experiments show that the baseline agent struggled to reach the goal (Fig. 2B), while VLOG agents stably solved the task after learning. To check how using VLOG affected the learned latent representation, we visualize the latent representation of both VLOG and baseline models with principal component analysis (PCA, Pearson F.R.S. (1901)). In Fig. 2C, we map path distance to goal dg to color and plot the scores of the first 2 PCs of z (computed using executor observation) for VLOG and the corresponding hidden state for baseline3 (“hidden layer” in Fig. 1C). The latent state of VLOG showed a relatively smoother and interpretable representation of distance-to-goal comparing to that of baseline. We then plot the latent representations for different positions in the maze in Fig. 2D. The latent state of VLOG more clearly represented dg , consistent with the result (Fig. 2C).
In particular, we looked into a rectangle region (denoted by rectangles in Fig. 2D) inside which the left 2 grids and right 2 grids are segregated by a wall. We found the corresponding areas in the latent
2https://github.com/MattChanTK/gym-maze 3Since the baseline did not always succeed in reaching the goal, we used the successful trials only.
PC space and circled them in Fig. 2C. While these 4 grids are close in (x, y) (executor observation), their distances-to-goal (oracle observation) are highly distinguished. By leveraging oracle guiding using VLOG, the agents can clearly differentiate the left 2 grids and the right 2 grids in latent space as shown in Fig. 2C, D, left (note that the latent state z here of VLOG was computed using executor observation only). By contrast, the latent representations of these grids were overlapped for the baseline model, which did not utilize the oracle observation (Fig. 2C, D, right). In sum, we demonstrated with a toy example that VLOG effectively helped the latent space to couple with oracle state useful for the task. The following sections will transfer to experiments on more complicated tasks and discuss how VLOG can improve practical performance.

heading: 5.2 NOISY MINATAR
text: To evaluate how VLOG scales with more high-dimensional state space, we tested it on a set of MinAtar videos games. MinAtar (Young & Tian, 2019) is a test platform for AI agents, which implements 5 miniaturized Atari 2600 games with discrete actions (seaquest, breakout, space invaders, freeway and asterix). MinAtar is inspired by Arcade Learning Environment (Bellemare et al., 2013) but simplifies the environments for efficiency. The observations are 10×10 pixels with multiple channels indicating different objects.
In real-world, the observation usually contains some noise. Thus it is natural to consider the noisy observation as the partially-observable executor observation, and the original, non-noisy observation as the oracle one. Suppose that at each frame, each pixel may “break” randomly with an independent probability of 1/8 (Fig. 3A). The original observation at a broken pixel is erased and is replaced by a different value in all channels. We consider such noisy MinAtar environments with the noisy pixels as the executor observation and the original pixels as the oracle observation.
The network structure was the same as that for Maze, but the encoder was replaced by a CNN (Appendix C). We ran experiments on all the 5 environments of MinAtar with VLOG as well as baseline, oracle and alternative oracle guiding methods (see Appendix A for details). Baseline model always uses executor observation as network input (Fig. 1C). Oracle is the same as baseline except that it always receives oracle observation (i.e., cheating, we ran the experiments of oracle for reference). VLOG-no oracle is an ablation of VLOG where we use the executor observation
as the input to posterior encoder in VLOG (oracle observation is not used). Suphx-style oracle guiding is the oracle guiding method used in the Mahjong AI Suphx (Li et al., 2020), in which the executor observation and the dropout-ed oracle observation (with dropout probability pdropout) are concatenated as the input to the network. With training proceeds, pdropout is gradually increased from 0 to 1, thus the trained network does not need oracle observation for input (Appendix A). OPD-style oracle guiding is the oracle guiding method used in oracle policy distillation (OPD) (Fang et al., 2021). OPD-style oracle guiding first trains a teacher model using oracle observation as input, and then trains the executor model using an auxiliary loss, which is the error between the executor and the teacher models’ estimation for value function (Appendix A)..
The results show that oracle performed usually the best, as expected. We normalized the performance of non-oracle models using oracle model as reference, for more clear comparison (Fig. 3B). Among all the oracle guiding methods (VLOG, OPD-style and Suphx-style), VLOG consistently performed the best. It is notable that VLOG and VLOG-no oracle performed surprisingly well in seaquest. This can be explained by that Seaquest is a task with a local optimum4, while the stochasticity in hidden states of VLOG helped exploration in latent space to escape from the local optimum (a similar idea was proposed in Fortunato et al. (2018), but their noise was added to the network weights). Except in Seaquest, VLOG-no oracle did not show significant performance different with baseline, showing that the performance gain of VLOG in this task set mainly came from leveraging oracle observation for shaping latent distribution; and the usage of variational Bayesian model was, at least not harming the performance when there is no helpful oracle information.

heading: 5.3 OFFLINE LEARNING ON MAHJONG
text: Mahjong is a popular tile-based game with hundreds of millions of players worldwide (here we consider the Japanese variant). The game is like many other card games (but using tiles instead of cards), in which multiple (usually four) players draw and discard tiles (totally 136 tiles) alternatively to satisfy winning conditions. It is a highly challenging game characterized with (1) imperfect information in executor observation (a player cannot see opponents’ private tiles and the remaining tiles to be drawn), (2) stochastic state transitions as in many card games, and (3) extremely high game complexity (i.e., the number of distinguished, legal game states). Complexity of Mahjong is much larger than 10166 (Appendix E.1). For reference, complexity of Go is ∼ 10172 (Silver et al., 2016) and complexity of no-limit Poker is ∼ 10162 (Johanson, 2013). In Mahjong, it is hard to make optimal decisions based on executor observation because the outcomes heavily depend on invisible information, and complexity of invisible state space is as high as 1048 on average (Li et al., 2020). In response to this challenge, Li et al. (2020) introduced suphx-style oracle guiding and demonstrated a performance gain. Thus, we consider Mahjong as a promising test platform for oracle guiding methods. Since the number of possible states in Mahjong
4In Seaquest, the agent drives a submarine diving into the sea to shoot at enemies and rescue divers to earn scores. However, the submarine has limited oxygen. It must surface to replenish the oxygen before running out for surviving, by doing which it temporarily cannot earn scores. A typical local optimum is to use the last remaining oxygen for diving instead of surfacing.
is extremely large. It is costly to explore with random actions in an online RL manner, and no previous work could train a strong Mahjong AI with purely online RL. Also, we would like to examine the effectiveness of VLOG in offline RL settings. For these reasons, we transferred to offline RL (Levine et al., 2020) for the Mahjong task using expert demonstrations.
We processed about 23M steps of human experts’ plays from the online Mahjong game platform Tenhou (https://tenhou.net/mjlog.html) to a dataset for offline RL (data were augmented using the symmetry in Mahjong, see Appendix E). Also, we created a simulator of Mahjong as the testing environment. Though there are sophisticated ways to encodes the state and action space of Mahjong (Li et al., 2020), we attempts to make simplifications with reasonable amounts of approximations since our goal is to not create an strong Mahjong AI, but to use Mahjong as a platform to study oracle guiding problems. In our case, the action space is composed of 47 discrete actions covering all decisions in Mahjong. An executor observation is a matrix encoding public information and the current player’s private hand; An oracle observation concatenates the executor observation with the information of the opponents’ private hands. (see Appendix E). We used an 1-D CNN as the encoder as done in Mahjong AIs commonly (Li et al., 2020), and the size of zt and decoder network width was increased to 512 and 1024, respectively (Appendix C).
Note that although Mahjong is a 4-player game, using offline RL data to train an agent does not involve multi-agent RL (Zhang et al., 2021) because the offline dataset is fixed: the opponents have fixed policies and thus can be considered as parts of the environment. Our experiments focused on single-agent RL to avoid the complexity caused by considering multi-agent RL. We investigated two kinds of offline RL settings. The first is conservative Q-learning (CQL) (Kumar et al., 2020). Our CQL setting differed from the online RL setting in previous sections by adding auxiliary CQL loss (Kumar et al., 2020) to the Q-learning loss function (Appendix B.1.2).
The other is behavior cloning (BC). Although VLOG was designed for value-based RL, we could straightforwardly incorporate VLOG with BC by letting the network predict action (and the decoder does not need action input). The learning was conducted by minimizing the cross entropy between the output and the target action (demonstration) as in a classification problem. Note that we did not test OPD-style oracle guiding in BC setting because it was equal to the baseline since we can directly use demonstration actions as oracle policy for distillation.
Because Mahjong is a zero-sum, four-player game, we tested the performance of trained models in two scenarios: playing with the (trained) baseline model (Table 1) and playing with each other (fight each other, Table 2). In the first scenario, four agents were playing the same matches on the game table, where two of them were being tested agents and the other two were the baseline models. Although each agent played for itself and there was no communication between players, we simply added up the payoff of the two being tested agents, and consider they won a match if one of them ranked top (so the match win rate will be 50% if equally strong as baseline), for statistics (Table 1).
For CQL, the results (Table 1 left and 2 upper) show that VLOG substantially outperformed the baseline and alternative methods (because Mahjong is a highly random game, 55.7% match win rate indicates a large skill gap). Interestingly, VLOG was even comparable to oracle. This can be explained by that VLOG also benefited from its Bayesian property, which is consistent with that VLOG-no oracle showed a significant performance gain over the baseline model (Table 1 left). Still, the oracle model learned to reduce deal-ins (i.e., player discards a tile and another player wins the game by picking up this tile to compose a winning hand) since it could explicitly see the opponents’ private tiles, showing a much lower deal-in rate than other non-cheating models (Table 2 upper).
In BC setting, the agents did not learn a value function, but tried to predict human experts’ actions. Therefore, the training procedure did not involve reasoning the relationship between playing outcome and oracle observation, but just imitating human behaviors. This can be seen from the results that oracle did not substantially outperform baseline in BC (Table 1 right and 2 lower). However, VLOG and VLOG-no oracle still showed performance gain, thanks to the stochastic modeling.

heading: 6 SUMMARY
text: We have proposed VLOG – a variational Bayesian learning framework for leveraging oracle observations to facilitate DRL especially in partially observable environments. VLOG is available for any RL problem in which there is oracle observation that may help the executor to make decision.
We first introduced a latent vector z to represent the environmental state. The prior and posterior distribution of z is modeled using executor and oracle observation, respectively. Then, we derived a variational lower bound (Eq. 2) by maximizing which we can optimize the executor model using the oracle observations. We developed the corresponding methodology for DRL, which can be incorporated with most RL algorithms that need to estimate a value function.
If oracle observation contains more information to retrieval the true environmental state (or, it is the true environmental state), VLOG’s oracle guiding in latent space helps to shape a latent representation in neural networks closer to the true one. We demonstrated this advantage of VLOG using the maze task. Then, we scaled VLOG up to solve image-based videos games, and compared it with alternative oracle-guiding methods. Though all oracle-guiding methods showed performance gain over the baseline model, VLOG performed consistently the best. Finally, we transferred to offline RL domain using a challenging tile-based game Mahjong in which an executor plays with hidden information and random state transitions, and observed VLOG achieved best overall performance.
We also conducted ablation study of VLOG (VLOG-no oracle) in which posterior model did not receive oracle observation, but executor one. VLOG-no oracle demonstrated performance gain in the tasks that may benefit from the stochasticity; otherwise, it performs similar to the deterministic baseline. This clarified that the source of VLOG’s promising performance is two-fold: oracle guiding and stochastic modeling. Finally, we publish the dataset of Mahjong for offline RL and the corresponding RL environment so as to facilitate future research on oracle guiding.

heading: REPRODUCIBILITY STATEMENT
text: We included the source code of VLOG and alternative models in the supplementary files. The environments and dataset we used are public online.

heading: ETHICS STATEMENT
text: We declare no conflict of interests. We tried to use friendly colors to people with color recognition disabilities (Fig. 2C, D) and distinguishable markers for performance curves of different models (Fig. 2B, Fig. 3 and Fig6). Our Mahjong dataset was generated using downloadable, public game replay data from Tenhou.net with post-processing. The dataset contains no private information about players. Since VLOG is a general framework for leveraging the oracle information, we cannot foresee any direct application of VLOG to malicious purposes. However, any new RL algorithm might confer increased autonomy on an agent, and eventually lead to a completely autonomous agent, which can be used for malicious purposes, e.g., fully autonomous soldiers.

heading: B RL ALGORITHMS AND HYPER-PARAMETERS

heading: B.1 RL ALGORITHMS

heading: B.1.1 DUELING DOUBLE DQN FOR MAZE AND MINATAR TASKS
text: As we discussed in Sec. 5, we used double DQN with dueling network architecture (van Hasselt et al., 2016; Wang et al., 2016) as the base RL algorithm, because it work relatively well (Hessel et al., 2018) without introducing additional hyper-parameter.
The Dueling architecture of DQN (Wang et al., 2016) is defined as follows (see Appendix C for hidden layer size): c l a s s DuelingQNetwork ( nn . Module ) :
def i n i t ( s e l f , i n p u t s i z e , ac t ion num , h i d d e n l a y e r s ) : super ( DuelingQNetwork , s e l f ) . i n i t ( ) s e l f . i n p u t s i z e = i n p u t s i z e s e l f . a c t i o n n u m = a c t i o n n u m s e l f . h i d d e n l a y e r s = h i d d e n l a y e r s s e l f . ne twork modu le s = nn . Modu leL i s t ( ) l a s t l a y e r s i z e = i n p u t s i z e f o r l a y e r s i z e in h i d d e n l a y e r s :
s e l f . ne twork modu le s . append ( nn . L i n e a r ( l a s t l a y e r s i z e , l a y e r s i z e ) ) s e l f . ne twork modu le s . append ( nn . ReLU ( ) ) l a s t l a y e r s i z e = l a y e r s i z e
s e l f . v a l u e l a y e r = nn . L i n e a r ( l a s t l a y e r s i z e , 1 ) s e l f . a d v a n t a g e l a y e r = nn . L i n e a r ( l a s t l a y e r s i z e , a c t i o n n u m ) s e l f . ma in ne twork = nn . S e q u e n t i a l (* s e l f . ne twork modu le s )
def f o r w a r d ( s e l f , x ) : h = s e l f . ma in ne twork ( x ) v = s e l f . v a l u e l a y e r ( h ) . r e p e a t i n t e r l e a v e ( s e l f . ac t ion num , dim = −1) q0 = s e l f . a d v a n t a g e l a y e r ( h ) a = q0 − t o r c h . mean ( q0 , dim = −1 , keepdim=True ) . r e p e a t i n t e r l e a v e (
s e l f . ac t ion num , dim = −1) q = v + a re turn q
Double deep Q-learning (van Hasselt et al., 2016) was used to compute Qtarget in Fig. 1 A (one can use any other algorithm to compute Qtarget without changing other parts). In particular, as in Wang et al. (2016), we have
Qtargett = rt + γQ(zt, argmax a′ Q(zt+1), a ′; θ); θ−),
where rt is the reward at step t, γ is the discount factor (Table 3), θ denotes the parameters of the Q network (MLP decoder) for computing the Q-function (Appendix C). Note that z is given by the posterior decoder with oracle observation x as input, since the oracle prediction error term Eq(zt|x̂t) [logP (v(zt) = vtart |zt)] in Eq. 1 is the expectation over the posterior distribution q(z|x̂). Like normally done in RL (Mnih et al., 2015; Wang et al., 2016; van Hasselt et al., 2016), we used a target Q network with the same structure as the original Q network, of which the parameters are denoted as θ− (Table 3). Every 1,000 steps, the target Q network copies the parameters from the original Q network (Table 3). Then the first term of the VLOG loss function (Eq. 2) is simply given by the mean square error between Qtarget and the output of Q network (MLP decoder) for the Maze and MinAtar tasks.

heading: B.1.2 DUELING DOUBLE DQN WITH CONSERVATIVE Q LEARNING FOR MAHJONG
text: In Mahjong, as we transfer to offline RL domain (Sec. 5.3), directly using off-policy RL algorithm usually results to very unsatisfying performance (Levine et al., 2020).
Therefore, we complement the loss function of VLOG (Eq. 2) with an auxiliary conservative Qlearning (CQL) loss (Kumar et al., 2020), which is an state-of-the-art offline RL algorithm:
JCQL = αEx̂,a∼D [ log
∑ a′∈A exp(Q(z(x̂), a′, θ))− [Q(z(x̂), a, θ)]
] ,
where D is the offline dataset we used for Mahjong and α = 1. The combined loss function used for Mahjong (CQL) is J βVLOG + JCQL.

heading: B.1.3 BEHAVIOR CLONING IN MAHJONG
text: We also tested behavior cloning with VLOG (Sec. 5.3) for Mahjong. we implement VLOG with BC by letting the decoder network predict action (and the decoder does not need action input). The learning was conducted by minimizing the cross entropy between the output and the target action (demonstration) as in a classification problem.

heading: B.2 HYPER-PARAMETER SELECTION
text: We summarize the hyper-parameters in Table 3.
B.3 SENSITIVITY ANALYSIS FOR β
As we mentioned in Sec. 4.2, the coefficient β is important to the learning in VLOG. If we fix the value of β throughout training, a too large or too small β will result in worse performance. The corresponding results are shown in Fig. 6.

heading: C NETWORK STRUCTURES
text: For simplicity, we tried to use the same hidden layer size for all fully connected layers in the same model, where the hidden layer size is 256 for maze and MinAtar, and 1024 for Mahjong.

heading: C.1 ENCODER
text: Since we targeted at various tasks, we used different encoder network for each type of environment. The prior and posterior encoder has the same structure except different sizes of input features/channels.
For the maze task, the encoder was a 2-layers MLP with ReLU activation. The output size is also equal to hidden layer size.
For the MinAtar tasks, we used 2-D CNN encoder defined as follows:
import t o r c h . nn as nn c n n m o d u l e l i s t = nn . Modu leL i s t ( ) c n n m o d u l e l i s t . append ( nn . Conv2d ( n c h a n n e l s , 16 , 3 , 1 , 0 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . Conv2d ( 1 6 , 32 , 3 , 1 , 0 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . Conv2d ( 3 2 , 128 , 4 , 2 , 0 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . Conv2d ( 1 2 8 , 256 , 2 , 1 , 0 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . F l a t t e n ( ) ) c n n m i n a t a r = nn . S e q u e n t i a l (* c n n m o d u l e l i s t )
where n channels is the number of channels of executor or oracle observation. The output size is equal to hidden layer size.
For Mahjong, because the second dimension of observation (tile ID) has local contextual relationship (Li et al., 2020), we used 1-D CNN (convolution along the tile ID dimension) as the encoders, defined as follows:
c n n m o d u l e l i s t = nn . Modu leL i s t ( ) c n n m o d u l e l i s t . append ( nn . Conv1d ( n c h a n n e l s , 64 , 3 , 1 , 1 ) ) c n n m o d u l e l i s t . append ( nn . Conv1d ( 6 4 , 64 , 3 , 1 , 1 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . Conv1d ( 6 4 , 64 , 3 , 1 , 1 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . Conv1d ( 6 4 , 32 , 3 , 1 , 1 ) ) c n n m o d u l e l i s t . append ( nn . ReLU ( ) ) c n n m o d u l e l i s t . append ( nn . F l a t t e n ( ) ) cnn mahjong = nn . S e q u e n t i a l (* c n n m o d u l e l i s t )
The output size is 1088, close to hidden layer size.

heading: C.2 LATENT LAYER
text: For VLOG and VLOG (no-oracle), the size of z layer is half of hidden layer size because we need to estimate both the mean and the variance of z. For all other models, the latent layer is one fully connected layer with size hidden layer size and ReLU activation.

heading: C.3 DECODER
text: The decoders for all models were 2-layers MLPs with size hidden layer size. Except BC on Mahjong, the input of decoder was the output of the latent layer and the action, and we used the dueling Q-network structure (Wang et al., 2016) to output a scalar Q value.
For BC on Mahjong, the input of decoder was the output of the latent layer. The outputs of decoder was logit of actions, and the actions could be obtained using softmax.

heading: D TASK-AGNOSTIC LATENT BOTTLENECK CONTROL
text: As we discussed in Sec. 4.2, the coefficient β of the regularization term in the VLOG loss function (Eq. 2) is adaptive regularized by Eq. 3, provided with DtarKL, which is another hyper-parameter. While our experiments demonstrated the effectiveness of this approach, the follows discuss more thoughts behind this design choice.
In principle, replacing a hyper-parameter with another does not always make training easier. However, in practice (especially deep RL), the performance will be highly sensitive to some hyperparameters (e.g., the entropy coefficient α in the original soft-actor critic algorithm (Haarnoja et al., 2018) need to be tuned in each robotic task. This is because the reward magnitude is different among tasks.). As a result, it will be beneficial to replace a sensitive hyper-parameter with another one that does not need fine tuning. For example, in the follow-up paper of soft-actor critic, the authors let the entropy coefficient α to be adaptive by introducing another hyper-parameter, the entropy target (Haarnoja et al., 2019). They empirically found that it is good to set the entropy target equal to the negative of the degree of freedom of the agent, thus to avoid tuning α.
Our idea of replacing β with DtarKL is due to similar reasons. One obvious problem is that the magnitude of the “oracle prediction error” term (Eq. 2) relies on the reward magnitude of the task. Therefore β should also be adjusted to match with the magnitude of task reward. However, DtarKL is only relevant to the magnitude of prior z and posterior zt, which does not differ too much among tasks (usually at the order of 1). In practical we found that a single value for the target KLD, DtarKL = 50, work well for all tasks including Maze, MinAtar and Mahjong.
Another way of regularizing β is to employ a linear or exponential scheduler. For example, in Burgess et al. (2017), the authors used a linear scheduler for the target KL-divergence and got good
results. However, using a scheduler introducing more hyper-parameters (at least two: initial β and final β), which is against our willing to reduce the impact of hyper-parameters.

heading: E MAHJONG

heading: E.1 ESTIMATION OF GAME COMPLEXITY OF MAHJONG
text: We consider 4-players Japanese Mahjong5. Although there are minor variants of rules, the following estimation applies to general cases.
For easier computation, we make two major simplifications (i.e., we estimate a lower bound of the game complexity): (1) melding from discard6 is not considered and (2) the information other than tiles, such as results of contextual games, points of players, is not considered.
There are 34 types of tiles, each with 4 duplicates (so totally 136 tiles). We further restrict our estimation to the last turn (i.e., the last tile is drawn). Among 136 tiles, 53 tiles are in someone’s hand (the 4 players have 14, 13, 13, 13 tiles in hands, respectively). Permutation of tiles in one’s hand does not make a different, while the permutation matters if not in one’s hand. The number of distinguishable configurations of 136 tiles thus can be computed as 136!(13!)3×14!×(4!)34 ∼ 10 145
Meanwhile, for each discarded tile, it is important to know whether it was discarded immediately after being drawn or not. For 70 discarded tiles, the number of possibilities is simply 270 ∼ 1021. Therefore, the lower bound of game complexity of Mahjong is estimated as
10145 × 1021 ∼ 10166.
If considering the being simplified information, the state space could be much larger. For example, let’s consider the current points of each player. The most common rule is that each player starts with 25,000 points, with 100 points as the minimal unit (totally 1,000 units), and the game terminates if someone got negative points. So the number of possibilities can be converted to the answer of “how many ways to distribute 1000 candies to 4 kids”, which is (1000+1) (4−1)
(4−1)! ∼ 10 8.

heading: E.2 DETAILS OF OBSERVATION SPACE AND ACTION SPACE ENCODING
text: In our Mahjong environment, the action space is composed of 47 discrete actions (Table 5). Because not all actions are available at a certain step, we also provide the set of valid actions among all 47 actions at each step according to the rules, which can be used during playing and learning.
The oracle observation has the shape of 111 × 34 (executor observation is a part of oracle observation, with shape 93 × 34) (Fig. 5). The first dimension corresponds to 111 features (channels). The second dimension of observation (with size 34) corresponds to 34 Mahjong tiles (the order is Character 1-9, Dot 1-9, Bamboo 1-9, East, South, West, North, White, Green, Red). We used 1-D CNNs with convolution along the second dimension for the encoders. Suppose the current player is player 0, and other players are numbered 1, 2, 3 counter-clockwise. The value of any element in an observation is 1 or 0, explained in Table 4.

heading: E.3 DATA AUGMENTATION
text: In (Japanese) Mahjong, there are 3 suit sets of (characters, dots, bamboos) tiles, 4 wind tiles and 3 dragon tiles. The 3 suit sets are symmetric and thus exchangeable with each other7. So are the 4 wind tiles and the 3 dragon tiles. According to such symmetry, we augmented the offline RL dataset by randomly exchanging the 3 suit sets, the 4 wind tiles and the 3 dragon tiles, respectively.
5https://en.wikipedia.org/wiki/Japanese Mahjong 6A meld is a specific pattern of three or four tiles. A player can pick up a discarded tile from others to form a meld by displaying the meld to public, if certain condition is satisfied. 7There is one exemption of winning hand pattern “all green”. Since it is an extremely rare case, we simple ignore it.

Review summaries:
In this paper, reinforcement learning (RL) with training-time privileged information is studied. This scenario is the relaxation of traditional POMDP by allowing additional observations to be available during training time. A variational Bayes approach is proposed under this scenario. The main idea lies in learning a joint latent space for both testing-stage partial observations and the training-stage full observations.  Experiments over several benchmark tasks, including Mahjong, a large-scale multi-player game with imperfect information are conducted to verify the performance of the approach.  

This paper tackles reinforcement learning scenarios when the raw observation obtained from the executor is not very descriptive and there are high quality oracle observations available for the learning process to guide the policy training. The proposed method aims to derive a representative latent state via incorporating a variational inference perspective. Specifically, the oracle and raw observations are processed by two individual encoders and then the two latent spaces are fused and fed to a shared decoder. The latent distribution obtained from the oracle observations serve as the posterior distribution and that from the raw observation serves as the prior distribution. The distance two distributions is minimized via minimizing a KL divergence term. 

This paper proposed a variational-based model VLOG for solving the oracle-guiding RL problem. By minimizing the KL-divergence between the latent features from the oracle observations and the executor observations, the model can apply the oracle signals to boost the execution performance. The author(s) derived a lower bound objective as the training loss and empirically demonstrated the performance of their method under three different types of environments.