Mutual Information State Intrinsic Control
Authors: Rui Zhao
Yang Gao
Pieter Abbeel
Volker Tresp
Wei Xu
Venue: ICLR 2021 Spotlight (2021)
Keywords: Intrinsically Motivated Reinforcement Learning
Intrinsic Reward
Intrinsic Motivation
Deep Reinforcement Learning
Reinforcement Learning

Abstract: Reinforcement learning has been shown to be highly successful at many challenging tasks. However, success heavily relies on well-shaped rewards. Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the self-consciousness concept in psychology, we make a natural assumption that the agent knows what constitutes itself, and propose a new intrinsic objective that encourages the agent to have maximum control on the environment. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state under the current agent policy. With this new intrinsic motivation, we are able to outperform previous methods, including being able to complete the pick-and-place task for the first time without using any task reward. A video showing experimental results is available at https://youtu.be/AUCwc9RThpk.

Published as a conference paper at ICLR 2021
MUTUAL INFORMATION STATE INTRINSIC CONTROL
Rui Zhao 12; Yang Gao², Pieter Abbeel', Volker Tresp".2 1.2, Wei Xu 'Ludwig Maximilian University of Munich Siemenn AG TTinnghaa University *University of California, Berkeley 'Horizon Robotics
ABSTRACT
Reinforcement learning has been shown to be highly successful at many challeng- ing tasks. However, success heavily relies on well-shaped rewards. Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the consciousees concept in psychology, we make a natural assumption that the agent knows what constitutes itself, and propose a new intrinsic objective that encourages the agent to have maximum control on the environment. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state under the current agent policy. With this new intrinsic motivation, we are able to outperform previous methods, including being able to complete the pick-and-place task for the first time with- out using any task reward. A video showing experimental results is available at https::/youtu. wtw.:/tww.....//...t.........
INTRODUCTION
Reinforcement learning (RL) allows an agent to learn meaningful skills by interacting with vironment and optimizing some reward function, provided by the environment. Although RL has achieved impressive achievements various tasks (Silver al. 2017 Mnih et al. 2015 Berner iet al. 2019), it is very expensive to provide dense rewards for every task we want the robot to learn. Intrinsically motivated reinforcement learning encourages the agent to explore by providing an "internal motivation" instead, such as curiosity (Schmidhuber 1991 Pathak et al.) 2017 Burda al. 2018). diversity (Gregor et al., 2016, Haarnoja et al., [0118] Eysenbach
al. 2019) and em- powerment (Klyubin et al. 2005, Salge et al. [014] MMhamed & Rezende 2015]. Those internal motivations can be computed on the fly when the agent is interacting with the environment, without any human engineered reward. We hope to extract useful "skills" from those internally motivated agents, which could later be used to solve downstream tasks, or simply augment the sparse reward with those intrinsic rewards to solve given task faster.
Most of the previous works in RL model the environment as a Markov Decision Process (MDP). In an MDP, explicitly distinguishing the agent itself from its surrounding. However, in the physical world, there clear boundary between an intelligent agent and its surrounding. The skin of any mammal is an example of such boundary. The separation of the agent and its surrounding also holds true for most of the man-made agents, such as any mechanical robot. This sgrnn-nurrounndin separation has been studied for a long time in psychology under the concept of selffconssioussess. Self-consciousness refers that subject knows itself is the object of awareness (Smith) (020). effectively treating the agent itself differently from everything else. Gallup (1970) has shown that self-consciousness widely exists chimpanzees, dolphins, some elephants and human infants. To equally emphasize the agent and its surrounding, we name this separation as suret-surrounding separation in this paper. The widely adopted MDP formulation ignores the natural agent-surrounding separation, but simply stacks the agent state and its surrounding state together single state vector. Although this formulation is mathematically concise, we argue that it is sier-simplistic, and as a result, it makes the learning harder.
single state vector to describe the current state of the whole environment, without
With this sically motivated RL algorithm. We propose new intrinsic motivation by encouraging the agent to
separation in mind, we are able to design a much more efficient intrin-
"ooreesonndence Rui Zhao (zhaorui in.
iartaaruiiii.ggrmana....eee ger.any@ggmai w.aaai.ccoo com).

---

Published as a conference paper at ICLR 2021
perform actions such that the resulting agent state should have high Mutual Information (Ml) with the surrounding state. Intuitively, the higher the MI, the more control the agent could have on its surrounding. We name the proposed method "MUtual information-base or MUSIC in short. With the proposed MUSIC method, we are able to learn many complex skills in an unsupervised manner, such learning pick up object without any task reward. We can also augment a sparse reward with the dense MUSIC intrinsic reward, to accelerate the learning process.
State Intrinsic Control",
Our contributions are three-fold. First, we propose novel intrinsic motivation (MUSIC) that encourages the agent to have maximum control on its surrounding, based on the natural agent- surrounding separation assumption. Secondly, we propose scalable objectives that make the MUSIC intrinsic reward easy to optimize. Last but not least, we show MUSIC's superior performance, by comparing it with other competitive intrinsic rewards on multiple environments. Noticeably, with our method, for the first time the pick-and-play task can be solved without any task reward.
2 PRELIMINARIES
For environments, we consider four robotic tasks, including push, slide, pick-and-place alace, and naviga- tion, shown in Figure The goal in the manipulation task is to move the target object to desired position. For the navigation task, the goal is to navigate to a target ball. In the following, we define some terminologies.
2.1 AGENT STATE, SURROUNDING STATE, AND REINFORCEMENT LEARNING SETTINGS
In this paper, the agent state 8 means literally the state variable of the agent. The surrounding state s* refers to the state variable that describes the surrounding of the agent, for example, the state variable of an object. For multi-goal environments,
use the same assumption as previous works (Andrychowicz et al.) 2017;; Plappert et al., 2018), which consider that the goals can be represented as states and we denote the goal variable as g. For example, in the manipulation task, goal is particular desired position of the object in the episode. These desired positions, i.e., goals, are sampled from the environment.
The division between the agent state and the surrounding state is naturally defined by the agent- surrounding separation concept introduced in Section I From a biology point view, a human can naturally distinguish its own parts, like hands legs from the environments. Analog to this, when we design a robotic system, we can easily know what is the agent state and what is its surrounding state. In this paper, lower case letter, such as s, to represent the values of random variables.
upper letters, such S, denote random variables and the corresponding
We assume the world is fully observable, including set of states S, set of actions A, distribution of initial states p(5o), transition probabilities p(*++1 8t, az), reward function r: x A - IR, and discount factor
[0, 1]. These components formulate Markov Decision Process represented as a tuple, (S, A. r, 7). We use 7 to denote trajectory, which contains a series of agent states and surrounding states. Its random variable denoted as T.
METHOD
We focus on agent learning to control its surrounding purely by using its observations and actions without supervision. Motivated by the idea that when an agent takes control of its surrounding, then there is high MI between the agent state and the surrounding state, we formulate the problem of learning without external supervision as one of learning a policy na | 8t) with parameters 0 to maximize intrinsic MI rewards, 1 S*). In this section, we formally describe our method, mutual information- based state intrinsic control (MUSIC).
3.1 MUTUAL INFORMATION REWARD FUNCTION
Our framework simultancously learns a policy and an intrinsic reward function by maximizing the MI between the surrounding state and the agent state. Mathematically, the MI between the surround-
2

---

Published as a conference paper at ICLR 2021
ENVIRONMENT
Learned
Algorithm 1: MUSIC
p( 1
Fixed
while not converged do
Sumple Inajectorics
Gr
Estimatoe predicts MI from states. Update estimator to
Sample an initial state so p(so).
from environmennts
for t <- 1 to steps per eepisode do
Sample action Oe To(a; 8t).
AGENT
MI ESTIMATOR
Step environment $141 ~ p( S(5t+1 8t, oz).
de (o sx)
s IT)
Sample transitions T from the buffer.
Set intrinsic reward r ; T').
Updats policy to maximize the munual information reward
Update policy (0) via DDPG SAC.
I TT)
Update the MI estimator (Φ) with SGD.
Figure MUSIC Algorithm: We update the estimator to better predict the MI, and update the agent to control the surrounding state to have higher MI with the agent state.
ing state random variable S and the agent state random variable S" is represented as follows:
S0) = (L(Ps S
Ps & Psa)
(1)
sup Ep Pagaga ga [T] log BP s4 (e"))
(2)
T-2
sup ga [To) log [ Sp (e=a]) To (S
6E
where g- g- the joint probability distribution; Pg. @ Pgo is the product of the marginal distri-
butions Ps- and s=::; KL denotes the Kullback-Leibler (KL) divergence. MI is notoriously difficult to compute in real-world settings (Hjelm et al.] 2019). Compared to the variational information
maximizing-baas
approaches (Barber Agakov 2003 Alemi et al., 2016 Chalk al.) 2016
Kolchinsky et al. (2017). the recent MINE-based approaches have shown superior performance (Bel-) ghazi al.J [2018) Hjelm al.) 2019 Velickovic et al.) 2019). Motivated by MINE (Belghazi al., 2018), we use a lower bound to approximate the MI quantity ; First, we rewrite Equa- tion I the KL formulation of the MI objective, using the Donsker- Varadhan representation, to Equation (2) (Donsker & Varadhan 1975). The input space Ω is compact domain of IRd, i.e., Ω CRd, and the supremum is taken over all functions T such that the two expectations are fi- nite. Secondly, we lower bound the MI in the Donsker- Varadhan representation with the compres- sion lemma in the PAC-Bayes literature and derive Equation (3) (Banerjee) (006)) Belghazi et al.) 2018). The expectations in Equation (3) estimated by using empirical samples from F and Ps. s Psu.. The statistics model To is parameterized by deep neural network with parameters
h, whose inputs are the empirical samples.
3.2 EFFECTIVELY COMPUTING THE MUTUAL INFORMATION REWARD IN PRACTICE
Lemma 1. There is monotonically increasing relationship between (s";; | Ep (S [')], mathematically.
T) and
L (T) 14 [T'],
(4)
where Sa, S", and T denote the surrounding state, the agent state, and the trajectory, respectively. The trajectory fractions are defined as the adjacent state pairs, namely T' {St, S The symbol x denotes monotonically increasing relationship between two variables and 6 represents the parameter of the statistics model in MINE, Proof. See Appendix
We define the reward for each transition a given time-step the mutual information of the pair of adjacent states at that time-step, see Equation 44) Ranh Side (RHS). However, in practice, we find that it is not very efficient to train the MI estimator using state pairs. To counter this issue, use all the states in the same trajectory in a batch to train the MI estimator, see Equation (4) Left- Hand Side (LHS), since more empirical samples help to reduce variance and therefore accelerate learning. In Lemma I] we prove the monotonically increasing relationship between Equation (4) RHS and Equation (4) LHS.
In more detail, we divide the process of computing rewards into two phases, i.c., the training phase and the evaluation phase. In the training phase, we efficiently train the MI estimator with large
3

---

Published as a conference paper at ICLR 2021
batch of samples from the whole trajectory. For training the MI estimator network, we first randomly sample the trajectory т from the replay buffer. Then, the states of used for calculating the product of marginal distributions are sampled by shuffling the states of from the joint distribution along the temporal axis I within the trajectory. We use hack-propagation to optimize the parameter (o) to maximize the MI lower bound, see Equation 4 LHS.
For evaluating the MI reward, we use pair of transitions to calculate the transition reward, see Equation 44) RHS and Equation (5), instead of using the complete trajectory. Each time, to calculate the MI reward for the transition, the reward is calculated over a small fraction of the complete trajectory 7', namely r (o S T'). The trajectory fraction, r', is defined as adjacent state pairs, r' = {s,, 1. and 'I' represents its corresponding random variable.
The derived Lemma [ brings us two important benefits. First, it enables us efficiently train the MI estimator using all the states in the same trajectory. And a large batch of empirical samples reduce the variance of the gradients. Secondly, it allows us to estimate the MI reward for each transition with only the relevant state pair. This way of estimating MI enables us to assign rewards more accurately at the transition level.
Based on Lemma
D we calculate the transition reward as the MI of each trajectory fraction, namely
Ta(at, st) == ; = 0.5 To - log - T)())
(5)
where 5f) ~ PS 5. ~ PSarT', and 7' {ss, $t+13. In case that the estimated MI value is particularly small, we scale the reward with a hyper-parameter and clip the reward between 0 and 1. MUSIC can be combined with any off-the-shelf reinforcement learning methods, such as deep deterministic policy gradient (DDPG) (Lillierap et al. (016) and soft actor-critic (SAC) (Haarnojal
cet al. 2018). We summarize the complete training algorithm in Algorithm [T and in Figure]
The introduced MUSIC method is unsupervised rein- forcement learning approach, which is denoted as "MUSIC-u", where "-u" stands for unsupervised learning. We propose three ways of using MUSIC to accelerate learning- The first method is using the MUSIC-u pretrained policy as the parameter initialization and then fine-tuning the agent with the task rewards. We denote this variant as "MUSIC-F", where "-f" stands for fine-tuning. The second variant is to use the MI intrinsic reward to help the agent to explore more efficiently. Here, the MI re- ward and the task reward are added together. We name this method as "MUSIC-r", where "-r" stands for reward. The third approach is to use the MI quantity from MUSIC to prioritize trajectories for replay. The approach is similar to the TD-error-based prioritized experience replay (PER) (Schaul et al. 2016). The only difference is that we use the estimated MI instead of the TD-error as the priority for sampling. We name this method as "MUSIC-p". where "-p" stands for prioritization.
MUSIC Variants with Task Rewards:
Skill Discovery with MUSIC and DIAYN: AYN (Eysenbach et al., 2019), introduces an information-thhorrtiial objective FDIAYN, which learns diverse discriminable skills indexed by the latent variable Z. mathematically, FDIAYn I(S: Z) + H(A 5, 2). The first term, I(S; Z). in the objective, FDIAYN, is implemented via a skill discriminator, which serves Agakov] 2003) Eysenbach et al.] 2019).. The skill discriminator assigns high rewards to the agent, it can predict the skill-options, Z. given the states, S. Here, we substitute the full state S with the surrounding state S" to encourage the agent to learn control skills. DIAYN and MUSIC can be combined as follows: -MUSII-DIAYN = [SS" S*)) I(S"; 2) + H(A S, Z). The combined version enables the agent to learn diverse control primitives via skill-conditioned policy (Eysenbach et al., 2019 in an unsupervised fashion.
One of the relevant works on unsupervised RL, DI-
variational lower bound of the original objective (Barber &
Comparison and Combination with DISCERN: Another relevant work is Discriminative Embed- ding Reward Networks (DISCERN) (Warde-Farley et al. 2019). whose objective is to maximize the MI between the state S and the goal G, namely T(S; GJ. While MUSIC's objective is to maximize the MI between the agent state S and the surrounding state S", namely I(S"; S*). Intuitively, DISCERN attempts to reach a particular goal in each episode, while our method tries to manip- ulate the surrounding state to any different value. MUSIC and DISCERN can be combined as FMUSIC-DISCERN I[S": St) + I(S: Optionally, we can replace the full states S with Ss, since it performs better than with S empirically. Through this combination, MUSIC helps DISCERN to leam its discriminative objective.
4

---

Published as a conference paper at ICLR 2021
Figure 2: Fetch robot arm manipulation tasks in OpenAl Gym and navigation task based on the Gazebo simulator: FetchPush, -Pa--lott--aardnnddd
FetchSlide,
4 EXPERIMENTS
Environments: To evaluate the proposed methods, we used the robotic manipulation tasks and a navigation task, see Figure 2 (Brockman al. 2016 Plappert et al. 2018). The navigation task based on the Gazebo simulator. the ball, otherwise, the task reward is O. Here, the agent state the robot car position and the surrounding state is the red ball. The manipulation environments, including push, pick-and-place, and slide. have a of predefined goals, which are represented as the red dots. The task for the RL agent is to manipulate the object to the goal positions. In the manipulation task, the agent state is the gripper position and the surrounding state is the object position.
the navigation task, the task reward is if the agent reaches
First, we analyze the control behaviors learned purely with the intrinsic reward, i.e., MUSIC-U. Secondly, we show that the pretrained models can be used for improving performance in conjunction with the task rewards. Interestingly, we show that the pretrained MI estimator can be transferred among different tasks and still improve performance. We compared MUSIC with other methods, including DDPG (Lillicrap et al.] (2016), SAC (Haarnoja et al. 2018)), DIAYN (Eysenbach
Experiments:
ctt al. 2019), DISCERN "Warde-Farles et al. (2019), PER (Schaul et al., 2016). VIME (Houthooft al. 2016). ICM (Pathak et al. (017). and Empowerment (Mohamed & Rezende 2015). Thirdly,
we show some insights about how the MUSIC rewards are distributed across a trajectory. The experimental details are shown in Appendix G Our code is available at https: / com/
rui wouzzhaoiit// tositt/musicci and https:////tt. /ttp:/gitttt //ggtb.t..bb..../ggg com
wwwmruiieaeecccccc..moooooo.oo.o rui zhacgitaaaaaaaaaa
Question 1. What behavior does MUSIC-u lear??
We tested MUSIC-U in the robotic manipulation tasks. During training. the agent only receives the intrinsic MUSIC reward. In all three environments, the behavior of reaching objects emerges. In the push environments, the agent learns to push the object around on the table. In the slide environment, the agent learns to slide the object to different directions. Perhaps surprisingly, in the pick-and-place environment, the agent learns to pick up the object from the table without any task reward. All the observations are shown in the supplementary video
Question 2. How does MUSIC-u compare to Empowerment or ICM?
We tested our method in the navigation task. We combined our method with PPO (Schulman et al.] (017) and com- pared the performance with ICM (Pathak)
Sodia Bos PaySPaayrreeeee
Socia MayCrrarddC
Neagn Taek Reward
CV/ 87%
NUSiC-u
3.
et al., (017)) and Empowerment (Mo-
3.15
hamed Rezende (213)). During train-
ing, only used of the intrinsic
wards such as MUSIC, ICM, or Empow- erment train the agent. Then, we used the averaged task reward as the evaluation metric. The experimental results are shown in Figure (left). The y-axis represents the mean task (right) shows that the MUSIC reward signal ISS", S*) is relatively strong compared to the Empowerment reward signal / (A,, S*). Subse- quently, high MI reward encourages the agent to explore more states with higher MI. A theoretical connection between Empowerment and MUSIC is shown Appendix B The video starting from 1:28 shows the learned navigation behaviors.
100
253
300
400
220
100
230
300
400
590
Figure 3: Experimental results
reward and the x-axis denotes the training epochs. Figure 3
Question 3. How does MUSIC compare DIAYN?
5

---

Published as a conference paper at ICLR 2021
FelchPush-v1
Fole:FFikkAndPaacee
Felch-8le-v11
Mean Success Rate
SAC: 4.21%
0.4
SAC+MUSIC+ 36, 85%
SAC+MMBBC--
SAC+MUSIC C 20 22'5
S:C-MUSIC-:: 52.29%
0.3
SAC-MUSIC-S 25 87%
SAC+MUBIC A--S 25.0%
SAC-MUSIC-F
SAC 50 50.30%
0.3
SACHMUSIC+
0.4
0.4
0.3
0.2
022
0.0
10
10
20
30
10 20
30
Epoch
Figure 4: Mean success rate with standard deviation: The percentage values after colon (:) rep- resent the best mean success rate during training. The shaded area describes the standard deviation. A full comparison is shown in Appendix D Figure
We compared MUSIC, DIAYN and MUSIC+DIAYN in the pick-and-place environment. For im- plementing MUSIC+DIAYN, we first pre-train the agent with only MUSIC, and then fine-tune policy with DIAYN. After pre-training, the MUSIC-trained agent learns manipulation behaviors such as, reaching, pushing, sliding, and picking up an object. Compared to MUSIC, the DIAYN- trained agent rarely learns pick up the object. It mostly pushes flicks the object with the gripper. However, the combined model, MÜSIC+DIAYN, learns to pick up the object and moves to differ- locations, depending the skill-option. These observations 0:46. From this experiment, we can see that MUSIC helps the agent to learn the DIAYN objective. DIAYN alone doesn't I succeed because DIAYN doesn't It start learn any skills until it touches the object, which is rare in the first place. This happens because the skill discriminator only encourages the skills to be different.
shown in the video starting from
Question 4. How does MUSIC+DISCERN compare DISCERN?
The combination of MUSIC
Table Comparison of DISCERN with and without MUSIC
and DISCERN, encourages the agent to learn to control the ob- ject via MUSIC and then move the object to the target position via DISCERN. Table shows that DISCERN+MUSIC signif- icantly outperforms DISCERN.
Method
Push (%)
Pick & Place (%)
DISCERN
7.94% + 0.71%
4.23% + 0.47%
R (Task Reward)
11.65% 1.36%
4.21% + 0.46%
R+DISCERN
21.15% 5,49%
4.28% + 0.52%
R+DISCERN+MUSCE 95.15% + 8.13%
48.91% = 12.67%
This is because that MUSIC emphases more state-control and teaches the agent to interact with an object. Afterwards, DISCERN teaches the agent to move the object to the goal position each episode.
Question 5. How can we use MUSIC to accelerate learning?
We investigated three ways, including MUSIC-F, MUSIC-p. and MUSIC-r, of using MUSIC to accelerate learning in addition to the task reward. We combined these three variants with DDPG and SAC and tested them in the multi-goal robotic tasks. From Figure 4 we can see that all these three methods, including MUSIC-F, MUSIC-p. and MUSIC-r, accelerate learning in the presence of task rewards. Among these variants, the MUSIC-r has the best overall improvements. In the push and pick-and-place tasks, MUSIC enables the agent to learn in a short period of time. In the slide tasks, MUSIC-r also improves the performances by a decent margin.
We also compare our methods with their closest related methods. To be more specific, we compare MUSIC-F against the parameter initialization using DIAYN (Eysenbach et al., 2019); MUSIC-p against Prioritized Experience Replay (PER), which uses TD-errors for prioritization (Schaul et al.) 2016): and MUSIC-r versus Variational Information Maximizing Exploration (VIME) (Houthooft et al. 2016). The experimental results are shown in Figure 5 From Figure 5 (1st column), can see that MUSIC-f enables the agent to learn, while DIAYN does not. In the 2nd column of Figure 5] MUSIC-r performs better than VIME. This result indicates that the MI between states is crucial quantity for accelerating learning. The MI intrinsic rewards boost performance significantly compared VIME. This observation consistent with the experimental results of MUSIC-p and

---

Published as a conference paper at ICLR 2021
FetchSild-v11
FelchFushh-r1
1.0
MUBIC-F. 22%
-UBICCK:
MUSIC:p: 93.08%
Mean Success Rate
0.30
DLYY:
3.5
PER: 58.46%
0.25
0.16
0.3
0.4
0.10
0.08
0.1
0.00
0
10
60
80
10
20
30
40
0
10
Epoch
Performance comparison: We compare the MUSIC variants, including MUSIC-F, MUSIC-r, and MUSIC-p, with DIAYN, VIME, and PER, respectively, A full comparison is shown in Appendix D Figure 10
Figure 5:
PER, shown in Figure (3nd column), where the MI-based prioritization framework performs better than the TD-error based approach, PER. On all tasks, MUSIC enables the agent to learn the benchmark task more quickly.
Question 6. Can the learned MI estimator be transferred to new tasks?
would be beneficial if the pretrained MI estimator could be transferred to
Fass/Pushwr
FeschSidevrt
new task and still improve the perfor- mance (Pan al. 2010, [Beniio] 2012) To verity this idea, we directly applied the pretrained MI estimator from the pick-
Mean Success Rate
DDPO 9 75%
DOPGHMUS C- 06%
0.5
DOPOHMUBICC
DSPG 27 5%
0/8
DDPG 600
04
20POHNUSIC-
0.4
and-place environment to the push and slide environments, respectively, and train the agent from scratch.
Epoch
We denote this transferred method "MUSIC-"", where "-t" stands for transfer. The MUSIC reward function trained in its corresponding environments is denoted as "MUSIC-r". We compared the performances of DDPG, MUSIC-r, and MUSIC-t. The results are in Figure which shows that the transferred MUSIC still improved the performance significantly. Furthermore, as expected, MUSIC-r performed better than MUSIC-t. We can see that the MI estimator can be trained in task-agnostic (Finn et al. 2017) fashion and later utilized in unseen tasks.
Figure 6: Transferred MUSIC
Question 7. How does MUSIC distribute rewards over a trajectory?
To understand why MUSIC works, we vi- sualize the learned MUSIC-U reward Figure 7 We can observe that the MI re- ward peaks between the 4th and 5th frame, where the robot quickly picks up the cube from the table. Around the peak reward value, the middle range reward values are corresponding to the relatively slow move- ment of the object and the gripper (see the 3rd, 9th, and 10th frame). When there is contact between the gripper and the cube (see the 1st & 2nd frames), or the gripper holds the object still (see the 6th 8th frames) the intrinsic reward remains nearly zero. From this example, we that MUSIC distributes positive intrinsic rewards when the surrounding state has correlated changes with the agent state.
MUSIC
Figure 7: MUSIC rewards over a trajectory
Question 8. How does MUSIC reward compare to reward shaping?
Here, we want to compare MUSIC and reward shaping and show that MUSIC cannot be easily replaced by reward shaping. We consider simple L2-norm reward shaping, which is the distance between the robot's 's gripper and the object. With this aand-engineered reward, the agent learns to move its gripper close to the object but barely touch the object. However, with MUSIC reward, the
7

---

Published as a conference paper at ICLR 2021
agent reaches the object and moves it into different locations. MUSIC automatically induces lot of hand-engineered rewards, including the L2-norm distance reward between the gripper and the object, the contact reward between the agent and the object, the L2-norm distance reward between the object and the goal position and any other rewards that maximize the mutual information between the agent and the surrounding state. From this perspective, MUSIC can be considered as meta- reward for the state-control tasks, which helps the agent to learn any specific downstream tasks that falls into this category.
Question 9. Can MUSIC help the agent to learn when there are multiple surrounding objects?
When there are multiple objects. the agent is trained to maximize the MI between the surrounding objects and the agent via MUSIC. the case that there is a red and blue ball on the ground, with MUSIC, the agent learns to reach both balls and sometimes also learns to use one ball to hit the other ball. The results are shown in the supplementary video starting from 1:56.
Summary and Future Work:
We can see that, with different combinations of the surrounding state and the agent state, the agent is able to learn different control behaviors. We can train a skill- conditioned policy corresponding to different combinations of the agent state and the surrounding state and later use the pretrained policy for the tasks at hand, see Appendix [F] "Skill Discovery for Hierarchical Reinforcement Learning". In some cases, when there is no clear agent-surrounding separation or the existing separation is suboptimal, new methods are needed to divide and select the states automatically. Another future work direction is to extend the current method the partially observed cases. For example, we can combine MUSIC with state estimation methods and extend MUSIC to the partially observed settings.
5 RELATED WORK
Intrinsically motivated RL is a challenging topic. We divide the previous works in three categories. In the first category, intrinsic rewards are often used to help the agent learn more efficiently to solve tasks. For example, Jung et al. (20IT) and Mohamed & Rezende (0013) use empowerment,
which is the channel capacity between states and actions. A theoretical connection between MU- SIC and empowerment is shown in Appendix B VIME (Houthooft et al.] 2016) and ICM (Pathak)
al. 2017) use curiosity as intrinsic rewards encourage the agents explore the environment more thoroughly. Another category of work on intrinsic motivation for RL is to discover mean- ingful skills, such as Variational Intrinsic Control (VIC) (Gregor et al. 2016). DIAYN (Eysenbach) iet al. 2019). and Explore Discover Learn (EDL) (Campos et al. 2020). In the third category, in- trinsic motivation also helps the agent to learn soal-conditioned policies. Warde-Farley et al. (2019) proposed DISCERN, a method to learn MI objective between the states and goals. Based on DIS- CERN,Pong et al. (2019) introduced -it, which adapts maximum entropy strategy to sample goals from the replay buffer (Zhao et al. (019) in order to make the agent learn more efficiently in the absence of rewards. However, these methods fail to enable the agent to learn meaningful interaction skills in the environment, such as in the robot manipulation tasks. Our work is based on the surooudirg separation concept and drives an efficient state intrinsic control objective, which empowers RL agents to learn meaningful interaction and control skills without any task ward. A recent work (Song et al.] 2020) with similar motivation, introduces mega-reward, which aims to maximize the control capabilities of agents on given entities in given environment and show promising results in Atari games. Another related work (Dilokthanakul et al. (011)) proposes feature control as intrinsic motivation and shows sfate-of-the-art results in Montezuma revenge.
In this paper, we introduce MUSIC, a method that uses the MI between the surrounding state and the agent state as the intrinsic reward. In contrast to previous works on intrinsic rewards (Mohamed & Rezende 2015 Houthooft et al. 2016 Pathak et al., 2017 [Esenbachh et al.) [019] Warde-Farley let al. 2019). MUSIC encourages the agent to interact with the interested part of the environment, which is represented by the surrounding state, and learn to control it. The MUSIC intrinsic reward is critical when controlling a specific subset of the environmental state is the key to complete the task, such the case in robotic manipulation tasks. Our method is complementary to these previ- ous works, such as DIAYN and DISCERN, and can be combined with them. Inspired by previous works (Schaul et al., 2016 Houthooft et al. 2016 Eysenbach et al. 2019),, we additionally demon- strate three variants, including MUSIC-based fine-tune mee-tunngg rewarding, and prioritizing mechanisms, to sig significantly accelerate learning in the downstream tasks.
8

---

Published as a conference paper at ICLR 2021
6 CONCLUSION
This paper introduces Mutual Information-base State Intrinsic Control (MUSIC), an unsupervised RL framework for learning useful control behaviors. The derived efficient MI-based theoretical objective encourages the agent to control states without any task reward. MUSIC enables the agent to self-learn different control behaviors, which are non-trivial, intuitively meaningful, and useful for learning and planning- Additionally, the pretrained policy and the MI estimator significantly accelerate learning in the presence of task rewards. We evaluated three MUSIC-based variants in different enviromments and demonstrate substantial improvement in learning efficiency compared to state-of thetttrrrrt methods.
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint 1rXiv:1612.00410,, 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Arindam Banerjee. On bayesian bounds. Proceedings of the 23rd international conference on Machine learning, pp. 81-88. ACM, 2006.
David Barber and Felix V Agakov. The im algorithm: variational approach to information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, pp. 531-540. PMLR, 2018
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML workshop on unsupervised and transfer learning, pp. 17-36, 2012.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912: 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of auriosity-driver learning. arXiv preprint arXiv:1808.0433 00..888.0055,.. 2018.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres, Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceed- ings of the 37th International Conference on Machine Learning. PMLR, 2020.
Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational informa- tion bottleneck. In Advances in Neural Information Processing Systems, pp. 1957-1965, 2016.
Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, and Murray Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. JEEE transactions on neural net- works and learning systems, 30(1 11:3440933444 2019.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications Pure and Applied Mathematics, 28(1):1-47, 1975.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- fions, 2019. URL htRlhttpss:. /ttttps://openrr//// www.:eee/ffrru net/fforum21d=50x63

---

Published as a conference paper at ICLR 2021
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th Interationaal Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Gordon G Gallup. Chimpanzees: self-recognition. Science, 167(3914):86-87) 1970.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1r 1.07507, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchillon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference on Learning Representations, 2019. URL htttss://ppenrev///
net Fttp//oooommni11111111
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 1109-1117, 2016.
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous agent-environment systems. Adaptive Behavior, 19(1):16-39, 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint wrXiv:1412.69880 2014.
Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress Evolutionary Computation, volume 1, pp. 128-135. IEEE, 2005.
Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. arXiv preprint arXiv: 17t.::247000034366 2017.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6)306, 2004.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2115-2133. 2015.
Sinno Jialin Pan, Qiang Yang, al. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10)1345-1355, 2010.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning (ICML), volume 2017, 2017.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 094644 2018.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint 1XXX:1903300669 2019.
10

---

Published as a conference paper at ICLR 2021
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment- an introduction. In Guided Seif- Orfaanrgaaiztionn: Inception, pp. 67-114. Springer, 2014.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, 2016.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-buildng neu- ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-221 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017.
Joel Smith. Self-Coonccousneess. In Edward N. Zalta (ed.), The Stanford Encyclopedia Philosophy. Metaphysics Research Lab, Stanford University, summer 2020 edition, 2020.
Yuhang Song, Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, Shangtong Zhang, Andrzej Wojei- cki, and Mai Xu. Mega-reward: Achieving human-level play without extrinsic rewards. In AAAI, pp. 5826-5833, 2020.
Petar Velickovic, William Fedus, William Hamilton, Pietro Liō, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR (Poster), 2019.
David Warde-Farles Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through con-aaamettic discriminative rewards. In In- ternational Conference on Learning Representations, 2019. URL
9K'
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy- multi-goal reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 7553- 7562. PMLR, 2019.
APPENDIX
A MONOTONICALLY INCREASING RELATIONSHIP
Lemma 2. There a monotonically increasing relationship between (S mathematically.
T) and
Ep, (S [T)],
(T) S
1605"" [T'],,
(6)
where Ss, So, and T denote the surrounding stale, the agent stale, and the trajectory, respectively. The trajectory fractions are defined as the adjacent state pairs, namely T' = {St, S.+1). The symbol denotes monotonically increasing relationship between two variables and 6 represents the parameter of the statistics model in MINE.
Proof. The derivation of the monotonically increasing relationship shown as follows:
: T)
5o [Tp]
log log(Ep
84 Ie) s
x E se 5 [Tp]
EE> 8- 8m le
(8)
Er Ep s. [Tp]
Ex
8P
le To)
xE ,,
E s [Tl log(Ep
so [e (e
87
Ho TT),
(10)
where T3 represents a neural network, whose inputs are state samples and the output scalar. For simplicity, the symbol to denote monotonically increasing relationship between two

---

Published as a conference paper at ICLR 2021
variables, for example, log(2) means that the value of x increases, the value of log(z) also
increases and vice versa. To decompose the lower bound Equation 7 into small parts, we make
the following derivations, see Equation [8 10
Deriving from Equation 7 to Equation (
use the property that log(z) & a. Here, the new form, Equation (8). allows us to decompose the MI estimation into the expectation over MI estimations of each trajectory fractions, Equation (9). To be more specific, we move the implicit expectation over trajectory fractions in Equation (8) to the front,
and then have Equation (9). The quantity inside the expectation over trajectory fractions the MI
estimation using only each trajectory fraction, see Equation (9). We use the property, log(z) & x, Equation (TO).
again derive from Equation
B CONNECTION TO EMPOWERMENT
The state S contains the surrounding state S and the agent state S For example, in robotic tasks, the surrounding state and the agent state represents the object state and the end-effector state, respectively. The action space the change the gripper position and the status of the gripper, such open or closed. Note that, the agent's s action directly alters the agent state.
Here, given the assumption that the transform, S = F(A), from the action, A, to the agent state, S" is smooth and uniquely invertible mapping (Kraskov et al., 2004), then MUSIC objective, (S" 5*), is equivalent the empowerment objective, [(A, S*).
prove that the
The empowerment objective (Klyubin et al. 2005 Salge et al. 2014; Mohamed & Rezende 2015) defined as the channel capacity in information theory, which means the amount of information contained in the action A about the state S, mathematically:
E A(S,)).
(1I)
Here, we replace the state variable S with the surrounding state St, we have the empowerment objective as follows,
E = 1(8" A).
(12)
The MUSIC objective, [AA S* given the assumption that the transform, Sa = F(A), is a smooth and uniquely invertible mapping:
Theorem 3.
equivalent the empowerment objective,
I(S", = I(A,5")
(13)
where 5t, S0, and denote the surrounding state, the agent state, and the action, respectively.
Proof.
/ ds ds p(s"' s') p(s", s*)
I(S"
(14)
p(sa)p(ss
/ do ds
0A aas
a4 p(a, 8*)
p(a, s*) log
(15)
04 p(a)p(s*)
/ / ds dss JA(s") J s²) log
JA(s" JA(as)p(a,s")
(16)
JA(s") (((ss))(a)p(s))
/ Lads"p(a, 3)) p(a,s*)
(17)
p(a)p(s*)
S*)
(18)
12

---

Published as a conference paper at ICLR 2021
C MUTUAL INFORMATION NEURAL ESTIMATOR TRAINING
Algorithm 2: MINE (Belghazi al. 2018))
<- initialize network parameters
repeat
Draw minibatch samples from the joint distribution:
(x(1), ())), ).... (x((), , (x))) ~ Pxzz
Draw samples from the Z marginal distribution:
2(), 200) ~ Pz
Evaluate the lower-bound:
V(0) + 5 Σ , log( E. ( (())
Evaluate bias corrected gradients (c.g., moving average):
(()) < (())
Update the statistics network parameters:
< + G())
until convergence
One potential pitfall of training the RL agent using the MINE reward is that the MINE reward signal can be relatively small compared to the task reward signal. The practical guidance to solve this problem is to tune the scale of the MINE reward to be similar to the scale of the task reward.
D EXPERIMENTAL RESULTS
The learned control behaviors without supervision are shown in Figure as well as in the supple- mentary video. The detailed experimental results are shown in Figure and Figure 10
Reach
Push
Slide
Pick-up -up
Figure 8: Learned Control behaviors with MUSIC: Without any reward, MUSIC enables the
agent to learn control behaviors, such as reaching, pushing, sliding, and picking up an object. The learned behaviors are shown in the supplementary video.
13

---

Published as a conference paper at ICLR 2021
FeichPush-v1
FelcsiickknnPPaaeev
FeichSlide-v1
0.5
DOPG:
COPG+MUSIC F. 9226
0.5
DOPG-MUSIC F: 12.20%
COPS+MUSIC 85.0%
-CPG-MUSIC-p: 98.05%
DOPG-MUSIC-P-2 21.1-2
27.51%
DOPC-MUSIC-r 08%
DDPG+MUSICH
COPG-MUSIC 35.8%
0.5
0.2
Mean Success Rate
0.0
0
10
20
30
10
20
30
40
10
20
30
40
FetchPush-vv
FutchPickArrPPacevr
FotchSlid-v1
SAC 11.85%
SIAC: 4.21%
SAC: 19.69%
SAC+MUBIC-1
SAC-MUBIC-p: 29%
SAC-MUUSIC-: 87%
SAC-MUBIC- 25.0%
0.6
SAC+MRUSICHT 94.89%
SAC-MUSIC-T 50. 58%
SAC+MMU9CC..B8.00
0.4
0.2
Epoch
Figure 9: Mean success rafe with standard deviation: The percentage values after colon rep- resent the best mean success rate during training. The shaded area describes the standard deviation.
E COMPARISON OF VARIATIONAL MI-BASED AND MINE-BASED IMPLEMENTATIONS
Here, we compare the variational approach-based (Barber & Agakov 2003) implementation of MU- SIC and MINE-based implementation (Belghazi et al. (018)) of MUSIC in Table 2 All the exper- iments are conducted with different random seeds. The performance metric is mean success rate (%) + standard deviation. The "Task-r" stands for the task reward. From Table 2
that
Table Comparison of variational MI (y)-based and MINE (m)-based MUSIC
Method
Push (%)
Pick & Place (%)
Task-r+MUSICVv)
94.9% + 5.83% 94.83% + 4.95%
49.17% + 4.9%
MUsK-)+MUSIC())
50.38% + 8.8%
the performance of these two MI estimation methods similar. However, the variational method introduces additional complicated sampling mechanisms, and two additional hyper-parameters. i.e.. the number of the candidates and the type of the similarity measurement (Barber & Agakov 20033 Eysenbach et al. 2019: Warde-Farla et al., 2019). In contrast, MINE-style MUSIC is easier to implement and has less hyper- peper-parameters to tune. Furthermore, the derived objective improves the scalability of the MINE-style MUSIC.
F SKILL DISCOVERY FOR HIBRARCHICAL REINFORCEMENT LEARNING
In this section, we explore the direction Hierarchical Reinforcement Learning based on MUSIC.
For example, in the Fetch robot pick-and-place environment, have the follow states the observation: grip-pos, object object.pos, ob jejcttvelp, object.rot, object_velr, where the abbreviation "pos" stands for position; "rot" stands for rotation; "velp" stands for linear velocity, and "velr" stands for rotational velocity.
The gripipos is the agent state. The surrounding states ob jectt-oos, ob ject_velp, object rot, ob jeject velr. In Table 3 we show the MI value with different state-pair com- binations prior training and post training. When the MI value difference high, it means that
14

---

Published as a conference paper at ICLR 2021
FetchPush-v
FetenPockAndPaceev
FetchSlide-v1
MUSICC 10%
-USIC+4 57.5%
MUSICH 20
3.65%
0.25
0.4
0.15
0.2
02
0.10
0.0
0.0
0.00
FotchPush-vi
FotenPiccAndPacce v1
FotchSlide-vl
Mean Success Rate
MUSIC-: 06%
51.20%
0.5
MUSIC
4.75%
21.77%
0.8
0.4
0.9
0.3
0.4
0.2
0.1
0.0
FotchPush-vrt
FetenFiddAndPPaze--11
FatchSlide-v1
-MSICC 90.08%
MUBIC-p PER: 0.34%
0.35
MMSIC 27.81%
0.8
PER: 58 68.44%
0.25
PER: 20.91%
0.20
0.25
0.15
0.4
0.16
0.10
0.06
0.06
Epoch
Figure 10: Performance comparison: We compare the MUSIC variants, including MUSIC-F, MUSIC-r, and MUSIC-p, with DIAYN, VIME, and PER, respectively.
the agent has a good learning progress with the corresponding MI objective. From Table first row,
Table 3: Mutual Information estimation prior and post to the training
Mutual Information Objective
Prior-train Value
Post-train Value
MI(grip-post object.pos)
0.003 + 0.017
0.164 + 0.055
Mlggrip_pos object.rot)
0.017 0.084
0.461 = 0.088
MI(grip pos; sejjectvvelp
0.005 + 0.010
0.157 + 0.050
Ml(grip-pos; object_velr)
0.016 0.083
0.438 + 0.084
we can see that with the intrinsic reward MI(grip-por ob ject.poos), the agent achieves a high MI after training. which means that the agent learns better control the object positions using its gripper. Similarly, in the second row of the table, with Mi(grip pos; object ro)), the agent learns to control object rotation with its gripper.
From the experimental results, we can see that with different combination of state-pairs of the agent and surrounding state, the agent can learn different skills, such as manipulate object positions or rotations. We can connect these learned skills with different skill-options (Eysenbach et al. 2019) and train a meta-controller to control these motion primitives to complete long-horizon tasks in hierarchical reinforcement learning framework (Eysenbach et al. 2019). We consider this future research direction, which could be a solution in solving more challenging and complex long-horizon tasks.
15

---

Published as a conference paper at ICLR 2021
G EXPERIMENTAL DETAILS
We ran all the methods in each environment with 5 different random seeds and report the mean success and the standard deviation. The experiments of the robotic manipulation tasks in this paper use the following
Actor and critic networks: layers with 256 units each and ReLU non-linearities
Adam optimizer (Kingma & Ba 2014) with 10 for training both actor and critic
Buffer size: 10 transitions
Polyak-averaging coefficient: 0.95
Action L2 nomn coefficient: 1.0
Observation clipping: |-200, 200]
Batch size: 256
Rollouts per MPI worker: 2
Number of MPI workers: 16
Cycles per epoch: 50
Batches per cycle: 40
Test rollouts per epoch: 10
Probability of random actions: 0.3
Scale of additive Gaussian noise: 0.2
Scale of the mutual information reward: 5000
The specific hyper-puune parameters for DIAYN follows:
Number of skill options:
Discriminate skills based on the surrounding state
The specific hyper- hyper-parameters for VIME are follows:
Weight for intrinsic reward ry: 0.2
Bayesian Neural Network (BNN) learning rate: 0.0001
BNN number of hidden units: 32
BNN number of layers: 2
Prior standard deviation: 0.5
Use second order update: True
Use information gain: True
Use KL ratio: True
Number updates per sample:
The specific
for DISCERN are follows:
Number of candidates calculate the contrastive loss: 10
Calculate the MI using the surrounding state
The specific hyper-parameters for PER are follows:
Prioritization strength 0.6
Importance sampling factor 3: 0.4
The specific typer-parameter for SAC following:
Weight of the entropy reward: 0.02
16


Reviews:
[{"review_id": "lPebmHHcSoQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose MUSIC, an RL algorithm for learning controllers in a unsupervised way. They key idea of the proposed algorithm is to separate the state of the robot like joint angles from the state of the environments such as location of an external object and optimize the mutual information between the two set of states. By maximizing this mutual information, the resulting policy learns to better control the environment and can be used to train downstream tasks. The paper experimented with different ways of training the downstream tasks and demonstrated favorable results compared to prior methods.\n\nI think the paper introduces an interesting idea for training unsupervised skills for manipulation tasks. The results also seem very promising. However, I have the following concerns that I hope the authors could help address:\n\n1. The exposition of the paper can be improved. For example, Eq. 4 is referred and discussed multiple times in the text before it is defined. In the first sentence of 3.2, what does adjacent state mean? Does it mean S^s and S^a? What's the difference between \\Tau and \\Tau' in Eq. 4? In general, although the proposed algorithm seems reasonable, the derivation is a bit confusing.\n2. The agent-surrounding separation seems to apply to the full state space, but I'm not sure how we can apply it to observation spaces that contains partial information of the full state. For example, if the robot is equipped with a camera is tasked to train policies with vision input, how does the proposed algorithm handle such cases?\n3. I feel that it would be more interesting if the agent-surrounding separation idea could be generalized to be separating the observation space into two subsets where one is optimized to control the other. For example, if we are dealing with a legged robot, which is trained to move forward. The current framework wouldn't be able to handle it because there is no environment state. However, if we separate it into the space of joint angles and velocities, and the space of base position and orientations, would the current framework be able to obtain a policy that can take the robot to a large variety of positions and orientations? If so, a related example would greatly strengthen the paper in my opinion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ceKs9niPx0Z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces MUSIC, a reinforcement learning approach that separates the agent state from its surrounding state and trains the agent to maximize the mutual information between the two. This implies that the agent has control over the surrounding state. The approach is evaluated within four environments and compared to multiple baselines.\n\n\nThis paper is well-motivated and the approach is interesting. The paper is mostly well-written, though I found parts to be somewhat confusing. Code is provided as well as hyperparameters so the approach seems reproducible. The experiments are strong as the approach is evaluated within multiple environments with extensive comparisons to relevant baselines.\n\n\nMUSIC is shown to achieve very good performance on simulated robotic tasks, and was able to improve performance when combined with other intrinsic reward and RL methods. I think this is an interesting direction and it does make sense to separate out the agent’s state from the environment state. For these reasons I do think the paper should be accepted.\n\n\nHowever, I found the description of the methodology in section 3.2 to be very confusing. The equations are referred to before they are introduced which was unexpected. Hence, this section would be greatly improved by some rearranging. I also did not understand what exactly T was. What does this function output and how is it trained? \n\n\nComments:\n\n\n- Some other related works are [1] which uses an intrinsic reward to maximize the controllable entities in the environment and [2] which learns an intrinsic reward that maximizes controllable features.\n\n\n-  Question 3 in the paper does not refer to any figure (does this correspond to figure 5?). Where are the MUSIC + DIAYN results?\n\n\n- Is the reward in Question 8 the negative L2 norm?\n\n\n- How does MUSIC alone perform in Table 1? This should be included here as well. \n\n\n[1] Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards. Song et al. \n\n\n[2] Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning. Dilokthanakul et al. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UBWBELoBSA4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper proposes the use of state control as intrinsic motivation. It does so by separating the overall state into an agent state and a surrounding state. The idea then is to maximize the mutual information between the agent's internal state and the environment state. This mutual information is given as a reward to the agent which it attempts to maximize using policy gradient algorithms. Experiments attempt to validate the usefulness of such an intrinsic reward signal for pre-training an agent, or for augmenting a given task reward.\n\n## Positives\n+ The proposed idea of tying an agent's internal state to the surrounding environment is an interesting new metric. A practical use-case for embodiment is extremely interesting and this paper champions such a use-case fairly well.\n+ Adapting the use of MINE for computing the MUSIC reward is a useful contribution.\n+ Experiments sufficiently showcase the viability of the idea in various manipulation tasks.\n+ The video showcasing the various behaviors learned using MUSIC rewards in different environments, as well as the different skill learned when combined with DIAYN give a clear understanding of the effect of this intrinsic reward.\n+ Comparison with various exploration bonuses, skill learning priors, and prioritization schemes gives a clear idea of how MUSIC compares with other similar techniques.\n\n## Drawbacks:\n- While I find the idea and the experiments quite interesting, I do feel there is some improvement necessary in presenting the idea and comparing it with the idea that is most related to MUSIC, which is empowerment.\n- My main gripe with the paper is that the background of mutual information, estimating mutual information using MINE, and the adaptation in this paper in order to compute mutual information for agent trajectory is not sufficient or clear. While an interested reader can go to the referenced work and glean more information, the idea itself should be sufficiently understood by reading this paper. I would argue this is not true with the current state of the draft.\n- While there is an experiment for comparison with empowerment, more explanation in the related work is necessary for a reader unfamiliar with that body of work. \n- While the paper deals with the idea that an agent uses its knowledge of itself and the environment to affect the environment as much as possible, it does not discuss sufficiently how such an agent-environment divide might be possible or not possible in various scenarios where RL would be a useful solution mechanism. For example, would it be possible to use MUSIC in Atari, or for optimizing a recommendation system? From my understanding, it would not be straightforward. But this is a question that seems like it deserves more discussion in the paper. \n\n## Questions:\n- It is unclear why computing the mutual information between the agent and surrounding state needs information of the trajectory. Is it just a practical issue, as referenced in Section 3.2?\n- Related to the above (and quite possibly answered by it), why are all the states in a trajectory used for computing the MI? Is correlation in the samples not an issue? Why not?\n- In the experiments with the Fetch robot, the off-policy algorithm used by HER is labeled as SAC (Figure 4). However, the experimental details in the appendix as well as the code provided with the supplementary material seems to be specific to DDPG. Is there a reason for this discrepancy?\n- Where can we expect MUSIC to not provide informative rewards? Will it work in scenarios where agent actions will change the surrounding state in a temporally offset manner? What happens when the surrounding state changes due to some change in agent state, but delayed. For example, in the Fetch Slide task, the agent must slide the block across the table, but after the initial push any movement the robot makes does not affect the block's trajectory. How does this dynamic affect MUSIC rewards, and subsequent performance on the task? I understand if this particular question is out of the scope of the paper. But curious as to the authors' views.\n\n## Conclusion\nMUSIC rewards are an interesting idea and would be a good addition to intrinsic motivation literature. A more well-rounded explanation of the techniques used in this paper and the capabilities and use-case for MUSIC would elevate the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "O5CqDcWPtC8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces MUSIC, a framework for intrinsically motivated RL, where the intrinsic reward comes from maximizing the mutual information between the agent's state and the surrounding environment's state. The authors motivate and describe this approach, explain its incorporation into various training modes, exhaustively characterize its properties, and compare to numerous related past approaches. Although it is somewhat specific to particular environment domains, MUSIC offers a compelling addition to the family of intrinsically motivated RL algorithms based on concepts of mutual information.\n\n### Clarity\nThis paper is, for the most part, a model of clarity. The proposed algorithm follows from a clear intuition and the results present a detailed and organized characterization/validation. The authors could further improve clarity by adding a bit more exposition around the training of the mutual information estimator. This is only a minor issue, since the mutual information framework itself is well established by now. Still, it would help to more concretely describe the implementation details covered in Section 3.2, perhaps in another small section of the Appendix.\n\n### Quality\nThis paper is very high quality. The experiments are thorough and well organized, addressing an impressive number of (literally enumerated) questions. This also serves to demonstrates an impressive versatility of MUSIC, the proposed technique, while simultaneously enabling comparison to a **wide** range of past methods and showcasing a variety of potential uses. However, there is a lack of guidance around practical challenges facing this technique and its potential pitfalls.  For example, Campos et al. (_ICML_ 2020) describe some failure modes associated with simultaneously learning networks for estimating mutual information and using them to train policies. It would be valuable to know if MUSIC has similar (or otherwise noteworthy) failure modes.\n\n### Originality and Significance\nAs the paper describes, there is a large body of work on intrinsically motivated RL as well as intrinsic rewards derived from mutual information-based objectives. In many cases, the types of behaviors those alternative approaches would hope to encourage are the same as those learned by MUSIC. From what the paper demonstrates, it seems like MUSIC is a more successful iteration of these attempts. That is not meant to diminish its significance -- it's a very hard problem! Ostensibly, MUSIC incorporates the right inductive bias through its decomposition of the state. I expect that the field will find both the technique to be useful as well as the general insights brought about through this work. My only concern regarding the significance is whether MUSIC is only applicable to a relatively narrow set of domains. Even if we assume that the state can be cleanly decomposed into the agent/surrounding constituents, how might MUSIC handle things like partial observability or other types of uncertainty that may affect estimations of mutual information? The paper may benefit from a brief discussion around generality and, if appropriate, how future work may address any issues therein.\n\n**Pros**\n- Simple extension of an existing framework leading to clear and versatile improvements within a challenging problem\n- Paper is exceptionally clear and well-organized\n- Impressively thorough experimental characterization/validation\n- Code is provided to help improve reproducibility and external adoption\n\n**Cons**\n- Some gaps in practical guidance and discussion of potential pitfalls\n- Generality of the approach (with respect to environment/task setting) is a bit unclear", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces MUSIC, a method for unsupervised learning of control policies, which partitions state variables into exogenous and endogenous collections and maximizes mutual information between them. Reviewers were uniformly positive, agreeing that the  approach was interesting and well-motivated, and the experiments convincing. Some concerns were raised as to clarity, which were addressed through several revisions of the manuscript. I am happy to recommend acceptance.
Decision: Accept (Spotlight)