Self-Supervised State-Control through Intrinsic Mutual Information Rewards
Authors: Rui Zhao
Volker Tresp
Wei Xu
Venue:  (2020)
Keywords: Intrinsic Reward
Deep Reinforcement Learning
Skill Discovery
Mutual Information
Self-Supervised Learning
Unsupervised Learning

Abstract: Learning to discover useful skills without a manually-designed reward function would have many applications, yet is still a challenge for reinforcement learning. In this paper, we propose Mutual Information-based State-Control (MISC), a new self-supervised Reinforcement Learning approach for learning to control states of interest without any external reward function. We formulate the intrinsic objective as rewarding the skills that maximize the mutual information between the context states and the states of interest. For example, in robotic manipulation tasks, the context states are the robot states and the states of interest are the states of an object. We evaluate our approach for different simulated robotic manipulation tasks from OpenAI Gym. We show that our method is able to learn to manipulate the object, such as pushing and picking up, purely based on the intrinsic mutual information rewards. Furthermore, the pre-trained policy and mutual information discriminator can be used to accelerate learning to achieve high task rewards. Our results show that the mutual information between the context states and the states of interest can be an effective ingredient for overcoming challenges in robotic manipulation tasks with sparse rewards. A video showing experimental results is available at https://youtu.be/cLRrkd3Y7vU

Under review as a conference paper at ICLR 2020
SELF-SUPERVISED STATE-CONTROL THROUGH INTRINSIC MUTUAL INFORMATION REWARDS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning useful skills without a manually-designed reward function would have many applications, yet is still a challenge for reinforcement learning. In this paper, we propose Mutual Information-based State-Control (MISC), a new self- supervised Reinforcement Learning approach for learning to control states of in- terest without any external reward function. We formulate the intrinsic objec- tive as rewarding the skills that maximize the mutual information between the context states and the states of interest. For example, in robotic manipulation tasks, the context states are the robot states and the states of interest are the states an object. We evaluate our approach for different simulated robotic manip- ulation tasks from OpenAI Gym and navigation task in the Gazebo simula- tor. We show that our method is able to learn to manipulate the object, such pushing and picking up, purely based on the intrinsic mutual information re- wards. Furthermore, the pre-trained policy and mutual information discriminator can be used accelerate learning to achieve high task rewards. Our results show that the mutual information between the context states and the states of interest can be an effective ingredient for overcoming challenges in robotic manipulation tasks with sparse rewards. A video showing experimental results is available at pettps://youtu beakt/////e//KKKWaW......
INTRODUCTION
Reinforcement I Learning (RL) (Sutton & Barto
1998) combined with Deep Learning (DL) (Good-
fellow et al.) 2016) has led to great successes in various reward-driven tasks, such as playing video
games (Mnih et al. 2015), learning continuous control (Ng et al.) [0066] Peters & Schaal 2008
Levine et al., 2016 Chebotar et al.) (017). navigating in complex environments (Mirowski et al.
2017 Zhu et al. (017). and manipulating objects (Andrychowicz al. 2017 2018).
Despite these successes, RL agents that learn only from reward signals differ in the manner that humans learn. In the case of learning to manipulate objects, human agent not only attempts to accomplish the task but also learns to master the controllable aspects of the environment (Lake) eet al. 2017). Notably, human agent can quickly learn the correlation between its own action and the state change of an object, even without supervision, to later use the acquired skill to manipulate the object into the desired state.
The ability to fully autonomously learn to control the states of interest has many benefits. First, it would make learning possible in the absence of hand-engineered reward functions or manually- specified agent goals. It is known that designing reward function that ensures the agent to learn the desired behaviors is challenging (Hadfiel-
al. 2017). Secondly, to learn to "master" the environment potentially helps the agent to learn to achieve goals in sparse reward settings. Thirdly, the policy of controlling states of interest can be quickly adapted to unknown tasks. It is currently an open challenge to design RL agents that automatically learn useful skills to control the states of the environment, without rewards (Warde-Fatte Farley et al. 2019).
In this paper, skill is policy that changes the state of the environment in consistent way. The policy can be a single unconditioned policy (Peters & Schaal 2008) or a conent-condiottdd pol- icy (Eysenbach et al. 2019). One way to learn skills is to simultaneously train discriminator to discern the skill-options of the agent (Szepesvari et al. 2014) based on the states of the trajec- tory (Eysenbach et al. 2019). In this way, the agent should be able learn diverse set of skills.

---

Under review as a conference paper at ICLR 2020
Figure Fetch robot arm environments and navigation task based on the Gazebo simulator:
FetchPushe FetchPickAndP Lace, FetchhSiddd CocialBot-Playzzon
Another way is learn environment model encourage the agent to explore the states, which are relatively unpredictable (Houthooft et al. 2016 Pathak et al.) 2017 Ha & Schmidhuber (0188) These methods more focused on efficiently exploring novel states instead of controlling states of interest.
We propose a new supe-sspe reinforcement learning method, Mutual Information-base State- Control (MISC), which is an approach that learns skills to control states of interest without any reward signal. States of interest are the states of the environment that we are interested Context statey are the states of the environment excluding the states of interest. The states of interest and the context states are given by the user before training the model. The idea of our method to encourage the agent find skills that maximize the mutual information between the context states and the states of interest. We first divide the observation states into two sets, the context states and the states of interest. In the case of robotic manipulation tasks, Figure the context states the robot states; the states of interest are the states of an object. During the learning process of the agent, a discriminator learns evaluate the mutual information between the context states and the states of interest. The agent receives high intrinsic rewards from the discriminator when there is high mutual information. Our hypothesis is that if the agent learns to control the object, then the mutual information of the agent states and the object states should be relatively high. Interestingly, this automated RL scheme bears similarities to the self-supervised learning of feature representations in computer vision tasks, where neural network is trained to predict a part of the image given another part of the same image (Doersch et al., 2015). During the process, the neural network captures the mutual information among different parts of the image. Similarly, in our work, we want to capture the mutual information among different sets states.
This paper contains the following five contributions. First, we introduce a new self-supervised RI. method, Mutual Information- based StateCContro, for learning to control states of interest. Secondly, evaluate the developed framework in the robotic simulations of OpenAl Gym and navigation task in Gazebo and demonstrate that MISC enables the agent to learn skills, such as reaching, push- ing, picking up, and sliding the object without rewards. Thirdly, we show that the pre-trained policy can be quickly adapted to the specific tasks with external sparse reward signal. Fourthly, the pre- trained mutual information discriminator also improves the learning process of the agent, either as intrinsic rewards or as priorities for experience replay, in addition to the sparse rewards. Finally, we show that the learned mutual information discriminator can transferred among different tasks and still improves the performance.
PROBLEM FORMULATION
In the RL setup, an agent interacts with an environment. The environment is fully observable and includes a set of states S, set of actions A, distribution of initial states p(su), transition proba- bilities p(s(+1 St. az). reward function r: A + R. and a discount factor 7 E [0, 1]. These components formulate Markov decision process represented as a tuple, (8, p,r,,, A policy æ maps a state to an action, क S - A. The goal of the agent to maximize the accumulated reward, i.c., the return, R₁- > 0c expected return, [2,
2i=t 7 -rrrr, over all episodes, which is equivalent to maximizing the
We consider robotic manipulation tasks, like the robotic simulations provided by OpenAl Gym Todorov et al.) 2012 Plappert et al., 2018), where a robot arm attempts to manipulate an object to pushing, picking & placing, or sliding, as shown in Figure I We also consider navigation task, where the robot should navigate to the target, which is a ball. as shown in Figure [ TTe states of the environment consist of positions, orientations, linear and angular velocities of all
goal position
2

---

Under review as a conference paper at ICLR 2020
ENVIRONMENT
Leamed
Algorithm 1: MISC
1 plst41 Sty (1e
Fixed
while not converged do
Sample traeectories 04
Discrininator ustimaties
Sample initial state ~ p(so).
from unviromments
$1:7
from states. Update discri-
for t 4- 1 to steps.per-episodle do
Sample action de (ax st).
AGENT
MI DISCRIINATTOR
Step environment 8141 ~ p(5+1 81, 4.). Compute 1,(+he41, 96.41) with (6).
0, Ta (z s:)
To(s112 .i.1)
Set intrinsic reward P2 = (s (e++1)
Updale policy to maximize the mutua] information reward
Update policy (0) via DDPG or SAC.
Ta 11 $14+1)
Update MI discriminator (Φ) with SGD.
Figure 2: MISC Algorithm: We update the discriminator to better estimate the mutual information (MI), and update the agent to control states of interest to have higher MI with the context states.
robot joints and of an object. We divide the states s into two sets, the robot states s" and the object states s". For manipulation tasks, are interested in the states of the object. Therefore, define the states of interest s² as the object states, = s"; the context states 8 as the robot states, 8 8" For simplicity, we represent the states of interest using only the object positions, s (2", =0), and the context states using only the gripper positions, sf = (x yy, 29).
In this paper, we focus on learning to control states of interest purely using the agent's observations and actions, without any reward function. Motivated by the idea that an agent capable of controlling states of interest sf to have high mutual information with its context state s has "mastered" the environment, we formulate the problem of learning without external supervision one of learning a policy No (a se) with parameters f maximize intrinsic mutual information rewards, T(S"; 8"). Here, we use upper letters, such S to denote random variables and the corresponding lower case letter, such as s", represent the values of random variables.
3 METHOD
In this section, we formally describe our method, including the mutual information reward function and the self-supervised RI. framework.
3.1 LEARNING A MUTUAL INFORMATION REWARD FUNCTION OF STATES
Our method simultancously learns a policy and an intrinsic reward function by maximizing the mutual information between the states of interest and the context states. Mathematically, the mutual information between the state of interest random variable S' and the context state random variable S is represented as:
I(S;; = =H(S)) H -
= DKs Ps: & Ps.)
where ss is the joint probability distribution; Pss: @ P the product of the marginal distri- butions Ps.: and Ps=; DxL denotes the Kullback-Leibler (KL) divergence. The first line in Equa- tion (1) tells us that the agent should maximize the entropy of states of interest, ((*)), and concur- rently, minimize the conditional entropy of states of interest given the context states, H(S 59). When the conditional entropy, HS$ St), ), small, it becomes easy to predict the states of in- terest based the context states. In this case, for example in robotic manipulation tasks, we say that the robot can control the object because easy to predict the location of the object S given the robot states 5 The second line in Equation [ gives the mutual information in the KL. divergence form.
Mutual information is notoriously difficult compute in real-world settings (Hjelm et al. 2019). Motivated by Belghazi et al.] (2018), we use a lower bound to approximate the mutual information quantity, T( First, rewrite the KI. formulation of mutual information objective, Equa-
3

---

Under review as a conference paper at ICLR 2020
tion using the Donsker- Varadhan representation (Donsker & Varadhan 1975)
S05 (Pus(ss's P @ Ps==)
ge [7] log( of Fe (e7])
(2)
sup T:Q-
where the input space is a compact domain of TR. i.e., Ω C RR; the supremum taken over all functions Tsuch that the two expectations are finite. Secondly, we lower bound the mutual information in Donsker- Donsker-Varadhan representation with the compression lemma in the PAC-Bayes literature (Banerjee 2006), mathematically,
1($)
sup Bp gig [T] log &7 (e"1)
T:Q-3
sup sege [To] log( 57 le sc
= ( S*)
The expectations Equation 3 are estimated using empirical samples from Psi and IPg: @ P-s- or by shuffling the samples from the joint distribution along the batch axis (Belghazi et al. (018). The mutual information reward function, r = Is (S", 5*), can be trained by gradient ascent. The statistics model Ts is parameterized by deep neural network with parameters d 4, which aims to estimate the mutual information with arbitrary accuracy.
3.2 MUTUAL INFORMATION-BASED STATE-CONTROL.
We want train an agent to "master" the states of interest in self- self-suuer reinforcement learning fashion. The agent aims to maximize the mutual information between the states of interest and the context states. the beginning of each episode, the agent takes actions at following a partially random policy, such as e-greedy, to explore the environment and collects trajectories into a buffer for later replay. The trajectory contains a series of states, {s1, .T......,,,,; where T is the time horizon of the trajectory. Each state s, consists of the states of interest si and the context states st where in robot manipulation tasks, we use the object position as st. and the gripper position as si-
For training the mutual information discriminator network, we first randomly sample trajectories of states $1:7 = {ss, $2...., st.....,,) $T} from the replay buffer. Here, we use Sty to indicate subsequence of trajectory, for instance, $1. refers to {s1, .2... $.). Then, we evaluate the mutual information lower bound, (s )i.r), mathematically,
(o) log 2=11
Σ To(o).)) T
1
(4)
T
where the states &; are sampled by shuffling the states of along the temporal axis t within each trajectory. After evaluating the lower bound, To(si.r si.r), we back propppropagttion optimize the parameter (Φ) to maximize the mutual information lower bound. Here, we calculate the mutual information using the samples from the same trajectory. If the context states the states of interest do not change within the trajectory, then the mutual information of these states is zero. Therefore, the agent does alter the object states during the episode, then the mutual information between the agent states and the object states remains zero.
We define the transition mutual information reward as the mutual information estimation of trajec- tory fraction, sta+1. mathematically,
H+1 Σ t=t
++1
= ( clip
(s), - log
e To(s).55) Σ 2
0, tran
(5)
2
where Irre is the predefined maximal transition mutual information value. The clip function limits the transition mutual information value in an interval of [0, [maa]. The lower threshold 0 forces the mutual information estimation to be non-negative. In practice, to mitigate the influence some particular large transition mutual information, find it useful to clip the intrinsic reward with the threshold value Imar. This clipping trick makes the training stable. The threshold value can be considered hyper-parameter.
4

---

Under review as a conference paper at ICLR 2020
We implement MISC with both deep deterministic policy gradient (DDPG) (Lillicrap et al. 2016) and soft actor-critic (SAC) (Haarnoja et al. 2018) to learn policy Ta(a 8) that aims to control the states of interest. In comparison to DDPG and SAC, the DDPG method improves the policy in a more "greedy" fashion, while the SAC approach is more conservative, in the sense that SAC incorporates an entropy regularizer H(A S) that maximizing the policy's entropy over actions. We found empirically that DDPG works better when the agent starts learning quickly, while SAC explores the environment more thoroughly in general.
Overall, the agent is rewarded for controlling states of interest to have higher mutual information with its context states, which is considered to gain degree of mastery of the environment. We summarize the complete training algorithm in Algorithm and in Figure
4 EXPERIMENTS
To evaluate MISC, we use the robotic manipulation environments provided by OpenAI Gym and also navigation task, see Figure I (Brockman et al. 2016 Plappert et al. 2018). First, we an- alyze the skills learned purely with the intrinsic reward. We show that the agent is able learn skills, such as reaching, pushing, picking up, and sliding an object, see Figure 31 We also compare our method with "Diversity is All You Need" (DIAYN) method (Eysenbach ondly, we show that the pre-trained models, including the policy and the MI discriminator, can be used for improving performance conjunction with the rewards. There are several ways to im- prove performance, such via policy initialization, with the MI intrinsic rewards, and using the MI value for prioritized experience replay. Interestingly, we also show that the pre-trained MI dis- criminator can be transferred among different tasks and still improves performance. We combine or compare our method with other methods, including DDPG (Lillicrap et al. 2016), SAC (Haarnoja et al. 2018), PPO (Schulman al.) 2017), DIAYN (Eysenbach al.) 2019), 9, PER (Schaul et al.; 2016), VIME (Houthooft et al. 2016)) ICM (Pathak et al. 2017). and Empowerment (Mohamed & Rezende 2015). Thirdly, we show some insights about how MISC rewards are distributed across trajectory. The detailed experimental details are shown in the Appendix. Our code is available at https: /
al. 2019). Sec-
4.1 ANALYSIS OF LEARNED SKILLS
Question 1. What skill does MISC learn?
We tested MISC multiple robotic manipulation tasks, including push, pick & place, and slide, Figure [ In the push and slide tasks, the gripper is fixed to be closed. In the pick & place task, the agent has the control of the gripper to be open or closed. Unlike in push and pick & place environments, the surface of the table in the slide task has very low friction so that the object can slide on these surface. In all the environments, the object is randomly placed on the table at the beginning of each episode. During training, the agent only receives the intrinsic MISC reward, see Section 3.2 In all three environments, the behavior of reaching objects emerges from the self- supervised training, for example, as shown in Figure 3 (1st row). Furthermore, the push and pick & place environments. the agent learns to push the object around on the table, see Figure 3 (2nd row). In the slide environment. the agent learns to slide the object into different directions. see Figure 3 (3rd row). Perhaps surprisingly, in the pick & place environment, the agent learns to pick up the object from the table without any reward, which means taking full control of the object states, see Figure 3 t4th row).
We implemented the MISC with both DDPG and SAC and ran the experiments with different random seeds. To compare DDPC+MISC and SAC+MISC, we ran 20 trials using the learned policy in the pick & place environment with each seed. We observed that, in all 5 random seed settings, SAC+MISC learns the picking-up skill, while DDPG+MISC learns to pick up an object in only 1 out of 5 random seed settings. Mostly, the agent learns to push, flip, or grip the object. These observations show that the entropy bonus, H(A S), of SAC can incorporate with MISC and helps the agent to better explore the skill space.
Question 2. Can we use learned skills to directly maximize the task reward?
5

---

Under review as a conference paper at ICLR 2020
Reach
Push
Slide
Pick-up
Manipulation skills: Without any reward, MISC learns skills for reaching, pushing, sliding, and picking up an object. The video the learned skills is shown at https: //yutt.. be/ ee/155aaYWWu70 It is time-consuming manually design reward functions that enforce these behaviors.
Figure 3:
We tested method, MISC, in the navigation task, SocialBot-PlayGrom
Soc BecaBooPaayGrurdd
(CM 2.5%%
Maen Task Rlwardd
M4
MeanMM Estimation
0.25
which is based on the Gazebo simulator,
shown in Fig- ure [ In this navigation task, the external task reward is if the agent reaches the ball, otherwise, the task reward We combine our method with
0:15
022
6.10
00005
(Sc.8) 31.11%
(A S1: 20.30%
0.00
0
100
200
300
400
500
100
200
300
400
500
Epoch
Figure 4: Experimental results in the navigation task
PPO (Schulman et al. (017))
and compare the performance with ICM (Pathak et al.) 2017) and Empowerment (Mohamed & Rezende 2015). During training, we only use one of the intrinsic rewards such as MISC, ICM, or Empowerment train the agent. Then, The experimental results are shown in Flgure (left). The y-axis represents the mean task reward and the x-axis denotes the training epochs. From Figure f (left), method, MISC, has the best performance. The Empowerment has the second-best performance. The intuition why MISC, I(S St), ) is superior Empowerment, [(A, St), is that, the mutual information between the object states, S", and the robot states, S is relatively high compared to the mutual information between agent's action and the state of interests St, as shown Figure (right). Subsequently, higher mutual information reward encourages the agent to explore more states, which have high mutual information. theoretical connection between Empowerment and MISC is shown in the Appendix. Furthermore, the ICM method does not enable the agent to navigate the ball because it only seeks novel states instead of controlling these states. A video showing the MISC-trained agent is available at http:////oouuu..eeaaaaaaYtaa
the averaged task reward as the evaluation metric.
that the proposed
Question 3. How does MISC compare DIAYN (Eysenbach al. 2079?
The most related prior work on unsupervised skill learning. DIAYN, introduces an information- theoretic objective FDWYS, which learns diverse discriminable skills indexed by the latent variable

---

Under review as a conference paper at ICLR 2020
FeichPush-v1
FelusiickArdPPaaev1
FeichSlide-v1
DOPG: 9.75%
DCPG:4.668
0.5
DCPG: 27. 27.5%
MOPG+MBCC 55 92%
0.5
DOPG+MBBCH: 25%
DOPGIMISCH 29
DOPG+MSC.pp 08%
DOPG+MISC-A- 21.12%
DOPG+NIBC-A: 27.81%
DSPQ+MISCH 08%
DOPG+MISCH
0.6
0.2
Success Role
0.0
0
10
20
30
10
20
30
40
10
20
30
40
50
FetchPush-v1
FutchPiccArdPPace-r
FetchSide-v1
SAC: 11.65%
SAC: 4,21%
SAC: 19.06%
SAC-MISCIP:
SAC-MIEC-I:
SAC-MISCIR
SAC-MISC-P p: 92.9222220 29%
SAC-MISG-p: 87%
SAC-MISG p:
0.5
SAC-MISCI r. 84.83%
SAG-MISC r. 50.38%
SAC-MIEC 23.0%
0.4
0.2
Epoch
Figure 5: Mean success rate with standard deviation: The percentage values after colon (:) rep- resent the best mean success rate during training. The shaded area describes the standard deviation.
2, mathematically.
FDIAYN [(S; + S) I(A; = I(S;))+++ H(A| S,
(6)
> [==p(())s-==(=)) [oog 90(2 [z] -ogglo](())] H(A S,
The first term, I(S; 2). in the objective, FDIAYN, is implemented via skill discriminator, which is proven to variational lower bound the original objective (Barber & Agakov 2003 Eysenbach et al. 2019). The skill discriminator assigns high rewards to the agent, if it can predict the skill- options. Z. given the states, S. The second term, H(A S, 2), is implemented through SAC (Haarnoja et al. 2018) conditioned skill-options (Szepesvari et al. (014)). In the DIAYN paper, the authors mainly evaluated the algorithm on navigation and locomotion environments (Todoroy al. 2012 Brockman al., 2016). We adapt DIAYN manipulation tasks by replacing the full states, S. with states of interest, S", i.e., the object states, as I (Si; Z). In comparison, our method MISC proposes maximize the mutual information between the context states and the states of interest, i.e., ; S*), so that the agent learns to control the states of interest in self-supervised fashion. These two methods can be combined follows:
FMISC+DIAYN ISS S*) I(S 2) H(AI
We compare MISC, DIAYN and MISC+DIAYN in the pick & place environment. For implementing MISC+DIAYN, we first pre-train the agent with only MISC, and then fine-tune the policy with DIAYN. After pre-training, MISC learns manipulation behaviors such as, reaching, pushing, sliding, and picking up. as shown in Figure 3 Compared to MISC, DIAYN rarely learns to pick up the object. We found that DIAYN does not fully control the object. The MOAYN-trrinne agent mostly pushes or flicks the object with the gripper. However, the combined model, MISC+DIAYN, learns to pick up object and moves it to different locations, which depends on the conditioned skill-option. These observations are shown in the video at https: /ywutuuuuu... be 15kaYJWWy 70?t=47] MISC helps the agent to learn the DIAYN objective. The agent first learns to control the object with MISC, and then discovers diverse manipulation skills with DIAYN. The combination of MISC and DIAYN can be used for learning motion primitives via skill-conditioned policy for hierarchical reinforcement learning (Eysenbach et al. 2019).
4.2 ACCELERATING LEARNING WITH MISC
Question 4. How can we use the learned skills or the trained MI discriminator to accelerate leam- ing new tasks?

---

Under review as a conference paper at ICLR 2020
FeletPPanv
Faldi PidkAndPPPeoww
Faich
N15C4 36 85%
MBC. 37.55%
MISC 20.22%
0.8
DIYYN: 7.48%
0.5
DAYN 3.65%
DIYNN 0.54%
0.4
6.25
0.5
0.2
0.
6.10
0.1
FetchPast-11
FeichhSide-11
10
MISC- 94 DFN
MISC-r 51 51.2%%
MISC- 25 8%
Mean Success Rate
VIME: 10 98%
VIME: 4.70%
VIME 21 77%
0.8
0.3
0.4
0.2
0.1
10
20
FetchPushye ushv1
FetchFRccdrrPoooee1
v1
FonchSiidevv
1/2
MISC 3:c-st:P 93.00%
MIGC p: 12%
MISC y: 27 37.81%
0.8
PER: 58.44%
PERL 8.54%
PER: 20.31%
0.20
0.25
0.8
6.20
0.4
0.10
0.2
10
Epoch
Figure 6: Performance comparison: We compare the MISC variants, including MISC-f, MISC-r, and MISC-p, with DIAYN, VIME, and PER, respectively.
We investigate three ways of using the pre-trained policy and the MI discriminator to accelerate learning in addition to the task reward.
The first method is using the MISC pre-trained policy as parameter initialization and fine-tune the agent with rewards. We denote this variant as "MISC-f", where "-f" stands for fine-tuiing. The second variant is use the MI intrinsic reward to help the agent explore high mutual information states. We name this method as "MISC-r", where "rr" stands for rewarding. The third approach is to use the mutual information quantity from MISC to prioritize trajectories for replay. We name this method as "MISC-p", where "-p" stands for prioritization.
We combined these three variants with DDPG and SAC and tested them in the robotic simulations provided in OpenAl Gym. The environments, including push, pick & place, and slide, have a set of predefined goals, which are represented as the red dots, as shown in Figure RL agent is to manipulate the object to the goal positions. We ran all the methods in each environ- ment with 5 different random seeds and report the mean success rate and the standard deviation, as shown in Figure 3 The percentage values alongside the plots are the best mean success rates during training. Each experiment is carried out with 16 CPU-cores.
The task for the
From Figure 5 we can see that all these three methods, including MISC-f, MISC-p, and MISC-r, accelerate learning in the presence of task rewards. Among these variants, the MISC-r has the best overall improvements. In the push and pick & place tasks, MISC enables the agent to learn in short period of training time, as shown in Figure performances by decent margin.
the slide tasks, MISC-r also improves the
We also compare our methods with more advanced RL methods. To be more specific, we compare MISC-f against parameter initialization using DIAYN (Eysenbach et al. 2019): MISC-p against Prioritized Experience Replay (PER), which uses TD-errors for prioritization (Schaul et al. 2016);
MISC-r versus Variational Information Maximizing Exploration (VIME) (Houthooft et al. 2016
The experimental results are shown in Figure 6, From Figure 6 (1 row), we can see that MISC
enables the agent to learn, while DIAYN does not. The reason that DIAYN is more focused on
8

---

Under review as a conference paper at ICLR 2020
learning diverse set of skills instead of learning control the states. In the manipulation tasks, controlling states is more important, therefore MISC outperforms DIAYN. In the 2nd row of Fig- ure MISC-r performs better than VIME. This result indicates that the mutual information between states a crucial quantity for accelerating learning. The mutual information intrinsic rewards hoost performance significantly compared to VIME. This observation is consistent with the experimental results of MISC-p and PER, as shown in F row), where the mutual information-besn oritization framework performs better than the TD-error based approach, PER. On all tasks, MISC enables the agent to learn the benchmark task more quickly.
pri-
4.3 TRANSFER LEARNING WITH MISC
Question 5. Can the learned MI discriminator be transferred different tasks?
would be beneficial the pre- trained MI discriminator can be trans- ferred to new tasks and still im- proves the performance (Pan et al.) 2010 Bengio 2012). To verify this idea, we directly applied the pre- trained MI discriminator from the pick & place environment the push and slide environments, respectively. We denote this transferred method as "MISC-t", where "-t" stands for transfer. The MISC reward function trained in its corresponding environments is denoted as "MISC- r". We compare the performance of DDPG baseline. MISC-r, and MISC-t. The results are shown in Figure 7 Perhaps surprisingly, the transferred MISC still improves the performance significantly. Furthermore, expected, MISC-r performs better than MISC-t in both tasks. We MI discriminator can be trained in a task-agnostic fashion and later utilized in unseen tasks.
FanesPush-rt
detch
DDPG: 9 75%
0.5
D3PG 5%
Meen Success Rate
DDPG-MISC 94.0%%
DOPG-MSO-
DDPO-NIBC-L
DDPO-MBC4
0.5
0.4
0.2
02
0.1
0.0
30
Epoch
Figure 7: Transferred MISC
that the
4.4 INSIGHTS AND MORE
Question 6. How does MISC distribute rewards across trajectory?
To understand why MISC works and how MISC distributes rewards, we visualize the learned MISC rewards, defined in Section 3.2 in Figure and in video at htttts://youut
be/ L5KaYJNWu702t-33 92 From Figure we can observe that the mutual information reward peaks between the fourth and fifth frame, where the robot quickly picks up the object from the table. Around the peak reward value, the mid- dle range reward values correspond- ing to the relatively slow movement of the object and the gripper, the third, ninth, and tenth frame. When there is no contact between the gripper and the object, see the first two frames in Fig- ure & or the gripper holds the object still, see the sixth to eighth frames, the intrinsic reward remains nearly zero. From this example, we see that MISC distributes positive intrinsic rewards, when the object states, s¹, has correlated changes with the robot state, sf.
MBC-
0.59
0.00
Figure 8: MISC rewards over a trajectory
Question 7. Can MISC help the agent to learn skills when there are no object states? And what happens if there are multiple objects?
In the navigation task, we demonstrate that the agent learns to balance itself and run in a straight line, see video at https: defined to maximize the mutual information between its left wheel and its right wheel.
when the MISC objective is
When there are multiple objects interest, for example red and a blue ball on the ground. We can define the MISC objective follows:
FMSsC I(S); ((S)) ())
(8)
9

---

Under review as a conference paper at ICLR 2020
With this objective, Equation (8), the agent learns to reach both balls and sometimes also learns to use one ball to hit the other ball. The results are shown in the video at 15KaYJwwu707tee
bbe/
From these examples, we can see that, with different pairs of state of interests and the context state, the agent is able to learn different kinds of skills. When there is no specifie state of interests, can train a skill-conditioned policy corresponding to different combinations of the two sets of states and later use the pre-trained policy for the tasks at hand.
5 RELATED WORK
Unsupervised skill learning is a challenging topic in RL. Variational Intrinsic Control (VIC) (Gregor et al., (0166) proposes an
objective (Barber & Agakov 2003) to jointly max- imize the entropy of a set of options while keeping the options distinguishable based on the final states of the trajectory. Recently, [Eysenbach et al. (2019) introduce DIAYN, which maximizes the mutual information between a fixed number of skill-options and the entire states of the trajectory. Eysenbach al.) (2019) show that DIAYN can scale to more complex tasks compared to VIC and provides a handful of low-level primitive skills as the basis for hierarchical reinforcement learning.
Another trunk of work on skill learning is based on goal-conditioned policies. WWrde-Farley et al. (201)) propose, DISCERN, a method to learn a mutual information objective between the states and goals, which enables the agent learn to achieve goals in environments with continuous high- dimensional observation spaces. Based on DISCERN, Pong et al. (219)) introduce Skew-fit, which adapts maximum entropy strategy to sample goals from the replay buffer (Zhao & Tresp 2019, Zhao et al. 2019) in order to make the agent learn more efficiently in the absence of rewards. More recently, Hartikainen et al. (2019) propose to automatically learn dynamical distances: a measure of the expected number of time steps to reach given goal state, which can be used rewards for learning to achieve new goals.
Similar to the skill learning methods, intrinsic rewards are also often used to help the agent learn more efficiently to solve tasks. For example, Dung et al. (2011) and Mohamed & Rezende (2015) use empowerment, which is the mutual information between states and actions, for intrinsically moti- vated agents. A theoretical connection between MISC and empowerment is shown in the Appendix. VIME (Houthooft et al. 2016) and ICM (Pathak al. 2017) use curiosity as intrinsic rewards to encourage the agents to explore the environment more thoroughly.
Based on a similar motivation previous works, we introduce MISC, method that uses the mutual information between the states of interest and the context states (Doersch et al. 2015)) as an intrin- sic reward function. MISC enables the agent learn manipulation skills without supervision. Our method is complementary to the previous skill learning works, such as DIAYN, and can be com- bined with them. The idea of MISC to encourage the agent to learn to control states of interest, as one step forward towards mastery of the environment. Inspired by previous works (Schaul et al.) 2016 Houthooft et al., 2016: Zhao & Tresp 2018 |Esenbach er al. (209), we additionally demon- strate three variants, including MISC-based fine-tuning, rewarding, and prioritizing mechanisms, to accelerate learning in the case when the task rewards are available.
CONCLUSION
This paper introduces Mutual Information -aased State-Control (MISC), a self-supervised reinforce- ment learning framework for attaining useful skills. MISC utilizes a mutual information the- oretic objective to encourage the agent to control states of interest without any reward function. In robotic manipulation tasks, MISC enables the agent to self-learn manipulation behaviors, like reaching, pushing, picking up, and sliding an object. In the navigation task, the MISC-trained agent learss to navigate to the target. Additionally, the unsupervised pre-trained policy and the mutual information discriminator accelerate learning in the presence of task rewards. We evaluate three MISC-base based variants in different tasks and demonstrate a substantial improvement in learning effi- ciency compared to state-of thee-arr methods.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Picter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, 5048-5058, 2017.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. arXiv preprint 0rXiv=1808.00177 2018.
Arindam Banerjee. On bayesian bounds. In Proceedings of the 23rd international conference on Machine learning, pp. 81-88. ACM, 2006.
David Barber and Felix V Agakov. The im algorithm: variational approach information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar. Sherjil Ozair, Yoshua Bengio. Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, pp. 531-540. PMLR, 2018.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML workshop on unsupervised and transfer learning, pp. 17-36, 2012.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 3381-3388 IEEE, 2017.
Carl Doersch, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-4) 1975.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- tions, 2019. URL htttt:::: // wpw.revieeeeeeeeeeeeeeeeee net forum? peprumid==S.ne -xtiixds=sxxxxpeee
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. preprint arXiv:1611.075000 2016.
Variational intrinsic control. arXiv
David Ha and Jürgen Schmidhuber. World models. arXiv preprini 180. 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870, PMLR, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances neural information processing systems, pp. 6765-6774, 2017.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, Sergey Levine. Dynamical distance learning for unsupervised and sumi-supervised skill discovery. arXiv preprint arXiv: arXiv:1907.08225, 2019.
Devon Hjelm, Alex Fedorov, Samuel Marchiidnr Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference on Learning Representations, 2019. URL
https::/ net wttp:/oorummmmdmkk133333

---

Under review as a conference paper at ICLR 2020
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 1109-1117, 2016.
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous agent-envirrommet systems. Adaptive Behavior, 19(1):16-39, 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412v 6980, 2014.
Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135. IEEE, 2005.
Alexander Kraskov, Harald Stöghauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6):066133) 033.06666188666 2004.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-177) 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, In Inter- national Conference on Learning Representations, 2016.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning navigate in complex environments. In International Conference on Learning Representations, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2132 2015.
Andrew Ng. Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Bric Berger, and Eric Liang. Autonomous inverted helicopter flight via reinforcement learning. In Experimental Robotics IX. pp. 363-372. Springer, 2006.
Sinno Jialin Pan, Qiang Yang, et al. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10))1345-1333 2010.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017, 2017.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21 21(4))682-63 2008
Matthias Plappert. Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv: htt.:/802.0466 2018
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint 1rXiiv1903.0300 2019.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment- an introduction. In Guided Self- -ref-iaganizttion: Inception, pp. 67-114. Springer, 2014.
12

---

Under review as a conference paper at ICLR 2020
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000 2017.
Richard Sutton and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT
press Cambridge, 1998.
Csaba Szepesvari, Richard S Sutton, Joseph Modayil, Shalabh Bhatnagar, al. Universal option models. In Advances in Neural Information Processing Systems, pp. 990-998, 2014.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Infelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
David Warde-Fariee Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. ternational Conference Learning Representations, 2019. hwtttttt///peerreeetttt
bett/forum?id=rle/Mee
Rui Zhao and Volker Tresp. Energy-based hindsight experience prioritization. Proceedings of the
2nd Conference Robot Learning, pp. 113-122, 2018.
Rui Zhao and Volker Tresp. Curiosity-drinn experience prioritization via density estimation. arXiv preprint arXiv:1902. 08039, 2019.
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularizze multi-goal reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 7553- 7562. PMLR, 2019.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
13

---

Under review as a conference paper at ICLR 2020
A CONNECTION TO EMPOWERMENT
We divide the states S into the states of interest 5 and the context states 5 In robot manipulation tasks, define the states of interest as the object states, and the context states as the gripper states of the robot. The action space is the change of the gripper position and the status of the gripper, such as open or closed. Note that, the agent's 's actions directly control the context states.
Here, given the assumption that the transform, S = F(A). from the actions, A. to the context states, S is smooth and uniquely invertible mapping (Kraskov et al. 2004), then we can prove that the MISC objective, , S, is equivalent to the empowerment objective, T(4,5*)
The empowerment objective (Klyubin al. 2005 Salge et 2014; Mohamed & Rezende 2015) is defined as the channel capacity in information theory, which means the amount of information contained in the actions A about the states 5, mathematically:
& ((S,A).
Here, we replace the states variable S with sates of interest 8" we have the empowerment objective as follows,
E I(S A).
(10)
Theorem 1. The MISC objective, I(S S*), is equivalent to the empowerment objective, I(A, St), given the assumption that the trunsform, S = F(A), is a smooth and uniquely invertible mapping:
1(S StA
(11)
where St, S", and A denotes the states interest, the context states, and the actions, respectively. Proof.
/ ds ds' pss, s') log
p(se, st) p(ss)
I(S s()
/ ds ds
0A
BA p(a, st) p(a)p(s')
pla, s log
0S
das=
/ / ds ds JA(s))(((a log
(A pp(a,s
(12)
(s()))(()p(()))
= [
sta,
dadss dads'p(a, 5²) log p(a)p(sf)
I(A, St)
14

---

Under review as a conference paper at ICLR 2020
B EXPERIMENTAL DETAILS
The experiments of the robotic manipulation tasks in this paper use the following hyper-parameters:
Actor and critic networks: layers with 256 units each and ReLU non-linearities
Adam optimizer (Kingma Ba 2014 with 10 for training both actor and critic
Buffer size: 10 transitions
Polyak-averaging coefficient: 0.95
Action L2 norm coefficient: 1.0
Observation clipping: 2-000,
Batch size: 256
Rollouts per MPI worker: 2
Number of MPI workers: 16
Cycles per epoch: 50
Batches per cycle: 40
Test rollouts per epoch: 10
Probability random actions: 0.3
Scale of additive Gaussian noise: 0.2
All hyper-parameters are described in greater detail at this link (ttps://githuj com/
mi hLLc:ppciiet//llee/e////attaaaap
15


Reviews:
[{"review_id": "SkxED7_nKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I take issue with the usage of the phrase \"skill discovery\". In prior work (e.g. VIC, DIAYN), this meant learning a skill-conditional policy. Here, there is only a single (unconditioned) policy, and the different \"skills\" come from modifications of the environment -- the number of skills is tied to the number of environments. This is not to say that this way of doing things is wrong, but rather that it is misleading in the context of prior work. Skill discovery in this context implies being able to have a single agent execute a variety of learned skills, rather than having one agent per environment with each environment designed to elicit a specific skill.\n\nRather than \"skill discovery\", I suggest the authors position MISC relative to earlier work on empowerment, wherein a single policy was used to maximize mutual information of the form I(a; s_t | s_{t-1}). Modifying the objective to incorporate domain knowledge (as done in your DIAYN baseline) yields I(a; s_i | s_{t-1}) and is amenable to maximization by either of the lower bounds considered here. Indeed, your DIAYN baseline with skill length set to 1 and the number of skills equal to the number of actions (or same parameterization in the case of continuous actions) should recover this approach. I believe this would be a much more appropriate baseline, and I'd be curious to hear the intuition for why I(s_c ; s_i) should be superior.\n\nApart from this missing baseline, the experimental results seem convincing. However, it is unclear whether or not VIME and PER were modified to incorporate domain knowledge (i.e. s_i/s_c distinction). Indeed, an appendix would be greatly appreciated, as many experimental details were omitted. Ideally, an experimental setup with previously published results (e.g. control suite for DIAYN, Seaquest for DISCERN) would be considered, but I can understand why this wasn't done as incorporating domain knowledge is the main contribution of the paper. That said, the claims should be weakened to reflect this gap, and domain knowledge should be mentioned more prominently (e.g. states of interest vs context are given, not learned).\n\nRebuttal EDIT:\n\nThe language around skills and the extent of prior knowledge still downplays things a bit too much for my liking. Needing new environment variations to obtain new skills is a large step backwards from things like DIAYN (the MISC/DIAYN combination needs more evidence to be considered a possible solution), and the s_i/s_c distinction is non-trivial to specify or learn for harder problems (e.g. pixel observations).\n\nThat said, in the sort of settings under consideration (low dimensional state variables and environmental variations are simple to create) MISC does appear to be superior to prior work. The empowerment baseline is much appreciated, and while modifications of PER and VIME that incorporate prior knowledge would've also been nice, the experimental results pass the bar for acceptance in my view.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJxy9Ndy5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a self-supervised reinforcement learning approach, Mutual Information-based State-Control (MISC), which maximizes the mutual information between the context states (i.e. robot states) and the states of interest (i.e. states of an object to manipulate). For this, they first split the entire state into two mutually exclusive sets of the context states and the states of interest. Then, the neural discriminator is trained to estimate the (lower-bound of) mutual information between the two states. The (mutual-information) intrinsic reward is computed by the trained neural discriminator, which is used for policy pre-training. Experimental results show that MISC helps to improve the performance of DDPG/SAC and the learned discriminator can be transferred to different environments.\n\n\nDetailed comments and questions:\n\n- In the paper, the states are represented by only object positions (x, y, z). Is this sufficient? (e.g. velocity is unnecessary?)\n\n- For MISC, the additional assumption is required: the agent should know that which parts of the states are its own controllable state and object's state respectively. Is this additional assumption realistic enough and has it been adopted in other previous works? Is there any way to discriminate robot states and object states automatically?\n\n- Can MISC deal with the problems where the number of objects of interest is more than two? In this case, how can we define mutual information?\n\n- In Eq. (4), T(x_1:N, y_1:N) is assumed to be decomposable into the sum of T(x_t, y_t) / N. Can this make the lower bound (Eq. (3)) arbitrarily loose since the class of functions becomes very limited?\n\n- Detailed experimental setups are missing. e.g. network architecture, hyper-parameters (e.g. I_tran^max), and how they were searched.\n\n- Similarly to the problem of sparse reward, if the robot and the object are far apart and it is difficult to reach the object with random exploration, it would also be difficult to train the mutual information discriminator. How was the discriminator trained? How many time steps were used to train MI discriminator?\n\n- It seems that the MI discriminator learns to estimate the 'proximity' between the robot and the object. Compared to using just a very simple dense reward (e.g. negative L2 distance between the robot and the object), what would be the advantage of using MI discriminator? It would be great to show the comparison between using simple dense reward and MI discriminator for each Push, Pick&Place, and Slide task.\n\n- For the MISC+DIAYN, what if we train the agent using MISC and DIAYN at the same time, instead of pre-training MISC first and fine-tuning DIAYN later?\n\n- It is unclear how MISC-p is performed. Please elaborate on how MISC-p works for prioritization.\n\n- Also, for MISC-r experiments, the weights between the intrinsic reward bonus and the extrinsic reward are not specified in the paper.\n\n- It seems that MISC is beneficial when the robot should get closer to the object for the success of the task. Then, how about the opposite situation? What if the task requires that the robot should 'avoid' the object of interest? Does MISC still work? Is it helpful for the improvement of sample efficiency?\n\n- In order to pre-train the discriminator network, additional (s,a,s') experiences are required, thus it seems difficult to say that it is better for exploration than VIME.\n\n- In section 4.3, what happens if we transfer the learned discriminator to Pick&Place from Push that has a gripper fixed to be closed, rather than the opposite direction (i.e. from Pick&Place to Push)? Does the MISC-t still well work? Can the learned MI discriminator be transferred to different tasks even when the state space is different?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkgcm01WqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper paper proposes a mutual information maximization objective for discovering unsupervised robotic manipulation skills. The paper assumes that the state space can be divided into two parts - the state of the robot (“context states”) which is controllable via actions and the state of an object (“states of interest”) which must be manipulated by the robot. Given these two categories of states, the proposed algorithm maximizes a lower bound on the mutual information between the two categories of states such that a policy is learnt that is able to manipulate the object with the robot meaningfully.\n\nI vote for weak reject as (1) the paper makes a strong assumption about the availability of both the robot and object states which is not realistic in typical robotic manipulation applications and (2) the objective in the paper will not work if there is no notion of an “object” or object-state e.g.: this algorithm will not learn skills for a robot trying to control itself; hence, it is not truly a general purpose skill discovery algorithm but rather a skill discovery algorithm specifically meant for robot-object manipulation tasks.\n\nMy main concern with the paper is its limited applicability to robotic manipulation tasks with a clear divide between states of interest vs others. The paper does not talk about settings where states of interest are not known, so all of the experiments are based on this strong assumption. It doesn’t seem like a surprising discovery that maximizing the mutual information between the robot state and object state will lead to skills that actually make the robot move the object.\n\nGiven that object manipulation is the specific application of interest, the comparison with DIAYN and the combined objective with DIAYN is interesting but little motivation or discussion has been provided in the paper. Can the authors elaborate on why this choice should intuitively be better than the proposed method alone?\n\nThe paper does not talk about how these skills can be used as primitive actions by a higher level controller (in a hierarchical RL setup), which would help in demonstrating the usefulness of these skills - e.g.: are these skills sequentially composable such that they can solve a complex task?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers a setting where the state of a (robotics) environment can be divided roughly into "context states" (such as variables under the robot's direct control) and "states of interest" (such as the state variables of an object to be manipulated), and learn skills by maximizing a lower bound on the mutual information between these two components of the state. Experimental results compare to DDPG/SAC, and show that the learned discriminator is somewhat transferable between environments.

Reviewers found the assumptions necessary on the degree of domain knowledge to be quite strong and domain-specific, and that even after revision, the authors were understating the degree to which this was necessary. The paper did improve based on reviewer feedback, and while R3 was more convinced by the follow-up experiments (though remarked that requiring environment variations to obtain new skills was a "significant step backward from things like [Diversity is All You Need]"), the other reviewers remained unconvinced regarding domain knowledge and in particular how it interacts with the scalability of the proposed method to complex environments/robots.

Given the reviewers' concerns regarding applicability and scalability, I recommend rejection in its present form. A future revision may be able to more convincingly demonstrate that limitations based on domain knowledge are less significant than they appear.
Decision: Reject