Title: Beyond Games: Bringing Exploration to Robots in Real-world
Authors: Deepak Pathak
Dhiraj Gandhi
Abhinav Gupta
Venue: ICLR 2019
Keywords: Exploration
curiosity
manipulation

Abstract: Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control. While there has been major advances over the years, most of these successes have been demonstrated in either video games or simulation environments. This is primarily because the rewards (even the intrinsic ones) are non-differentiable since they are function of the environment (which is a black-box). In this paper, we focus on the policy optimization aspect of the intrinsic reward function. Specifically, by using a local approximation, we formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization -- much like supervised learning instead of reinforcement learning. This leads to a significantly sample efficient exploration policy. Our experiments clearly show that our approach outperforms both on-policy and off-policy optimization approaches like REINFORCE and DQN respectively. But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module. See project videos at https://doubleblindICLR.github.io/robot-exploration/


Review discussion:
This paper presents an interesting way to reformulate intrinsic curiosity as a differentiable function. The authors compare the differentiable function against using prediction error via REINFORCE and DQN, showing that their intrinsic curiosity method results in more interactions with unseen objects than the other two methods. For DQN this is to be expected, but it shows that backprop through this function is more efficient than reinforce in getting to unseen state spaces. I think this is an interesting method/proposal and is a somewhat novel reformulation of intrinsic error, but I do have some concerns in comparisons/claims.
In the introduction, the authors say that the intrinsic curiosity method proposed by Pathak et al. is sample inefficient and isn’t tested in robots.
However, to my understanding the REINFORCE baseline isn’t really equivalent (though it may be possible that it is, it was unclear how exactly the loss was formulated in the baseline, did include the other components from Pathak et al.?). If the claim is that this method is more efficient, I think it should have compared against that method directly.
Moreover, I think the description of the experiments doesn’t provide enough information.
For example, the method says that different learning rates were used for the min-max game to stabilize it, but doesn’t say what they were.
Also, for the DQN baseline what were the parameters? Was there an epsilon greedy policy on top of the exploration reward? Was this annealed as in other work? Generally, I think more detail is needed throughout (even if it just refers to a more detailed appendix).
Overall, I think this work needs to be revised to include more details on hyperaparameters, details on the baselines, and describing differences between Pathak et al.’s method and the REINFORCE baseline.
Moreover, feedback from other comments on this work should be addressed which reflect in more detail my comments below on opinionated claims (e.g., https://openreview.net/forum?id=SkzeJ3A9F7&noteId=HJlFlZOa2X )
Comments/Thoughts:
+ I think in the introduction there are some statements that probably need citations.
For example, “But the same formulation from an optimization viewpoint, it suffers from all the bad properties of extrinsic rewards. The reward is a function of environment behavior with respect to the performed action. Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> Why is this true? Is there a citation that can back this? Do you prove it later in the paper?
+ “Yes, 54 environments but no real-world physical robots” —> this and the intro seems like a blogpost at times. That can be fine (some would argue it’s a good thing), but there seem to be some opinions without citations/backing, I suggest trying to back up statements wherever possible and avoid opinions.
For example in this statement, robots aren't a requirement for evaluating intrinsic motivation.
+ “Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> citation/backing? it might be nice to point to the experiment section here to back it (e.g., "As will be shown in Section X and in \citet{something}, REINFORCE can be quite sample inefficient")
+ “In practice, the existing on-policy algorithms, e.g., A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) etc. are deployed off-the shelf -> This is confusing, so is this using REINFORCE or PPO/A3C? what is this statement referring to?
+ “regress to rti to learn value estimates (i.e., off-policy) as discussed in the previous section” —> regress to \sum r_t{I} for a value estimate?? Value is the expected return so not sure if this is a typo or i missed something earlier
+ What is the actual loss function used for the baseline? Is it the same as Pathak et al.?
+ What are the hyper parameters for DQN exploration? What are all the hyper parameters for any/all the algorithms?
+ Was a variance-reducing baseline used in REINFORCE?
+ What is the variance representing in the graphs, std across several trials? Maybe I missed it, but how many trials represent this standard deviation?
+ “Hence, we train the forward predictor slightly faster than the policy by keeping higher learning rate to stabilize the learning process. “ —> what were the learning rates?
Linguistic/Typos:
Also, some minor, but frequent, grammatical issues/typos that I’ve added below could be fixed. I would ask that the authors please have the submission proof-read for English style and grammar issues. There are many minor mistakes, some of which I’ve tried to point out below.
+ “This leads to a significantly sample efficient exploration policy. “ —> significantly more (?) sample efficient ?
“Why is that? To understand the reason behind sample inefficiency of curiosity or intrinsic rewards, notice how the intrinsic rewards are given by agent” —> by the agent?
“Forward model fθF is trained to minimize its loss which amounts to minimizing rti with respect to θF” —> the forward model
“However, policy is optimized to maximize the objective” —> However, the policy
“We can also optimize  for policy parameters θP via differentiable loss function” —> We can also optimize for (the) policy parameters \theta via (a) differentiable loss function?
“To optimize policy to maximize a discounted sum “ —> To optimize the policy
“How good is Forward Prediction Model” —> How good is the forward prediction model
There are several other spots, but basically another pass over the paper might be worth it to check for these sorts of issues.
We would like to first thank the reviewer for the comments. Please see the meta response in the combined reply for the common concerns among reviewers. The reviewer found the formulation to be "interesting" and "novel" but has raised concerns about lack of details about the experiment hyper-parameters. We answer them below:
[R2]: REINFORCE baseline vs. Pathak et. al.? Is it the same as Pathak et al.?
=> Please refer to the combined response to all reviewers above. It is indeed the same.
[R2]: "different learning rates ..
but doesn't say what they were"
[R2]: "What were DQN parameters? Epsilon greedy"
[R2]: "include more details on hyper-parameters ..."
=> Yes, in our case, we can simply train the forward model faster than policy to stabilize the training. The learning rate for the model was 5e-4 and policy 1e-4 with an entropy loss coefficient of 1e-4. The optimizer used was Adam.
Yes, we used standard DQN algorithm with prioritized experience replay [Schaul et. al. 2016] and epsilon-greedy to maximize the curiosity objective [Pathak et.al. 2017]. Q-value was trained with a learning rate of 1e-4. In epsilon-greedy, epsilon was initialized at 0.5 then annealed over training to 0.1.
We agree that these details are important for the reproducibility of work, and we will include them in supplementary -- we apologize that we missed adding them in the supplementary. That being said, we will make our code, environment, and models public upon release.
[R2]: "Feedback from the other commentator should be addressed"
=> We have answered both the factual and stylistic comments with proper citations in our reply.
[R2]: "... This is confusing, so is this using REINFORCE or PPO/A3C? .."
[R2]: "it might be nice to point to the experiment section here to back"
=> With due respect, we believe reviewer has misunderstood the background here. REINFORCE is an operator that was proposed in (Williams, 1992), and then became the main ingredient of policy gradient algorithms. All policy gradient algorithms, whether PPO or A3C, use reinforce operator to update the policy. PPO, in addition to reward maximization, penalizes KL-divergence change while A3C relies on regularization from multiple workers, but both these algorithms use reinforce at the core of it.
[Schulman et.al. 2016] provides a nice discussion on how REINFORCE helps to optimize any black-box function in expectation using only samples from the function. More detailed and fundamentals of policy gradients are described in [Sutton & Barto, 1998]. We implemented the policy-gradient algorithm as optimizing the curiosity objective in [Pathak et.al. 2017] as a baseline which we call "REINFORCE". We will update the legend of graphs to say [Pathak et.al. 2017] in future to avoid confusion.
[R2]: "regress to rti .. discussed in the previous section ..
so not sure if this is a typo"
=> Thanks, it should be indeed summed.
However, note that in the previous section it was mentioned correctly. We will correct this typo.
[R2]: "Some minor, but frequent, grammatical issues/typos .."
=> Thank you. We will fix all these writing typos and make sure there no other ones.
"With due respect, we believe reviewer has misunderstood the background here. REINFORCE is an operator that was proposed in (Williams, 1992), and then became the main ingredient of policy gradient algorithms. All policy gradient algorithms, whether PPO or A3C, use reinforce operator to update the policy. PPO, in addition to reward maximization, penalizes KL-divergence change while A3C relies on regularization from multiple workers, but both these algorithms use reinforce at the core of it."
--> Sorry if I was not clear here. The difference between REINFORCE and PPO is self-evident, and that was not my point. I was trying to suggest that the language highlighted made it unclear as to why PPO is mentioned if the REINFORCE baseline is under test (e.g., the language suggests that some aspect of PPO/A3C is relevant here, but that doesn't seem to be the case since PPO nor A3C are compared against).
"We agree that these details are important for the reproducibility of work, and we will include them in supplementary -- we apologize that we missed adding them in the supplementary. That being said, we will make our code, environment, and models public upon release." --> I appreciate the addition of these important details and look forward to seeing them in revisions.
While the new revision is significantly more refined, I still don't think there is enough information about the experiments provided in the main text to judge the contributions properly in their current form (for example much of the information that was asked about above, whether REINFORCE in this paper uses a variance reduction baseline, learning rates, etc.
still remain missing).
Moreover, I agree with the concerns that other reviewers raised regarding the baselines.
For example, while it is understandable that the authors compare against optimizing this curiosity reward through REINFORCE vs. their method, it is unclear whether the REINFORCE method used here makes use of variance reduction methods which are known to boost performance significantly.
However, I have updated my original review rating because I believe this current draft is much more clear than the prior version.
We thank you for your earlier feedback and are grateful for your follow-up response. We would like to clarify the subtle misunderstanding here. To keep the changes minimal for the reviewers in our updated draft, we only included new experiments and analysis in the last two pages of the supplementary in the paper, which were not possible to include within the text of rebuttal responses. We are yet to include the information and details provided as part of our rebuttal responses in the main paper.
For instance, we described the learning rate, optimizer, epsilon-greedy, experience replay details etc. in our first response to your main review (above) and will include them in the final draft of the paper, posting here again just for reference:
"The learning rate for the model was 5e-4 and policy 1e-4 with an entropy loss coefficient of 1e-4. The optimizer used was Adam. We used standard DQN algorithm with prioritized experience replay [Schaul et. al. 2016] and epsilon-greedy to maximize the curiosity objective [Pathak et.al. 2017]. Q-value was trained with a learning rate of 1e-4. In epsilon-greedy, epsilon was initialized at 0.5 then annealed over training to 0.1."
Variance Reduction: Yes. Since our robotic pushing environment is not long term, the value-function baseline reduces to a simple constant-value in our scenario, which we used for variance reduction. It turned out to be sufficient to make the REINFORCE algorithm work efficiently in this environment using external rewards. We will add this to the paper. Please refer to the result in the Appendix Figure-4, which shows that our REINFORCE baseline indeed works well with an external reward.
In order to further demonstrate that our baselines are credible, we took the public implementation from the recent large-scale curiosity paper [Burda et. al. 2018]. We use the curiosity baseline provided in the paper and to demonstrate our objective function for exploration is meaningful, we add our objective to the original exploration objective. This comparison uses the official implementation of PPO with all the necessary state-of-the-art techniques for variance reduction. Please see results in Appendix Section-C of the updated paper.
Thank you again for the review. We look forward to the follow-up comments.
Summary:
This paper proposes a novel differentiable approximation to the curiosity reward by Pathak et al. that allows a learning agent to optimize a policy for greedy exploration directly by supervised learning, rather than RL.
The authors motivate this work with arguments about the sample-efficiency required by real robot learning, and demonstrate basic results using a real robot.
Comments:
The paper has serious style and tone issues that must be addressed before publication. The rest of my review will focus solely on the technical details.
The experimental details are lacking (learning rates? rollout lengths for REINFORCE? what are the inner and outer loops and what are their sizes? what are the plots measuring - extrinsic reward? intrinsic reward? what is "multi-step learning" in Table 2?).
Without these details, the results will be difficult to validate and reproduce independently.
The approach is compared only against very weak baselines. Why vanilla REINFORCE and not any of the modern policy-gradient algorithms (A3C, PPO, TRPO, DDPG, ...)? The ability to deal with this large action space is certainly impressive, but it is likely far too large for DQN or REINFORCE to work, so the comparison is questionable to begin with: you can make any algorithm fail if you give it an unnecessarily difficult interface. Did the authors try a smaller, more traditional action space?
The authors claim the sample-efficiency improvements by many existing exploration approaches are insignificant. In what way are the results in this paper more significant? Table 1 shows very minor improvements to a MPC planning task, Appendix Figure 1 shows barely any improvement over the baseline, and Appendix Figure 4 shows that learning from extrinsic rewards using REINFORCE seems to work just fine. Why use intrinsic rewards at all in this case? It appears that maybe some of the results look significant because the baselines are so weak.
The paper contains many factual errors and unsupported claims.
For example:
- "the field of RL was born out of need to make our robots learn"
- "none of the recent advances have translated to success in the field of robotics" (see e.g. the proceedings of CoRL 2017 and 2018)
- "Building a good model will require enormous number of interactions" (see e.g. PILCO)
- "[our approach enables us to] for the first time ever, implement exploration on a real-world physical robot" (PILCO and many others)
- describing Pathak et al. curiosity as a "gaussian density model" in eq1; it's a deterministic forward model
- in sec3, "regress r^i_t to learn value estimates", this is probably meant to be the discounted sum of rewards
- also sec3, "[REINFORCE] gives no signal as to what action to take"; the signal has high variance but it works (see all policy gradient work)
These errors can be easily corrected.
However, the contribution of the paper is based on a more serious error:
- sec3.1, "If the policy could be optimized using direct gradients, the rewarder could ... inform the agent to change its action space in the direction where forward prediction loss is high."
This is incorrect. The paper is based on using the gradient of the forward model to directly optimize the policy to produce higher prediction errors, as in Pathak et al.
But in order to make the prediction error differentiable, it makes the severe assumption that the next state x_{t+1} is constant and does not depend on a_t, which is false and invalidates the idea of optimizing actions for prediction error.
As a result, the gradient obtained does not actually move the policy toward higher prediction errors.
To understand what the author's approximation actually does, consider a perfect forward model. No matter what actions the policy produces, the prediction error is always zero, but the authors' gradient is not.
So it can't be optimizing for higher prediction errors.
Instead of optimizing for high prediction errors as the authors claim, the policy is being optimized for state transitions that are maximally different from the observed state x_{t+1}.
This is an interesting objective to optimize. I can see how it could result in interesting exploration.
But it's not what the authors say they're proposing.
It's much more like a count-based exploration strategy, which prefers visiting states that are maximally different from the states visited so far. It is much less like the prediction-error based curiosity of Pathak et al. that the authors are motivated by.
I would like to see focused analysis of this particular objective.
For example, would this not result in the policy oscillating between different parts of the state space, since it's only optimizing for maximal difference to what it just saw, rather than long-term knowledge gain? This issue requires more discussion.
Finally, the approach is not really robot-specific despite the title and arguments in the paper. I recommend pursuing a more general investigation, because if this objective is truly as effective as the authors believe, then it should be applicable in a wide variety of domains (many of which are very easy to evaluate in: Atari, OpenAI Gym, DMLab, VizDoom, Mujoco, etc.).
Conclusion:
The paper proposes an interesting new objective, but it is motivated by a very naive approximation that completely changes the behavior of the exploration compared to what the authors want to approximate. The idea is novel and worth exploring, but the paper should be heavily rewritten to emphasize what the authors are actually doing with this new objective, and should include thorough analysis of its behavior, before I can recommend acceptance.
We would like to first thank the reviewer for the comments. These comments
[R3]: "The experimental details are lacking"
=> We agree that these details are important for the reproducibility of work, and we will include them in supplementary -- we apologize that we missed adding them in the supplementary. That being said, we will make our code, environment, and models public upon release.
The learning rate for the model was 5e-4 and policy 1e-4 with an entropy loss coefficient of 1e-4. The optimizer used was Adam. We used standard DQN algorithm with prioritized experience replay [Schaul et. al. 2016] and epsilon-greedy to maximize the curiosity objective [Pathak et.al. 2017]. Q-value was trained with a learning rate of 1e-4. In epsilon-greedy, epsilon was initialized at 0.5 then annealed over training to 0.1.
[R3]: What are the plots measuring - extrinsic reward? intrinsic reward?
=> The plots are measuring the how frequently the robotic arm touches the object which is a proxy for measuring how good the exploration is. Note that this reward was not used for training the policy, but just used to measure the quality of exploration at training. The robot was trained with an intrinsic reward only.
[R3]: "The approach is compared only against very weak baselines."
[R3]: "I recommend pursuing a more general investigation ..to evaluate in: Atari"
=> Inspired by reviewer's suggestion, we have added a comparison on games in the supplementary using the state-of-the-art implementation proposed couple of months ago in (Burda et. al. 2018) [2]. Please see the combined response above for more details.
Our paper is just a first step towards explicitly using the structure of the prediction model in training the policy in a supervised manner. It relies on the model being good which is true for short-horizon tasks, but learning good long-term models is still an active research area [Finn et.al., 2017, Ebert et.al. 2017].
Hence, our approach might provide diminishing gains for long-horizon tasks.
[R3]: "this large action space is certainly impressive, but it is likely far too large for DQN or REINFORCE"
=> Our fundamental goal is to test exploration algorithms on real robotics setup where it is crucial to experiment in presence of large input space as well as large action space. We believe working with larger action space should be seen as a positive feature, and not something made to fail baselines.
However, upon the reviewer's suggestion, we have added a comparison on games in the supplementary. Please see the combined response above for more details.
[R3]: "Table 1 shows very minor improvements to a MPC planning task"
=> As discussed in our combined response above, the baselines compared to in paper are same as the curiosity-driven exploration and our method achieves absolute 10% improvement in the mean accuracy for multi-object manipulation -- which we believe is a significant improvement.
[R3]: “Appendix Figure 4 shows that learning from extrinsic rewards using REINFORCE... the results look significant because the baselines are so weak."
=> Please note that the external reward curve shown in Figure-4 is just to show that the REINFORCE baseline works in our setting. The reward, in that case, was touching the objects, while our single exploration policy learns diverse behavior without rewards learns to push, pull, grasp objects allowing us to collect useful data for training the planning models. A more detailed discussion on the usage of extrinsic and intrinsic rewards is presented in (Burda et. al. 2018) [2].
=> We have now added comparisons to state-of-the-art off-the-shelf baselines on Atari games.
[R3]: "Building a good model require enormous number .." (see e.g. PILCO).
[R3]: " .. implement exploration on physical robot" (PILCO and many others)
=> These are great advances and strong motivation for our work. We already discussed them in our method section (see Section-3.1, first para). Note that these are not reinforcement learning methods and based classical optimal control using learned models.
However, those models are dynamics models in low-state-space of the agent learned using Gaussian processes and not from high dimensional raw image input. Even using deep networks, these methods have trouble generalizing to raw images as input (e.g. Deep PILCO [Gal et.al. 2016]) and rely on the learned model is quite good.
Moreover, PILCO is not an exploration approach but learned for an end-task with well-defined cost objective.
[R3]: "To understand what the author's approximation actually does, consider a perfect forward model … is always zero.. maximally different from the observed state"
=> If the prediction error is always zero, the action gradients will also be 0. The interesting scenario happens, when the error is low but not zero, in which objective encourages action that leads to states different from x_{t+1}. We have added a detailed clarification in Appendix Section-D.
Further, we re-emphasize that our local constancy approximation for x_{t+1} is only applied in a very local neighborhood of the executed action. It is only used for computing *local* gradients and nowhere else. This means when a completely different action a_t_new is applied at x_t, we will consider the new next state corresponding to a_t_new for computing gradients at a_t_new. One interesting consequence of our objective is that it has to be trained in an on-policy manner as off-policy violates the local-constancy approximation.
[R3]: "It's much more like a count-based exploration … much less like the prediction-error based"
=> We agree with the reviewer that our is not exactly the same as Pathak et.al., and we did not claim this as well.
However, we respectfully disagree that this objective is more-like a count-based strategy. This is so because our constant state approximation is only used in the small neighborhood of the executed action.
Thank you for bringing this point up. Inspired by this, we have added a detailed discussion in Appendix Section-D on understanding the behavior of objective.
[R3]: The paper has serious style and tone issues that must be addressed before publication.
=> We will update and soften the introduction of the paper as discussed in the combined response above.
Thank you for the clarification of the experimental details, and for the extra discussion of the new objective.
Unfortunately I'm still skeptical about the precise behavior of the objective: if you're always pushing the policy toward an outcome that's as different as possible than the previous outcome, does this result in non-convergent oscillation in policy space? In Pathak et. al the predictive model should eventually converge, so the intrinsic rewards should eventually go to zero once exploration is complete. For this proposed objective, I don't see a reason it should converge because there's always a direction to push the policy toward different outcomes, and the process has no memory mechanism to avoid repeating previous exploration. This requires investigation, or at least an explanation of why this is not a problem.
Once again, I think it could be an interesting objective to study, but I'd like to see a version of this paper that is more focused on analysis of this objective as an exploration strategy and less about the specific application of robotics, which seems unrelated to the exploration angle. I appreciate the experiments on the Atari games, but it looks like the improvements over the baseline are minor (with high variance) and the BeamRider results in the Burda et. al paper achieve roughly twice the score that you report from your baseline, which makes me wonder whether the performance reported here is representative.
As before I think there is some potential here; the idea of turning an exploration procedure from RL into a supervised task is interesting. At this point I still have concerns about convergence and oscillation, so my decision remains.
Thank you for the reply. We clarify the concerns here:
R3: "the BeamRider results in the Burda et al. paper achieve roughly twice the score that you report from your baseline, which makes me wonder whether the performance reported here is representative."
=> Curiosity results based on random features are the most stable and best-performing method in Burda et.al. on Atari.
Hence, we used random features for both our method as well as the Burda et.al. in Figure 7. The results indeed *exactly* matches the curve in Burda et.al. Please compare: orange curve in Figure 7 of our paper vs. green curve in Figure 2 of Burda et.al. -- they have the same performance.
=> We think the reviewer has mistakenly compared VAE to random-feature based curiosity results. It turns out the VAE features worked the better only in the BeamRider game and the simplest/best-performing method in Burda et al. is based on random-features which we used. We will try our method on top of VAE features as well.
R3: "If you're always pushing the policy toward an outcome that's as different as possible than the previous outcome"
=> We would like to emphasize that our objective does not *exactly* push the policy toward an outcome that's different. This is so because our constancy assumption only applies to compute gradient in only a very local region around the exected action. The reviewer would be correct if we were to make this assumption at all points, but we only use it while computing gradient only at the point when action is executed (i.e., pi(s_t)=a_t and a_t is applied to obtain the corresponding s_{t+1}) in an on-policy manner.
=> When applied locally, this assumption results into an interesting behavior which behaves like count-based *locally*, and prediction-based *globally*. This results into moving the agent's policy towards the actions which would result in a different outcome locally (like count-based because s_{t+1} is fixed) and towards the actions where prediction model is wrong globally (because we also update s_{t+1} using the new transitions obtained by updated agent's policy). Please see our example-2 in section-D for an illustration of how this idea especially plays out well in robotic manipulation settings.
"We think the reviewer has mistakenly compared VAE to random-feature based curiosity results"
My mistake. Thank you for the clarification on this, I was indeed looking at the best curve, rather than the random features.
However, given that the VAE features performed best on this game, I think it would be reasonable to compare to those instead. Or if you would like to compare to the random features, I think it would be fair to compare on another game where the random features do well (for example, what about Breakout, Pong, Montezuma, Mario, Qbert, Seaquest, Space Invaders, do you have results for those?).
"We would like to emphasize that our objective does not *exactly* push the policy toward an outcome that's different...
When applied locally, this assumption results into an interesting behavior which behaves like count-based *locally*, and prediction-based *globally*."
I don't understand the explanation for why this is the case. The gradient used to update the policy does exactly push in the direction that maximizes the difference in outcome. In what way does taking a difference-maximizing step imply a global trend toward maximizing prediction error?
I don't see the necessary connection between achieving Euclidean-distant outcomes and achieving outcomes that are hard to predict; I think that would need to be established rigorously in order to make the claim that's being made here.
Thank for the follow-up comments.
R3: "I don't see the necessary connection between achieving Euclidean-distant outcomes and achieving outcomes that are hard to predict; I think that would need to be established rigorously"
=> We describe our intrinsic reward r_t mathematically as follows:
r_t = || f(x_t, \pi(x_t)) - y_t ||
where
if \pi(x_t) == a_t:
y_t is set to x_{t+1}, i.e., T(x_t, a_t)  -- kept constant around a_t (local)
else \pi(x_t) != a_t:
y_t = T(x_t, \pi(x_t))             -- updated to the true next state (global)
- f is learned forward model
- T is true environment dynamics (unknown)
- \pi is the agent's policy
- {x_t, a_t, x_t+1}: a real transition executed in the environment
=> Here, only the first step of gradient enforces euclidean-distant outcome to y_t (i.e., count-based 'locally').
But as soon as the first gradient step is taken, the policy gets updated, and hence, \pi(x_t) != a_t.
Therefore, the target y_t will also get updated to the new ground truth next state, as shown in the else condition above.
=> Hence, next gradient step will not try to maximize the difference from original y_t but with the new next state obtained by T(x_t, \pi(x_t)). This implies that, in order to maximize the objective r_t in the global sense, the agent policy should take actions where f(x_t, a_t) will be far from *corresponding* ground truth x_t+1 -- which is same as prediction error being high.
=> We discuss the potential benefit of this simple *local* trick in Example-2 of Section-D in the appendix.
We again emphasize that we would rework the introduction and have shown results in simulation as well as real robot exploration in a high-dimensional input (images) and high-dimensional action space setup. Hopefully, the reviewer can take these points into consideration.
Beyond Games: Bringing Exploration to Robots in Real-world
This paper tackles the laudable goal of making an algorithm for efficient exploration in "real-world" RL.
To do this, they augment the "curiosity" algorithm of Pathak et al with a differentiable approximation to the reward prediction model.
They motivate this algorithm through several intuitive arguments together with a series of experiments where the algorithm outperforms vanilla DQN/REINFORCE.
There are several things to like about this paper:
- The problem of making "real-world" practical algorithms for exploration is clearly one of the biggest outstanding problems in reinforcement learning.
- The authors have sucessfully gone from ideas, to algorithm, to real robot and their algorithm really seems to outperform the baselines.
- The authors clearly make an effort to survey a wide variety of recent papers in the field
However, there are several important places where this paper falls down:
- In a paper that posits a new, groundbreaking, real-world application of "exploration" there is remarkably little discussion of the key issues of "efficient exploration".
Indeed, I don't think that this paper even presents a clear metric for how we can tell if something *is* a good method for exploration.
+ This is a huge shortcoming, since we know that it is possible to guarantee polynomial regret bounds for many settings (mostly tabular, but some with function approximation too... see UCRL2, PSRL and more)... there is no discussion of whether the proposed algorithm would also satisfy these bounds?
+ Of course, this is not a paper designed for "tabular MDPs", but we already have exploration algorithms like UCB / Thompson sampling that *are* widely used in online advertising...
so why is this method not compared/contrasted to these approaches?
- There is very little *science* in this paper, beyond the experiments pitting "improved algorithm" vs DQN/REINFORCE, which nobody ever claimed would be a good approach to exploration! I don't think it's possible to assess if their algorithm (which I don't think has a clear name beyond "sample-efficient exploration formulation") performs better than the myriad of other exploration approaches listed.
Although many intuitive arguments are presented, I did not find these convincing, and the overall narrative ends up being a little jumbled.
- A lot of the writing is generally imprecise, and alludes to claims/statements that make no sense to me:
+ "... most of these sucesses have been demonstrated in either video games or simulation environments. This is primarily becuase the rewards (even the intrinsic ones) are non-differentiable ..."
+ "Again these approaches have mostly been considered in context of external rewards and hence turn out to be sample inefficient"
I would suggest that each statement/claim is backed up by some material reasoning/statement/experiment unless extremely obvious - at the moment these are not!
- Nothing in this algorithm really seems specific to real-world... or at least nothing in the competing algorithms seems to preclude them from being run on a real-world robot... I think that the main issue is that if people want to iterate fast (or don't have a robot) they prefer to do things in simulation. If your point is really that findings from simulation don't translate to real robots, then I think that is really interesting, but I don't see any evidence for that in this paper.
Overall, it is clear that this is an interesting area to do work in.
The goal of making a practical algorithm for real-world exploration tasks is exciting.
However, in its current form, this paper falls well short of the level of science and insight I would expect for ICLR.
We would like to first thank the reviewer for the comments. Please see the meta response in the combined reply for the common concerns among reviewers. We answer other individual concerns below:
[R1]: Only comparison to "DQN/REINFORCE, which nobody ever claimed would be a good approach to exploration!"
=> We respectfully disagree with the reviewer as there seems to be some misunderstanding. Here, DQN/REINFORCE do not refer to the vanilla RL algorithm but are optimizing the curiosity exploration objective proposed in Pathak et.al. [2017]. Please see the combined response above for details. We hope that this answers the reviewer's major concern about the comparison to other exploration methods.
[R1]: Comparison to UCB / Thompson sampling; UCRL2, PSRL.
=> As the reviewer agrees, most of these algorithms have been primarily studied in tabular cases and there have been some extensions to function approximators. These algorithms do not scale to the high dimensional image input (320x320x4) and a high dimensional action space (location, angle, push, grasp) used in our paper.
Further, we respectfully believe there is a misunderstanding about the context in which the term "exploration" has been used in this paper. Our robotic agent explores the environment completely out of its own; using only its intrinsic reward and no external reward at all. UCB-like algorithms are parameter-space exploration and are successful in learning a policy in presence of external reward (even if sparse). One can not use UCB when there is no external reward as the value functions would collapse.
The algorithms UCRL2/PSRL integrate the prior over dynamics or reward function into learning a policy in a natural and efficient manner. These have been great advancement but not relevant to the problem setup or the goal of this paper because our agent's start completely from scratch without any prior, and has access to no external rewards.
As far as bounds are concerned, our algorithm draws a similarity to Experiment-Design literature from optimization. In a linear case and explicit policy, optimizing our objective is the same as finding a max-variance action which is an efficient way to discover the space satisfied by the model.
While this is out of the scope of the paper, we have mentioned the pointers in Appendix Section-D.1 and elaborate more to spur discussion for future work to build upon.
[R1]: "how we can tell if something *is* a good method for exploration".
=> It is difficult to measure exploration as an end in itself. There are several domain-specific metrics can be created that could act as a useful proxy to measure the quality of exploration.
For instance, we measure how frequently the robot touches the object kept on the table purely out its exploration, and found out that this frequency increases to almost perfect as our training proceeds (please see the video on the website link provided).
Another proxy to measure exploration is to see if a purely exploration-driven learning without using any external reward can help the agent obtain an external reward, as discussed in (Burda et. al. 2018) [2].
However, this latter proxy is only applicable to games-like environments.
In this paper, we argue that the true measure of exploration is how well it is able to contribute to learning planning models later. Using the data collected by the exploration, we build models and report the success of those models for grasping, pushing etc. If the robot explored well during exploration, then it should have gathered good data for learning planning models for manipulation. This discussion is present in Section-5.1, 5.2 of paper. We will clarify it further.
[R1]: "Nothing ... seems specific to real-world... or at least nothing in the competing algorithms seems to preclude them from being run on a real-world robot…"
=> The REINFORCE comparison is indeed existing curiosity-driven exploration from Pathak et.al. [2017] and being run on the real robot (please refer to meta combined response above for details)
We further provide a comparison for long-term horizon dependency tasks in video games. Please refer to the combined response for details.
We thank reviewers for their time and useful feedback. It is always the authors' responsibility that the readers understand the context, the underlying technical approach, and the experiments. We believe we were short in all the three, but we also believe most of the content is there and all it needs is rewriting.
On top of that, to convince reviewers we have also performed additional experiments including in simulation to demonstrate our baselines are the baselines reviewers are asking for (we just did not present them properly).
Therefore, we urge both reviewers and AC to give us a second chance and consider the paper in light of the rebuttal.
Experiments -- REINFORCE baseline is the curiosity baseline [Pathak et al. 2017]
First, we would handle the experimental concerns because we agree if we cannot demonstrate empirically that our approach works, then the paper should not be accepted. Reviewers are concerned we are comparing with DQN and REINFORCE but not with other exploration approaches. We would like to point out that both DQN and REINFORCE are not using external rewards and hence they are exploration approaches.
Specifically, REINFORCE baseline is THE curiosity baseline and DQN is another optimization approach used for the same objective as curiosity baseline [Pathak et al. 2017]. One minor difference in the use of curiosity [Pathak et al. 2017] for our experiments is the use of pre-trained ImageNet feature. This feature is stationary which further helps the curiosity formulation as discussed in the follow-up [Burda et. al. 2018].
In order to further demonstrate that our baselines are credible, we took the implementation and curves from recent curiosity paper [Burda et. al. 2018]. We use the curiosity baseline provided in the paper and to demonstrate our objective function for exploration is meaningful, we add our objective to the original exploration objective. Note both these environments require long-term modeling and since using current approaches long-term models are hard to learn; we use our objective in conjunction with existing models. On both the environments, our objective seems to provide gain in efficiency. Hopefully, this confirms that the curiosity baselines used in the paper (referred to as REINFORCE) is a meaningful baseline. Please see results in Appendix Section-C of the updated paper.
Context
We agree that some of our context placement was too strong, and we would be happy to soften it.
But to put our paper in the right context.
First, we agree that current Deep-RL approaches have made significant advances but as most RL experts would agree: sample efficiency still is a huge bottleneck. Again almost all RL experts would agree, this has been one of the bottlenecks towards bringing RL to robotics (without using expert demonstrations).
One possible way is to train in simulation and then transfer to real-world robots; this might work but currently, in object manipulation, it is hard to transfer due to reality gap (contact dynamics are hard to simulate). As a research community, we should also investigate the alternative way: to train from scratch on the robot in real-world settings. This is where our paper comes in. Current RL-based exploration approaches (e.g., curiosity and visitation counts) use reward functions that are black-box in nature.
Hence the gradients are not that meaningful which reduces the sample complexity. In our paper, we propose a new exploration objective that can be optimized via supervised learning (instead of reinforcement learning). We do not claim that this new objective has all properties of the previous one, and we elaborate on this below.
Technical Insights -- What does this new objective actually do?
AR2 rightly asks the question: what does this new objective metric actually do? To answer this, and give intuition about the objective we have added a new section to the paper, see Appendix Section-D. We will move some part of it to the main paper in the final version.
To summarize, due to the differentiable nature of our objective and presence of local constancy approximation, our objective does not behave identically to the original curiosity reward which is optimized by policy gradients (REINFORCE). The main difference in behavior is in case of interactions which have the same repetitive outcome: our method explicitly forces towards the action with different outcomes, while original curiosity objective penalizes the actions for repetitive outcomes. We further add pointers to the Experimental-Design in convex optimization literature which bears a close similarity to our setup. Please refer to Section-D for details.
We now answer other individual comments in the direct replies to reviewers.

Meta-review:
role: AC
data: metareview: The authors propose implementing intrinsic motivation as a differentiable supervised loss coming from the error of a forward model, rather than the black box style of curiosity reward. The motivation is that this approach will lead to more sample efficient exploration for real robots. The use of a differentiable loss for policy optimization is interesting and has some novelty. However, the reviewers were unanimous in their criticism of the paper for poor baselines, unclear experiments and results, and unsupported claims. Even after substantial revisions to the paper, the AC and reviewers were unconvinced of the basic claims of the paper.

Decision: Reject