Title: #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Authors: Haoran Tang
Rein Houthooft
Davis Foote
Adam Stooke
Xi Chen
Yan Duan
John Schulman
Filip De Turck
Pieter Abbeel
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Games

Abstract: Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once.
Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. 

In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results.

Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

TL;DR: We improve exploration in deep reinforcement learning by simply hashing states and assigning bonus rewards according to state counts.

Review discussion:
I assume pixel-SimHash is to use the entire image as input to SimHash without any preprocessing. Is this correct? For learned embedding, from Eqn.4 it seems that from the optimization of AE, we already get a binary representation for each state.
So why there is an additional step applying SimHash on (almost) binarized code b(s)? From the context, Eqn. 4 needs to be optimized online while RL training is running, but Alg. 1 does not make it explicit. How often does Eqn. 4 gets optimized? I assume it is after every episode but is it true? From the text it seems that the binary code needs to be consistent during the training, which is not required if the code is updated between episodes.
1) "pixel-SimHash" indeed means using only the current frame as input to SimHash (resized in the same as policy inputs)
2) The binary code b(s) is fed to SimHash to improve the code consistency, as a shorter code is less likely to change during AE training (since SimHash maps similar input to similar output).
3) Eq. (4) is updated every 3 iterations.
Although the code for a particular image does not remain the same in all cases, it is mostly varying at the beginning of the training procedure. Once the training has gone through some "burn-in" time, the code remains relatively consistent. This is due to
a) the combination of the second term in Eq. (4), which forces the code to be 0/1;
b) the SimHash procedure which projects the code to a lower dimension and thus filters out small perturbations;
c) the saturating nonlinearity (sigmoid/tanh) which makes "converged" codes less prone to change.
We will improve the clarity of Section 2.3, which details the AE training procedure.
Apologies for the (very) late questions!
For the learned embedding:
1. U(-0.3, 0.3) ? Why the magnitude of 0.3, and why uniform? This seems arbitrary.. did you try others? Is this what people have used in the past? Is this domain(Atari)-dependent?
2a. Can you explain p(s_i) in the autoencoder loss of Equation (4)? Normally MSE is used for reconstruction, so I don't know how to interpret p(s_i) here, and it is never defined.
2b. After some digging, I suspect there is a softmax distribution per pixel? Is the output of these distributions a probability per pixel value (0-255)? And is p(s_i) then a sum of likelihoods over all the pixels? If so, this should be explained in more detail here. PixelCNN is still quite new so p(s_i) should either be defined formally or at the very least described in a sentence.
3. "Because the code dimension often needs to be large in order to correctly reconstruct the input, we apply a downsampling procedure to the resulting code, either through random projection to a lower-dimensional space or through grouping bits together." Ok, so which one is it: random projection or grouping bits together? Can you elaborate and give some details here? This section is quite lacking in details.. it might be difficult to reproduce this. If the details come later in the paper, I recommend referring to the section here.
Otherwise, it could leave the reader confused.
Thanks for your questions. Please see below for our answers.
1) Adding some uniform noise to force continuous variables to behave like discrete ones is studied in multiple context [1-3], and we adopt this methodology without much tuning on the 0.3 hyperparameter, but we expect that higher values like 0.5 should also work.
Ultimately, the noise width should be sufficiently high in order to force the continuous sigmoid outputs to take on binary values (this happens since it is the only way to correctly reconstruct the inputs).
In addition, we believe it is possible to use other recently proposed continuous relaxation techniques [4,5] to achieve similar binary embeddings.
2) Sorry for the confusion, log p(s_i) is merely meant to be a generalization of normal MSE loss in a typical autoencoder, and indeed the specific form used here assumes a discretized pixel value space, which is modeled by a softmax (as in [6]). We will improve clarity of this section.
3) In the final experiment, random projections were used (as in Section 2.2). In the next revision, we will include a more detailed training procedure and we plan to open-source our code, so that precise setup can be inspected.
[1] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, Shimon Whiteson, "Learning to Communicate with Deep Multi-Agent Reinforcement Learning", arXiv preprint arXiv:1605.06676 (2016).
[2] Geoffrey Hinton, Ruslan Salakhutdinov, "Discovering binary codes for documents by learning deep generative models", Topics in Cognitive Science 3.1 (2011): 74-91.
[3] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio, "Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1", arXiv preprint arXiv:1602.02830 (2016).
[4] Chris J. Maddison, Andriy Mnih, Yee Whye Teh, "The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables", arXiv preprint arXiv:1611.00712 (2016).
[5] https://openreview.net/forum?id=rkE3y85ee
[6] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu, "Conditional Image Generation with PixelCNN Decoders", arXiv preprint arXiv:1606.05328 (2016).
This paper proposed to use a simple count-based exploration technique in high-dimensional RL application (e.g., Atari Games). The counting is based on state hash, which implicitly groups (quantizes) similar state together. The hash is computed either via hand-designed features or learned features (unsupervisedly with auto-encoder). The new state to be explored receives a bonus similar to UCB (to encourage further exploration).
Overall the paper is solid with quite extensive experiments. I wonder how it generalizes to more Atari games. Montezuma’s Revenge may be particularly suitable for approaches that implicitly/explicitly cluster states together (like the proposed one), as it has multiple distinct scenarios, each with small variations in terms of visual appearance, showing clustering structures.
On the other hand, such approaches might not work as well if the state space is fully continuous (e.g. in RLLab experiments).
The authors did not answer my question about why the hash code needs to be updated during training. I think it is mainly because the code still needs to be adaptive for a particular game (to achieve lower reconstruction error) in the first few iterations . After that stabilization is the most important. Sec. 2.3 (Learned embedding) is quite confusing (but very important). I hope that the authors could make it more clear (e.g., by writing an algorithm block) in the next version.
Dear Reviewer,
We incorrectly understood your initial question, sorry about that! The hash code (in case it is learned through the autoencoder) needs to be updated to ensure that the codes remain different for significantly different states.
For example in Montezuma’s Revenge, when the agent enters a completely new room, there would be many code collisions, therefore the autoencoder is updated to differentiate between states in this new room. It is indeed true that most of the training is needed in the first few iterations, since the environment is completely novel to the agent. Hereafter, as you suggested, the consistency of the code becomes more important.
Thanks for pointing out that Section 2.3 could be improved in terms of clarity, especially since many details are deferred to the Appendix. We have polished and improved the structure (e.g., through addition of algorithmic pseudocode in Algorithm 2) of this section in the updated manuscript.
The paper proposes a new exploration scheme for reinforcement learning using locality-sensitive hashing states to build a table of visit counts which are then used to encourage exploration in the style of MBIE-EB of Strehl and Littman.
Several points are appealing about this approach: first, it is quite simple compared to the current alternatives (e.g. VIME, density estimation and pseudo-counts).
Second, the paper presents results across several domains, including classic benchmarks, continuous control domains, and Atari 2600 games.
In addition, there are results for comparison from several other algorithms (DQN variants), many of which are quite recent. The results indicate that the approach clearly improves over the baseline. The results against other exploration algorithms are not as clear (more dependent on the individual domain/game), but I think this is fine as the appeal of the technique is its simplicity.
Third, the paper presents results on the sensitivity to the granularity of the abstraction.
I have only one main complaint, which is it seems there was some engineering involved to get this to work, and I do not have much confidence in the robustness of the conclusions. I am left uncertain as to how the story changes given slight perturbations over hyper-parameter values or enabling/disabling of certain choices.
For example, how critical was using PixelCNN (or tying the weights?) or noisifying the output in the autoencoder, or what happens if you remove the custom additions to BASS? The granularity results show that the choice of resolution is sensitive, and even across games the story is not consistent.
The authors decide to use state-based counts instead of state-action based counts, deviating from the theory, which is odd because the reason to used LSH in the first place is to get closer to what MBIE-EB would advise via tabular counts. There are several explanations as to why state-based versus state-action based counts perform similarly in Atari; the authors do not offer any. Why?
It seems like the technique could be easily used in DQN as well, and many of the variants the authors compare to are DQN-based, so omitting DQN here again seems strange. The authors justify their choice of TRPO by saying it ensures safe policy improvement, though it is not clear that this is still true when adding these exploration bonuses.
The case study on Montezuma's revenge, while interesting, involves using domain knowledge and so does not really fit well with the rest of the paper.
So, in the end, simple and elegant idea to help with exploration tested in many domains, though I am not certain which of the many pieces are critical for the story to hold versus just slightly helpful, which could hurt the long-term impact of the paper.
--- After response:
Thank you for the thorough response, and again my apologies for the late reply.
I appreciate the follow-up version on the robustness of SimHash and state counting vs. state-action counting.
The paper addresses an important problem (exploration), suggesting a "simple" (compared to density estimation) counting method via hashing. It is a nice alternative approach to the one offered by Bellemare et al. If discussion among reviewers were possible, I would now try to assemble an argument to accept the paper.
Specifically, I am not as concerned about beating the state of the art in Montezuma's as Reviewer3 as the merit of the current paper is one the simplicity of the hashing and on the wide comparison of domains vs. the baseline TRPO. This paper shows that we should not give up on simple hashing. There still seems to be a bunch of fiddly bits to get this to work, and I am still not confident that these results are easily reproducible.
Nonetheless, it is an interesting new contrasting approach to exploration which deserves attention.
Not important for the decision: The argument in the rebuttal concerning DQN & A3C is a bit of a straw man. I did not mention anything at all about A3C, I strictly referred to DQN, which is less sensitive to parameter-tuning than A3C.
Also, Bellemare 2016 main result on Montezuma used DQN.
Hence the omission of these techniques applied to DQN still seems a bit strange (for the Atari experiments). The figure S9 from Mnih et al. points to instances of asynchronous one-step Sarsa with varied thread counts..
of course this will be sensitive to parameters: it is both asynchronous online algorithms *and* the parameter varied is the thread count! This is hardly indicative of DQN's sensitivity to parameters, since DQN is (a) single-threaded (b) uses experience replay, leading to slower policy changes.
Another source of stability, DQN uses a target network that changes infrequently. Perhaps the authors made a mistake in the reference graph in the figure? (I see no Figure 9 in https://arxiv.org/pdf/1602.01783v2.pdf , I assume the authors meant Figure S9)
Dear Reviewer,
* State counting v.s. state-action counting
Our choice for state counting is motivated by experimental results: We could not notice a significant performance difference between state and state-action counting (the comparison is added in Appendix A.6 of the revised manuscript).
Moreover, the bonus coefficient for state-action count needs certain tuning, as different games have different number of available actions. Only taking into account the state space is consistent with similar approaches such as the pseudo-count paper of Bellemare et al (2016). Our belief is that due to 1) the large number of samples drawn from the environment, 2) the relatively low-dimensional action space of the environments, 3) the fact that the state count-based reward bonus decays sufficiently slowly, the agent’s policy’s randomness is sufficient for exploring most actions in a particular state.
As such, adding an additional bonus to underexplored actions does not cause significant improvements. We could imagine a difference in case of high-dimensional combinatorial action spaces.
* Application to other RL algorithms
Indeed, our method is also applicable to other RL algorithms, including DQN and A3C. We have noticed that DQN and A3C are sensitive to small changes in hyperparameters (also supported by Figure 9 in Mnih et al. (2016)), making comparisons difficult.
Therefore, we made use of natural policy gradient methods, in particular TPRO, due to its stability, minimal need for hyperparameter tuning, and speed.
Also compared to DQN, it was beneficial to make use of an RL algorithm capable of performing both in discrete and continuous state-action spaces for consistency reasons.
* Hyperparameters
It is true that specific architectural choices can make a difference, however, the results reported were only subject to mild hyperparameter search. Architectural decisions of the autoencoder were made to make sure it achieves high reconstruction accuracy as quickly as possible (quickly reducing the autoencoder reconstruction loss helps stabilizing the code fast, making the bonus more consistent). The specifics of the BASS alterations are of lower importance since its inclusion in the paper is mostly a proof-of-concept of incorporation of domain knowledge. We will improve the experiments regarding the granularity of the hash function, in order to draw more meaningful conclusions. To accommodate your inquiry about robustness to hyperparameters changes, we have run additional experiments studying the performance sensitivity to changing hyperparameter. The results (now shown in Appendix A.6 of the updated manuscript) indicate that the performance changes smoothly with varying hyperparameters k and beta, highlighting the robustness of the proposed hashing method.
* Experiments on Montezuma’s Revenge
We would like to advertise the section on Montezuma’s Revenge since it highlights a limitation of naive counting strategies: Due to complicated structure of image observations, it is difficult to extract informative elements. It also emphasizes that adding limited prior knowledge to the exploration strategy might be a great leap forward in performance in case purely model-free RL methods fail to work well.
Thanks you for your thorough response, and follow-up additions to the paper.\
I have now editted (finalized) my review, adding a section on comments post-rebuttal.
This paper introduces a new way of extending the count based exploration approach to domains where counts are not readily available. The way in which the authors do it is through hash functions. Experiments are conducted on several domains including control and Atari.
It is nice that the authors confirmed the results of Bellemare in that given the right "density" estimator, count based exploration can be effective. It is also great the observe that given the right features, we can crack games like Montezuma's revenge to some extend.
I, however, have several complaints:
First, by using hashing, the authors did not seem to be able to achieve significant improvements over past approaches. Without "feature engineering", the authors achieved only a fraction of the performance achieved in Bellemare et al. on Montezuma's Revenge. The proposed approaches In the control domains, the authors also does not outperform VIME.
So experimentally, it is very hard to justify the approach.
Second, hashing, although could be effective in the domains that the authors tested on, it may not be the best way of estimating densities going forward. As the environments get more complicated, some learning methods, are required for the understanding of the environments instead of blind hashing. The authors claim that the advantage of the proposed method over Bellemare et al. is that one does not have to design density estimators.
But I would argue that density estimators have become readily available (PixelCNN, VAEs, Real NVP, GANs) that they can be as easily applied as can hashing. Training the density estimators is not difficult problem as more.
Dear Reviewer,
* Density-based methods
The density-based modeling approach is interesting, and in fact our approach could be interpreted as a form of hash-based density estimation, albeit with a particular structure, as also mentioned by Marc Bellemare as an openreview.net comment. We shorty touch on the connection at the end of Section 4.This is especially so in case an autoencoder is used to obtain a learned code from data, as explained in Section 2.3 (with experimental results in Table 1 and Figure 3, under TRPO-AE-SimHash. Some of the learned hash codes are depicted in the appendix A.4, Figures 6 and 7).
* Scalability
We believe the learned hash code approach would also scale to significantly more complex domains, since the complexity is captured in the autoencoder structure/parameters. It essentially extracts the most salient features in its hidden layer, which directly form the elements of the hash code.
* Improvement over past approaches
The approach significantly outperforms the baseline, and in certain cases (Frostbite and Solaris) gets near state-of-the-art results. The results on Montezuma’s Revenge could be explained by the fact that TRPO as a policy gradient method is not suited for this type of task. This is supported by the related method A3C, in which the pseudo-count method in Bellemare et al. (2016) also leads to only minor improvements (A3C+), despite Bellemare et al. (2016) achieving state-of-the-art with DQN.
* Comparison to VIME
The sparse reward continuous control benchmark tasks were originally designed to measure whether or not an RL method is capable of reaching the reward through exploration, rather than comparison in terms of average return over time.
As such, the performance on these tasks should be viewed more or less as binary (Houthooft et al. (2016)): Is the agent capable of reaching target or not? From this perspective, both VIME and the hash-based method are capable of solving the sparse rewards tasks.
Moreover, due to the proposed approach’s simplicity, it is much faster to run than VIME (see also Achiam & Sastry (2016) at https://openreview.net/forum?id=Bk8aOm9xl).
We would like to convince you that the strength of this method is not its performance, but its simplicity. It can be implemented in a few lines of code, while achieving great performance boosts over baseline approaches. More advanced density-based approaches are promising methods for building advanced exploration strategies, but are many times more to stabilize and tune correctly, and can be much slower to run.
We have updated the paper with the following changes
(1) The distinction between static hashing (e.g. LSH) and learned hashing is made clear. They are now discussed separately in sections 2.2 and 2.3. BASS is moved to the experimental part (section 3.2), since it is specific to Atari.
(2) The text in section 2.3 is improved to clarify details of training the autoencoder. A separate algorithm block (Algorithm 2) clarifies that the autoencoder is retrained periodically.
(3) Robustness of SimHash and and the comparison between state counting and state action counting are included in Appendix A.6.

Meta-review:
role: PC
data: comment: The paper proposes a simple approach to exploration that uses a hash of the current state within a exploration bonus approach (there are some modifications to learned hash codes, but this is the basic approach). The method achieves reasonable performance on Atari game tasks (sometimes outperformed by other approaches, but overall performing well), and it's simplicity is its main appeal (although the autoencoder-based learned hash seems substantially less simple, so actually loses some advantage there). 
 
 The paper is likely borderline, as the results are not fantastic: the approach is typically outperformed or similarly-performed by one of the comparison approaches (though it should be noted that no comparison approach performs well over all tasks, so this is not necessarily that bad). But overall, especially because so many of these methods have tunable hyperparameters it was difficult to get a clear understanding of just how these experimental results fit.
 
 Pros:
 + Simple method for exploration that seems to work reasonably well in practice
 + Would have the potential to be widely used because of its simplicity
 
 Cons:
 - Improvements over previous approaches are not always there, and it's not clear whether the algorithm has any "killer app" domain where it is just clearly the best approach
  
 Overall, this work in its current form is too borderline. The PCs encourage the authors to strengthen the empirical validation and resubmit.

Decision: Reject