Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introdues NoisyNets, that are neural networks whose parameters are perturbed by a parametric noise function, and they apply them to 3 state-of-the-art deep reinforcement learning algorithms: DQN, Dueling networks and A3C. They obtain a substantial performance improvement over the baseline algorithms, without explaining clearly why.

The general concept is nice, the paper is well written and the experiments are convincing, so to me this paper should be accepted, despite a weak analysis.

Below are my comments for the authors.

---------------------------------
General, conceptual comments:

The second paragraph of the intro is rather nice, but it might be updated with recent work about exploration in RL.
Note that more than 30 papers are submitted to ICLR 2018 mentionning this topic, and many things have happened since this paper was
posted on arxiv (see the "official comments" too).

p2: "our NoisyNet approach requires only one extra parameter per weight" Parameters in a NN are mostly weights and biases, so from this sentence
one may understand that you close-to-double the number of parameters, which is not so few! If this is not what you mean, you should reformulate...

p2: "Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent."
Two ideas seem to be collapsed here: the idea of diminishing noise over an experiment, exploring first and exploiting later, and the idea of
adapting the amount of noise to a specific problem. It should be made clearer whether NoisyNet can address both issues and whether other
algorithms do so too...

In particular, an algorithm may adapt noise along an experiment or from an experiment to the next.
From Fig.3, one can see that having the same initial noise in all environments is not a good idea, so the second mechanism may help much.

BTW, the short section in Appendix B about initialization of noisy networks should be moved into the main text.

p4: the presentation of NoisyNets is not so easy to follow and could be clarified in several respects:
- a picture could be given to better explain the structure of parameters, particularly in the case of factorised (factorized, factored?) Gaussian noise.
- I would start with the paragraph "Considering a linear layer [...] below)" and only after this I would introduce \theta and \xi as a more synthetic notation.
Later in the paper, you then have to state "...are now noted \xi" several times, which I found rather clumsy.

p5: Why do you use option (b) for DQN and Dueling and option (a) for A3C? The reason why (if any) should be made clear from the clearer presentation required above.

By the way, a wild question: if you wanted to use NoisyNets in an actor-critic architecture like DDPG, would you put noise both in the actor and the critic?

The paragraph above Fig3 raises important questions which do not get a satisfactory answer.
Why is it that, in deterministic environments, the network does not converge to a deterministic policy, which should be able to perform better?
Why is it that the adequate level of noise changes depending on the environment? By the way, are we sure that the curves of Fig3 correspond to some progress
in noise tuning (that is, is the level of noise really "better" through time with these curves, or they they show something poorly correlated with the true reasons of success?)?

Finally, I would be glad to see the effect of your technique on algorithms like TRPO and PPO which require a stochastic policy for exploration, and where I believe that the role of the KL divergence bound is mostly to prevent the level of stochasticity from collasping too quickly.

-----------------------------------
Local comments:

The first sentence may make the reader think you only know about 4-5 old works about exploration.

Pp. 1-2 : "the approach differs ... from variational inference. [...] It also differs variational inference..."
If you mean it differs from variational inference in two ways, the paragraph should be reorganized.

p2: "At a high level our algorithm induces a randomised network for exploration, with care exploration
via randomised value functions can be provably-efficient with suitable linear basis (Osband et al., 2014)"
=> I don't understand this sentence at all.

At the top of p3, you may update your list with PPO and ACKTR, which are now "classical" baselines too.

Appendices A1 and A2 are a lot redundant with the main text (some sentences and equations are just copy-pasted), this should be improved.
The best would be to need to reject nothing to the Appendix.

---------------------------------------
Typos, language issues:

p2
the idea ... the optimization process have been => has

p2
Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent.
=> you should make a sentence...

p3
the the double-DQN

several times, an equation is cut over two lines, a line finishing with "=", which is inelegant

You should deal better with appendices: Every "Sec. Ax/By/Cz" should be replaced by "Appendix Ax/By/Cz".
Besides, the big table and the list of performances figures should themselves be put in two additional appendices
and you should refer to them as Appendix D or E rather than "the Appendix".





review_title: A good paper, despite a weak analysis

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, a new heuristic is introduced with the purpose of controlling the exploration in deep reinforcement learning. 

The proposed approach, NoisyNet, seems very simple and smart: a noise of zero mean and unknown variance is added to each weight of the deep network. The matrices of unknown variances are considered as parameters and are learned with a standard gradient descent. The strengths of the proposed approach are the following:
1 NoisyNet is generic: it is applied to A3C, DQN and Dueling agents. 
2 NoisyNet reduces the number of hyperparameters. NoisyNet does not need hyperparameters (only the kind of the noise distribution has to be defined), and replacing the usual exploration heuristics by NoisyNet, a hyperparameter is suppressed (for instance \epsilon in the case of epsilon-greedy exploration).
3 NoisyNet exhibits impressive experimental results in comparison to the usual exploration heuristics for to A3C, DQN and Dueling agents.

The weakness of the proposed approach is the lack of explanation and investigation (experimental or theoretical) of why does Noisy work so well. At the end of the paper a single experiment investigates the behavior of weights of noise during the learning.  Unfortunately this experiment seems to be done in a hurry. Indeed, the confidence intervals are not plotted, and probably no conclusion can be reached because the curves are averaged only across three seeds! It’s disappointing.  As expected for an exploration heuristic, it seems that the noise weights of the last layer (slowly) tend to zero. However for some games, the weights of the penultimate layer seem to increase. Is it due to NoisyNet or to the lack of seeds? 

In the same vein, in section 3, two kinds of noise are proposed: independent or factorized Gaussian noise. The factorized Gaussian noise, which reduces the number of parameters, is associated with DQN and Dueling agents, while the independent noise is associated with A3C agent. Why? 

Overall the proposed approach is interesting and has strengths, but the paper has weaknesses. I am somewhat divided for acceptance. 

review_title: The proposed approach is interesting and has strengths, but the paper has weaknesses. I am somewhat divided for acceptance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: A new exploration method for deep RL is presented, based on the idea of injecting noise into the deep networks’ weights. The noise may take various forms (either uncorrelated or factored) and its magnitude is trained by gradient descent along other parameters. It is shown how to implement this idea both in DQN (and its dueling variant) and A3C, with experiments on Atari games showing a significant improvement on average compared to these baseline algorithms.

This definitely looks like a worthy direction of research, and experiments are convincing enough to show that the proposed algorithms indeed improve on their baseline version. The specific proposed algorithm is close in spirit to the one from “Parameter space noise for exploration”, but there are significant differences. It is also interesting to see (Section 4.1) that the noise evolves in non-obvious ways across different games.

I have two main concerns about this submission. The first one is the absence of a comparison to the method from “Parameter space noise for exploration”, which shares similar key ideas (and was published in early June, so there was enough time to add this comparison by the ICLR deadline). A comparison to the paper(s) by Osband et al (2016, 2017) would have also been worth adding. My second concern is that I find the title and overall discussion in the paper potentially misleading, by focusing only on the “exploration” part of the proposed algorithm(s). Although the noise injected in the parameters is indeed responsible for the exploration behavior of the agent, it may also have an important effect on the optimization process: in both DQN and A3C it modifies the cost function being optimized, both through the “target” values (respectively Q_hat and advantage) and the parameters of the policy (respectively Q and pi). Since there is no attempt to disentangle these exploration and optimization effects, it is unclear if one is more important than the other to explain the success of the approach. It also sheds doubt on the interpretation that the agent somehow learns some kind of optimal exploration behavior through gradient descent (something I believe is far from obvious).

Estimating the impact of a paper on future research is an important factor in evaluating it. Here, I find myself in the akward (and unusual to me) situation where I know the proposed approach has been shown to bring a meaningful improvement, more precisely in Rainbow (“Rainbow: Combining Improvements in Deep Reinforcement Learning”). I am unsure whether I should take it into account in this review, but in doubt I am choosing to, which is why I am advocating for acceptance in spite of the above-mentioned concerns.

A few small remarks / questions / typos:
- In eq. 3 A(...) is missing the action a as input
- Just below: “the the”
- Last sentence of p. 3 can be misleading because the gradient is not back-propagated through all paths in the defined cost
- “In our experiments we used f(x) = sgn(x) p |x|”: this makes sense to me for eq. 9 but why not use f(x) = x in eq. 10?
- Why use factored noise in DQN and independent noise in A3C? This is presented like an arbitrary choice here.
- What is the justification for using epsilon’ instead of epsilon in eq. 15? My interpretation of double DQN is that we want to evaluate (with the target network) the action chosen by the Q network, which here is perturbed with epsilon (NB: eq. 15 should have b in the argmax, not b*)
- Section 4 should say explicitly that results are over 200M frames
- Assuming the noise is sampled similarly doing evaluation (= as in training), please mention it clearly.
- In paragraph below eq. 18: “superior performance compare to their corresponding baselines”: compared
- There is a Section 4.1 but no 4.2
- Appendix has a lot of redundant material with the main text, for instance it seems to me that A.1 is useless.
- In appendix B: “σi,j is simply set to 0.017 for all parameters” => where does this magic value come from?
- List x seems useless in C.1 and C.2
- C.1 and C.2 should be combined in a single algorithm with a simple “if dueling” on l. 24
- In C.3: (1) missing pi subscript for zeta in the “Output:” line, (2) it is not clear what the zeta’ parameters are for, in particular should they be used in l. 12 and 22?
- The paper “Dropout as a Bayesian approximation” seems worth at least  adding to the list of related work in the introduction.
review_title: Good paper but lack of empirical comparison & analysis

Meta-review: The paper proposes to add noise to the weights of a policy network during learning in Deep-RL settings and finds that this results in better performance on DQN, A3C and other algorithms that use other exploration strategies. Unfortunately, the paper does not do a thorough job of exploring the reasons and doesn't offer a comparison to other methods that have been out on arxiv for several months before the submission, in spite of reviewers and anonymous requests. Otherwise I might have supported recommending the paper for a talk. 
Decision: Accept (Poster)