Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Disclaimer: 
This is an emergency review that I got assigned on Nov 5th, so I unfortunately had only a few hours to assess the paper.

Summary:
This paper proposes a new mechanism for “search control”, that is, choosing the planning-start-states from which to roll out the model, in the context of Dyna-style updates. The key idea is to favour updates to states where the the value functions changes a lot, because those need more information to become accurate.
The topic is very interesting, and not studied enough, the proposed method looks novel, and the experimental results look promising. However, the clarity of the paper is not great, the theoretical result (Theorem 1) seems to be incorrect, the narrative from Claude Shannon to Joseph Fourier to gradient norms to practical choices is somewhat confusing, the empirical results are not really conclusive, and despite a large appendix there are missing implementation details. I think this paper is currently one or two major revisions away from the acceptance threshold.

Comments:
1. Theorem 1, equation (3): consider a constant function f(x) = K. Then its derivative is zero, so the left-hand side is zero, yet the right-hand side is the product of two strictly positive factors (each of them proportional to K^2), so how can the equation hold?
2. Sine experiment: How can you reasonably do *linear* regression onto a sine function? What are your input features?
3. Figure 2 is odd: panels (b) and (c) look very similar in how they bias the sampling toward the left side, yet the performance difference (a) is very stark, how come? Also, how can there be such a high contrast in (b)/(c) if 60% of all samples are chosen uniformly, as stated in the text?
4. The paper has numerous grammatical mistakes, to the extent it becomes difficult to read. I know this can happen in deadline mode, but please revise the draft thoroughly (special call-out to the dozens of missing definite/indefinite articles and plural forms). Also, use “\citep” where appropriate.
5. The objective g as the sum of a gradient norm and a Hessian norm seems odd, as these terms have completely different scales, so usually one of them will dominate, can you explain and motivate this further, and compare empirically to the two terms in isolation?
7. For the prioritizedER baseline, what variant and hyper-parameters are you using?
8. Please describe “out of boundary” mentioned in Algorithm 1.
9. Section 5.2 states that the “variance of the evaluation curve” is smaller, indicating robustness, yet Figure 3(a) appears to have high instability in the (early) learning curve?
10. Section 5.2 states that Figure 5 should show that the bottleneck areas are sampled more densely, but that’s a dubious claim. Please quantify the claim or drop it.
11. Figure 4(b) has oddly wide error-bars for DQN-Value, which looks suspiciously like a single failed run. Can you add a plot to the appendix with median/quantiles instead of mean/std statistics?


--------------
Update Nov 17: most of my concerns have been addressed, and I have thus increased my rating.




review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new way to select states from which do do transitions in dyna algorithm (which trains policy from model experience as if it was a real experience). It proposes to look for states where frequency of value function as a function of a real valued state is large, because these are the states where the function is harder to approximate. The paper also shows that such frequency is large where the gradient of the function is large in magnitude which allows for finding such states in practice. In more detail, similar to previous algorithms, this algorithm keeps both an experience replay buffer as well as another buffer of states (search-control queue) and uses a hill climbing strategy to find states with both higher frequency and higher value. The paper tests the algorithm on toy domains - the mountain car and a maze with doors.

The idea of using the magnitude of the gradient as an exploration signal is not new - “go to places where agent learns more”. In this paper, such signal is not used as a reward but for finding states from which to do training updates. It is also nice that the paper provides a good relation (with explanation) between this signal and the frequency of the value function. The paper is clearly written. One drawback is that the main computations are only tractable in toy domains - it would be good if they discussed how to use this with general neural model with large state spaces (e.g. states obtained with an RNN).

Detailed comments:
- In the abstract it says “…searching high frequency region of value function”. At this point it is not clear what function we are considering - what is on the axes? - a time, state, value? (value on y axis and a real valued state on the x as it turns out later).
- Same at line end-7 on page 2
- End of section 2: and states with high value (as you do later).
- A demonstration experiment: Why do you fit linear regression to a sin function? Why not at least one layer NN?
- Page 5 top: You reason that Hessian learns faster - why not just squaring gradient norm?
- Section 4: Hessian is intractable for general neural network unless you have a toy domain - does it work with just the gradient? This is an important point if this is to be scaled. May be you can also discuss how to compute the gradient of the norm of the gradient.

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper basically built upon [1]. The authors propose to do sampling in the high-frequency domain to increase the sample efficiency.  They first argue that the high-frequency part of the function is hard to approximate (i.e., needs more sample points) in section 3.1. They argue that the gradient and Hessian can be used to identify the high-frequency region. And then they propose to use g(x)=||gradient||+||Hessian || as the sampling metric as illustrated in Algorithm 1. To be noticed that, they actually hybrid the proposed metric (6) and the value-based metric (7, proposed in [1]) in their algorithm.

Strength:
Compared to [1], their experiment environment seems more complicated (MazeGridWorld vs. GridWorld). 
Figure 3 shows that their method converges faster than Dyna-Value.
Figure 5 is very interesting. It shows that their method concentrates more on the important region of the function. 

Weakness:
In footnote 3: I don't see why such an extension is natural.
In theorem 1, why the radius of the local region has to be?
Theorem1 only justifies the average (expectation) of gradient norm is related to the frequency. The proposed metric $g$, however, is evaluated on a single sample point. So I think if adding some perturbations to $s$ (and then take the average) when evaluating $g$ will be helpful.
The authors only evaluate their algorithm in one environment, MazeGridWorld. 
I would like to see the experiment results of using only (6) as the sampling rule. 
What kind of norm are you using? (||gradient||, ||hessian||)
Why $g$ is the combination of gradient norm and hessian norm? What will be the performance of using only gradient or hessian?
Figure 4(b), DQN -> Dyna

Reference:
[1] Hill Climbing on Value Estimates for Search-control in Dyna
review_title: Official Blind Review #3

Meta-review: The reviewers are unanimous in their evaluation of this paper, and I concur.
Decision: Accept (Poster)