Query The Agent: Improving Sample Efficiency Through Epistemic Uncertainty Estimation
Authors: Julian Alverio
Boris Katz
Andrei Barbu
Venue: Submitted to ICLR 2023 (2023)
Keywords: goal-conditioned reinforcement learning
reinforcement learning
goal-conditioned
goal
model-free
sample efficiency
deep reinforcement learning

Abstract: Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency. We propose a novel algorithm, Query The Agent (QTA), which significantly improves sample efficiency by estimating the agent's epistemic uncertainty throughout the state space and setting goals in highly uncertain areas. Encouraging the agent to collect data in highly uncertain states allows the agent to improve its estimation of the value function rapidly. QTA utilizes a novel technique for estimating epistemic uncertainty, Predictive Uncertainty Networks (PUN), to allow QTA to assess the agent's uncertainty in all previously observed states. We demonstrate that QTA offers decisive sample efficiency improvements over preexisting methods.

Under review as a conference paper at ICLR 2023
QUERY THE AGENT: IMPROVING SAMPLE EFFICIENCY THROUGH EPISTEMIC UNCERTAINTY ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency. We propose a novel algorithm, Query The Agent (QTA), which significantly improves sample efficiency by estimating the agent's 's epistemic uncertainty throughout the state space and setting goals in highly uncertain areas, Encouraging the agent to collect data in highly uncertain states allows the agent to improve its estimation of the value function rapidly. QTA utilizes novel technique for estimating epistemic uncertainty, Predictive Uncertainty Networks (PUN), to allow QTA to assess the agent's uncertainty in all previously observed states. We demonstrate that QTA offers decisive sample efficiency improvements over preexisting methods.
INTRODUCTION
Deep reinforcement learning has been demonstrated to be highly effective in diverse array of se- quential decision-making tasks (Silver et al.) 2016 Berner et al.) 2019). However, deep reinforcement learning remains challenging to implement the real world, in part because of the massive amount of data required for training. This challenge is acute in robotics (Sünderhauu et al., 2018: Dulac-Arold al. 2019). in tasks such as manipulation (Liu a and in self-driving cars (Kothari al.).
Existing curriculum methods for training goal- conditiond reinforcement learning (RL) agents suffer from poor sample efficiency Arnold al.) (2019) and often fail to consider agents' specific deficiencies and epistemic uncertainty when selecting goals. Instead, they rely on poor proxy estimates of epistemic uncertainty or high-level statisties from rollouts, such as the task success rate. Without customizing learning according to agents' epistemic uncertainties, existing methods inhibit the agent's 's learning with three modes of failure. Firstly, a given curriculum may not be sufficiently challenging for an agent, thus using timesteps inefficiently. Secondly, given curriculum may be too challenging to an agent, causing the agent to learn more slowly than it could otherwise. Thirdly, curriculum may fail to take into account an agent catastrophically forgetting the value manifold in a previously learned region of the state space. Curriculum algorithms need a detailed estimate of the agent's epistemic uncertainty throughout the state space in order maximize learning by encouraging agents to explore the regions of the state space the agent least understands (Kaclbling 1993, Plappert et al. 2018).
We propose a novel curriculum algorithm, Query The Agent (QTA), to accelerate learning in goal- conditioned settings. QTA estimates the agent's epistemic uncertainty in all previously observed states, then drives the agent to reduce its epistemic uncertainty quickly as possible by setting goals in states with high epistemic uncertainty. QTA estimates epistemic uncertainty using a novel neural architecture, Predictive Uncertainty Networks (PUN). By taking into account the agent's s epistemic uncertainty throughout the state space, QTA aims to explore neither too quickly nor too slowly, and revisit previously explored states when catastrophic forgetting occurs. We demonstrate in a 2D continuous maze environment that QTA is significantly more sample efficient than preexisting methods. We also provide detailed analysis of how QTA's approximation of the optimal value manifold evolves over time, demonstrating that QTA's learning dynamics are meaningfully driven by epistemic uncertainty estimation. An overview of QTA and maze environments shown in [
We further demonstrate the importance of utilizing the agent's 's epistemic uncertainty by extending QTA with modified Prioritized Experience Replay (Schaul al. 2016) (PER) buffer. This modified

---

Under review as a conference paper at ICLR 2023
Cangg shates pracicie Liteei
Aaeggigg wineery CONAEE DINGRION
Figure 1: Left: An overview of our goal-selection method. Threshold filter is only applied when sampling the first goal of episode. Right: An example of 1-period square-wave maze, and M-maze. Starting locations are sampled uniformly in the blue regions and goal locations are sampled uniformly in the green regions. Environment described in detail in section
PER buffer assigns sampling priorities to transitions based on states' estimated epistemic uncertainty. QTA augmented with modified PER buffer outperforms QTA, while QTA using standard PER buffer does not. This demonstrates the benefits of integrating detailed epistemic uncertainty estimation into reinforcement learning curricula.
Our contributions are:
1. Query the Agent (QTA): novel curriculum algorithm that adapts goals the agent's epistemic uncertainty and pushes an agent to improve rapidly by collecting data in highly uncertainty states the environment.
2. Predictive Uncertainty Networks (PUN): A new technique, broadly applicable to all Q learning approaches, for measuring an agent's epistemic uncertainty throughout the state space by using the agent's own latent representation. We demonstrate implementation with DDPG.
3. An analysis, including ablation experiments, on how QTA estimates epistemic uncertainty throughout state space and how QTA evolves an agent's 's understanding of its environment over time.
RELATED WORK
Recent advances build upon Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) by designing curricula that allow the agent to cleverly select goals accelerate learning. None of these appropriately take into account the agent's 's epistemic uncertainty. We will categorize methods according to which information about the agent they utilize to select goals, then discuss them.
Pitis et al. (2020) does not directly take into account any information regarding the agent's epistemic uncertainty performance. Their method simply fits kernel density estimate over the set of previously visited goals, then draws samples as goal candidates and selects the sample with the lowest likelihood under the kernel density estimate. This method benefits from reliably producing
2

---

Under review as a conference paper at ICLR 2023
increasingly difficult goals for the agent and being immune to problems such as a goal generator network destabilizing. There are multiple drawbacks. Firstly, if an agent suffers from catastrophic forgetting, the curriculum won't 't be able to correct the problem effectively. Secondly, if the incremental goals are insuffcieetll//over challenging for the agent, the curriculum won't properly adapt. Pitis et al. (2020) shares some commonalities with pseudocount techniques in setting goals exclusively based on visitation frequency.
Similarly, count-based and pseudocount-based algorithms (Tang et al. Bellemare et al.) rely on counting the number of times that a state or a region of state space has been visited. In high dimensional or continuous environments, these algorithms rely on various compression or learned similarity metrics (Machado et al., [Schmidhuber] 2008) or basic statistics in order to identify which regions of the state space are considered similar and adjacent to one another. Intrinsic exploration bonuses can be awarded and goals can be set based on visitation frequencies to accelerate exploration. These methods, taking into account exclusively visitation frequency, do not directly into account the agent's 's epistemic uncertainty or performance and may set goals too ambitiously or lazily and won't 't account for catastrophic forgetting.
Other techniques, such as Bharadhwaj et al. 2020); Portelas et al. (2019); Florensa et al. (0018), only indirectly take into account the agent's 's understanding of the state space through approximations based on statistics regarding agents' historical ability to reach goals. Since such metrics offer minimal insight into the agent's 's understanding of the task, these methods select goals that may not be maximally informative the agent. The major weakness of these methods is that the goal-generation networks can destabilize, and are required to set goals with only very abstract data about the agent's 's understanding of the state space.
Another class of algorithms, such as Pathak et al. Zhang et al. (2020), rely on an external ensemble of networks and estimate epistemic uncertainty throughout the state space. Pathak al. maintains an external network of forward models, which observe the same data as the agent, and uses the disagreement among the forward models as an epistemic uncertainty estimate, [Zhang et al. (2020) does the same, but with three external critics. The major problem with these algorithms is that they are estimating the epistemic uncertainty of other networks, not the agent's 's network. Because these external networks' understanding of the state space will drift away from the agent's 's over time, the epistemic uncertainty estimate the external networks is not representative the agent's epistemic uncertainty estimate. Additionally, Zhang al. (2020) requires privileged access to the entire state space in order to stabilize.
Other curiosity-reaated related methods, such random network distillation (Burda et al.),, use different formulations of a similar mechanism; estimating the uncertainty of networks disconnected from the agent's computation graph. Random network distillation relies on using the agent's experiences to train an external network to predict the output of a frozen, randomly initialized network known as the target network. As an agent visits particular state more, the external network improves its prediction of the output of the target network, making it an effective exploration technique to provide intrinsic reward bonuses proportionate to the size of the external network's 's prediction error. However, this approach relies on estimating the agent's 's error in the random network prediction task to gauge uncertainty in the Q-learning task. These are two very different tasks with different difficulties and will learn at different speeds, thus this technique provides a poor estimate of epistemic uncertainty.
number of techniques (Chane-Sane et al., 2021 Nachum et al., 2018 Christen et al. [021] OpenAl let al. (022]) have used multiple reinforcement learning agents in order to solve long- Long-horizon tasks, by either using the agents in an adversarial fashion or in hierarchy. Using multiple reinforcement learning agents incurs a significant computation expense, and comes along with significant technical challenges such as nonstationarity between agents. Consideration of these techniques is outside the scope of this work, as this work focuses on maximizing the learning of a single agent.
3 METHODS
When learning to navigate a goal-conditioned reinforcement learning environment, we seek to collect much useful information as possible with each rollout, based on which areas of the state space the agent is least certain about. In each episode, QTA measures the agent's 's uncertainty and samples goal in highly uncertain region of the state space in order gather state transitions that will be
3

---

Under review as a conference paper at ICLR 2023
maximally informative to the agent when performing gradient updates. We ensure that the most important transitions are sampled immediately for gradient updates through a modified version of prioritized experience replay (PER), in which the priorities are determined by the estimated epistemic uncertainty of a state in the replay buffer.
In this section, first introduce Predictive Uncertainty Networks (PUN) for predicting epistemic uncertainty. Next, we introduce QTA's mechanism for using epistemic uncertainty estimates to select goal. Lastly, we introduce extension of QTA in which our customized PER buffer allows for further sample efficiency improvements.
3.1 UNCERTAINTY ESTIMATION
Predictive Uncertainty Networks (PUN) PUN utilizes a critic ensemble to enable effective epistemic uncertainty estimation. PUN uses G critics @ie(I....G). G target critics Ptarg,iE (I...,G)++ and in this implementations a standard policy wy. In our PUN implementation we G = 3. Having multiple crities providing estimates is essential for assessing the epistemic uncertainty of the network. In order to ensure the stability of an ensemble of critics, we train them with joint loss function. Given mini-batch B {(s, a, r, s' 9.9g)))))))))) ) inspired by Chen et al. (2021) sample a set of two indices M from (1...., G) then compute the target:
V/ + 7 (min Qerarg ( g')) d' ~ no(-ss)
Then update o; with gradient descent using
Σ (8.0.0)ER
(s, - y)²
We update target networks using polyak averaging coefficient
Otarg, potarg, + p)dz
Lastly, the policy updated as in DDPG Lillicrap et al. 2016).
Predictive Network We assign a predictive network P; to each critic network, where each predictive network single ferd-f layer that takes input the final latent state of the critic and predicts the next hidden state the agent will observe. More specifically Queekbone,s a, g) yields final hidden state ε,, and the final layer of Q, maps Qi,fimai(E:) - Qs(s, a, g) while the predictive network maps (..) >> 8'. We train the predictive network by minimizing the objective:
1 B
L
( SEB
The critic's epistemic uncertainty regarding a particular state is simply the standard deviation of the three predictive heads' predictions. The small predictive networks are a component of the larger ensemble of critics and predictive networks that make up the Predictive Uncertainty Networks.
Note that we intentionally use the same & input to both the final head of the critic and input to the predictive network so that the quality of the two predictions will be tightly coupled. This allows the disagreement among the predictive networks be highly indicative of the eritic's epistemic uncertainty. When initially the critic backbone's latent representations are meaningless, the predictive networks' outputs will be inconsistent with one another and have a large standard deviation. As the critic network better learns to approximate the optimal value manifold, the latent representation will better encode the next state the agent will observe for the final critic layer to predict the next Q value as ((s,, a.) = Ft + y ((5 44+1). As the critic improves and the latent representation better encodes 1,+ the predictive heads' outputs 4 will improve and become more consistent, having a smaller standard deviation among each other. Lastly, we intentionally use an identical architecture for the predictive network as with the critic network ensure they have the same learning capacity and remain tightly coupled.
3.2 GOAL SELECTION
QTA samples previously seen goals from the replay buffer to serve as goal candidates. In order to get precise estimate of epistemic uncertainty for particular state s, we uniformly sample d random
4

---

Under review as a conference paper at ICLR 2023
actions from the action space and compute the epistemic uncertainty of a state be the mean of d uncertainty estimates for that state. More precisely:
c
=
G
P((Qbackoone,s(; Ojz (dessredd)
==
1=0
We use the desired goal sampled from the environment for these estimates, though the prediction is learned be agnostic the choice of goal. The distribution estimated epistemic uncertainties is normalized between 0 and before proceeding to goal selection. Each goal is sampled with probability linearly proportionate to estimated epistemic uncertainty. This simple linear sampling approach encourages QTA to sample goals with high epistemic uncertainty, allowing the agent to gather maximally informative transitions in the regions of the state space where is least certain. Given slope and y-intercept 6, goal candidate with epistemic uncertainty eN is selected as the agent's next goal with probability:
mar(me; + 6,0)
mar(me; * 0)
Whenever the agent successfully reaches its goal, a new goal is sampled using this same procedure. Estimating the agent's epistemic uncertainty only requires forward passes through the critic backbone and predictive head, so QTA is able to efficiently consider an arbitrarily large number of goal candidates by utilizing batch operations. In practice, d is small.
3.3 PRIORITIZED EXPERIENCE REPLAY
To demonstrate the importance of learning from state transitions with high epistemic uncertainty. we implement a modified version prioritized experience replay (PER). Our implementation ensures that our agent samples the transitions with greatest epistemic uncertainty when performing gradient updates. We make two key modifications to PER. Firstly, rather than using temporal difference error to establish priorities in the replay buffer, utilize the estimated epistemic uncertainty according to PUN to provide a more accurate measure of which transitions are most beneficial to sample.
Secondly, once transition is sampled its goal is probabilistically replaced with the goal of a transition later in its trajectory and its reward is recomputed, as in the standard hindsight experience replay (HER) (Andrychowicz et al. 2017) mechanism. That is to say, that in the sparse reward setting use the PER mechanism to sample from HER replay buffer instead of a standard replay buffer. This maintains the benefits of the HER replay buffer, while allowing transitions through the PER mechanism. Similarly to in Schaul et al. (2016), we sample a transition with priority pj with probability:
sample the most important
Schaul et al. (016) demonstrated that oversampling high-priority transitions introduces a bias that destabilize gradient updates. In order to anneal the bias created by oversampling transitions with high epistemic uncertainty, we multiply by importance sampling weights:
Pj
Where 3 is initialized to set value and gradually increased throughout training until / 1, at which point the bias fully compensated for.
Finally, it is important that the goals selected by QTA are achievable. To promote achievability, QTA eliminates invalid goal candidates from being selected as the first goal in a rollout through a simple threshold mechanism. Any goal candidates with mean Q value smaller than cT are eliminated, where the maximum length of the rollout. Similar to as in Pitis et al. (2020), this heuristic is not core to the algorithm, and could be replaced with, e.g., generative model designed to generate achievable goals (Florensa et al. 2018) or a discrminative model to quantify the achievability of goals.

---

Under review as a conference paper at ICLR 2023
14
17
20 105
23
25
35
23
Immestepe to solae mare (41000
OTA
Temsstops to solver maze (x40000
70-
QTA
VD5
12
36
19
23
25
23
13
17 Shortest path length
21
Shortest path length
Figure 2: Left: M-maze performance. Right: Square-war gquare-wave performance. Above we plot the number of timesteps required for various algorithms to attain evaluation performance of 1.0 across 12 seeds on various environments. Performance omitted for when seeds failed to converge for a particular environment. Lower on the graph represents better sample efficiency. Confidence intervals represent one standard deviation.
RESULTS AND ANALYSIS
Environments We utilize custom a suite of procedurally generated 2D mazes for dot agent with continuous action and state spaces. In all environments the agent receives reward of -1 unless it reaches, the goal to within a small radius, in which case it receives reward of 0. These mazes are carefully designed have minimum solving time T such that an optimal agent cannot solve the task in fewer than T timesteps. This reward structure and environment makes it such that the best total for reaching a goal is 11 * (T 1), which we use in our analysis below compute the optimal return.
These environments also allow us to evaluate the performance of QTA compared to baseline methods series of increasingly challenging environments, where each environment quantifiably more difficult than the last. Our first set of environments is set of square-ward wave mazes, in which the agent must traverse long path with single route forward. The square-wave -ave mazes allow us directly evaluate the most critical skills pertaining to sample efficiency: how quickly agent is able to explore new regions of state space and integrate information, while reintegrating information about any regions of the state space that catastrophically forgotten. Our second set of environments is an M-maze, which features a large, useless dead-end that takes up a large fraction of the state space and designed to waste that agent's time. While evaluating many of the same challenges, it additionally allows us to evaluate agents' sample efficiency when faced with an artificially enlarged state space.
Baseline algorithms We compare QTA's performance with two competitive curriculum baselines: MEGA (Pitis et al.] (020)) and VDS (Zhang et al., (020).. HER was not able to reach the goal in any of our environments within the time allowed. We used the authors' code and hyperparameters for MEGA and VDS, and tried to tune the hyperparameters to our environment. We were unable to improve performance further, likely because both algorithms were originally tuned for very similar 2D navigation environments.
As baselines for comparing PUNs performance, we evaluate QTA with two popular epistemic uncertainty estimation algorithms: Bootstrapped Q Networks (Osband et al., 2016), and Deep Ensembles (Lakshminarayanann
al., (017). We also evaluate PUN using the discrepancy among Q values as an epistemic uncertainty estimate in order to evaluate the efficacy of predicting next observed states with PUN.
By accurately estimating the epistemic uncertainty of the agent, QTA is able to navigate mazes using significantly fewer timesteps faster than MEGA, as shown in figure 2 MEGA, which ignores the epistemic uncertainty of the model and largely relies high-level statistics about visited states, did not induce learning as quickly as QTA's methodology of taking into account the epistemic uncertainty. Interestingly, horizons and large state spaces, MEGA begins to approach the performance of QTA, which may be due to the predictive network overfitting when predicting the agent's epistemic uncertainties.
Main results
that as the time horizon continues to expand to very long

---

Under review as a conference paper at ICLR 2023
20
23
25
35
41
14
23
32
38
temestege To 106R TiKe (X00000
OER+
Temestops to solee maze (A10000)
GTA
50
20
12
36
19
23
25
23
14
21 Shortest path length
17
Shortest path length
Figure 3: Left: QTA with and without our customized PER. Right: QTA vs standard PER. We plot the number of timesteps needed to attain evaluation performance of 1.0 across 12 seeds in M-mazes of various lengths.
We find that VDS is competitive first, but quickly becomes less sample efficient and then fails to solve many environments. We suspect that VDS is only competitive at the beginning because VDS has access to privileged information about the entire state space, including states not yet observed.
We also observe that all algorithms perform much worse and scale more poorly on square- wave- shaped environments, despite having much shorter mazes to solve with much smaller state spaces. The greater number of sharp turns required to solve the square-wave mazes hurts sample efficiency, as the optimal value function harder to approximate.
Epistemic Uncertainty for Sampling Replay Transitions We further demonstrate the importance of estimating the agent's epistemic uncertainty by showing that can used as signal to prioritize certain replay transitions, shown in figure In the left figure, we show that introducing PER offers some performance benefits with longer mazes before the predictive network begins to overfit. In the right figure. We demonstrate that standard PER with cunporal-differemt error for prioritization offers no significant performance benefits. Prioritizing based on temporal difference was not helpful to the agent. Prioritizing with epistemic uncertainty accelerated the agent's learning, especially in navigating longer mazes.
1.0
Test Accurecy
0.6
02
-0.2
20000
40000
60000
80000
100000
120000
140000
Timesteps
Figure 4: We add gaussian noise to PUN's uncertainty estimates. Blue: no noise. Orange: standard deviation=0.01. Green: standard deviation=0.03. Red: standard deviation=0.05
Verifying QTA's Approach Next, we verify that QTA is forcing the agent to minimize uncertainty and not simply choosing goals that are incrementally further away or following any other strategy. To demonstrate this, as shown in figure we add small amounts of gaussian noise to PUN's uncertainty estimates and find that performance suffers significantly and ultimately collapses with even a tiny amount of added noise. We demonstrate that the performance of QTA hinges on following highly precise estimates of epistemic uncertainty and that QTA selects goals strictly based on epistemic uncertainty.
7

---

Under review as a conference paper at ICLR 2023
Tirnntampe
Valuee Fanctien
Absslitee Sirer
Matue Poavtion
Optimal Pundtton
Pred ctive Eeeer
Goal Campllnng
Top 200 Camples
-0.925
1,78
38429
0.107
0.070
20K
0.099
1348
23K
0.0OD
Figure 5: Analysis of the ensemble of critics for one seed on a 12x5-unit M-maze, From left to right, in each column we show: 1) the value function for all states averaged across all critics 2) the absolute value the error between the average value function and the optimal value funetion, 3) the optimal value function 4) the average error among the three predictive heads 5) the probability of state being sampled as the agent's next goal 6) 200 sampled states based on epistemic uncertainty. We emphasize to the reader that across time all color scales are the same across time except for the predictive head error, which changes over time to make the contrast more visible.
Analysis of critic networks In figure we visualize the critics of one of our agents learning over time in an M-maze. The agent starts in the bottom-left inlet, shown in figure We demonstrate that our neural networks are performing as we would expect. Firstly, upon the analysis the value function,
that the map generally gets lighter over time, indicating that the estimated Q values increase over time as the agent's 's competency increases. Naturally, some areas of the value manifold are better learned than others.
Comparing this learned value function with the optimal value function yields the absolute error map shown in the second column. We show that in general the error starts high and uniform, and decreases
8

---

Under review as a conference paper at ICLR 2023
Area of stade SOACE
PUN iours!
soatshrpped DOlo
Deep Ensomitts
QTA
(41000)
03000
62000
00
40000
20
10
17
Shartend pathi tmrgth
Shartest
Figure 6: Left: Performance of QTA using PUN uncertainty vs using disagreement among Q values for PUN on various M-mazes. Right: QTA with PUN other critic ensemble architectures across M-mazes of varying lengths. All experiments averaged from 12 random seeds. Error bars indicate one standard deviation.
over time. As expected, the error is greater in states further away from the starting point, as those states have been visited less frequently and been sampled fewer times in gradient updates than those closer to the start.
By analyzing the estimated epistemic uncertainty in the fourth column, we gain insight into how the agent selects goals. The fourth column contains a spatial representation of the agent's epistemic uncertainty where brighter colors indicate greater uncertainty. We simultaneously discuss the fifth column, showing the likelihood of any particular point being sampled as a goal for the agent. We from the average predictive head errors that after only five-thousand iterations, the network already understands and has explored a large region of the maze. We can see that the network least understands the furthest regions of the maze. This is demonstrated by the network sampling goals only in the regions of the maze furthest from the start. As time progresses, the network overall becomes more certain and the magnitude of the average forward error decreases significantly but is punctuated by occasional catastrophic forgetting events. We point out to the reader that the visual scale of the fourth column is unique to each image, as indicated to the left of the maze graphic, in order to make the contrast more visible to the reader. We see that the goal sampling likelihoods in the fifth column track very closely with the predictive head errors in the fourth column, demonstrating that QTA samples goals in the most epistemically uncertain locations for the agent.
In the final column we have representative sampling of 200 goals, and see that they tend to be targeted toward areas of the maze that are far away from the origin, tracking very closely with the goal sampling likelihoods.
Q value disagreement PUN utilizes the disagreement among predictive networks' outputs to estimate epistemic uncertainty. A more intuitive choice might be to use the disagreement among Q values. We experimented with using the disagreement among values rather than the predicted next states as in but found it to an inferior estimator of epistemic uncertainty, leading to dramatically worse performance. By analyzing the sources of noise, prediction to he less noisy and thus more stable than the prediction for the Q value. By the definition of the value ((s:) Fr Q((5411 at+1), there are three sources of noise in the estimation of the Q value: 87 de 1 and d,Q(+/, 1.#1+1) When predicting s', the predictive network's 's output has only one source of error: 5.r. Naturally, the signal-to-noise ratio will be much higher with the predictive heads' output. A very low signal-to-noise ratio is consistent with the analysis in ZZhnn al. 2020).. Further, we intuitively anticipate 3' to be a good predictor of the quality of the critic's latent representation since the accuracy 3/41 is essential for accurately predicting QSt,, at).
that would anticipate the next state
Epistemic Uncertainty Estimation Techniques In figure we compare QTA with PUN to analo- gous implementations using bootstrapped Q networks and deep ensembles. All are implemented with next state prediction discrepancies uncertainty estimates. QTA performs best when it uses the epistemic uncertainty estimates from PUN, demonstrating that estimates from PUN the most accurate of the approaches available.
three crities. With all algorithms

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement We take the reproducibility of our work with the utmost seriousness. For this reason we include all of our code, along with instructions for running our code, in the supplementary materials submission. All experiments were with 12 random seeds and all plots include confidence intervals to make our results clear. We also provide implementation details and hyperparameter settings in the Methods section and Appendix.
REFERENCES
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abs/1912. 2019.
Homanga Bharadhwaj. Animesh Garg, and Florian Shkurti. Leaf: Latent exploration along the frontier. CoRR, abs/2005. 10934, 2020.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Elliot Chane-Sanc Cordelia Schmid, and Ivan Laptev. Goal-conditioned Reinforcement Learning with Imagined Subgoals. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In ICLR, 2021.
Sammy Christen, Lukás Jendele, Emre Aksan, and Otmar Hilliges. Learning functionally decomposed hierarchies for continuous control tasks with path planning. IEEE Robotics Autom. Lett., 6(2): 3623-3630, 2021.
Gabricl Dulac-Amold, Danicl J. Mankowitz, and Todd Hester. Challenges of real-worll reinforcement learning. CoRR, abs/ htss/1994.1220 12901, 2019.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Proceedings of the 35th International Conference on Machine Learning, pp. 1515-1528. 2018.
Leslie Pack Kaelbling. Learning to achieve goals. In Proc. of IJCAI-93, pp. 1094-1098, 1993.
Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http::tt::///c htRL///Epps//ppccceeiggg...
cc/ cc/paper/20i 2010 1010011//ieeee htt////1eeeefffefe 9ef2ed4b
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, and Birgitta Langle.. Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review.
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. Count- based exploration with the successor representation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020.
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben DaS Arthur Petron, Henrique Ponde de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojeiech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation, 2021.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. CoRR, ab2/1602.04421, 2016. URL
wwwpab..rrgaab...ooo 1602
04621
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pp. 1062-5071. PMLR.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning. In Proceedings of the 37th International Conference Machine Learning, 2020.
Matthias Plappert, Marcin Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs// 802.09642 2018
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning deep RL in continuously parameterized environments. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.). 3rd Annual Conference Robot Leurning, 2019, Osaka, Japan, October 30- November 1, 2019, Proceedings, volume 100, pp. 835-853, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. CoRR, abs/0812.4360, 2008. URL httt:tt/aaaxiii org/ abs// 08120 4360
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanetot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, 1lya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016. URL http: html
www. www.nnac. nature.
Niko Sünderhauf, Oliver Brock, Walter J. Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Uperoft, Pieter Abbeel, Wolfram Burgard, Michael Milford, and Peter Corke. The limits and potentials of deep learning for roboties. CoRR, abs/1804.06 1148806006777 2018.
11

---

Under review as a conference paper at ICLR 2023
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic Curriculum Learning through Value Disagreement. In Advances Neural Information Processing Systems, volume 33, 2020.
A
APPENDIX
A.1 HYPERPARAMETERS
Parameter
Value
Batch size
1024
Network hidden size
256
HER replay k (Andrychowicz al. 2017
Discount factor
0.99
Actor learning rate
le-3
Critic Learning Rate
2e-3
Predictive Network Learning Rate d (random actions)
5e-3
8 (16 for deep ensembles and bootstrapped DQN)
Initial random steps
1e3
0.2
Prandomaction
0.3
с (threshold)
-1.6
Linear sampling M
626
Linear Sampling
-591
0.95
PER Binstial
0.3
PER 8 steps
2c5
PER a
1.0
12


Reviews:
[{"review_id": "sFJJ-zekIvw", "reviewer": "Reviewer_beUS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In recent years, goal conditioned reinforcement learning approaches have received a lot of attention in the literature. However, curricula for selecting goals during training have predominantly been based on heuristics, which still lead to poor sample-efficiency in realistic scenarios. The paper proposes using curricula based on the agent's epistemic uncertainty. An agent's epistemic uncertainty is it's lack of knowledge. Using this epistemic uncertainty to select goals provides a natural curriculum where the agent gradually acquires knowledge. The authors propose predictive uncertainty networks, a novel uncertainty estimation method for quantifying the agent's epistemic uncertainty. The uncertainty estimates are used as scores in a prioritized replay-based mechanism for making gradient updates. Additionally the replay buffer also involves hindsight experience-like goal selection. Finally the authors implement the proposed algorithm in a procedurally-generated 2D mazes.", "summary_of_the_review": "in summary,  while the presented approach is interesting and is accompanied by a through and insightful empirical analysis, the significance of the approach is limited by simple environments and limited novelty. Results on more diverse environments, for instance from bsuite [1], would improve the claims of the paper significantly. I encourage the authors to incorporate the feedback during the discussion and rebuttal. \n\n[1] Behaviour Suite for Reinforcement Learning", "strength_and_weaknesses": "**Strengths**\n\n- Leveraging the epistemic uncertainty of the learner to establish a curriculum of goal for training is a neat idea and is a natural way to characterize the learning process - the agent gradually learns to achieve goals that it has high uncertainty over - leading to improved sample efficiency. \n- The empirical analysis in the paper is quite thorough and well done. \n\n**Weaknesses**\n\n- In my view the main weakness of the paper is the limited experimental setups considered. The experiments are done only on 2D mazes which do not seem sufficient as evidence for the claims made in the paper. \n- Using the uncertainty estimates directly has the potential to suffer from the noisy-TV problem. That is, states of higher-entropy might be selected as goals even though they do not serve as informative goals for the agent. \n- The idea of using epistemic uncertainty to guide exploration in reinforcement learning is not particularly novel and has been explored in the past [1, 2]. [2] also addresses the previous point as instead of simply using the uncertainty to weigh the points, they define an acquisition function to quantify the information gain. \n\n[1] - DEUP: Direct Epistemic Uncertainty Prediction\n\n[2] - An Experimental Design Perspective on Model-Based Reinforcement Learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mTkM_68FP14", "reviewer": "Reviewer_tn6E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new reinforcement learning algorithm: Query the Agent.\nThis algorithm uses uncertainty estimates derived from an ensemble actor-critic setup to bias replay and learning around more uncertain states and actions.\nThe authors support their claims mainly through evaluation on an \"M-maze\" experiment, where QTA outperforms baselines.", "summary_of_the_review": "This paper presents a new RL algorithm designed to leverage epistemic uncertainty estimates for improved learning.\nOverall, I think there are several things to like about the paper, but I'm left with more of a feeling that these improvements are more incremental than transformative.\nThis could be a reasonable paper in the conference, but I think it could benefit from more clear and decisive results and/or comparisons to existing work.", "strength_and_weaknesses": "There are several things to like about this paper:\n- The overall quality of writing and polish is pretty high.\n- The core QTA algorithm seems reasonable, and some of the pieces of network design/modifcations to ensemble uncertainty to form \"PUN\" seem useful.\n- The experiments and analysis are quite well presented.\n- The accompanying code and reproducibility appears to be extremely high.\n\nHowever there are a few places where the paper could be improved:\n- The improvements relative to baselines (MEGA, BootDQN, ...) all seem to be more incremental than transformative. Typically, when people look at these difficult maze-style domains you really care about *scalability* rather than pure performance (since these are just toy problems after all)... in terms of scalability, it does not seem very significant differences as the problems grow.\n- Related to the point above, these experiments seem a little \"adhoc\"... I would have liked to see something like a regret analysis/proof for tabular settings and/or an evaluation on some kind of scalable domain like DeepSea (behaviour suite for reinforcement learning).\n- The discussion of related work, while pretty comprehensive, seems quite piecemeal.. I think it would be possible to consolidate and digest some of the related approaches (note that many of these are essentially ensemble-based uncertainty). It would be nice to get some kind of bigger-picture evaluation of what types of uncertainty and techniques really are what make QTA valuable versus these other approaches. Looking at something like \"Deep Exploration via Randomized Value Functions\" (Osband et al) might be a good overview of the type of approximate posterior sampling that many of these algorithms instantiate..\n- Some pieces of the algorithm/design could benefit from being spelled out more explicity. For example, action selection is only described via reference to the DDPG paper... I think this should be made more explicit... what type of exploration policy is used in action selection? I think this is unusual if it relies on the same kind of boltzmann/dithering as DDPG... could not scale to large domains that require deep exploration.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hT6wLUbHyg", "reviewer": "Reviewer_6NMf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\"Query The Agent\" is an improvement on curriculum learning for deep reinforcement learning. If I understand, to remedy the data inefficiency of RL, this approach applies \"curricula\" in the form of intermediate learning goals that are informed by the epistemic uncertainty of the agent, e.g. the agent's lack of understanding of parts of the state space.  The agent benefits by exploring (trying to achieve) appropriate goals -- those not to easy or too hard. Uncertainty is represented by running multiple critics.  \"Epistemic uncertainty\" is estimated by running multiple critics, based on a neural network predictor called a \"predictive uncertainty network\" and looking at their standard deviation.   \n\nThe modifications tested in the paper show measurable improvement compared to recent \"MEGA\" and \"VDS\" methods. ", "summary_of_the_review": "The high level impression from the paper is of an minor improvement made among several related recent approaches in a simulated environment.  I appreciate the value of simple experiments to illustrate a point, but wonder where the applicability might be to less specific and more realistic problems.  To truly evaluate the paper one needs to understand the related \"baseline\" work from a few specific papers from the past few years to which the current work is compared. ", "strength_and_weaknesses": "Granted no familiarity with the few recent papers on which this work depends, my sense is that this is a study of an incremental exploration of a set of proposed modifications to RL critics demonstrated in a toy domain.   It's not clear what the larger relevance and value of this work is.  The paper may be \"correct\" but the take-aways appear marginal if any. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AiCoZiPwGdU", "reviewer": "Reviewer_dsJh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new goal selection algorithm for training curricula of goal-oriented reinforcement learning agents. The algorithm, called query the agent, estimates epistemic uncertainty of the agent in the state space and sets goals in areas with higher epistemic uncertainty. The algorithm is evaluated on a set of generated 2-dimensional mazes.", "summary_of_the_review": "Interesting research direction, however the results as presented are too early for  publication.", "strength_and_weaknesses": "Strengths: the paper considers application of techniques of adaptive active learning to goal-conditioned reinforcement learning problems. The idea is explained in great detail, compared with other ideas to account for uncertainty in the literature. The supplementary material is provided, with the code to reproduce all experiments in the paper.\n\nWeaknesses: the paper claims that\n\n1) goals should be set with regions with high epistemic uncertainty;\n2) states are described by real vectors;\n3) the mean of standard deviations of each component of the state vectors is a good measure of epistemic uncertainty.\n\nThe first claim seems to be a feasible option, but unsupported. An alternative option might be, for example, setting goals such that paths pass through regions of high uncertainty (rather than end in them). I do not argue that  a different option can be better, I just point that the choice to set goals where uncertainty is high seems to be rather arbirtrary, unless supported at least by some discussions.\n\nThe second claim seems to cover many applications of deep reinforcement learning, however it must be stated explicitly. There are models with other state representations.\n\nThe third claim is problematic. There is extensive research in quantify sample-based uncertainty (just one citation for example, https://link.springer.com/article/10.1007/s10994-021-06003-9).  Even if  second-order approximation of the distribution is used, the covariance matrix is only poorly characterized by the sum (or the mean) of the diagonal elements.\n\nThe ideas behind the algorithm seem to possibly usable, but unsupported theoretically. An alternative would be to support than experimentally. However, the experiments are performed only on a set of relatively small 2-dimensional mazes. One does not really need deep neural networks with complicated architecture to solve those mazes, so while they can be used to confirm that the idea basically works, they are not a sufficient empirical evaluation for the above heuristic claims. One obvious concern, but there are others, is that mean standard deviation of components will probably work in 2 dimensions, but it will have no chance to work in 20 or more. In high dimensions L2 norm between points in the space vanishes to be informative. It is called 'the curse of dimensionality'.\n\nFinally, judging by both the text of the paper and by the code in the supplementary material, the algorithm has dozens of hyperparameters, (probably set GIVEN the data set of mazes, btw). It is not clearly why those values were chosen, and how to chose the best values in general. Without that, the algorithm is not going to be generalizable.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject