Empirical Sufficiency Featuring Reward Delay Calibration
Authors: Yixuan Liu
Hu Wang
Xiaowei Wang
Xiaoyue Sun
Liuyue Jiang
Minhui Xue
Venue:  (2021)
Keywords: Deep Reinforcement Learning
Reward Calibration
Empirical Sufficiency
Overfitting.

Abstract: Appropriate credit assignment for delay rewards is a fundamental challenge in various deep reinforcement learning tasks. To tackle this problem, we introduce a delay reward calibration paradigm inspired from a classification perspective. We hypothesize that when an agent's behavior satisfies an equivalent sufficient condition to be awarded, well-represented state vectors should share similarities. To this end, we define an empirical sufficient distribution, where the state vectors within the distribution will lead agents to environmental reward signals in consequent steps. Therefore, an overfitting classifier is established to handle the distribution and generate calibrated rewards. We examine the correctness of sufficient state extraction by tracking the real-time extraction and building hybrid different reward functions in environments with different levels of awarding latency. The results demonstrate that the classifier could generate timely and accurate calibrated rewards, and the rewards could make the training more efficient. Finally, we find that the sufficient states extracted by our model resonate with observations of human cognition.

Under review as a conference paper at ICLR 2021
EMPIRICAL SUFFICIENCY FEATURING REWARD DE- LAY CALIBRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Appropriate credit assignment for delay rewards fundamental challenge various deep reinforcement learning tasks, To tackle this problem, we introduce a delay reward calibration paradigm inspired from classification perspective. We hypothesize that when an agent's 's behavior satisfies an equivalent sufficient condi- tion to be awarded, well-represented state vectors should share similarities. To this end, we define an empirical sufficient distribution, where the state vectors within the distribution will lead agents to environmental reward signals in consequent steps. Therefore, overfitting classifier is established to handle the distribution and generate calibrated rewards. We examine the correctness of sufficient state ex- traction by tracking the real-time extraction and building hybrid different reward functions in environments with different levels of awarding latency. The results demonstrate that the classifier could generate timely and accurate calibrated re- wards, and the rewards could make the training more efficient. Finally, we find that the sufficient states extracted by our model resonate with observations of hu- man cognition.
INTRODUCTION
Reinforcement learning (RL) approaches have made incredible breakthroughs in various do- mains (Silver et al., 2016 Mnih et al.] 2015) OpenAl et al.] 2019] Vinyals et al. (019), where the performance exceeds people's i's expectations. The reinforcement learning theoretical basis mod- els sequential decision tasks as dynamic programming processes to maximize expected accumulated rewards. Given that environmental rewards generally cannot entirely reflect the contribution of each action in step, existing approaches commit to distributing different credits to individual decisions, known as credit assignment (Sutton & Barto, 1998). Bellman equation- based architecture calculates value of a state based on the gathered rewards in the future, which times assigns an unreason- able value to prior states. This problem becomes even more intractable when reward signals are extremely sparse severely delayed.
In this paper, we formulate an overfitting classification mechanism to extract empirical sufficient conditions of acquiring desired environmental signals. We refer to this extraction formulation as an Empirical Sufficient Condition Extractor (ESCE) to fairly assign delayed rewards to corresponding states. In doing, we first propose classification mechanism identify empirical sufficient states. To train a classifier with partially labeled data, we label the state vectors with matched environmental signals. We then train the classifier with two phases, wherein novel overfitting training process is conducted. In addition to existing value-based estimation, the ESCE provides concrete predictions. We equip the ESCE with Asynchronous Advantage Actor Critic (A3C) algorithm (Mnih al. 2016) and measure the performance on six Atari games, most of which have delayed discrete rewards. We comprehensively examine the extraction correctness by formulating different reward functions, and further track the accuracy/recall of ESCE on the fly. The results show that agents guided by our empirical efficiency achieve significant improvements in convergence, especially in the scenarios with delayed rewards. Furthermore, we constructively modify the environment to render the rewards even to be more delayed, termed as hindsight rewards settings. The results show that equal calibrated rewards could lead agents acquire well-learned target policies as if rewards not delayed. In addition to quantitative experiments, we screenshot the identified sufficient states, showing high similarity with human cognition. Our contributions can be summarized as follows:

---

Under review as a conference paper at ICLR 2021
We introduce overfitting classification model extract empirical sufficient conditions, where overfitting mechanism could significantly reduce uncertainty.
We formulate calibrated reward signal in line with the environmental targets tackle the reinforcement learning reward delay issues, such that the rewards are provided when empirical sufficient conditions are satisfied.
The experimental results show -allbbbaee agents are able to learn decent target poli- cies in the scenarios where rewards have been severely delayed. The identified sufficient conditions empirically resonate with the "true" environment targets.
2
RELATED WORK
2.1 INTRINSIC MOTIVATION
Intrinsic rewards (Singh et al. 2004 Ryan & Deci, 2000), inspired by intrinsic motivation, are primarily introduced to encourage exploration. The rewards mechanism is largely independent of environmental rewards. Intrinsic Rewards in the exploration-oriente mechanism are generally cor- related to the novelty or informative acquisition of new arrival states (Pathak et al. 2017 Burda et al.) 2019 Houthooft et al. 2016 Zhang et al., 2019). Due to the awarding mechanism does not depend on environmental rewards, as exchange, the policy may not align with the environmental target. In addition to exploration, intrinsic rewards can often be found in hierarchical frameworks (Kulkarni al., 2016, Vezhnevets al. 2017 Frans et al.) 2018). Moreover, intrinsic rewards also used to assist agents to more directly learn optimal or near-optimal policies (Wang et al.] [2020) iZheng al.) 22018: 2019). Likewise, following down this branch, the ESCE developed this paper could generate empirical intrinsic rewards to learn better policies, without encouraging exploration.
2.2 CREDIT ASSIGNMENT FOR DELAYED REWARDS
Most evaluation mechanisms in reinforcement learning rely on Bellman equation, where the en- vironmental signals are passed across states in sequences (Lee al.) 2019; Arjona-Medina et al.
2019 Ng et al. 1999 Marom & Rosman 2018). To make the training more efficient, one effective direction is build an extra mechanism
capture critical states and to emphatically regress on
these states (Sutton et al. 2016 Ke et al. 2018 Hung et al. 2018). Ideologically, Irpan et al. (2019) introduce binary classification into evaluation, and positive-unlabeles learning (Kiryo al. al.] 2017) adopted to distinguish promising and catastrophic states. Our work evaluates states by discriminat- ing states as simple binary classification problem without relying on Bellman equation. Specially, accurately differentiate states between "sufficient for success" and "insufficient for success". by
developing a new overfitting classifier (see Sections 3.3 and 3.4).
REWARD DELAY CALIBRATION
3.1 LEARNING WITH HYBRID REWARD FUNCTIONS
As an independent module, the proposed Empirical Sufficient Condition Extractor (ESCE) can be incorporated into multiple mainstream reinforcement learning frameworks. Calibrated rewards are provided by ESCE when state meets the empirical sufficient condition. We denote ( 0p) as the learned policy, where 0p is the set parameters of the policy network; rf is the calibrated reward generated by ESCE and rf an environmental reward from the environment, at time step t. Both reward signals have the same scale. The total reward function is synthesized with calibrated signals and environmental signals, rt arf + Br where and 3 are the weight coefficients of corresponding rewards. Our baseline. optimized by environmental rewards, has coefficients a = O and B I. The policy network optimized maximize expected accumulated rewards, as shown below:
max (T((, Verr. 0p
(I)
2

---

Under review as a conference paper at ICLR 2021
R
Pluse oce Minibash
Randenty
Dowwing
Prodina laneling
Poutra Pook
Endrcoment
Phase-one
Binary
Vegatioe Post
Ragarn
Deedring Trakiig
Phac-tan Mimbarzh
Rasdorb
Dunvay
Sanplo
ISCE
repall
ESCO
Updasing
Ageent
Figure 1: The overall framework of a RI. agent equipped with ESCE. Agents receive environmental rewards and calibrated signals from ESCE as its total rewards. The policy network and ESCE are two independent models. The ESCE examines each state and provides calibrated rewards when the state is considered empirical sufficient state. Within the ESCE training process, state vectors are automatically labeled with our proposed novel labeling method, and are then stored in corresponding pools. The ESCE network is updated with overfitting training, where phase one is binary classification with data from pools and Ruegative pools, and phase two is the proposed overfitting training updated with Raegative data alone.
The overview our framework illustrated Figure | Whenever an empirical sufficient state identified by ESCE, a positive reward is offered. The calibrated reward is available after receiving an environmental signal. Meanwhile, these states are labeled and stored in pools. The training of the ESCE and policy network proceeds alternately. We take input raw image pixels without any data preprocessing.
3.2 EMPIRICAL SUFFICIENT DISTRIBUTION (ESD)
The occurrence of a partieular event's sufficient condition always followed by the occurrence of its corresponding event. Inspired by this, we try to resolve the reward delay issue in reinforcement learning from a similar perspective. We consider a set of particular environmental signals as tar- get events, and utilize the information from state feature vectors to yield the sufficiency to incur these target events. We hypothesize some vectors are closely distributed to each other, if: (1) the state vectors contain all critical information, and (2) the state vectors include the information of same sufficient condition. We therefore use this hypothesis to proceed with the classification evaluation.
Definition 3.1. Empirical Sufficiency. We define empirical sufficiency if there exists bounded space such that all reachable state vectors inside always imply a particular environmental reward signal.
Given a stable environment, let Rpositive be desired environmental signals, such as positive rewards. Conversely,
Rnegative represent undesired environmental signals, including negative wards, agent's 's deaths, game endings, etc.
We define Policy-based Empirical Sufficient Distribution (ESD-policy) Agents explore the vironment with a specific policy; if an agent reaches a state s invariably acquiring R Rpositive. we take state s as policy-based empirical sufficient state acquire Rpositive- all reachable states in bounded distribution are empirical sufficient states, then we consider the distribution as Policy-based Empirical Sufficient Distribution of Rpositive. In this case, the ESD-policy effective if: (1) the environment remains stable, and (2) the policy remains stable. If the policy is upgraded, corresponding ESD-policy may change as well.
3.3 LABELING
We implement ESCE training with binary classification mechanism and take individual raw pixels of states as input. For labeling, we automate this process to extend environmental signals as labels for classification training.

---

Under review as a conference paper at ICLR 2021
We first cut long sequence of states into rounds along with environmental signals, and use these labels as break points of rounds. Furthermore, these environmental signals are adopted as the labels for states within round. Specifically, label for states starting from the last environmental signal to the newly received signal. The labeling procedure is shown in Figure] We generalize all desired environmental signals Rpositives and all undesired environmental signals as Anegative. Correspondingly, Rpositive and Rnegative pools are created for collecting labeled pixels.
use newly received environmental signal mutual
As we adopt nearby environmental signals as labels, another problem is label ambiguity. For the states whose endings are determined yet, the labels can be converse different episodes, such as beginming states. To put it differently, the state vectors of two different labels might be densely mixed, and there might not a clear boundary between them, as opposed supervised training, where every sample has only one corresponding label. Our classification training procedure is in- spired by the label ambiguity. Regarding Definition [3.1] ESD should only contain states that lead to one particular environmental signal. This indicates that ESD should only include states with unique labels. We formulate ESCE training as an overfitting training process to exclude state space with ambiguous labels.
3.4 OVERFITTING TRAINING
We formulate two-phase training process extract pure distribution. In phase one, we expect ESCE to assign a dominant probability to real empirical sufficient states since ambiguous states may fool the classifier. In phase two, exclude insufficient states, an overfitting training mechanism is adopted to update the decision boundary.
We measure performance and maturity of ESCE with Precision and Recall on Rpositive. The Recall roughly indicates the ratio of identification coverage on those samples associated with Rpositives and the Precision roughly shows how accurate the identification is. We set the number of rounds that include identified states as Nident, and let Naufficient denote the number of rounds that contains identified empirical sufficient states and leads Rpositive- Additionally, we set the total number of samples leading to Rpositive acquired Npos.
The phase one is carried out by binary classification training. The training data are from both Reositive and Rnegative pools, where all Rpossiive samples (spos) have desired signal labels (Tpos), and Rnegative samples (5 have undesired signal labels (Tneg). We use both types of samples to prudently maximize Recall the basis that ESCE parameters dg are trained with an appreciable amount of representative samples (Figure 2(a)). Let the learning function f generate estimates of foreseeable reward with ESCE network parameters ÓF. where pos and fneg are respectively generated with Spos and &nog. as defined below:
ppos = of {Spoosi 01s)
(2)
fneg f (sneg: BE)
(3)
To reflect the real network identification of samples, the output is returned by the maximum likeli- hood estimate of the soft-max distribution rather than sampling. The loss of phase one measures the discrepancy between estimates and foreseeable signals, which are defined follows:
Nsufficient Nident
max (Precisionpasitivee dg
(4)
min BE
Lope (fpass Ppos) + (Fneg, Fneg)
In phase two, we maximize Precision with complete Rnegative samples. The classification boundary updated to acquire a distribution of pure Rpositive samples by excluding all insufficient samples. The signal that the ESCE has sufficiently excluded insufficient distribution appears when the Recall of Rnogative sample batch reaches 100% (σ) (see Figure ((b).. The loss of phase-two process is defined as follows:
4

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure 2: 2(a) A schematic diagram of decision boundary after phase-one overfitting training: the classifier will try to make most of the samples correctly distributed both sides of the boundary. However, there still quite a few promiscuous samples falsely allocated duc to label ambiguity. [(b) A schematic diagram of decision boundary after phase-two overfitting training: all state vectors labeled with Rnegasive are excluded from Regaative boundary, which matches the definition of ESD-policy. [Tc] A schematic diagram of sensitive Sampling: the actual ESD-policy may change with the policy's 's updating. The state vectors inside the gray region insufficient states, and then turn into empirical sufficient states. The ESCE primarily updated with these dynamic samples.
Nautficient
max (Recall 6g
= max 0g
N
pas
(5)
~ min o L
two
(fueg Tneg).
Ug
Once phase-two process is terminated, the states recognized as Rpositive should only include states with label Rpositive- In other words, all states identified as ros should have resulted in Rpositive in line with the definition of empirical sufficient distributions. If the network updated with the policy and the sampling is representative, the distribution identified would be the Dolicy-based Empirical Sufficient Distribution (ESID-policy). Theoretically, if stochastic policies are adopted without any bias, Task-Specific Empirical Sufficient Distribution (ESD-task) may be acquired. We show the ESCE architecture in Algorithm I, which can be found in the Appendix.
The optimization for overall architecture could be derived through combining Equations
and
(- En(ss0p)) L + toL two
min 0p.0g
(6)
3.5 SENSITIVE SAMPLING
Given ESD-policy is based on a particular policy defined in Section 3.2 with corresponding policy's ' updating. To efficiently update the ESCE network, we adopt a sensitive sampling strategy for overfitting training.
ESD-policy might change
For ESD-policy, the empirical sufficient distribution of Rpositive should change as the agent is up- dated. For instance, if an agent becomes more skillful, may stably acquire rewards which were previously unattainable. As result, the physical volume of ESD- ESD-policy would inflate. An efficient way to update the ESCE parameter 0g focus on the evolving data distribution (see Figure [ Accordingly, we build two sensitive state pools. One pool is for missed identification, indicating which rounds reach Rposiiive. without identified states in the sequences; the other pool is built for false identification, which collects the rounds containing identified pixels that did not actually reach Reositive. These two state pools force ESCE to focus on the variation of policy, either new strategy learned strategy forgotten due to the network update. Approximately 75% of training data imported from two sensitive pools.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison with baselines hindsight rewards settings: = 1 .ri 0..r. Rewards offered after a Anegative signal or the end of episodes; the real time Precision and Recall of Rpositive are synchronized with time steps, which are referred to as the right vertical axis.
Agents Environmental Rewards
Agents with Customized Rewards
Precision
Recall
FishingDerby-v00
Breakout-vO
Pong-Vi V0
Max
5.0
EXPERIMENTAL RESULTS
We primarily measure the ESCE component by building different reward functions, and compare to the baseline A3C+LSTM agents six Atari games. The original A3C LSTM agent only optimizes with environmental rewards, written as re = 0 0.rg +1++1 1f. The detailed experiment settings are laid out in Appendices and Further, examine agents' performance modified experimental settings, termed as hindsight rewards settings, where the summation of environmental rewards is provided after short episode.
Since it is unusual and risky to schedule overfitting as a part of the training, the intuition behind the design that the spatial expansion of the Rnegative identification potentially makes the space of Rpositive inevitably shrink. It is worth worrying how overfitting affects the recognition on Rpositive samples. We therefore try raise two research questions in our experiments:
Does the severely overfitting network perform well, and what are the results of the extraction?
RQI:
RQ2: How do calibrated rewards affect RL training?
We attempt to answer RQI in Sections 4.1 and 4.2 and answer RQ2 in Sections 4.3 and 4.4
4.1 PRECISION AND RECALL OF EMPIRICAL SUFFICIENT STATE TRAINING
To answer RQ1, we first examine the training process of empirical sufficient extraction statistically. The Recall indicates how widespread the identification is, and the Precision indicates how accurate the prediction is.
It is rare, although almost unlikely, that the boundary trained perfectly matches the "true" boundary of the sufficient distribution. This means the Recall and Precision of positive samples are the pair of trade-offs. Empirically, we fine-turned the hyper-parameter from 0.81 to which in turn keeps Recall and Precision both high. The changes of these two indices are recorded in Tables I and 2 in which the right vertical axes correspond the two indices. We also identify that Precision has strong positive correlation with the policy. Since the policy networks are randomly initialized at the beginning, the erratic performance makes the prediction difficult. In most games, both Recall and Precision could reach high values when the sampling policy becomes stable.
For Breakout-vo shown in Table 2 the Recall significantly reduces when agents on average get more than 40 marks in the game as the bricks hit by the pellet could be vastly different in every episode. Thus, when compared with other games, there are more state variance in Breakout-v0 environment. Given that the overfitting mechanism is adopted in our model, its ability to eliminate negative samples differentiates across models. Therefore, when compared to other networks that do not account for overfitting, the ESCE model is more sensitive state change.

---

Under review as a conference paper at ICLR 2021
15
ICAE Fred cisn
Cowrserrrnll Saverts
Value Furction Trained winh NICE
Value Turction Truined with Entroementa Rewarce
100
12
50
(a)
(b)
(c)
(d)
The calibrated rewards and environmental rewards are encoded and yellow, and value estimation (Critic) trained with calibrated rewards and environmental rewards are encoded in green and red, respectively. The blue line displays that ESCE identifies a empirical sufficient state when the agent hit the pellet with the edge of the bat (right bat), which significantly increases transverse velocity for the agent win the game, and makes unsolvable for opponent. However, value of baseline (red) increases continuously until receiving the reward returned by the environment, which far away from the decisive state. To the best of our knowledge, prior approaches could not make such precise prediction (bluc) on critical states. 3(b) In FishingDerbby-vl, most states identified when the agents' hook close to fishes when the pellet is close the bat the bat is the pellet's
Figure 3: 3(a)
fish is already hooked.
3(c] For Breakout-vO, most recognition polential trajectory. Fod Pong-vO, recognized states show the opponent is about to miss, or agents hit the pellet with the edge of the bat, which gives pellets has quick vertical velocity.
4.2 EMPIRICAL EFFICIENCY IDENTIFICATION
To further verify the correctness of ESCE, screenshot the extracted states three games. Most states identified exhibit high correlations with rewards acquisition and are visually close to human cognition. With the evolving of the agent policy, ESD-policy keeps altering, which results in the dynamically changed observations. In the initial episode, the empirical sufficient checkpoints recognized only a few steps away from the states where the actual rewards are given; with the policy becoming stronger and stabler, more and earlier states can be identified. We thus infer that well-trained ESCE model is capable of making accurate predictions through the deep understanding of the policy and environment. We visualize the screenshots of extracted states in Figure
4.3 REWARD DELAY CALIBRATION HINDSIGHT REWARDS SETTING
The latency in realistic environments be highly unpredictable. Since current experiments still largely unable to reflect the performance of different models on reward delay environments, further modify the environments force it to offer more delayed rewards, which we termed as hindsight rewards. In this setting, rewards are only provided an episode ends or after negative en- vironment signal. We compare the performance of two agents trained with environment rewards and calibrated rewards in the hindsight rewards settings, respectively. The results of the three games are shown Table In this modified scenario, agents guided by environment rewards can hardly make any progress, whereas the ones updated with calibrated rewards are able to learn distinctive target policies.
The reason behind this phenomenon is that the ESCE model helps to calibrate the reward by iden- tifying those states that meet the empirical sufficient condition. Given that the value is gained from the acquired rewards, rewards received from inappropriate states would thus mislead the value esti- mation. Figure ((a) illustrates the difference agent value estimation learned with environmental rewards and calibrated rewards. As seen from Figure 3[a). the value function (Critic) trained with environmental rewards and calibrated rewards are encoded red and green, respectively. The cal- ibrated reward (blue) is offered beforehand compared with the environment reward (yellow). This

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison between different reward functions and original environmental rewards. Row-one: 0.3 B-rri + 1. 1.rr. Row-two: ri = 1. + 0-rğ. The real time Precision and Recall of Hpositiva are synchronized with time steps. which are referred to as the right vertical axis. For Bowling-vo v0, the data size too small to provide valid Precision and Recall statistics.
Agents with Environmental Rewards
Agents with Customized Rewards
Predision
Recall
DerhingDerby
Breakout
Pong
Boxing
Asterix
Bowling
because the ESCE identifies an empirical sufficient state at the moment when the agent hits the pellet with the edge of the bat.
4.4 FULLY- AND SEMI-CALIBRATOO REWARDS
In addition modified hindsight rewards settings, we also build two reward settings to explore how the calibrated rewards affect the RL training process. The first is the semi-calibrated rewards setting, where the calibrated reward coefficient is set to 0.3 and the environmental reward coefficient is 1, written altogether T; 0.3 rf 1 rf. The second is the fully-calibrated rewards setting, where the reward function is r = 11.177 r + 0 O. The results are shown in Table
For the semi-calibrated rewards setting, the results show that small number of calibrated rewards could accelerate the training process. It is also clear that higher Recall and Precision are essential prerequisites ensure the effectiveness of calibrated rewards toward environmental targets. In Breakout-v0 and Boxing-vO,, the Recall can barely reach a high value, and we believe this is due to the large variance in their states (see Section 4.2). Essentially, for Boxing-vO the two players could be found in any position of the screen, while for Breakout-v0 the remaining bricks could be vastly different each episode.
For the fully-calibrated rewards setting, only calibrated rewards are provided when reaching an empirical sufficient condition. This ablation comparison examines the rationality of the time of awarding. The results show that agents trained with calibrated rewards can beat our baseline model in multiple different games. To acquire rewards FishingDerby-vl. agents need to move hook to catch fish, and then reel back the line before a shark eats the fish. It is common that the shark steals the fish if the agents have not learned strategies to quickly pull the hook up. In the first column of Table 2 the calibrated rewards significantly boost the convergence of the model, causing spike. This is because when the agent learns a stable policy of pulling the hook up, the empirical sufficient condition simply becomes hooking the fish. The ESCE provides instant calibrated rewards to the action triggering the state of successfully hooking the fish. This training process also resembles the human learning strategy by breaking down complex missions into easier sub-tasks.
5 CONCLUSION
In this paper, we formulate an approach to calibrating delay rewards from a classification perspec- tive. Due to the overfitting mechanism, the proposed ESCE model is capable of accurately extract- ing the critical states. Accordingly, the agents trained with calibrated rewards could assign supreme values to the critical states. In addition, the results show that agents trained with calibrated rewards could learn distinctive target policies in environments with extremely delayed rewards. Furthermore, the screenshots of extracted states show a strong correlation with environmental rewards, visually approaching the observations of human cognition.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. RUDDER: return decomposition for delayed rewards. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'lche-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13544-13555, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 02.001 540, 2016.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random net- work distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Kevin Frans, Jonathan Ho. Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5- 10, 2016, Barcelona, Spain, pp. 1109-1117, 2016.
Chia-Chun Hung, Timothy P. Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. CoRR, abs/1810.06721,, 2018
Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off off-policy classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neurul Information Processing Systems 32: Annual Conference on Neural Informa- lion Processing Systems 2019, NeurlPS 2019, 8- 14 December 2019, Vancouver, BC, Canada, pp- 5438-5449 5449, 2019.
Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C. Mozer, Chris Pal. and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through remind- ing. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 7651-7662, 2018.
Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive- unlabeled learning with non-negative risk estimator. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Infor- mation Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 1675-1685, 2017.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3675 3683, 2016.
Su Young Lee, Sung-lk Choi, and Sae-Young Chung. Sample-efficient deep reinforcement learn- ing via episodic backward update. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Informatinn Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 21 2110-2119, 2019.

---

Under review as a conference paper at ICLR 2021
Ofir Marom and Benjamin Rosman. Belief reward shaping in reinforcement learning. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.). Proceedings of Thirty Seerty-Secondda AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI- 18), and the Sth AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3762-3769 AAAI Press, 2018
Volodymyr Maih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540))552--33 533, 2015. doi: 10. 1038000ntnr 14236.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19- 24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928 1937. JMLR.org, 2016.
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. In Ivan Bratko and Saso Dzeroski (eds.), Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27- 30, 1999, pp. 278-287, Morgan Kaufmann, 1999.
OpenAI, Iige Akkaya, Marein Andrychowiez, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113, 2019. URL hettp://arxiv.oet wtap////////////
abs 0tbs//9900 071113
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney. NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMIR, 2017.
Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54-67, 2000.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):4844485 489, 2016. doi: 10. 1..1008/nature166611
Satinder P. Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforce- ment learning. In Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pp. 1281-1288 2004.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive com- putation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6.
Richard S. Sutton, Ashique Rupam Mahmood, and Martha White. An emphatic approach to the cemporal-difference
problem of off- oof-co temporal- difference learning. J. Mach. Learn. Res., 17:73:1-7::29, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Pro- ceedings of Machine Learning Research, pp. 3540-3549. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czameckii Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Damihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom I.. Paine, Çaglar Gülçehre, Ziyu Wang. To- bias Pfaff, Yuhuai Wu, Roman Ring. Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782)3350 354, 2019. doi: 10. 10.1038/s415860
Hu Wang, Qi Wu, and Chunhua Shen. Soft expert reward learning for vision-and -aan-ann--aanuage naviga- tion. CoRR, abs/2007. 10s/2007.10835, 2020.
Jingwei Zhang, Niklas Wetzel, Nicolai Dorka, Josehka Boedecker, and Wolfram Burgard. Scheduled intrinsic drive: A hierarchical take on intrinsically motivated exploration. CoRR, abs/19033 1es/1990307400, 2019.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 4649-4659, 2018.
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, and Satinder Singh. What learned intrinsic rewards capture? CoRR, 2019.
11

---

Under review as a conference paper at ICLR 2021
A APPENDICES
A.1 SPARSE AND DELAY REWARD SETTING
We evaluate our model and the comparing models on 6 Atari 2600 games on OpenAl Gym (Brockman et al. 2016). and uniformly choose vO of all games, where actions are not repeated. Positive rewards are defined as Rossiivee while the negative rewards, deaths, and game endings are Inegative- In some games, the environmen- tal rewards are offered a few sleps after empirical sufficient state. For instance, in game Bowling-vO, the environmental rewards are determined when the character throws the ball. Reinforcement Learning algo- rithms are inevitably facing the low sample efficiency issue, while the delayed rewards worsen the problem. The longer gaps in time between empirical sufficient state and reward are, the more challenge the agent has learn the "correct" policies.
A.2 A3C AND ESCE ARCHITECTURE
We adopt A3C LSTM framework as our backbone architecture. The original RGB value 210 160 image frames are converted to x 80 gray-sealc frames. Four continuous frames are stacked as the input. In A3C architecture, four convolution layers and max-pooling layers are adopted. An LSTM layer with 512 units is followed with two heads policy head and a value function head.
Considering the policy is evolving the fly, the policy-basse empirical sufficient conditions should be updated accordingly. To ensure the extracted empirical sufficient distribution up to date with policy, we train the ESCE model with data sampled from the latest episodes. Subject to this reason, the maximum capacity of the datasets for the training of empirical sufficient state classifier should be flexibly decided. On the other hand, training models in model-free settings request vast samples for convergence. Thus, it is necessary to expand datasets
cases. After trading-off between covering collected sample points, the capacity of Rpoative Anegative pools is set from 20,000 frames to 80,000 frames, sampled by 24 workers.
and minimizing the variance of
Algorithm 1: Empirical Sufficient Conditions Extractor (ESCE)
Initialize Extractor network and policy 0; Initialize Rpusitive pool and Rnegative pool;
repeat
Initialize temporary storage; while any state pool is not full do
if state is identified as Rpositive and calibrated reward have been given the
agent in this round then
Assign calibrated reward the agent;
end
Push states to temporary storage;
Update the policy network with the parameters 0;
environmental signal is not null then
Push temporary storage to pool based on positive/negative environmental signals;
Clear temporary storage;
Reset calibrated awarding status (Start round);
end
end
Update Extractor with states from both pools;
while Recall of Rmegative do
Update ESCE parameters with data from Rnegative pool;
end
Clear all pools;
until Converged;
12


Reviews:
[{"review_id": "OTWW0fFNwKQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes 1) classifying states according to their observed reward roll-outs and 2) using the resulting classifier for reward shaping.\n\nThis reviewer finds the work difficult to read.  For example, the term \"overfitting classifier\" exhibits cognitive interference with standard usage without explanation.\n\nThe use of binary classification seems most appropriate for environments with sparse rewards (both positive and negative).  Otherwise the concept of \"desired environmental signals\" and \"undesirable environmental signals\" from section 3.2 is ill-specified.  (While the reviewer can imagine a policy improvement scenario where much-better-or-worse-than-the-baseline is related to desirable/undesirable, this is not discussed.)  The experimental section is not informative with respect to the issue of defining desirable or undesirable.\n\nA better version of this paper would justify the technique analytically.  For instance, the optimal critic would provide the true Q-values as reward.  Under what conditions does a logistic binary classifier output a probability which is close to the true Q-value (e.g., with very sparse reward rollouts)?  Or alternatively under what conditions does the classifier induce a policy gradient [equation (1)] which is a descent direction?\n\nThe experiment section at the moment is structured as \"it works on these 6 atari games\", but this reviewer does not find this compelling (e.g., what about the other games?  do these 6 games have an obvious desirable/undesirable reward boundary?).  However, the claims about robustness to reward delay are interesting.  This reviewer encourages the authors to try their techniques on the RWRL challenge benchmark, specifically the challenges related to reward (and observation and action) delay.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9yrgrtP7zCM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with RL problems with the reward is either very sparse or delayed, in which case the Q-values would don't get updated. Honestly, I found the write-up to be really confusing, partly because of the structure of the paper and partly because of frequent carelessness in wording. Something that would really help is to take a difficult RL problem and just use it as a running example when discussing the different concepts in the paper. For instance, how does this notion of empirical sufficiency manifest itself in an image-based example and something like grid world?\n\nAs for the results, the authors conduct experiments on 6 Atari games, which I didn't find very convincing, in particular since games like Montezuma's Revenge, which are known to be difficult because of their reward sparsity are not included. Also, there has been a fair bit of recent work in this area, so using just A3C as a baseline is hard to justify. For instance, the authors can look at https://arxiv.org/abs/1606.01868 and the many papers that followed.\n\nIn short, I think there are some interesting ideas in the paper, but both the writing and the experimental results need significant improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7FDivoS_zs1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an experimental approach to tackle the problem of appropriate credit assignment under the delayed reward scenario and proposes an empirically sufficient condition to calibrate the reward policy in a Deep Reinforcement Learning framework.\n\n**Pros**\n    - The problem is interesting.\n\n**Cons**\n    - The paper does not explain the pre-processing in sufficient depth. In the experiments (neither in section 4 nor in A.1 or A.2) the state representation is not clear. \n    - The approach presented in this paper seems to be ad-hoc, and it lacks rigorous experimentation to validate the contribution of building blocks of the architecture. In particular, **ablation tests** are very much necessary to clarify the contribution of different components in the proposed architecture, but no such study has been provided in the paper. Further, there is little theoretical backup and intuition behind the proposed idea. \n    - The authors define Empirical Sufficiency (Definition 3.1) but do not specify the structure of the space. Specifically, it is not clear what is the underlying set, and if there is a metric associated with it? Now, depending on the structure of the space, it could give rise to many topological properties. Hence, the definition is not complete. Further, the intuitive explanation presented in the paper is not sufficient.\n    - The proposed method is not rigorously compared against the existing well-known DRL techniques. The authors indeed present some experiments to highlight the improvement accomplished by their architecture w.r.t. to some hind-sight results (Table 1). However, the end-goal of solving a task (e.g. playing games) is to maximise the total reward. Hence, it is essential to compare the total reward obtained by the proposed method with that obtained by the latest DRL based algorithms, and it is missing in this paper.\n\n**Overall impression**\nThe paper attacks an exciting problem but lacks rigorous experimentation and theory to validate the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3qgZ6P23otS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\nThis paper proposes a method to tackle credit assignment for delay rewards. The method is based on building a classifier that can detect states that will lead to environmental rewards in the future and assigning an additional reward to these detected states. \n\n**Pros:**\n- The overall idea is simple and intuitive.\n- The visualizations in Fig 2 and 3 are clear and give good intuitions about the model.\n- The results in Table 1 significantly improve over the baseline.\n\n**Cons:**\n1. The paper is not particularly easy to read. Here are some of the elements that caused me confusion and could be improved:\n\n- Many terms introduced in the paper are not defined clearly enough (e.g. Empirical Sufficiency, Policy-based Empirical Sufficient Distribution,  Task-Specific Empirical Sufficient Distribution, Sensitive Sampling). I would suggest minimizing the introduction of new terms to the ones really required to explain the method.\n\n- Other terms are slightly vague (e.g. pure distribution, dominant probability). \n\n- Some terms have names that are not very intuitive (e.g. rounds, overfitting mechanism). \n\n- Related to the previous point. In what sense is the word calibration used here? This term has a particular definition in the literature that should be refer to and justified.\n\nIn the section on Overfitting training:\n- Equation (4) is not clear enough. Please define $L_E^one(\\hat r_pos, r_pos) + (\\hat r_neg, r_neg)$\n- Are $N_{ident}$ and $N_{sufficient}$ number of rounds or a number of samples?\n- What does it mean “we maximize Precision with complete $R_{negative}$ samples”?\n- Is the phase 1 and 2 done in sequence or in parallel as Eq 6 suggests? In the latter case, is this the same as increasing the loss for the negative class?\n\nIn the Results section:\n- Result order: The modified environment result is presented before the regular one, although in the first paragraph of the results the order is swapped.  \n\n- In Table 2, what is the difference between the red trace in top and bottom row?\n\n2. I’m not sure the Atari levels selected for the experiments have a particularly delayed reward. With the possible exception of Bowling, the other levels usually have a short period between action and reward. Some levels that could be more interesting for testing credit assignment could be Skiing, Seaquest, Solaris or Venture (justified in e.g. Arjona-Medina et al. 2019 or Puigdomènech Badia 2020). It would be interesting to include the justification for this particular selection of levels in the paper.\n\n3. The results are not particularly strong. From Table 2, it seems that the only significant improvement is made on FishingDerby. Is this because of the choice of levels? I would have expected a positive result in Bowling. Could the authors include a comment on why this is not the case? The results are positice in Table 1, in the modified environments, but I would like to see results in an original level with more delayed rewards (which already exist in Atari). \n\n4. How does this method cope with intermediate (distracting) rewards in between the key action and the ultimate reward? (as in Hung et al., 2018). Is the way of selective states for the positive pools a problem to learn this type of credit assignment?\n\nMinor: \n- Tables 1 and 2 should be Figures.\n- Fig3. caption “ The calibrated rewards and environmental rewards are encoded in red and yellow“  should be blue?\n- “resonate with human cognition” seems a bit over-claimed to me. I would say that the visual inspections make intuitive sense. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work tackles sparse or delayed reward problem in reinforcement learning. The key idea is to build a classifier to detect states that will lead to high rewards in the future and provide a bonus to those states. All the reviewers liked the idea but had issues with the execution of empirical results. The approach is evaluated only in a few Atari games skipping many sparse reward games and missing comparison to many exploration baselines. Furthermore, many reviewers found the writing confusing and hard to follow. The authors provided the rebuttal and addressed some of the concerns. However, upon discussion post rebuttal, the reviewers decided to maintain their score. Reviewers believe that the paper will immensely benefit with improved writing, evaluation on all atari games, and comparison to exploration baselines. Please refer to the reviews for final feedback and suggestions to strengthen the future submission.
Decision: Reject