Graph-Enhanced Exploration for Goal-oriented Reinforcement Learning

Abstract: Goal-oriented Reinforcement Learning (GoRL) is a promising approach for scaling up RL techniques on sparse reward environments requiring long horizon planning. Recent works attempt to build suitable abstraction graph of the environment and enhance GoRL with classical graphical methods such as shortest path searching; however, these approaches mainly focus on either graph construction or agent exploitation, but leave the exploration lack of study. This paper proposes Graph-enhanced GoRL (G2RL), a new GoRL framework for effective exploration and efficient training based on the state-transition graph. We first introduce the optimal goals for exploration on the graph and then use them as supervised signals to train the goal generator in G2RL in a hindsight manner. Furthermore, we define relevant trajectories of a state based on its graph neighborhood and show that giving high priority to these trajectories would lead to an efficient policy learning. In addition to the theoretical results regarding optimal goal generation, our empirical results on standard discrete and continuous control benchmarks show that leveraging the state-transition graph is beneficial for GoRL to learn an effective and informative exploration strategy and outperform the state-of-the-art methods.

Reviews and author responses:

 We sincerely thank again all reviewers for their constructive comments and suggestions. As reviewers also mentioned, our responses, new experiments, and revised version have adequately addressed all the questions raised by the reviewers . We hope the reviewers will champion our paper for acceptance.

 Thanks again for your instructive comments. We will address these minor issues in the final version. We are glad that our responses address your concerns and we hope you will consider increasing your score.

 Thanks again for your instructive comments and for increasing the score. 

 Thanks very much for the detailed response and the extensive revision. My concerns are addressed adequately, and the additional experiments look promising and make sense to me.     

Minor things:
-	Figure 3 may have included too many algorithms, and it could be helpful to consider moving some of the baselines and comparison to Appendix to improve the readability of the figure.  
-	(Typos) Impact of Graph Boundary seems to be Figure 4(d) and Impact of Discretization is 4(c).    

In this work, the authors describe a novel graph-augmented RL method for goal conditioned tasks. The results shown outperform a number of baseline methods. The method is well motivated and theoreticaly grounded.



 Main Review:
Strengths:

The method is well motivated and clearly written.
The theoretical analysis is detailed and provides performance gaurantees.
A detailed abalation study compares the performance of the approach in number of experiment configurations.

Weaknesses:

1. While the method may be able to generalize to new goals in the same environment configuation, I do not see how this method would be applied when generalizing the unseen environments which where much Deep RL research is currently applied. 
2. The environments where this method is evaluated are rather simplistic, it would be good to see an evaluation in a photo realistic simulator such as Habitat / Gibson / Matterport.
3. How does the method account for visual aliasing in continuous environments?

Other comments: SPTM is mislabelled in figure 4. I stated this in my previous review and it is still incorrect. I had the fortune to review this paper in my NeurIPS batch, unfortunately it did not pass. There were a number of things the authors stated they would include in the next revision, such as comparisons in a photo realistic simulator such as Habitat, which they have not done. 
I look forward to the authors explaination of why this is the case in the rebuttal. 

UPDATE AFTER REBUTTAL: I thank the authors for their detailed replies and additional experiments, they have demonstrated that this approach is applicable in photorealistic 3D environments and have addressed many of my concerns. I raise my rating to 6. 


 We sincerely appreciate for increasing your score, valuable comments, and your willingness to further discuss. We've used them to greatly improve the paper. We hope our responses have addressed all your concerns. Please let us know if you have more questions.

> In particular, Section 2.4 introduces the concept of group division. However, it is not explicitly stated why this is needed. In particular, could the method be applied at all if such group division operation was not done? If not, I would strongly suggest taking more space to explain why.

We further explain the motivations of group divisions as follows: As Theorem 1 says in order to generate goals within the graph boundary, one straightforward way is to directly generate the goals within the graph boundary without the group division operation. However, it is impractical to achieve, because *the graph boundary itself varies at each episode* due to the inherent dynamics of state-transition graphs. To address this issue, our proposed group division constructs *a fixed number of groups* (i.e., a hyper-parameter $N$) containing *a dynamic number of states*. 

We also provided two ways to construct these groups by adding different states into groups, namely *Neighbor States* and *Uncertain States*, whose motivations are summarized as follows: 

1) The motivation of adding the neighbor nodes (i.e., *Neighbor States*) is (i) as $s_\text{last}$ is likely to be valuable, its neighbor states/nodes might be valuable, (ii) as $s_\text{last}$ represents a direction of the previous exploration, we can gradually adjust the directions by assigning its neighbor states/nodes as the goals.
2) The motivation of adding the uncertain states/nodes (i.e., *Uncertain States*) is (i) as the certainty of each node/state represents its exploration potential, it's intuitive to group the states in terms of their certainties (ii) at different stages of goal-oriented RL, the agent needs to switch its focus between exploration and exploitation, which can be explicitly modeled by the group selection over the groups built by *Uncertain States*.

These discussions are already available in Section 2.4 and Appendix 4.5. We will further highlight them in the paper.

> Moreover, the definition of Neighbour States and Uncertain States could be written simply in words, or expressed visually.

That is a good suggestion, thanks. We describe the definition of *Neighbour States* and *Uncertain States* in words as follows.

To ensure all the groups are non-empty, we first include the last visited state in the previous episode in each group (denoted as $s_\text{last}$), and then extend each group by adding states within the group boundary in the following ways: (i) *Neighbour States* is to add $n$-hop neighbor states of $s_\text{last}$ into group $n$, (ii) *Uncertain States* is to add the states whose certainties equal to $|A|-n$ into group $n$; where $n=1,2,\ldots, N-1$, and group $N$ consists of all the rest states in the graph boundary. 

> In Figure 4(b), there seems to be not much difference between each of these methods, and if I understand correctly, only UNCERT3 is used in the experiments. Given this information, why present two methods? 

Good question. While these two methods achieve comparable results (as shown in Figure 4(b)), we present both methods mainly due to their different use cases. As shown in Section 2.4 and Appendix 4.5, the time complexity of *Neighbour States* is $O(d_{\partial\mathcal{G}^e}^{N-1})$ where $d_{\partial\mathcal{G}^e}$ denotes the maximum degree of states in $\partial\mathcal{G}^e$, and the time complexity of *Uncertain States* is $O(|S_{\partial\mathcal{G}^e}|)$ where $|S_{\partial\mathcal{G}^e}|$ denotes the number of states in $\partial\mathcal{G}^e$. Therefore, *Neighbour States* is efficient on the graph boundary with low maximum out-degree (i.e., $d_{\partial\mathcal{G}^e}$), while *Uncertain States* is efficient on the graph boundary with small number of states (i.e., $|S_{\partial\mathcal{G}^e}|$). We apologize for the confusion of the motivation. We will include the above explanation in the revision.

> If only UNCERT3 is used, I would strongly suggest to only present this method and to take the time to present it clearly.

Thanks for your instructive suggestion. We agree that only presenting *Uncertain States* in the main text is more clear and more friendly for the readers. Therefore, we will only include the motivation, definition and analysis of *Uncertain States* in Section 2.4, and move the motivation, definition and analysis of *Neighbour States* into Appendix 4.5 in revision. 

> Section 2.3 could present the different existing definitions of certainty of a state/node before stating Proposition 1 that adds arguments to the definition the authors choose.

Thanks for your suggestion. That is an easy and simple fix. We will include the different existing definitions of certainty of a state/node before Proposition 1.

 > If the authors can mention environments where K bins is not practical or not efficient? Such limitation would be important to be discussed, as it can shed some light on the generality of the method and suggest future work.

We summarize the environments where K-bin discretization is practical and the environments where K-bin discretization can not be directly applied as follows.
1) K-bins discretization is applicable to the low-dimensional (discrete and continuous) environments (e.g., Maze, Ant Maze in Figure A6(a)(b)).
2) K-bins discretization can not be directly applied to the environments (e.g., VizDoom and Matterport3D in Figure A6(e)(f)) with high-dimensional inputs (e.g., images), since there is no explicit spatial distance metric. So, we follow [1] and [2] to first pre-train a ResNet-18 [3] and Mask-RCNN [4] to obtain representations from RGB images, which can then be used to generate a graph given a pre-defined similarity threshold. And in these cases, the K-bins discretization is applied to reduce the scale of the graphs, not to build the graphs.

These discussions are also available in Section 2.2 and Appendix 4.1. We will further highlight them in the paper.

[1] Semi-parametric topological memory for navigation. Savinov, et al., ICLR 2018.

[2] Object goal navigation using goal-oriented semantic exploration. Chaplot, et al., NeurIPS 2020.

[3] Deep residual learning for image recognition. He et al., CVPR 2016.

[4] Mask R-CNN. He et al., ICCV 2017.

 Thank you for your detailed feedback and suggested experiments. We hope you will consider increasing your score after seeing our response and new experiments.

> While the method may be able to generalize to new goals in the same environment configuation, I do not see how this method would be applied when generalizing the unseen environments which where much Deep RL research is currently applied.

We first want to emphasize that the main components of G2RL are (i) differentiable goal generators for effective goal generation and (ii) relevance sampling technique for policy learning. As no learning is allowed when generalizing to unseen environments, we only discuss the generalizability of our goal generator as follows:
1) Our definition of the graph boundary providing the candidate states for the goal generation can seamlessly generalize to the unseen environment, as it can re-compute the certainty of each state and construct the graph boundary by uncertain states.
2) The group division based on the graph boundary can also seamlessly generalize to the unseen environment, since the number of groups (i.e., $N$) is a hyper-parameter.
3) The group selection over $N$ groups can directly generalize to unseen environments by freezing the parameters (i.e., $\phi$) in the attention model (i.e., $\mathtt{ATT}_\phi$).

Following [1], we designed two test environments (named TEST1, TEST2 illustrated in Figure A7) based on Ant Maze environment. We evaluated the performances of G2RL along with the baselines by freezing all the parameters in the policy network and goal generation (if applicable). Results reported in Table A1 verify the superiority of G2RL on unseen environments with the same task as the training environments (e.g., the task of TEST1, TEST2 environments is navigation, same to TRAIN environment). Please refer to revised Appendix 6.2 for details. 

> It would be good to see an evaluation in a photo realistic simulator such as Habitat / Gibson / Matterport.

Per your suggestion. We included Matterport3D environment [2] based on Habitat simulator [3] (please see Appendix 5.1 for description). Results reported in Table A1 show that our method can consistently outperform all the baseline methods in terms of success rate and successes weighted by the path length. Please refer to revised Appendix 6.2 for details. 

> How does the method account for visual aliasing in continuous environments?

As discussed in Appendix 4.1, for the environments with image inputs, we can not directly apply the K-bins discretization to build the graph. Therefore, we follow [1] and [4] to first pre-train a ResNet-18 [5] and Mask-RCNN [6] to obtain representations from RGB images for VizDoom and Matterport3D respectively. In this regard, the power of dealing with visual aliasing images largely depends on the expressive power of ResNet-18 and Mask-RCNN, which is not the core problem of our work. From Figure 3(f) and Table A1, we can see that G2RL can consistently outperform all the baselines using the same pre-trained models and image inputs. 

> Other comments: SPTM is mislabelled in figure 4.

Thanks for pointing it out. We corrected them in the updated PDF.

[1] Semi-parametric topological memory for navigation. Savinov, et al., ICLR 2018.

[2] Matterport3D: Learning from RGB-D data in indoor environments. Chang et al., 3DV 2017.

[3] Habitat: A platform for embodied research. Savva, et al., ICCV 2019.

[4] Object goal navigation using goal-oriented semantic exploration. Chaplot, et al., NeurIPS 2020.

[5] Deep residual learning for image recognition. He et al., CVPR 2016.

[6] Mask R-CNN. He et al., ICCV 2017.

The paper proposes a goal generation procedure in order to improve goal-oriented reinforcement learning (GoRL) by relying on the proposition that the degree of exploration of a state/node can be related to its out-degree. They propose a solution that leverages this proposition while takin into account that we do not have access to the full graph. The paper also proposes to a method sample from the replay buffer, named relevance sampling, with the goal of improving the sample efficiency. The experimental setup considers continuous control domains as well a a visual navigation experiment on VizDoom. In these experiments, modest improvements are presented when compared to other GoRL methods. The authors also provide an ablation study to evaluate the effects of different components of the algorithm. Strengths
------------
-Goal generation method that explicitly considers only having access to a partial graph of the MDP

-Comparison with many baselines is provided



Weaknesses
-----------------
-Clarity and presentation of the method is hindered by heavy notation and theory

-The derivation relies on quite a limited definition of uncertainty and strong assumptions on the MDP

-Ablation analysis doesn't show critical improvement

Details
---------
GoRL is an important are of RL and has to the potential to improve exploration as well as the sample efficiency of learning agents. The papers proposes to work on this problem by considering explicitly the underlying graph of the MDP. Although this can be intractable in practice and might require approximation, for the specific environments presented in this work, the method seems to be well adapted. The idea is intuitive and could work in various settings, although the presentation itself is convoluted and can definitely be improved. Section 2.2 and 3.1 includes definitions, propositions which really don't add much to clarity and in fact hide some of the fundamental assumptions. For example, the definition of state uncertainty is really limited in scope and very specific to the particular setting the authors tackle. Similarly, regarding the whole derivation, it is only later in the remarks that the fundamental assumption of deterministic dynamics is revealed. This should be clearly stated at the beginning. 

The goal generation idea is intuitive and can be explained clearly, I am wondering why the authors choose not to do so. In my opinion that would add to the quality of the paper rather than diminish it. That being said, the paper considers a specific setting, which is also okay, however I would appreciate to see a simple toy example to make more clear in what setting would the method work, and also a toy example where it would not work (when some of the assumptions are violated). Some of the details of the method, such as "extending in scales in different perspective" for the groups is very obscure and would clearly gain in importance through a toy example.


The experimental results show modest improvements in terms of final performance, but show good improvements in terms of sample efficiency. In particular, in the stochastic ant maze domain, the methods works better than in ant maze, which goes against the assumption of the theoretical derivation. Could the authors expand on this difference?

With respect to previous work, there seems to be three differences: a way to generate the graph, a way to sample the goals and a way to sample from the replay buffer. However, when looking at the ablation study, its unclear which of these contributions is fundamental to the performance of the agent. There is indeed a slight drop in performance in Fig5abc, but its not that conclusive. What exactly brings improvement then? Do the baselines also use positional observations to construct the graph? The authors mention that previous work introduces distance function, but the current method also seems to rely on a distance function for the VizDoom domain. 

Finally, the authors do cite some important recent work on GoRL, but seem to miss some important landmark papers such as (Schaul et al. 2015, Kaelbling, 1993) only to name a few. Additionally, important recent work on graph generation is also missing (Jiang et al. 2021, Klissarov et al., 2020). Each of these papers proposes with different strategies to approximate the graph. In particular the latter is related to the current work as it also tries to improve sample efficiency. I would be interested in seeing a comparison with it, although I don't think its necessary as the authors include many baselines already.

Minor points:
1. What happens if there are very valuable states a few layers after the graph boundary, would the algorithm be able to find these?
2. When generating episode e+1, is the agent using the policy from episode e? 


Schaul et al. 2015, Universal Value Function Approximators

Leslie Pack Kaelbling 1993, Learning to achieve goals.

Jiang et al., 2021 Grid-to-Graph: Flexible Spatial Relational Inductive Biases for Reinforcement Learning

Klissarov et al. 2020, Reward Propagation Using Graph Convolutional Networks
 The presentation of the paper is convoluted and could really be improved. Much of the theory is unnecessary to understand the contribution/algorithm and in fact seems to inject assumptions that are hard to satisfy. The experiments show modest and good improvements, however the ablation study doesn't really shed light on what really helps. Some important related work is missing.


 I would like to thank the authors for taking the time to write the rebuttal and perform additional experiments, in particular 4d is very interesting. The rebuttal has shed some light on the contributions and the revised paper adds a lot of clarity to the ideas. Considering these changes I will be raising my score to a 6.

I think the clarity of the paper could be further improved. In particular, section 2.4 introduces the concept of group division. However, it is not explicitly stated why this is needed. In particular, could the method be applied at all if such group division operation was not done? If not, I would strongly suggest taking more space to explain why.  Moreover, the definition of Neighbour States and Uncertain States could be written simply in words, or expressed visually. In Figure 4b), there seems to be not much difference between each of these methods, and if I understand correctly, only UNCERT3 is used in the experiments. Given this information, why present two methods? If only UNCERT3 is used, I would strongly suggest to only present this method and to take the time to present it clearly. 

To a less important degree, section 2.3 could present the different existing definitions of certainty of a state/node before stating the Proposition 1 that adds arguments to the definition the authors choose.

I would also be curious to know if the authors can mention environments where K bins is not practical or not efficient? Such limitation would be important to be discussed, as it can shed some light on the generality of the method and suggest future work.

 > What exactly brings improvement then? 

We designed a new ablation to investigate the impact of graph boundary in Figure 4(d) in revised Section 4.2. We then summarize the findings of ablation studies as follows.

1) Figures 4(a) and A8(a) show that *the relevance sampling algorithm* can seamlessly incorporate with goal-oriented RL and bring *slight improvement*.
2) Figures 4(b) and (c) show that there are no significant gaps among using different group selection and K-bins discretization, as these two techniques are mainly proposed to scale up G2RL to the environments with large-scale and dynamic state-transition graphs.
3) Figure 4(d) shows a significant gap between ours and randomly sampling nodes/states to construct the graph boundary, which indicates that our *graph boundary construction* (including the proposed *certainty of states*) brings *major improvement*.

In conclusion, the performance gain of G2RL mainly from our differentiable goal generation algorithm and slightly from our relevance sampling algorithm. We clarified this in revised Section 4.2.

> Do the baselines also use positional observations to construct the graph? 

Yes. We clarified this in revised Appendix 5.1.

> The authors mention that previous work introduces distance function, but the current method also seems to rely on a distance function for the VizDoom domain.

As discussed in revised Appendix 4.1, we apply K-bins discretization and do not need to introduce distance function on low-dimensional (discrete and continuous) environments, where there naturally exists a spatial distance metric for these positional inputs. However, for the environments with high-dimensional inputs (e.g., images), we can not directly employ the K-bins discretization since there is no explicit spatial distance metric, and thus we follow [2] and [3] to first pre-train a ResNet-18 [4] and Mask-RCNN [5] to obtain representations from RGB images, which can then be used to generate a graph given a pre-defined similarity threshold. And in this case, the K-bins discretization is applied to reduce the scale of the graphs, not to build the graphs. We clarified this in revised Section 2.2 and Appendix 4.1.

> Some important related work is missing.

Thanks for pointing it out. We include these reference in the updated version.

> In particular the latter [6][7] is related to the current work as it also tries to improve sample efficiency. I would be interested in seeing a comparison with it.

Per your suggestion. We selected the recent one [6] (named GTG) as a baseline and made comparison with it in various environments. Results reported in Figures 3 and A8(b) and Table A1 show that our method consistently outperforms GTG, which further verifies the superiority of G2RL. Please refer to Appendix 5.3 for configuration and Figures 3, A8(b) for results. We also included [7] as related work. At the same time, we argue that [7] is not as relevant to our work as [6], since [7] mainly investigates to leverage the sampled graph structure to do reward propagation instead of exploration. 

> What happens if there are very valuable states a few layers after the graph boundary, would the algorithm be able to find these?

It depends on both our algorithm and the dynamics of the environments. We further show that there are two designs of G2RL that makes our algorithm more likely to find these states than other algorithms:
1) We limit the goal generation within the graph boundary, which encourages the agent to explore towards the graph boundary, and thus it is likely for the agent to explore around the graph boundary. Please refer to Theorem 1 and Section 2.4 for details.
2) After our group divisions, G2RL first selects an appropriate group using Eq. (3) and then generates the goals according to Eq. (4). Note that Eq. (4) encourages G2RL to select the state with high value (namely valuable states) as the goals, and thus it is likely for the agent to explore valuable states. Please see Section 3.1 for details.

In conclusion, it is very likely for the agent following G2RL to explore the valuable states around the graph boundary, and further find these states. 

> When generating episode e+1, is the agent using the policy from episode e?

Yes, as shown in Algorithm 1.

[1] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. Eysenbach, et al., NeurIPS 2019.

[2] Semi-parametric topological memory for navigation. Savinov, et al., ICLR 2018.

[3] Object goal navigation using goal-oriented semantic exploration. Chaplot, et al., NeurIPS 2020.

[4] Deep residual learning for image recognition. He, et al., CVPR 2016.

[5] Mask R-CNN. He, et al., ICCV 2017.

[6] Grid-to-graph: Flexible spatial relational inductive biases for reinforcement learning. Jiang, et al., AAMAS 2021.

[7] Reward propagation using graph convolutional networks. Klissarov, et al., NeurIPS 2020.

 Thank you for your detailed feedback and suggested experiments. We hope you will consider increasing your score after seeing our response and new experiments.

> Clarity and presentation of the method is hindered by heavy notation and theory. For example, the definition of state uncertainty is really limited in scope and very specific to the particular setting the authors tackle. Similarly, regarding the whole derivation, it is only later in the remarks that the fundamental assumption of deterministic dynamics is revealed.

Thanks for your suggestion. We polished the presentation as follows:
1) We established a new subsection (i.e., Section 2.3) to package the theoretical part and started the subsection with a sentence to summarize the theoretical findings (including definition and theorem).
2) We added the assumption of deterministic MDPs at the beginning of Proposition 1 and Theorem 1.
3) We moved the method part (i.e., Section 3) after the group division part, as all the content above Section 3 can be regarded as the pre-processing procedure of our key methods: differentiable goal generator and relevance sampling technique. 

> I would appreciate to see a simple toy example to make more clear in what setting would the method work, and also a toy example where it would not work (when some of the assumptions are violated).

We showed an example of a deterministic MDP using the Ant Maze environment in Figure 3(b). This example shows how our method works. Note that this example satisfies the assumption of deterministic MDPs in Theorem 1 and provides an accurate state-transition graph. We further used Low-Stochastic Ant Maze in Figure 3(c) as an example to show that G2RL can work with low stochasticity, and High-Stochastic Ant Maze and Noisy Ant Maze in Figure A8(b)(c) as examples when the performance of this method drops. Note that these settings either violate the deterministic MDPs or provide an inaccurate state-transition graph. And our experimental results in Figures 3(b) and A8(b) also reveal that the performance of other goal-oriented RL such as SoRB [1] drops when running in such environments. Therefore, we think that the limited performance on high stochastic or noisy environments is a common bottleneck for goal-oriented RL using the state-transition graphs.

> Some of the details of the method, such as "extending in scales in different perspective" for the groups is very obscure and would clearly gain in importance through a toy example.

The above sentence means that there are two different ways to add states/nodes to each group. We clarified this in the revised Section 2.4. We also provided an illustrated example in revised Appendix 4.5.


> In particular, in the stochastic ant maze domain, the methods work better than in ant maze, which goes against the assumption of the theoretical derivation. Could the authors expand on this difference?

From comparison between Figure 3(b) and (c), we can see the asymptotic performance of G2RL in Ant Maze (i.e., around 91\% success rate) is significantly better than the performance of G2RL in Low-Stochastic Ant Maze (i.e., around 81\% success rate). Besides, the sample efficiency of the latter one might be comparable to the former one and we provide one possible explanation as follows. As discussed in revised Appendix 5.1, the stochastic version of Ant Maze might accelerate the exploration by creating a shortcut from the start point to the target point. Therefore, we re-run and design the following experiments:
1) Low-Stochastic and High-Stochastic Ant Maze environments: two stochastic variants of Ant Maze where the connectivity between each pair of neighbor cells changes with 5\% and 10\% probabilities in each episode, respectively. Please see Figures 3(c) and A8(b) for results.
2) Noisy Ant Maze environment: a variant of Ant Maze where the noises are generated from Gaussian distribution and added to the adjacency matrix of the state-transition graphs. Please refer to Appendix 5.1 for description and Figure A8(c) for results. 

Comparing Figures 3(c) and Figure A8(c)) to Figure 3(b), we can easily conclude that G2RL works better on deterministic environments with accurate state-transition graphs than stochastic environments or environments with inaccurate state-transition graphs, which is consistent with the assumptions of our Theorem 1. And our experimental results in Figures 3(b) and A8(b) also reveal that the performance of other goal-oriented RL such as SoRB [1] drops when running in such environments. Therefore, we think that the limited performance on high stochastic or noisy environments is a common bottleneck for goal-oriented RL using the state-transition graphs.

 Thank you for your detailed feedback and suggested experiments. We hope you will consider increasing your score after seeing our response and new experiments.

> Does this method assume that every state would have same number of out-degree? What if this method is applied to environments with obstacles - where a state node near or containing the wall will not have the same out-degree as a state node in the middle of a corridor? How would the goal proximity be calculated in this case?

Every state has the same number of actions (i.e. same action space), thus every state has the same out-degree. When applying to the environments with obstacles, the state-transitions will draw self-loops on the states near the obstacles (see Figure A2 for an illustrated example). In this regard, the goal proximity can be calculated in the same way as other states. 


> In the appendix, the goal generated are visualized and compared to that with HER. But not discussed with SoRB.

We added SoRB as a baseline for visualization in Appendix 6.3, and depicted the results in Figure A9. Figure A9 shows that the generated goals of SoRB are located at the middle of that of HER and G2RL. One possible explanation is that comparing to HER, both SoRB and G2RL leverage the state-transition graphs, but G2RL could gain further improvement from the proposed certainty of states and the differentiable goal generator. 

> Can you please clarify the novelty of using the graph boundary? 

Our definition and usage of the graph boundary is novel for the following reasons:

1) Existing and related concepts (i.e., landmark in [1], frontier in [2]) are constructed by the farthest points of visited states. Instead, our graph boundary is built based on the proposed certainty of states which considers the exploration status of each state (i.e., the number of untaken actions). Please see revised Appendix 4.3 for detailed discussion.
2) Existing approaches [1][2] greedily explore or sample farthest states in the landmark and frontier. Instead, we design a *differentiable* goal generator with the hindsight graph supervisions to smartly select an appropriate state in the graph boundary as the goal. Please see revised Section 3.1 for details.

> How this theorem motivates the proposed group selection (as the group selection extending to neighbor nodes seems counter-intuitive to the theorem)?

We summarize the motivations of the proposed group selection as follows:

1) Theorem 1 only motivates to generate goals *within the graph boundary*.
2) Directly generating the goals within the graph boundary is hard to achieve, because the graph boundary itself varies at each episode which motivates the implementation of the group division. With the group division, the goal generator can scale up to dynamic and large-scale state-transition graphs.
3) The motivation of extending to the neighbor nodes is (i) as $s_\text{last}$ is likely to be valuable, its neighbor states/nodes might be valuable, (ii) as $s_\text{last}$ represents a direction of the previous exploration, we can gradually adjust the directions by assigning its neighbor states/nodes as the goals.

Notably, the selected nodes/states by extending to neighbor nodes must be within the graph boundary and thus is not counter-intuitive to the theorem. We have clarified this in revised Section 2.4. 

> Is it regarding the hyperparameter choice in group selection? How does it make the method more scaleable?

Yes, the number of groups is hyperparameter $N$. As the number of states is always changing and sometimes too large to directly build a differentiable model upon these states. Therefore, our group division can make the goal generator works well on dynamic graphs, as the number of groups is fixed and much smaller than the number of states.

> What does next episode graph mean in the Definition 2 of Optimal Goal?

It means the state-transition graph on the next episode $e+1$, namely $\mathcal{G}^{e+1}$. We clarified this in the revised Definition 1, as Definition 1 in the original version is now Definition 2 in the updated version.

> "We then build these groups by extending scales in two different perspectives to include more states" - what does "scales" mean here?

"scales" means the numbers of nodes in the groups, and the above sentence means that there are two different ways to add states/nodes to each group. We clarified this in the revised Section 2.4.


[1] Mapping state space using landmarks for universal goal reaching. Huang et al., NeurIPS 2019.

[2] A frontier-based approach for autonomous exploration. Yamauchi et al., CIRA 1997.

 > It would be more interesting to see how well G2RL can handle different types of stochasticity. For instance, one could introduce Gaussian noises (with certain probability) to state transitions of the MDP.

Per your suggestion, we included the following experiments and environments:

1) Low-Stochastic and High-Stochastic Ant Maze environments: two stochastic variants of Ant Maze where the connectivity between each pair of neighbor cells changes with 5\% and 10\% probabilities in each episode, respectively. Please see Figures 3(c) and A8(b) for results.
2) Noisy Ant Maze environment: a variant of Ant Maze where the noises are generated from Gaussian distribution and added to the adjacency matrix of the state-transition graphs. Please refer to Appendix 5.1 for description and Figure A8(c) for results. 

We showed an example of a deterministic MDP using the Ant Maze environment in Figure 3(b). This example shows how our method works. Note that this example satisfies the assumption of deterministic MDPs in Theorem 1 and provides an accurate state-transition graph. We further used Low-Stochastic Ant Maze in Figure 3(c) as an example to show that G2RL can work with low stochasticity, and High-Stochastic Ant Maze and Noisy Ant Maze in Figure A8(b)(c) as examples when the performance of this method drops. Note that these settings either violate the deterministic MDPs or provide an inaccurate state-transition graph. And our experimental results in Figures 3(b) and A8(b) also reveal that the performance of other goal-oriented RL such as SoRB [1] drops when running in such environments. Therefore, we think that the limited performance on high stochastic or noisy environments is a common bottleneck for goal-oriented RL using the state-transition graphs.

> I wonder how the performance of G2RL would compare against HRAC [2]?

We employed HRAC as a baseline, and make comparisons on all the environments. The new experiments show that HRAC can achieve the best performance among all the baselines in most of the environments and  G2RL can consistently outperform HRAC in all the environments. Please refer to Appendix 5.3 for description of HRAC, and Figures 3, A8(b) for results.

> Minor comments about Eq. (1) and Eq. (12).

Thanks for pointing it out. We corrected them in the updated PDF.

[1] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. Eysenbach, et al., NeurIPS 2019.

[2] Generating adjacency-constrained subgoals in hierarchical reinforcement learning. Zhang et al. NeurIPS 2020.

 Thank you for your detailed feedback and suggested experiments. We hope you will consider increasing your score after seeing our response and new experiments.

> How is Certainty of State defined for problems with continuous action spaces? 

We define the certainty of a discretized node by the average certainty of its state. We clarified this in revised Section 2.3.

> If K-bins discretization is applied, will that result in too many states being uncertain? and too many possible states being considered in the boundary $\partial\mathcal{G}^e_t$? Any issues observed in the experiments?

As the certainty of each state is measured by its untaken actions, two different cases can happen: 

1) The action space is small (e.g., forward, left, right in navigation task), and then it's unlikely to have too many states being uncertain. 
2) The action space is large (e.g., continuous action space), and then we can apply K-bins discretization on the action space to reduce it to a small action space.

Also, we further show that the potential effects from too many uncertain states are limited:

1) For the differentiable goal generator, our group division will divide these states into a fixed number of groups, and the supervision signals of the goal generator are on groups instead of states. Hence, the effect of too uncertain states is limited.
2) For the relevance sampling technique, there is no explicit effect as the relevant states are determined by the neighborhood structure, regardless of whether the nodes are uncertain or not.

We clarified this in revised Appendix 4.3. 

> Why not simply using the state/node visitation counts? What might be wrong with using visitation counts? It would be helpful to compare the proposed Certainty of States with the simple visitation count.

1. Note that the aim of the exploration is to discover new states. Comparing to the visitation counts, certainty of each state (measured by the number of taken actions of each state) is more direct to represent the potential of each state to explore new states. For example, considering a high-frequently visited state $s$, if we greedily take the same action of $s$ in all the visitations, then we might find new states by taking other actions, which indicates that this state is not well explored. Please refer to revised Appendix 4.3 for detailed discussion and illustration. 
2. We established a series of variants to use the different state/node visitation counts to define the graph boundary (e.g., 5\% most-infrequently visited states make up the graph boundary). We empirically evaluated their performance in Section 4.2, and reported results in Figure 4(d). Our new experiment on Ant Maze verifies the superiority of the proposed certainty of states.

> In Definition 2, how to select the goal $g_{e+1}$?

$g_{e+1}$ is the generated goal in episode $e+1$, whose computation is later introduced in Eq. (4). Please see revised Definition 2, as Definition 1 in the original version is now Definition 2 in the updated version.

> How the features for each group are encoded. Could you elaborate more on this (or provide one example) and add to the paper?

The features are one-hot embeddings of group ID. We provide a detailed description in Appendix 2.2.

> Equation (5) confused me and unfortunately simply saying "$V(s,g_{e-1})$ used here is consistent with $V(s,g_{e+1})$ in Eq. (2)" is not very helpful to me.

It means that comparing Eq. (4) to Eq. (2), we can see that G2RL uses $V(s,g_{e-1})$ to generate $g_e$, and uses $V(s,g_{e+1})$ to supervise $g_e$. Namely, $V(s,g_{e-1})\stackrel{\text{generate using Eq. (4)}}{\longrightarrow}g_e\stackrel{\text{supervise using Eq.(5)}}{\longleftarrow}g^*_{e}\stackrel{\text{generate using Eq. (2)}}{\longleftarrow}V(s,g_{e+1})$. We re-organize it as follows: G2RL uses $V(s,g_{e-1})$ to generate $g_e$ (following Eq. (4)), and compute and use $V(s,g_{e})$ to generate $g^*_{e-1}$ (following Eq. (2)), which are further used to supervised $g_{e-1}$. Namely, $V(s,g_{e-1})\stackrel{\text{generate using Eq. (4)}}{\longrightarrow}g_e\stackrel{\text{generate using Eq.(2)}}{\longrightarrow}g^*_{e-1}\stackrel{\text{supervise using Eq. (5)}}{\longrightarrow}g_{e-1}$. We have clarified this in the updated Section 3.1 and Algorithm 1. Note Eq. (5) in the original version is now Eq. (4) in the updated version. 

 We thank the reviewers for their feedback that we've used to greatly improve the paper. We have responded to the concerns of the reviewers as individual comments below.

The reviewers agree that our results and experiments are convincing and comprehensive, our method is well motivated, and theoretically grounded. To summarize our contributions: (i) we demonstrate that our goal-oriented RL formulation based on the state-transition graph is sufficient to perform well on current goal-oriented RL benchmarks, and (ii) our differentiable goal generator method with hindsight graph supervisions and a relevance sampling technique improves the performance of goal-oriented RL tasks. 

Per Reviewer 6obU’s, Reviewer Sjwy's, and Reviewer EMRL's suggestions and to provide further insight about our method, we included the following new experiments in the revised version:

1) Comparison with recently published HRAC [1] and GTG [2] baselines. For full details, please see Appendix 5.3 and Figures 3, A8(b).
2) Results on a photo realistic simulator, Matterport3D navigation environment [3]. Please see Appendix 5.1 and Table A1. 
3) Ablation study on impact of the graph boundary. We designed a series of variants to use the visitation counts of states to define the graph boundary instead of the certainty of states (e.g., 5\% most-infrequently visited states make up the graph boundary). Please see description in Section 4.2 and results in Figure 4(d).
4) Robustness study on stochastic environments. We devised two stochastic variants of Ant Maze: Low-Stochastic and High-Stochastic Ant Maze where the connectivity between each pair of neighbor cells changes with 5\% and 10\% probabilities in each episode, respectively. Please see results in Figures 3(c) and A8(b).
5) Robustness study on noisy state-transition graphs. We evaluated our G2RL by introducing Gaussian noises (e.g., noise drawn from standard Gaussian distribution) on the state-transition graphs in the Ant Maze environment. Please see configuration in Appendix 5.1 and results in Figure A8(c).

The first two experiments show that our method consistently outperforms existing methods on standard benchmark and photo realistic simulators. The third new results verify the superiority of the proposed certainty of states. Comparing to the results of a deterministic Ant Maze environment in Figure 3(b), the last two experiments show that G2RL performs well with low stochasticity but its performance drops in the environments with high stochasticity or inaccurate state-transition graphs, which is consistent to our assumption of (accurate and) deterministic MDPs in Theorem 1. Our experimental results in Figure A8(b) also reveal the limited performance on high stochastic or noisy environments is a common bottleneck for goal-oriented RL (e.g., SoRB [4]) using the state-transition graphs.

In addition, we included a list of new discussions and clarifications:
1. Clarification on applying K-bins discretization technique on the environments with image input in  Appendix 4.1.
2. Comparison between the definition of graph boundary with related literature [5][6] in Appendix 4.3.
 
For your convenience, all modifications in the updated PDF are highlighted in blue color. 

[1] Generating adjacency-constrained subgoals in hierarchical reinforcement learning. Zhang et al., NeurIPS 2020.

[2] Grid-to-graph: Flexiblespatial relational inductive biases for reinforcement learning. Jiang et al., AAMAS 2021.

[3] Matterport3D: Learning from RGB-D data in indoor environments. Chang et al., 3DV 2017.

[4] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. Eysenbach, et al., NeurIPS 2019.

[5] Mapping state space using landmarks for universal goal reaching. Huang et al., NeurIPS 2019.

[6] A frontier-based approach for autonomous exploration. Yamauchi. CIRA 1997. 

This paper improves exploration and learning efficiency in Goal-oriented Reinforcement Learning (GoRL). In particular, a new framework graph-enhanced GoRL (G2RL) is proposed by leveraging the state-transition graph. (1) G2RL produces under-explored goals (or goal states) by planning on the “one-episode ahead” graph to conduct directed and effective exploration. In the actual implementation, the goal is generated by a differentiable generator trained in a hindsight manner.  (2) In addition, G2RL proposes to use “relevant trajectories” based on the graph neighborhood information to improve the efficiency of policy learning. By combining (1) and (2), G2RL demonstrated strong performance on various environments. Theoretical analysis is provided on deterministic MDPs to support the design of the algorithm.           Strengths:
+ Overall, the paper is well-written, organized and technically sound. The motivation is clear, and I believe exploration in GoRL is an important and practical problem. 
+ Defining the optimal goals to explore on the graph by looking one-step (or one-episode) ahead is a very interesting idea and makes sense to me. Major claims and the algorithmic design are well supported by both theoretical and empirical results. 
+ The paper provides comprehensive experiments on various settings to demonstrate the effectiveness of G2RL, including both discrete and continuous tasks, environments with vectorized state inputs and imagery state inputs, environments with static structure and changing structure. In addition, ablation studies are also included on various aspects of the algorithm. The empirical performance looks promising and convincing.   

Concerns:
1.	[Clarity] Some parts of the paper are not clear enough to me. Please see the list of items below for details. 
-	1.a. How is Certainty of State defined for problems with continuous action spaces? If K-bins discretization is applied, will that result in too many states being uncertain? and too many possible states being considered in the boundary $\partial \mathcal{G}_t^e$? Any issues observed in the experiments? 
-	1.b. I understand the motivation of using out-degree to define the boundary, but empirically why not simply using the state/node visitation counts? E.g., consider the most infrequently-visited 10% or 20% nodes/states in the graph as the boundary? What might be wrong with using visitation counts?   
-	1.c. In Definition 2, how to select the goal $g_{e+1}$? I assume it is selected based on $argmax_s V(s, g_{target})$ for $s \in \mathcal{G}^{e+1}$, but not 100% sure. It would be helpful to clarify this in the main paper. 
-	1.d. On page 5 last paragraph, it is not clear to me that how the features $f_1, …, f_N$ for each group $C_n$ are encoded. Could you elaborate more on this (or provide one example) and add to the paper?  
-	1.e. Equation (5) confused me and unfortunately simply saying “$V(s, g_{e-1})$ used here is consistent with $V(s, g_{e+1})$ in Eq. (2)” is not very helpful to me. Could you elaborate more? I would suggest add more clarification to Eq. (5) in the paper. 

2.	[Experiments + Baselines] 
-	2.a. I like the design of Stochastic Ant Maze that changes the structure of the problem. It would be more interesting to see how well G2RL can handle different types of stochasticity. For instance, one could introduce Gaussian noises (with certain probability) to state transitions of the MDP. It could be interesting to add as an additional experiment. 
-	2.b. As mentioned in 1.b., since visitation count (or its proxy) has been widely used in the RL community as intrinsic signals for exploration, it would be helpful to compare the proposed Certainty of States with the simple visitation count. To my understanding, it suffices to only change the candidate states in $\partial \mathcal{G}_t^e$ to include the 10% or 20% infrequently-visited states/nodes by counts, and the rest of G2RL can remain the same, for example either of the two grouping strategies may be used. It can be added as an additional baseline/ablation study. 
-	2.c. I wonder how the performance of G2RL would compare against HRAC [1]? (which is also cited several times in the paper). HRAC improves the reachability of goals by considering only an adjacent region, leading to improved learning efficiency for HRL. However, unlike G2RL, the exploration in HRAC is not directly guided towards the boundary. Comparing against HRAC in some of the environments can be helpful to further demonstrate the effectiveness of graph-enhanced guided exploration by G2RL.                                    

Minor comments: 
-	Eq. (1), $V^{\pi}(s, g)$ should be summed over $a \sim \pi$; if here $\pi$ represents a greedy deterministic policy, please specify before Eq. (1).  
-	Eq. (12), line 3, a typo, I believe should be $\mathcal{G}^{e-1}_1$.    
 

[1] Zhang, Tianren, et al. "Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning." arXiv preprint arXiv:2006.11485 (2020). 



 Exploration in goal-oriented RL is an important problem. The proposed framework G2RL improves exploration and learning efficiency by generating goals from the next episode graph, which is a new and interesting idea. Comprehensive results are provided to support the algorithm and major claims made in the paper. My main concerns are about the clarity in some parts of the paper and some additional baselines (please see Concerns above for details), which I believe can further improve the paper if addressed. Overall, I am satisfied with the submission and vote for accept.  







The paper presents a goal-oriented RL framework based on state-transition graph. Compared to prior work like SoRB, GoalGAN, HIRO, the proposed method G2RL demonstrates higher success in exploration on discrete and continuous control benchmarks. 

The paper addresses that inefficiency of randomly sampling past trajectories for policy learning by (1) modeling state-transition graph (2) relevance sampling of the nodes that are the current state or its neighbors (assuming that they have close goals or similar goal-oriented policies). The goal is sampled from the candidate set, selected by self-attention over sets of neighboring and uncertain nodes.

# Key ideas

1. Differentiable goal generator can be trained with optimal goals for supervision by planning algorithm,
2. (avg) out degree of the node in state-transition graph (also called certainty of a state in the paper) shows how much has the state been explored.
3. sampling relevant trajectories based on goal proximity.
 # Strengths


- Compares with existing work on Goal-oriented RL showing that the proposed approach learns faster and has at par asymptotic performance in different experimental setups such as Maze, AntMaze, Fetch push, Fetch obstacle push and Vizdoom.
- Ablation analyses to discuss the impact of relevance sampling, group selection, discretization. Also show generalization to larger sizes of AntMaze, complex AntMaze with obstacle, and unseen environments.
- Discusses generalization performance in larger sizes of AntMaze, complex AntMaze with obstacle, and unseen environments.

# Clarification

- Does this method assume that every state would have same number of out-degree? What if this method is applied to environments with obstacles - where a state node near or containing the wall will not have the same out-degree as a state node in the middle of a corridor? How would the goal proximity be calculated in this case?
- In the appendix, the goal generated are visualized and compared to that with HER. But not discussed with SoRB, which seems as the strongest baseline.
- The theorem implies that the "candidate set for each goal generation can be limited to the boundary, which can reduce the goal space with optimality preserved". This idea seems similar to frontier-based exploration ([B. Yamauchi](https://ieeexplore.ieee.org/author/37354780800) 1997 [https://ieeexplore.ieee.org/document/613851](https://ieeexplore.ieee.org/document/613851)). Can you please clarify the novelty of this? How this theorem motivates the proposed group selection (as the group selection extending to neighbor nodes seems counter-intuitive to the theorem)?
- Can you clarify in the introduction "In such a way, our method is scalable, as it only needs to build a fix number of groups."? Is it regarding the hyperparameter choice in group selection? How does it make the method more scaleable?

# Minor suggestions

- Typo in Theorem 1 (Goal Optimality ~~Preverved~~ Preserved in Graph Boundary)
- What does *next episode graph* mean in the definition 2 of Optimal Goal?
-  "We then build these groups by extending scales in two different perspectives to include more states" - what does "scales" mean here? I think that the paper provides impressive empirical performance as compared to the baselines on some continuous control tasks in locomotion in AntMaze, Vizdoom, and manipulation with Fetch in simulated scenarios. The paper is well-structured yet needs minor typing corrections and clarifications.
 

Meta-review: This paper proposes a new approach to goal-oriented RL (GoRL) which constructs a state-transition graph from experience and uses this to guide exploration. Compared to prior approaches, this work innovates on (1) how subgoals are selected and (2) how relevant experience is sampled from the replay buffer. This approach outperforms a number of baseline methods across a fairly wide range of environments. The paper also includes extensive ablation and generalization studies.

The reviewers agreed the problem tackled by the paper was important, and were impressed by the empirical evaluation. Several reviewers (6obU and Sjwy) appreciated the theoretical grounding of the method as well, and generally found the approach to the problem compelling, with Reviewer 6obU writing that “defining the optimal goals to explore on the graph by looking one-step (or one-episode) ahead is a very interesting idea” and Reviewer EMRL noting a strength of the paper was a “goal generation method that explicitly considers only having access to a partial graph of the MDP”. The main concerns raised by the reviewers had to do with clarity and the simplicity of the experiments; however, the reviewers felt these were sufficiently addressed by the rebuttal. I agree the approach seems well-motivated and interesting and recommend acceptance as a poster.

Decision: Accept (Poster)