Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes C-learning, which is an essentially a horizon aware Q-learning. In a nutshell, the authors proposes changing the Q function from Q(s, a) to C(s, a, h) where s is the state, a is the action and h is allowed time horizon i.e. the agent should get to the goal state using less than h states. Since C can be framed as a modified Q, the authors demonstrate that familiar Q-learning properties such as Bellman property and backprop can be used for C-learning. 

The paper is generally well-written and is easy to understand. The authors evaluated the performance of the proposed method on multiple hand crafted tasks which allow for multiple way of reaching to the goal with a varying hard time limit. The authors compared their method with GCSL (Ghosh et al., 2019) and HER (Andrychowicz et al., 2017) demonstrating an improved performance. The included website includes extra evaluations on a few robotic tasks.

The paper is suffering from multiple shortcomings:
1) Motivation. Although I agree that the proposed problem is important, the authors could provide more motivation. The listed examples demonstrate the problem however there is no real life example that visualizes the *importance* of the problem. 

2) Comparison with existing goal reaching methods. As mentioned by the authors, the reachability problem can be formulated as in an existing RL framework and therefore the proposed method can also be compared with existing RL methods. The comparison section tries to do this but unfortunately it falters. I believe the comparison section should include existing methods with modified RL formulation so they are horizon aware (e.g. by including the horizon limit as input and training them by randomly selecting it). Doing this will highlight *why* C-learning is required and a simple Q-learning method cannot solve the problem at hand. There are other issues in Section 4 as well. For example Fig 7 has incomplete curves, the metrics are not clearly defined and it is not clear how the baselines have been evaluated. 

3) Task selection. The paper evaluated the performance of their model on toy environments which is always a good idea. That enables the authors to clearly differentiate their proposed method from previous work (which is missing as I mentioned in 2). But, at the end of the day, it is important to visualize that the proposed method works on real challenges which is missing. I should mention that the website includes more realistic tasks and addresses this problem to some sense but the paper in its current form lacks.

Overall, I think the paper can use clearer motivation, comparison and evaluation.




review_title: Why no comparison with time horizon conditioned Q-learning?

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The paper proposes learning horizon-aware goal conditioned policies by integrating horizon dependency in value-based methods. The new formalism defines the value function (cumulative accessibility function) as the probability of reaching a certain goal from a given state within a fixed horizon. This allows trading-off speed and reliability.

### Main contributions
- The C-learning is a different approach to integrate temporal abstractions in value-based method.
- The derivation of the cumulative accessibility function in analogy to the standard Q-learning is principled and very interesting.
- The proposed approach is compatible with the algorithmic adaptations of Q-learning to continuous action spaces.

### Main concerns (and comments)
- The exploration problem has been ignored in this work while it is quite central. The standard Q-learning (if in the undiscounted return like your case) would go for the higher expected return which would correspond to the policy of the highest success probability, when provided with a satisfying exploration. In addition to this, alternative standard methods are horizon-penalized (e.g. through the discount factor). These point should be considered in this study and in the comparison to other methods.
- It is claimed that encoding monotonicity in h in the architecture hurt performance. A hypothesis is provided by not tested. More importantly, did the monotonicity emerge naturally from training ?
- Regarding C-function clipping. How is "if we know for a fact that g_i cannot be reached ..." implemented practically ? It is acknowledged that this requires "some knowledge of the environment, for example a metric over states". However, such a metric even when available does not always say how many steps separate the states (i.e continuous spaces).
- "[...] but also shown its monotonicity, a crucial component which facilitates neural network learning in the higher-dimensional goal-horizon space" : can you elaborate on this ? Is this claim referring to some property or known result on "neural network learning" ? If so any references ?
- "Surprisingly, GCSL learns to take the high-risk path, despite Ghosh et al.’s intention to incentivize paths that are guaranteed to reach the goal": Does your method uses a goal sampling strategy that provides more options to the agent (like a richer exploration would do) ? In other words, how much of the performance depends on the goal sampling strategy (ablation study) ?
- In Dubin's car, the set of reachable goals from a given state within some horizon does not have a trivial structure, how was the goal sampling implemented here ?
- Regarding the C-function clipping, can we imagine a goal conditioned Q-learning baseline that would be trained with a similar clipping, i.e how much of the performance is explained by this inductive bias (the C-clipping) ?
- "This is also impressive, as we are learning a more complicated objective, which contains information not recoverable by the alternatives." : This comment is misleading. Even if the objective is more complicated, the proposed method learns a more structured function and benefits (for free) from valuable information (like the C-function Clipping) while "alternatives" have to implicitly learn such information.
- "We did train for longer and observed decreased performance" : Does this mean the performance decreases for all the curves that were not plotted till convergence ? How do you explain this ? Also, Figure 7 shows averages without standard deviations.
- "We also note that, strangely, while C-learning dominates easy and medium goals in mini maze, it struggles to reach hard goals. We found this particularly puzzling given the good performance in the more challenging environments." : This deserves more attention and might suggest to consider the exploration problem (mentioned in the first comment) more seriously.

### Minor comment
- The objective (eq. 5) is quite confusing. Should it be minimized or maximized ?
- How does this compare to planning with options over different horizons ?
review_title: This submission calls for a more careful analysis of the proposed method components

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Update after the rebuttal

Thank you for including the comparison to TDMs - the results are now much more convincing.
I think this should be one of the main results in the paper (which should be presented in the main text rather than in the appendix).
Also, I'd suggest revising the paper to highlight new contributions compared to TDMs in the intro/method section. 
I increased the score assuming that the authors will reflect my final suggestions.

Regarding distributional RL and cross-entropy loss, distributional RL does allow a probability distribution as target (not just either 0 or 1 and just like yours). I'd suggest the authors to take a closer look at the paper and clarify differences (I still think the cross entropy loss is a special case of distributional RL with discount factor of 1).

# Summary 
This paper proposes cumulative accessibility functions that estimate horizon-aware value function for goal-reaching RL problems. Specifically, the proposed C-function estimates the probability of reaching the goal state within a time budget (say $h$ steps). The paper derives Bellman-equation for C-function along with several properties. The experimental results on several continuous control domains show that the proposed method can balance between reliability and speed at test-time. 

Pros
* The paper presents a new method that can balance between reliability and speed for goal-reaching RL problems.

Cons
* The novelty seems limited because the proposed method is quite similar to [1]. 
* Comparison to HER and GCSL is not fully convincing. 

# Novelty
* My major concern is that this paper does not cite or discuss the most relevant prior work [1]. The proposed C-function seems like a special case of the TD model proposed by [1], which also learns Q(s, a, g, t) as a distance estimation ||s-g||. The TD model has almost the same Bellman equation as Equation (3) and it can also trade-off reliability and speed at test-time. It seems like the only difference is the definition of the reward function (distance v.s. 1/0). It is important for this paper to compare against [1] throughout the paper.
* The cross entropy loss (Equation 5) is not particularly novel, because it can be viewed as Distributional RL without discounting. 

# Quality
* The empirical result shows the trade-off between reliability and speed nicely, which motivates the proposed method well. 
* The comparison to HER and CGSL was mostly conducted on non-standard domains. The claim about faster learning would be more convincing if the experiment was conducted on more standard domains (e.g., MuJoCo, robotics manipulation) 

# Clarity
* The motivation behind using a horizon-independent policy for behavior is not clear. 
* The problem considered in this paper is not clearly defined. Is this self-supervised learning setting (where there is no final goal) or goal-reaching RL setting? The Algorithm 1 sounds like the former, while the experiment seems like the latter.

[1] Vitchyr Pong et al., Temporal Difference Models: Model-Free Deep RL for Model-Based Control.
[2] Marc G. Bellemare et al., A Distributional Perspective on Reinforcement Learning.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper highlights a problem in existing goal-reaching RL agents, in that they do not explicitly allow for trading off speed (how fast you reach the goal) and reliability (how often you reach the goal). While this tradeoff is implicitly determined by the discount factor in training, the paper asserts that in practice the ability to more flexibly determine this during inference is more desirable. Given this shortcoming of existing work, the paper then proposes "C-Learning" which learns a policy conditioned on both a goal and a desired horizon (h) -- i.e., a time limit on the policy. The paper presents favorable results of C-Learning on a few simulated domains compared to existing goal-reaching RL agents.

Strengths:

-- The proposed problem in standard goal-reaching agents is convincing.

-- I appreciate the level of detail in the algorithmic description along with the pseudocode. It was mostly easy to follow, except in a few small places (see weaknesses).

-- The experiments appear to have been performed carefully, with a helpful demonstration of the trade-off enabled by the proposed algorithm.

Weaknesses:

-- I found the small section about "Horizon Independent Policies" very confusing. For example, in the definition of M, since we know C* is increasing with h, does the max_h just reduce to setting h=infty. And if so, isn't the maximal value simply either 0 or 1 (measuring reachability)? And how is this max efficiently computed? More generally, why is this specific gamma-conditioned behavior policy necessary?

-- The various algorithmic details in 3.1 are worrying, as they suggest that a number of things beyond the basic C-Learning paradigm are necessary in practice. While I understand that this is unavoidable in any algorithm, some of these algorithmic details are exceedingly specific to the domain. For example, "C-function clipping" relies on knowing reachability in the environment, which is arguably as difficult as learning a goal-reaching policy in many cases!

-- The introduction mentions evaluating on domains from Nachum 2018, Peng 2018, and Zhang 2020. However, as far as I can tell, none of these domains are actually evaluated on? If you're not going to evaluate on these domains, I suggest removing this sentence from the paper.

-- While I am convinced of the problem the paper claims to solve, I am not convinced that C-learning is necessarily the best solution. I can imagine a number of other approaches which may perform better, worse, or about the same. For example, why not simply learn a policy pi(a|s, g, gamma) for all possible gamma in [0, 1)? Or alternatively, why not use one of the risk-sensitive policy learning approaches in the safe RL or constrained MDP literature (e.g., Lyapunov Safe RL)?

-- Learning a horizon-conditioned Q function was also proposed in TDM (https://arxiv.org/abs/1802.09081). How does C-learning relate and compare to this existing technique?
review_title: Review

Meta-review: This paper introduces C-learning, an approach to integrate temporal abstractions to value-based methods. Specifically, it uses accessibility functions that estimate horizon-aware value functions for goal-reaching RL problems. Such an approach allows trading-off reliability and speed. After careful consideration I’m recommending the acceptance of this paper. The main weaknesses raised by the reviewers were addressed during the rebuttal, including the improvement of presentation and the introduction of new experiments and baselines. There were not many actionable criticisms left after the discussion and the reviewers acknowledged that the paper has improved since its first version.

For the final version of the manuscript, I recommend the authors to further take R2’s comments/suggestions into consideration. Further incorporating the discussion about TDMs in the main text will improve clarity, better position the paper, and increase its likelihood of having impact.

Decision: Accept (Poster)