Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes an hierarchical reinforcement learning algorithm based on discovering effects (mode normalized state space differences after acting) and using them to solve tasks.
Main review: The paper is full of new interesting ideas. In particular, that of using and learning effects rather than coarser level states or options for HRL. All of the paper, however, relies on a precise definition of effects: mode normalized *differences between states* after taking actions.

A big criticism of mine is that the algorithm in the tabular setting is essentially equivalent to using (random) states as goals, which is nothing new. In a deterministic tabular MDP, if s is a one-hot encoding description of the state, then e_t(s, a) = s' - s is just a +1 on s' and -1 on s. After mode-normalization, e_c(s, a) is +1 in s' and -1 in the s'(s, mode_a). Thus there are at least as many e_c as possible s', and thus the whole thing is essentially a UVFA with random states as goals.

The same criticism applies for high dimensional deep RL settings where we assume s' are essentially one to one with (s, a) pairs for non-terminal s. In this case we typically have a very bad reachability problem of using states as goals since for a given s in general it's impossible to reach a preselected s' (think of a video where we ask that we reach an exact image, this will generally be impossible since we can't control every pixel intensity that may change with the time of day / light / slight changes in motion).

Thus, the algorithm only makes sense for cases where we have a highly descriptive compositional state representation like (coordinate x, coordinate y, do I have the ball) and so forth. This means that for deep RL we are ignoring the representation learning issue, which is OK for a first paper on the subject. However, the fact that it doesn't work at all for tabular cases is in my opinion a bad omen.

I also think the authors should provide themselves this kind of analysis in the paper and be more honest about the limitations of their method.

Other fundamental criticisms I have are as follows:
 - since the e_t used are estimated by a neural network forward model, this is going to be incredibly hard to scale. Learning transition functions that are relevant is difficult in deep RL and typically miss very important information. See for instance https://arxiv.org/pdf/1507.08750.pdf and all this line of research.
- e_c^goals come from a vae which means they are learned vectors. How on earth do they match *exactly* the e_c^step that's used to match in algorithms 1 and 2. This seems very fishy and quite a limitation of the method. 

All in all, the paper is interesting and has a lot of new ideas, is clear, and I find the work significant. However, it has a lot of very strong problems that are not even adressed properly in the paper. If they provide a more proper analysis of the limitations in the paper I will increase my score.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper proposes a new hierarchical RL method, which uses finite differences in the state space (so-called effects) as sub-goals for the training agent. The authors propose an implementation of their method, and present a series of experiments in a grid world setting.
Main review: ### Originality

The proposed method seems original to me.

### Significance

The paper is not particularly relevant or significant to the CLeaR community, even if it tries to address an important problem, that of exploration and scalability in RL. While the authors try to relate their method to existing concepts in causality, I feel that the connection is very thin, and not really justified. In the end the proposed method is very similar to model-based RL, where the transition model is learned by residual learning ($f(s'-g(s'|s)|s,a)$ instead of $f(s'|s,a)$), and where the residues are used as goals for exploration. The proposed method might be interesting to the (hierarchical) RL community, but a comparison (methodological and experimental) to existing hierarchical RL methods is missing to judge of the significance of the proposed method.

### Technical quality

The paper is not technically sound. Several concepts appear to be ill-defined (see detailed comments), the assumptions are not clearly stated from the start, the experimental setup appears to be flawed (comparing a method with pre-training to a method learning from scratch) or irrelevant (comparing a learned/learning policy to a random policy), and the paper is poorly written in general (it is still unclear to me how the proposed method works, which parts of the bells-and-whistles are required for it to work, if those bells-and-whistles are also part of the baseline method, and finally what exactly is the experimental setup).

### Clarity

I found the paper to be poorly written overall, although it is well-written in terms of language. It is hard to parse what the core contribution of the paper is, and what are just implementation details thrown here and there (see detailed comments). I also found the motivation not well explained. How does eq.5 disentangle effects from the agents ?

### Detailed comments

p.1: we propose agents that learn by estimating the causal effect of an action on the
state -> Isn't that what model-based RL do already ? How does your approach differ from model-based RL ?

p.2 Fig.1: I would suggest improving that Figure and its caption. It took me q while to understand how (a) and (b) are related, and simply to understand what this figure describes (an RL environment).

p.3 Fig.2: How different is that from hierarchical RL ?

p.3: the definition for $e_t(s,a)$ seems rather abstract, and is not mathematically correct since $s'$ appears only one side. Unless you only consider deterministic environments where $s'=f(s,a)$ ? Also, what about state spaces that are not equipped with addition ? Or state spaces where addition does not make sense ?

p.3 eq.1: Here again, $\hat{x}$ only appears on the right-hand side.

p.4 eq. 5: $\text{mode}_{a_i \in A} e_t(s, a_i)$ -> this does not make sense. from what I understand $e_t$ has same dimensionality as the state space, and therefore can be multi-dimensional. How then do you decide on the mode of a multi-dimensional function ?

p.4 eq.6: $e_t$ is missing arguments here. I suppose you mean $e_t(s,a)$ ? Then you're basically learning a model of the world, via residual regression.

p.4: In our experiments, we assume [...] -> You should make it clear from the beginning what are the assumptions required by your method.

p.5: distribution of effects E -> is this distribution conditioned on a state $s$ ? I don't see how an effect makes any sense without the corresponding original state.

p.5: we store experiences of the form [...] -> why storing $e_c(s,a)$, when it can be deduced from $s'-s$ ? Also, $r'$ and $d'$ haven't been defined yet.

p.5: or a punishment $P$ otherwise -> why adding a punishment ? Is a necessary trick for the approach to work ? Or is there something more fundamental ?

p.5: Experiences are augmented [...] to each experience. -> These sound like a lot of bells and whistles. How is each of these components related to your contribution ? Are they necessary for your approach to work ? Are the observed experimental improvements due to these components, or to the fact that you model causal effects ? I suggest to either remove these components altogether, or perform a proper ablation study with those components on and off to help the reader in performing a proper credit assignment.

p.5: Distribution of effects -> I do not understand how this relates to your approach as described in Section 2, or how modeling a distribution of controlled effects $s'-\hat{s}'$ without conditioning on a state and action makes sense at all.

p.5 $1: $(e_c^{goal}|s,e_c)$ what is $e_c$ here ? Does it come from a specific action ? Which one ?

p.6: Task policy -> same comment here. I do not see how aiming for a change $s'-s$ hat does not consider the current stats $s$ makes any sense at all. Why not just learning a distribution over states then, and set those states as goals ? And why do you keep $s$ in the definition of $\pi_t$ on the left-hand-side, if in the end you don't use $s$ ?

p.6 alg.1: What do add_with_her or train_if_needed do ? Those routines are not defined.

p.7: CEHRL and the baseline -> what is the baseline ? It's not been described yet.

p.7: using the pre-trained models -> it seems your approach then is cheating ? Does the DQN baseline also have access to the samples you use for pre-training your models ?

p.8 Fig.4: the x-axis should report the number of interactions with the environment, not the training time, to measure sample-efficiency without bias.

p.8: random effect exploration performs [...] three times more often -> this is very much expected, if you compare a random policy to any policy that has learned a little bit on a non-trivial task, the random policy will achieve less complex task more often.

p8: we record [...] by each method -> which methods ? Do you record training samples ? Does your method start from scratch, or does it use pre-trained models ?

p.9: controlled effects reduce the effect space to model -> I don't see how. If I expand the controlled effect definition, I get $s'-\hat{s}$. How is the resulting space smaller than that of the total effect $s'-s$ ?
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: Hierarchical RL algorithm with very little insights or contributions to causal learning
Main review: The paper proposes an interesting hierarchical RL algorithm, that attempts to decompose a long-horizon sparse reward task into compositional sub-tasks. These are accomplished using (1) a task-specifying meta-controller; (2) a VAE to sample feasible tasks; (3) a task-conditioned low-level action policy. The only causal connection is in identifying the effect of an action -- this is done using essentially advantage functions, which are re-interpreted as causal effect estimates. There are other approaches in the literature that attempt to isolate the causal effect of actions from uncontrollable effects (e.g., the literature on intrinsic motivation, curiosity-driven exploration, solutions to the "Noisy-TV problem", or exogenous decompositions as in http://proceedings.mlr.press/v80/dietterich18a/dietterich18a.pdf) that are not discussed or compared to. The review below focuses only on the weaknesses of the paper.

- Please clarify that s is a p-dimensional vector, transitions are deterministic, and actions are discrete (before defining e_t(s,a) = s' - s). In Eqn5, mode_a e_t(s,a) is confusing -- how are ties broken? Why is this a reasonable "normative world"? What are the properties needed of beta_Y the normative world, and how does mode_a ... satisfy those properties?

- Eqn 3: Why is V(s) a reasonable choice for a normative world G_{tilde a}? V(s) depends on an agent's policy, while G_{tilde a} and beta_Y appear to be defined independent of policy. Can you please clarify why advantages can be viewed as causal effects (rather than the conventional view that they are variance-reduced estimates of cumulative returns Q(s,a))?

- Better figure: Fig2 helps to understand the main pieces of CEHRL, but does not spell out how it connects to Alg 1 and 2. Rather than Fig3, I would appreciate a figure that clarifies: two different timescales -- per-environment step and per-effect step, the timescales of experiences to train hat{e}_t (assuming it is per-environment step), E (per-effect step), pi_t (per-effect step), pi_e (per-environment step); switching from exploration phase to task-specific learning phase.

- Why does pi_t need e_c as input? Is identifying the next task a non-Markovian problem, and does e_c somehow convey sufficient information to make pi_t learnable?

- Eqn 5, 6: This approach of disentangling controlled effect e_c from total effect e_t requires more discussion. Prior work (e.g., https://pathak22.github.io/noreward-rl/) uses a state embedding s -> z which attempts to capture the controllable state factors. This is typically accomplished by looking at what aspects of the s-s' transition are predictive of the action taken. What are the benefits/drawbacks of learning a hat{e}_t model (which will suffer aleatoric uncertainty from uncontrollable factors) and then subtracting the mode (hoping that the uncontrollable factors affect both e_t and the mode in the same additive way); as opposed to learning a controllable minimal latent state representation?

- [Minor clarification] Mismatch between definition and use of e_c^goal: e_c^goal is sampled from E which is a VAE trained on all (s,a) pairs in the experience replay. e_c(s,a) from Eqn 5 gives per-environment-step controlled effect estimate. However, it is being used as a multi-step goal in Alg 1 (e_c^goal in line 5 is uniformly sampled from E and relates a goal state to s_0; has_achieved however compares e_c step which relates s_current-s_prev to e_c^goal which is s_desired - s_0?) Why is this "multi-step" usage of e_c^goal consistent with its one-environment-step definition?

- Clarify in Alg 1 pseudocode that pi_t is uniform (Eqn 8).

- Alg 2: Is the r' <- r' * (1-d) update supposed to happen before the "if h then" block? Also should it be r' * (1-h) [strange that we are rewarding a trajectory even when it terminates because t > K].

- (See comments above for other baselines to use in experiments). The approach of Badia et al 2020 seems especially important to benchmark against.

- Strange that the example using in Fig 1 is not revisited in the minigrid2D domain and tested empirically.

- Please discuss learnability issues: (this is hinted in the last paragraph about conditioning E such that only effects achievable within K are selected) An implicit assumption of Alg 1 and 2 is that pi_e can be learned for all the goals that E could emit. Why is this a reasonable assumption? How delicate is CEHRL to such an assumption?
review_title: null

Meta-review: The paper proposes a method for using and learning "controlled" effects for hierarchical reinforcement learning. The paper uses "controllable" states as goal. An interesting aspect of the proposed method is that it allows learning a model of the world that considers only controllable aspects of the environment and hence facilitating exploration and efficient generalization. Even though the reviews are mixed, I like the underlying idea. I encourage the authors to take into account the feedback by the reviewers and also include a "limitations" section discussing various limitations of the proposed method (stronger dependence on the compositional states), and other assumptions required by the proposed method.
Decision: Accept (Poster)