Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: this paper is proposing a new unsupervised reinforcement learning scheme for being able to train a more taskable agent that can reach a large base of goals in the environment. the motivation for this work is to not only collect the data and have the agent be able to explore but at the same time be able to train a goal condition to policy such that if we wanted the agents to reach a particular state in the world, we could specify that during test time. the method proposes to do a type of incremental search outward from the initial State distribution and slowly add states that the agent can learn how to reach in this case in an effort to be able to sample from the state distribution such that goals can be reached uniformly across the state space. the method does perform some analysis showing that qualitatively the agents appear to learn and display a more uniform type of visitation strategy across the state space while visiting goals. and the method is compared across a handful of mujoco-based robotic simulations.
strength_and_weaknesses: pros
- The paper proposes finding a more helpful policy that can be trained inside the environment compared to prior methods. Most prior curiosity-based methods do help the agents learn to discover different states that are more difficult to reach via exploration bonuses, but this method also seems to train a goal condition policy making it more taskable as well as prior methods.
- They are also able to show through some illustrative examples that their proposed method can or at least appears to explore the environment in a more uniform manner.

cons
- There are some limitations in the comparison and analysis that should be addressed. In particular, some prior methods are mentioned in the paper that would be important to compare to understand better if the proposed method is better than prior work. this also notes and is based on the concept that they're training a goal condition policy to explore the state distribution and not just a curiosity-based method.
- This method appears only to be applicable to simulators. As the authors note, it's also extremely data-hungry.
clarity,_quality,_novelty_and_reproducibility: Additional Comments on the paper:

- It's not discussed very clearly in the paper, but there are obvious practical challenges in being able to create a policy that can reach all of the states in a very uniform distribution. There will be more probability for states that are closer to the initial State distribution for the trained RL policy.
- What are the assumptions that go along with the proposed algorithm in section 2.1? From this somewhat difficult-to-understand analysis it seems that one additional assumption is that the environment is deterministic and can reach these states given similar paths and actions. Generally, the assumption and motivation that these methods will work really well in simulation can be a problematic hypothesis and can't imply that this method will not be helpful on any real-world problems that have data limitations, stochasticity and partial observation.
- It's not clear what is the goal of algorithm one. The first sentence after that algorithm states that it only requires NKT samples, but it doesn't say what these are required for.
- The stopping condition for individual episodes that depends on $$\epsilon$$, how was $$\epsilon$$ chosen for each of the environments? In particular, how was it chosen such that it does not bias the method proposed in the paper compared to other methods? 
- The appendix links in the paper appear to be broken.
- Why are the XYZ positions of everybody part added to the state in the simulation and not just the position information for the center of the agent? For example, the center of mass could be used or the agent's position of the root link.
- The beginning of section 3.2 on entropy upper bounds needs citations to inform the reader where these axioms are derived from.
- The results in figure 4 that compare different methods depending on the measure proposed for entropy over a prior distribution appear to be somewhat self-fulfilling. The method proposed in this paper may be optimizing almost this exact objective, while prior methods are not. how can we be sure that this is the most critical form of the objective to be optimized and that the other methods aren't performing much better, giving some other analysis of the entropy over the state space?
- Why is this method not compared to a PT and APS? both of those methods perform types of entropy maximization to explore states in the environment. this makes them appear to be very likely candidates for solving a similar problem.
- In addition, this method should also be compared to the UPSIDE algorithm.

Additional works that should be cited in the paper:
- Skew-Fit: State-Covering Self-Supervised Reinforcement Learning, Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, Sergey Levine
summary_of_the_review: The paper proposes a helpful algorithm to allow for training a goal-based agent to reach many goals. The work requires further details and comparisons to prior methods to understand if it is an improvement over prior work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a method called Entropy-Based Conditioned Continuous Control Policy Optimization (C3PO) that tackles the problem of learning a general goal-conditioned policy in two stages. In the first stage, using an exploration algorithm called ChronoGEM, an exploration dataset consisting of diverse goals is collected. In the second stage, the goals collected in the first phase are used as goal targets in goal-conditioned RL with soft-actor critic (SAC) as the base algorithm. The paper demonstrates the method’s effectiveness on Gym environments: walker, hopper, halfcheetah, and ant. It demonstrates that C3PO is not only able to reach a wider distribution of states compared to RND and SMM, training goal-conditioned policies on these states improves the ability of these policies to solve a range of goals.
strength_and_weaknesses: Strengths:
- The ChronoGEM algorithm appears empirically to generate more uniform state distributions compared to random walks, SMM, and RND exploration strategies on the maze and ant environments.  It is also conceptually simple to understand and implement.
The experimental results training goal-conditioned RL on the goals collected by ChromoGEN outperforms policies trained on the other goal distributions, which is a positive signal that the distribution of goals for ChromoGEM is relatively wide.

Weaknesses:
- The theoretical argument in section 2.1 does not seem convincing to me. The statement seems to be that the state visitation will eventually converge to be uniform over the entire state space, but the theorem in Appendix A only demonstrates that the states selected for the next step of the greedy procedure can be approximately uniform from KN sampled states. As the authors mention, playing uniform actions would not necessarily lead to a uniform sampling over the possible next states. So it seems like the “inductive” step here is missing. So I don’t agree with the statement in the conclusion that the method generates “high entropy behaviors, in theory (Theorem 1)”. 
- I think it would help the paper to have a comparison to a method like Skew-Fit[1] that has a similar exploration strategy to the one proposed in the paper by rebalancing the distribution of states to set as goals.
- There isn’t an ablation study conducted on the impact of the tolerance annealing component from SAC to see how much of the improved performance comes from that compared to the goals from ChronoGEM.
- The EWGA metric is an interesting way to evaluate the performance across evaluation sets that may have varying difficulties, but why not evaluate using another strategy like uniformly sampling goals across the [x, y, z] space? That seems like it would be less noisy, and that observation space has already been constructed.

[1] Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (Pong et. al, 2019)
clarity,_quality,_novelty_and_reproducibility: The paper is generally clearly written, and experiments are conducted thoroughly with multiple random seeds. The method is described with sufficient implementation details that it seems likely to be reproducible.

Questions: 
- Given that the entropy needed to be upper bounded in section 3.2, how is the entropy used for the computation of the EWGA metric?
- Is the entire observation of each environment replaced by the xyz positions, or are they appended to the default state spaces?

Other comments/nitpicks/typos:
- HalfCheetah has inconsistent capitalization throughout the manuscript
- I think the first sentence in Section 3.4 is overstated: while it does seem that ChronoGEM performs the best for the studied environments, the statement is rather general and this hasn’t been shown for all goal-conditioned setups.

summary_of_the_review: The proposed method has promising empirical results, but I find the current version of the theoretical justification unconvincing and the analysis of why the method works well could be more thorough. Therefore I do not think the paper should be accepted in its current form, but if the authors can address many of my concerns above, that would be very helpful.

I also want to leave a note that this paper seems to be improperly formatted: the margins are much smaller than in the original template, the ICLR header is missing, and the title / author list formatting is nonstandard.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a method that performs massive uniform exploration of simulated environments and then trains goal-conditioned SAC with an annealed success condition to achieve states discovered during exploration. They demonstrate this exploration procedure to be superior on locomotion environments to other methods.
strength_and_weaknesses: ## Strengths

**Clarity:** 

- The figures are, for the most part, very nice and clear.

**Experiments/Results:**

- ChronoGEM does work well as an exploration procedure, and GIFs in the supplementary demonstrate that their goal-conditioned policy can reach many arbitrary poses.
- C3PO is able to do well on goal distributions generated by other methods, verifying the fact that ChronoGEM is a good exploration algorithm.

## Weaknesses

**Clarity:** 

- Quite a few grammatical hiccups throughout. Please fix (some that I found are listed in the minor details heading)

****************Framing:****************

- I somewhat doubt the utility of this work. It requires a specific setting (a cheap simulator that can be sampled en masse in parallel, and reset to arbitrary states), and has extremely poor sample efficiency due to this assumption. Essentially, ChronoGEM tries to visit all of the states in the simulator. In anything other than locomotion tasks, (e.g. robotic manipulation) these states will mostly be useless and having so many states to try to achieve can hurt learning.
- What is the point of being able to achieve arbitrary positions and poses? The authors state in the abstract that it would allow for easier control and be re-usable as a key building block for downstream tasks. But they don’t show this. The experiments simply demonstrate that Goal-conditioned SAC can reach the position and poses given.
- Conclusion: “In the real world, no reward function is provided.” THis is true, however in the real world we also can’t run ChronoGEM. This goes back to how I think the paper is not framed too well.

**Experiments:** 

- Related to the above point about framing in achieving arbitrary positions and poses, the authors should show experiments where their method allows for better finetuning performance on some downstream tasks (maybe goal conditioned, maybe not) to validate the claims made in the abstract and intro.

****************************Minor details:****************************

- Grammar
    - Intro last paragraph: “its’ → its”
    - Sec 2.1: “necessary bounded” → “necessarily bounded”
    - “up-left room” → “top-left room”
    - “let it enough time” → “let it run for enough time”
    - “we says” → “we say”
- Questions
    - In chronoGEM, how exactly do you sample the ****next**** states? Do you just reset BRAX simulation to those exact states and continue from there?

**********************Formatting:**********************

- The formatting is incorrect. The margins are way too small. This is unfair to other papers who fit everything into the page limit without the small margins. Regardless of whether this was intentional or on accident, I am giving a strong reject no matter what, until the authors fix this issue during the rebuttal.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, of OK quality, incrementally novel, and seemingly reproducible provided sufficient computational resources.
summary_of_the_review: While some of the results are nice, I have 3 main issues with this paper, as detailed above: 1) Framing, 2) The formatting, and 3) The experiments.

Each of these is important and the formatting is critical (there are many papers already desk-rejected for incorrect formatting), and as such, I cannot recommend anything better than a strong reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method called C3PO that has two stages. In the first stage, the method explores the state space to reach different goals in the space. In the second stage, the collected data (states) are used as goals, and a goal-conditioned policy is trained to reach those states (goals). Ideally, the proposed method should explore all states uniformly during the exploration phase. 

Strengths:

- The problem of exploration is important, and the unsupervised nature of the proposed method makes it more interesting. 

- The exploration method is more effective compared to random walks, SMM, and RND for this particular set of tasks.

- The provided examples are nice and illustrative of what the method can achieve.

Weaknesses:

- There are concerns regarding the "infinite sample" budget that the method requires to achieve a uniform distribution.

- The proposed approach is quite sample inefficient, and its applicability is limited to "cheap" simulators. 

- There is no comparison with prior SOTA approaches on exploration. 

All three reviewers have carefully read the paper and the rebuttal. They did not find the rebuttal convincing and recommended rejection. The AC also read the reviews and the rebuttal and agrees with the reviewers' recommendation. 

justification_for_why_not_higher_score: The weaknesses (mentioned above) outweigh the positive aspects of the paper, so the paper cannot be accepted at this point. 
justification_for_why_not_lower_score: N/A
Decision: Reject