Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a way to generate auxiliary predictions through the generate and test mechanism. The predictions are randomly generated and are then evaluated for their utility by a proxy measure: by looking at the magnitude of the weights between the features and the the main task learner. The authors claim that this approach is somewhere in-between end-to-end learning, where sample efficiency is a concern and hand-specified tasks. The authors report improved sample efficiency on three environments that are smaller in scale. Ablation studies highlight important properties of the algorithm.
strength_and_weaknesses: # Strength
- Clear presentation and motivation
- Solid and trust-worthy empirical investigation
- A simple method that seems to do well on the environments tested.

# Weaknesses 
- Concerns in terms of scalability of the method beyond one-hot representations
- Additional baselines would be a very good addition to the paper
- A lot of additional hyperparameters
clarity,_quality,_novelty_and_reproducibility: The paper presents a clear problematic and a clear idea to tackle the approach. The paper is well situated with respect to previous work, although more connections to options would be important as auxiliary tasks (GVFs) and options are very closely related. It is mentioned a few times that generate and test could be combined with meta-learning, but this never happens in the paper so it is not clear why such references are made throughout the paper. 

The paper claims that meta-learning is not sample efficient, which can be true, but is not always the case. It really depends on what exactly we are meta-learning. For example meta-learning the whole update rule can be costly, but only learning a part of it is much more efficient. There are papers on both sides of the spectrum and it would be a more fair assessment of meta RL to make these references. 

The question of evaluating how good an auxiliary task is is a very difficult thing to do, as we can hardly find a good measure. The size of weights proxy is an interesting avenue, but how robust is it? The evaluated environments are smaller in size (especially FourRooms, isnt the standard FourRooms almost twice the size?), so it is not clear how this can scale up. More importantly, how can defining random task scale up beyond the one-hot representations? What about images are inputs? This could be done on these environments. A more detailed connection with Hindsight Experience Replay would also be important.
summary_of_the_review: The paper proposes an interesting way to generate and test random predictions and evaluate the sample efficiency. The experiments are informative, but still leave a bit of doubt in terms of scalability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a generate-and-test approach to discovering auxiliary tasks in RL. They form a multi-headed network where each head outputs a (general) value function for each task. At the last layer, the gradient update is only applied to the weights between each head and each dedicated set of features, and the rest of all the weights are used only in forward passes. The authors compute the magnitude of the value that each feature contributes to the value function for the main task and use it as a measure of how helpful that feature is to the main task. They evaluate their approach in the Four rooms, Maze, and Pinball environments.
strength_and_weaknesses: Strengths

- In the used environments and tasks, the proposed method seems to select helpful subgoals in the state space for subgoal-reaching auxiliary tasks.
- The description of the method is concise and easy to understand. Especially, Alg.1 and Fig.1 are quite helpful for an overview of the proposed approach.

Weaknesses

- I'm mainly concerned about the soundness of the proposed method. If I'm not mistaken, except for the features dedicated to the main task, all the features are connected with the value function for the main task via random weights that are not updated with backpropagation during the training. In my opinion, it means that some auxiliary features might contribute large-scale values to the main value function but with small relevance to the main task, so that some of the main features are trained to "undo" (or compensate for) the irrelevant contribution.
- I think the writing could be improved in some ways. For instance, the measurement of the usefulness of features based on the scales of the contributed values could be summarized and introduced as an overview in the abstract or at least Sec.1 (Introduction). There is a typo regarding the gradient notation in Sec.2. A typo "auxtasks" is in Fig.2.
- I see some issues in the presentation. In Fig.2, the Aux utility plots for "hallway aux tasks 3" and "corner aux tasks 3-4" are missing, which makes the figure less convincing. Also, presenting and comparing the heat maps of the performance improvements and the measured auxiliary utilities in the state space by testing many more subgoals could be quite informative.
- Empirically, I don't fully agree that the proposed algorithm "significantly outperforms" hand-designed tasks. One issue I see in this regard is that since subgoal-reaching auxiliary tasks are simple and easy to create unless the state space is huge, in the given environments and tasks, it may often be enough to form auxiliary tasks by hand.
clarity,_quality,_novelty_and_reproducibility: - In general, the presentation is somewhat clear, but I see some issues (please see my comment above).
- The overall quality could be improved in terms of supporting the main claim, presenting the alignment of the main task performance improvements and the measured auxiliary utilities (please see my comment above)
- In terms of the originality, I see some limited novelty in the measurement of the usefulness of each feature by using the scale of its contribution to the main value function.
summary_of_the_review: While I appreciate their concise description of the method and the meaningful empirical results, I have multiple concerns. My major concern is about the soundness of the method: each utility is measured by involving fixed random weights and their contributions to the main value function could be undone by some of the main features. I have some suggestions regarding the writing and presentation (especially of the experiments) as well.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on designing auxiliary tasks that could tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task. Previous work usually designs auxiliary tasks with human knowledge, which are computationally expensive and challenging to tune in practice.
In this paper, the authors propose continually generating new auxiliary tasks and preserving only those with high utility. They introduce a new measure of auxiliary tasks’ usefulness based on how useful the features induced by them are.
Experiments on the Gridworld and the Pinball environments show that their method outperforms random tasks, hand-designed tasks, and learning without auxiliary tasks.

strength_and_weaknesses: ### **Strength**

* This paper tackles a very interesting idea. Using auxiliary tasks to augment NN has shown improvement in the supervised learning area. This paper uses this idea to reinforcement learning.

* The writing of this paper is generally good. The idea is neat and easy to understand. Figure 1 helps me understand the framework.

### **Weaknesses**

* **Clarification.** 
    * Title is not concrete. According to Section 2, the paper uses the setting of MDP. I think the title should be more concrete by adding reinforcement learning or q-learning since the authors mention that they use q-learning as the base framework.
    * How to measure the influence of auxiliary tasks is the key to the entire paper. A high-level idea of this metric (in my understanding, using the magnitude of features for the main task as an indicator) should be mentioned earlier, for example, in the abstract or introduction.
    * Section 3 can be improved by using subsections to separate contents, for example, generate and test framework, tasks generator, tasks evaluation, etc. The current version put all things together, which makes it hard to find important information.

* **Generalization to other tasks.** At the end of Section 3, the authors discuss the design of the task generator. It seems that the generator just randomly selects sub-goals in the observation space (similar to some curriculum learning algorithms). How to extend this generator to other tasks? Do we still human to design the candidate task?

* **The correctness of the proposed metric.** Following last point, I also suspect that the proposed metric for evaluating the usefulness of tasks only works in limited cases. In the experiments of this paper, the features of different goals may be very different, thus using the magnitude is a good approximation. However, in more complicated situations, such as image input tasks or continuous control tasks, most of the features (representing dynamics or semantic objects) are shared across all tasks, including both good and bad ones. In these environments, I suspect that the magnitude does not make sense. More experiments on complicated environments should be done to investigate this point.


* **Questions to experiments.**
	* In the middle part of Figure 2, why does the curve of corner aux tasks finally reach the same level as the curve of hallway aux tasks? I expect that the bad aux tasks should harm the learning. I guess one explanation is that adding more goals always enables the agent to solve the sparse reward problem as stated in Hindsight Experience Replay (HER)[1].
	* In the middle part of Figure 3, the curve of generate-and-test is much better than the curve of hallway aux tasks. It is a little surprising since hallway aux tasks are reasonable middle points to the goal. 
	* In Figure 4, it seems that useful tasks are all close to the goal. In long-horizon problems, these tasks still have the sparse reward problem. The intuition behind these tasks is not clear.


---

[1] Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O. and Zaremba, W., 2017. Hindsight experience replay. Advances in neural information processing systems, 30.

clarity,_quality,_novelty_and_reproducibility: ### **Quality**
The quality of this paper is above average. However, the evaluation is not thorough enough. The environments are too simple to investigate the correctness of the proposed method in more complicated situations.

### **Clarity** 
The presentation of this paper is good. Section 3 may need to be improved with re-organization.

### **Originality**
As far as I know, using the weight of NN as an indicator to generate tasks for RL agents is an original idea.

### **Reproducibility**
The code is provided and the idea is easy to implement. I think the proposed method is reproducible.

summary_of_the_review: My biggest concern is the proposed method does not fit more complicated tasks. Using the magnitude of features seems to only work in limited situations. I suggest rejecting this paper unless more evidence is provided.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in proposing a generate-and-test approach to discovering auxiliary tasks in RL.

After reviewing the authors' rebuttal and an active discussion, the reviewers agree that this paper is generally well-written and easy to read. The proposed approach is simple, easy to understand, and performs well in the tested environments.

A major concern raised by all reviewers is the lack of demonstration of scalability of the proposed approach beyond the current simple environments to more complex ones. To some reviewers, such a concern remains after reading the authors' response.

The authors are encouraged to revise their work based on the reviewers' comments.
justification_for_why_not_higher_score: See the aforementioned major concern.
justification_for_why_not_lower_score: N/A.
Decision: Reject