Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an evolutionary strategy (ES)-based auxiliary loss search for training neural networks for RL. The main motivation is that while it is known that auxiliary losses can improve training by aiding useful representation learning, it is not clear which formulation of the auxiliary loss is most beneficial since the number of potential possibilities is very large. Therefore, it can be valuable to automate the search through this space in order to identify the most useful loss functions. The paper proposed to use binary-encoded selection variables for the inputs and targets (called *input elements*) of the auxiliary loss, with associated mutation operations for the ES. For the functional form of the loss itself (called *operator*), the paper does not use a search but instead uses simple random sampling to identify that the MSE loss works best when the inputs and targets selection are such that the auxiliary loss becomes forward dynamics. 

Fixing the loss to MSE and using the ES to search for the input elements, the authors identify an auxiliary loss using 3 image-based control environments for training, that leads to improved performance on multiple image-based test environments. It is also able to improve performance on 12 out of 18 tested non-visual RL tasks showing that it can generalize to a variety of tasks. The authors also present aggregated analyses of the losses evaluated during the search procedure to identify common useful patterns among loss functions that improve or reduce performance.
strengths_and_weaknesses: The main strength of this paper is that it is the first study to show the feasibility of automating the search of auxiliary loss functions for RL with results indicating that a useful and novel loss function was indeed discovered through the proposed method, with a relatively simple evolutionary strategy design.

I also appreciate that the authors attempted extensive generalization experiments on several unseen environments and compared to strong baselines.

I have some concerns that reduce my confidence in accepting this paper:

- The biggest concern is that it appears that the identified A2-winner loss is obtained with just 5 generations of the proposed ES (the term *stages* appears to be used in the paper, which I found a little confusing without explanation). This makes me wonder if the ES is actually necessary, and whether a random sampling of 4 or 5 times the population size would already produce equally good results, while being more parallelizable. The authors comment in Sec. 4.1 that "Judging from the trend, we believe the performances could improve even more if we had further increased the budget", but it unclear to me from Fig. 4 that there really is a trend of increasing performance, especially in terms of the best individual in the population improving across the three environments.

- In Table 2, many results for A2-winner at 500K steps are in bold, but their difference from CURL does not appear to be statistically significant.
 
- The main paper omits any discussion of how the target encoder parameters are obtained. This is left completely to the supplementary material, and confused me for the first couple of readings. 

- The paper is notoriously missing references from the ES literature, focusing only on references to some recent papers that apply ES to RL.
questions: Did I understand correctly above that A2-winner was obtained with 5 generations of the ES? If so, can the authors offer convincing arguments that the proposed ES is necessary, and address related comments above?

Is appears that at 500K, the performance with A2-winner might not be too different (statistically) from CURL. Is that correct?
limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to do automated search for reinforcement learning auxiliary losses. This is done by formulating a configuration for possible losses, which covers many of the common losses used in the field (e.g. contrastive, self-predictive), and searching using evolutionary search. This leads to a bi-level optimization: outer loop -- proposing, selecting and evolving candidate loss functions -- and inner loop, where the candidate loss function is evaluated by training an RL agent together with the candidate loss, and using the overall performance as a metric for the outer loop. 

The method is applied to continuous control DeepMind Control Suite, where an auxiliary loss (A2-winner) is found that consistently outperforms the baselines. Moreover, this loss has similarly strong results on different test environments, including e,g, some with different observation types or discrete control domain (Atari).
strengths_and_weaknesses: The paper has many strong aspects: clarity, thorough evaluation of the proposed candidate loss on many different test environments and originality in applying the AutoML framework to the space of auxiliary losses for RL and in, among others, proposing a loss configuration that covers most of the hand-engineered loss functions, as well as being interpretable (the proposed A2-winner seeming like a combination of reward prediction and forward dynamics). 

The method is expensive, requiring tens of thousands of GPU hours, and there is room for improvements in efficiency, which is also remarked by the authors. However, this does not represent a weakness with respect to the work's significance, as important insights can already be drawn from the presented empirical evidence and, further, the results of A2-winner loss seems to generalize to environments on which the search was not performed. 

However, presenting more information about the inner loop and its comparison with the baselines (hyper-parameters, choosing criteria) could be highly beneficial to leveraging the work in the future and answering questions such as: do these findings apply to value-based, as well as policy-based methods? 

Overall, as most aspects are highly positive, the paper meets the acceptance criteria. 
questions: In addition to the points above (including information about the inner loop algorithm in the main paper and adding a short discussion on how the baselines were chosen), the following aspects might be useful to consider:
- How does the method compare with other AutoML methods applied to the task? 
- Do the findings apply when tested in the setting of task generalization? 
- What values of lambda (eq 1) were tried and how do the results compare?
- Section 3.2.1 provides a very specific method to prune the search space, seemingly inspired by empirical evidence, rather than a general assumption. More discussion on results/generality could make its scope clearer.
- Does the search space include reconstruction losses at the moment? 
limitations: The authors have adequately noted and addressed the limitations of the proposed method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to use an evolutionary strategy to search the best auxiliary losses for reinforcement learning. The main technical contribution is the design of a search space that covers a set of common auxiliary losses such as the forward dynamics, inverse dynamics and reward prediction. Empirical experiments show that, a loss searched on 3 image-based DM Control tasks also brings better sample efficiency on other DM Control tasks and Atari games.
strengths_and_weaknesses: ### Strengths
* I like the overall direction of the paper. It did a good job exploring whether we can find a good set of auxiliary losses that transfer to different tasks. It also provides insights (through empirical experiments) what are important auxiliary tasks.
* The experiments are extensive well presented to support the main claims of the paper. The generalization of the searched auxiliary loss are demonstrated on different tasks as well as on different RL algorithms (SAC and Efficient Rainbow) and different input observation.

### Weakness
1. A main question that is not fully addressed in the paper is, to what degree is the searched auxiliary loss (i.e. A2-winnder) universal? When does the auxiliary loss transfer and when does it not? To provide two angles to explore in this aspect:
    * The searched loss in Eqn. 3 involves states up to 3 time steps into the future. Where does this horizon of 3 come from? In another environment where the time is discretized differently, either more frequently or less frequently, should this loss be adapted accordingly? One experiment that can be done is, for the currently tested environment, if we add action repeat i.e. repeat each action for two steps, would the optimal auxiliary loss change?
    * The tasks evaluated in this paper are all 2D tasks. How about the more complicated 3D manipulation tasks, such as meta-world [1], 
which is also commonly used for evaluating RL algorithms? 

2. Some related works are missing. There are a body of works on balancing multiple auxiliary tasks for reinforcement learning [2,3,4]. These works discuss the interplay between the auxiliary tasks and the main task, as well as among the auxiliary tasks. Specifically, Lin et al. [2] also considers a set of auxiliary tasks are also considered, including the forward and inverse dynamics. Lin et al. [2] proposes to tune the weights of the auxiliary tasks based on the main task performance. In a way, it provides a more efficient way of evaluating an auxiliary loss, and also provides an extention of the search space in the submitted paper by including the weights of the auxiliary losses.

I am willing to raise my score if these concerns can be sufficiently addressed.

[1] Yu, Tianhe, et al. "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning." Conference on robot learning. PMLR, 2020.

[2] Lin, Xingyu, et al. "Adaptive auxiliary task weighting for reinforcement learning." Advances in neural information processing systems 32 (2019).

[3] Shi, Baifeng, et al. "Auxiliary task reweighting for minimum-data learning." Advances in Neural Information Processing Systems 33 (2020): 7148-7160.

[4] Chen, Zhao, et al. "Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks." International conference on machine learning. PMLR, 2018. 
questions: Other suggestions:
1. Eqn. 1 seems a bit odd. Maybe it's better to write it as $\max_{\mathcal{L}_{aux}} \mathcal{R}(\min_w \text{losses})$

2. The normalized AUC score is not explained. Normalized w.r.t. what?
limitations: Please see above. Since searching over auxiliary losses is computationally expensive and unlikely for other researchers to search for each task they are interested in, I think it is important to state clearly when does the searched auxiliary task transfer and when it does not.
review_title: null

Meta-review: metareview: The paper introduces an interesting evolutionary scheme for black-box hyper-parameter optimisation for representation learning, looks like a useful and general tool for optimising RL-agent in case the compute is not a constraint.
confidence: Certain
Decision: Accept