Auxiliary task discovery through generate and test
Authors: Banafsheh Rafiee
Sina Ghiassian
Jun Jin
Richard S. Sutton
Jun Luo
Adam White
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks, hand-designed tasks, and learning without auxiliary tasks across a suite of environments. 

Under review as a conference paper at ICLR 2023
AUXILIARY TASK DISCOVERY THROUGH GENERATE- AND-TEST
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we explore an approach to auxiliary task discovery in reinforce- ment learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discov- ery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task. Our discov- ery algorithm significantly outperforms random tasks, hand-designed tasks, and learning without auxiliary tasks across suite of environments.
INTRODUCTION
The discovery question-whal should an agent learn about-remains an open challenge for Al re- search. In the context of reinforcement learning, multiple components define the scope of what the agent is learning about. The agent's 's behavior defines its focus and attention in terms of data collection. Related exploration methods based on intrinsic rewards define what the agent chooses to do outside of reward maximization. Most directly, the auxiliary learning objectives we build in, including macro actions or options, models, and representation learning objectives force the agent to learn about other things beyond reward maximizing policy. The primary question is where do these auxiliary learning objectives come from?
Classically, there are two approaches to defining auxiliary objectives that are the extremes of a spec- trum of possibilities. The most common approach is for people to build the auxiliary objectives in pre-defining option policies, intrinsic rewards, and model learning objectives. Although most empir- ically successful, this approach has obvious limitations like feature engineering of old. At the other extreme is end-to-end learning. The idea is to build in little inductive bias possible including the inductive biases introduced by auxiliary learning objectives. Instead, we let the agent's neural network discover and adapt internal representations and algorithmic components (e.g.. discovering objectives (Xu et al. 2020). update rules (Oh et al. 2020), and models (Silver et al. (017]) just through trial and error interaction with the world. This approach remains challenging due data efficiency concerns and in some cases shifts the difficulty from auxiliary objective design to loss function and curriculum design.
An alternative approach that exists somewhere between human design and end-to-end learning is to hand-design many tasks in the form of additional output heads on the network that must be optimized in addition to the primary learning signal. These tasks, called auxiliary tasks, exert pressure on the lower layers of the neural network during training, yielding agents that can learn faster (Mirowski et al. 2016 Shelhamer et al. 2016), produce better final performance (Jaderberg et al. 20166), and at times transfer to other related problems (Wang et al., 2022). This positive influence on neural network training is called the auxiliary task effect and is related to the emergence of good internal representations we seek in end-to-end learning. The major weakness of auxiliary task learning is its dependence on people. Relying on people for designing auxiliary tasks is not ideal because is

---

Under review as a conference paper at ICLR 2023
challenging to know what auxiliary tasks will be useful in advance and, as we will show later, poorly specified auxiliary tasks can significantly slow learning.
There has been relatively little work autonomously discovering auxiliary tasks. One approach is to use meta learning. Meta-learning methods are higher-level learning methods that adapt the pa- rameters of the base learning system, such as step-sizes, through gradient descent (Xu et al. 2018). This approach can be applied to learning auxiliary tasks defined via General Value Functions or GVFs (Sutton al. T by adapting the parameters that define the goal (cumulant) and termina- tion functions via gradient-deseen (Veeriah et al. 2019). Generally speaking, these meta-learning approaches require large amounts of training data and are notoriously difficult to tune (Antoniou et al. 2018).
An exciting alternative is to augment these meta-learning approaches with generate-and-or mech- anisms that can discover new auxiliary tasks, which can later be refined via meta-learning. This approach has produced promising results in representation learning where simple generate-and- test significantly improve classification and regression performance when combined with back-prop (Dohare et al., 202I). Before we can combine meta-learning and generate-and and-tes,, we must first develop the generate-and-test approach to auxiliary task discovery so that their combination has the best chance for success. Such an effort is worthy of an entire study on its own, so in this paper we leave combining the two to future work and focus on the generate- and-tesst approach.
Despite significant interest, it remains unclear what makes a good or bad auxiliary tasks. The meta- learning approaches do not generate human- numanninterpretablee tasks. Updating toward multiple previous policies, called the value improvement path (Dabney et al., 2020), can improve performance but limited to historical tasks. The gradient alignment between auxiliary tasks and the main task has been proposed as a measure of auxiliary tasks usefulness (Lin et al.) 2019 Du et al. 2018). However, the efficacy of this measure has not been thoroughly studied. Randomly generated auxiliary tasks can help avoid representation collapse (Lyle et al., (021) and improve performance (Zheng et al 2021). but can also generate significant interference which degrades performance (Wang et al. 2022).
In this paper we take step toward understanding what makes useful auxiliary tasks introducing a new generate-and-
method for autonomously generating new auxiliary tasks and a new measure of task usefulness to prune away bad ones. The proposed measure of task usefulness evaluates the auxiliary tasks based on how useful the features induced by them are for the main task. Our experimental results shows that our measure of task usefulness successfully distinguishes between the good and bad auxiliary tasks. Moreover, our proposed generate- tnd-test method outperforms random tasks, deninnesigned tasks, and learning without auxiliary tasks.
BACKGROUND
In this paper, we consider the interaction of an agent with its environment at discrete time steps t 1, 2, The current state is denoted by S, S. The agent's action A, E A is selected according to a policy π. : A X S + [0, 1). causing the environment to transition to the next state S.+1 emitting reward of R4+1 IR. The goal of the agent is find the policy x with the highest state- action value function defined as qu (s, a) = E, [G s, A, a) where G. = En= Tat R called the return with 7 [0, 1) being the discount factor.
To estimate the state-action value function, we use lemporal-differene learning Sutton 1988 Specifically, we use Q-learning (Watkins & Dayan 1992) to learn parametric approximation ((s, w) by updating a vector of parameters w R The update is as follows,
Wc+1 +- Wt + ad; V ((
where 6, = Re+1 q wt) g(S, A,; is the TD error, Vwt(Sp w) is the gradient of the value function with respect to the parameters Wt. and the scalar a denotes the step- size parameter. For action selection, Q-learning is commonly combined with epsilon greedy policy.
We use neural networks for function approximation. We integrate a replay buffer, target network, and the RMSProp optimizer with Q-learnin as is commonly done to improve performance (Mnih et al. 2013).
2

---

Under review as a conference paper at ICLR 2023
To formulate auxiliary tasks, common approach is to general value functions GVFs (Sutton et al. 2011). GVFs are value functions with a generalized notion of target and termination. More specifically, GVF can be written as the expectation of the discounted sum of any signal of interest:
20 h
Ux ( Em
E R=O ==1
((5+)))(((HH))))55 = =5, An:00 π.
where is the policy, is the continuation function, and is signal of interest and is referred to as the cumulant. Similarly, a generalized state-action value function ( (s, a) can be defined where the expectation is conditioned on A, where the agent attempts to learn a π to maximize the expected discounted sum of the future signal of interest (called control demon or control GVF in prior work).
well St A control auxiliary tasks is one
To learn these auxiliary tasks, multi-headed neural networks are commonly used where the last hid- den layer acts as the representation shared between the main task and the auxiliary tasks (Jaderberg) iet al., 2016). In this setting, each head corresponds to either the main task or one of the auxil- ary tasks and the auxiliary tasks make changes the representation alongside the main task via backpropagation.
3 AUXILIARY TASK DISCOVERY THROUGH GENERATE-AND-TE
We propose a new method for auxiliary task discovery based on a class of algorithms called generate- and-test. Generate-and-ter was originally proposed an approach to representation learning fea- ture finding. We can think of backprop with a large neural network as performing a massive parallel search in feature space (Frankle & Carbin 2018). Backprop greatly depends the weight initialization to find good features. The idea of generate and test is to continually inject randomness in the feature search by continually proposing new features using a generator, to mea- sure features usefulness using a tester, and to discard useless features. This idea has long history in supervised learning (Sutton al.) 2014] Mahmood Sutton 2013). and can even be combined with backprop (Dohare et al. 2021). The same basic structure can be applied to auxiliary task discovery, which we explain next.
the randomness in
We use generate-ande and-rtee--dd-estt for discovering and retaining auxiliary tasks that induce representation useful for learning the main task. That is, the goal is to find auxiliary tasks that induce positive auxiliary task effect. It is challenging to recognize which auxiliary tasks induce useful representa- tions. To so, we first evaluate how good each feature is based on how much it contributes to the approximation of the main task action-value function. Here define the features to be the output of neural network's 's last hidden layer after applying the activation function. We then identify which auxiliary task was responsible for shaping which features.
Our proposed generate-and-tert -nd- method for discovering auxiliary tasks consists of a generator and tester. The generator generates new auxiliary tasks and the tester evaluates the auxiliary tasks. The auxiliary tasks that are assessed as useful are retained while the auxiliary tasks that are assessed to be useless are replaced by newly generated auxiliary tasks. The newly generated auxiliary tasks will most likely have low utility. To prevent the replacement of newly generated auxiliary tasks, we calculate the number of steps since their generation and refer that as their age. An auxiliary task can only be replaced if its age is bigger than some age threshold. Every T time step, some ratio of the auxiliary tasks get replaced. We refer T as the replacement cycle and denote the replacement ratio by p. The pseudo-code for the proposed generate-and-ter method is shown in Algorithm []
Note that the proposed method does not generate-ande and-test on features but on auxiliary tasks. It, however, does assess the utility of features and derives the utility of the auxiliary tasks from the utility of the features that they induced.
We propose tester that evaluates the auxiliary tasks based on how useful the features induced by them are for the main task. When following the standard practice of jointly learning the main task and the auxiliary tasks, recognizing which feature was influenced the most by which auxiliary task is challenging. This is because all features are jointly shaped by all the tasks, both auxiliary and main. To address this issue, used by all tasks in the forward pass; however, each feature is only modified through the gradient backpropagated from one task. See Figure]] This learning strategy is similar to the Master-User
a strategy for learning the representation where all features
3

---

Under review as a conference paper at ICLR 2023
Algorithm Generate-and-test for auxiliary task discovery
Input: number of auxiliary tasks n., age threshold μ., replacement eycle replacement ratio p
2: Initialization:
3: generate auxiliary tasks using the generator
4: randomly initialize the base learning network
5: set age a; for each auxiliary task to zero
6: for Every time step do
7: do DQN step to update the base learning network
8: Increase a, by one for L....
9: update the utility of each auxiliary task uaux() (i) for i ..., n using the tester
10:
for Every T time steps do
11:
Find no auxiliary tasks with the lowest utilities such that 02 >
replace the np auxiliary tasks with new auxiliary tasks generated by the generator reinitialize the input and output weights of the features induced by the np auxiliary tasks reset a, zero for the np auxiliary tasks
13:
forward
stop gradient connections:
task backprop path
task 1 backprop path
features
teatures
features
main task
task action-value function
action-value
action-valuu
function
function
Input
input
task action-value function
task 1
task action-vvalue function
action-value function
task action-value
aux task 2 action-value function
task 2 action-valuu function
function
Figure 1: The forward pass, backward pass for the main task, and backward pass for auxiliary task 1 when using the Master-User strategy for learning auxiliary tasks alongside the main task. All fea- tures are used by all tasks in the forward pass but only modified through the gradient backpropagated from one task. The dotted arrows show stop-gradee nop-gradient connections. The gradients does not go back any further from these connections. When using the Master-User strategy, it is clear which auxiliary task was responsible for inducing which feature.
algorithm proposed for continual recurrent learning (Javed et al. learning strategy as the Master-User strategy. When using the Master-User strategy it is clear which auxiliary task was responsible for inducing which feature.
2021). Therefore, refer this
As we mentioned above, the proposed tester assesses the utility of an auxiliary task based on how useful the features induced by looks at the magnitude of the outgoing weights from the feature to the main task action-value func- tion for all actions. The magnitude of the weights represents how much the feature contributes the approximation of the main task action-value function. The greater the magnitude is, the more impor- tant the feature The tester also considers the magnitude each feature: the greater the magnitude of the feature is, the more it contributes to the approximation of the main task action-value function. Therefore, the instantaneous utility of feature f is defined as:
for the main task. To assess each feature, the proposed tester
ul w(ftt =
Σ
to ka
main
x fil
(I)
4

---

Under review as a conference paper at ICLR 2023
where fi is the kth feature shaped by auxiliary task i and u(f() is the instantaneous utility of feature fk- Instead of only looking at the instantaneous utility u(f(), the proposed tester considers trace of the past utilities using an exponential moving average of ('f
+ f (1-1) +u(fi)
where ((f() is trace of u u(ft) with the trace parameter denoted by This assessment method is similar to what has been used in generate-and-test on features (Mahmood & Sutton 2013).
After assessing the utility of the features, the utility of each auxiliary task is set to the sum of the utility of the features shaped by it: (1) = >n a(f()-
We combined the proposed tester with simple generator that randomly generates auxiliary tasks. The auxiliary task are formulated as aabboal-reaching GVFs where the continuation function returns 0 at the subgoals and elsewhere (similar y in epsiodic MDP). The cumulant is everywhere and the policy is greedy. In plain english, b The subgoals are randomly selected from the observation space, meaning the agent learning many policy reach different parts of the observation space in addition to solving the main task.
4 EXPERIMENTAL RESULTS
In this section, we provide empirical results supporting the efficacy of the proposed generate-and-test method for auxiliary task discovery. We include results two gridworld environments: four-rooms and maze. We also include results on the pinball environment (Konidaris & Barto 2009), which is widely used in skill chaining, option discovery, and recently model- planning (Lo al. 2022). We choose these environments so that we could easily visualize the discovered auxiliary tasks and casily design good and bad auxiliary task as baselines. All environments are episodic.
In the gridworld environments, the goal is to learn the shortest path from start state to goal. The start and goal states are denoted by S and G respectively in Figure 2 and At each cell. four actions are available: up, down, left, and right, which moves the agent one cell in the respective direction. The observation space is described with a one-hot representation with the index corresponding to the agent's position being 1. The reward is -1 1 on each time step. There is an episode cutoff of 500 steps.
In the pinball environment, a small ball should be navigated to the goal in maze-like environment with simplified ball physics. In Figure 3 the pinball environment is shown with the ball and goal shown by grey and yellow circle respectively. Collision with the obstacles causes the ball to bounce. The observation space is continuous and is described by x, y, I, ÿ. The start location and goal location at (0.8, 0.5) and (0.1, 0.1) respectively. The action space includes 5 actions of increasing or decreasing á or ý and no change to 4 and y. The reward is -5 at each time step. There no episode cutoff.
Note that in the original pinball environment, the agent receives a special reward of 10, 000 upon arrival at the goal. Instead we gave a reward of -5 (like every other step) so that the scale of the action-value function for the main task and the auxiliary tasks would not be too different. When learning multiple tasks in parallel, the contribution of each task is determined by the scale of the corresponding value function (Hessel et al., 2019). Therefore, when the scale of value functions are very different, we would need scale the reward of the main task and the cumulants of the auxiliary tasks appropriately. This issue requires an additional hyper parameter that would give our method an advantage if tuned. For this paper, we decided to focus on the case where the scale of the value function for the main task and the auxiliary tasks are similar.
We used DQN with RMSProp optimizer as the base learning system. We used a neural network with one hidden layer and tanh activation function. (We used tanh activation function so that the induced features would be all in the same range of (-1, 1); however, our proposed tester should work well when other activation functions For the girdworld environments, the one-hot observation vector was fed to the neural network. The hidden layer size for four-rooms and maze were 50 and 500 respectively. The replay buffer size for
used too. This can be investigated in future work.)
5

---

Under review as a conference paper at ICLR 2023
500
corner tasks
30
hallway aux tasks
corner
steps to goal
300
utility
20
hallway auxtasks
(30 runs)
aux asks
(30 run)
corner tasks
10
Hallway
2
100
corner aux tasks 2
hallway
0
0
50
100
150
200
0
50
100
150
200
episode
episode
Figure 2: Left: The four-rooms environment with the good and bad hand-designed auxiliary tasks shown in red and blue respectively. Middle: Hallway and corner auxiliary tasks improved and hurt the performance respectively. Right: The proposed tester evaluated the hand-designed auxiliary tasks well, giving higher utility to the hallway auxiliary tasks.
four-rooms and maze were 500 and 1000 respectively. For both four-rooms and maze, we used batch-size of and target network update frequency of 100.
For the pinball environment, the 4-dimensional observation was normalized and fed to the neural network. The hidden layer size was 128. We used a replay buffer of size 10, 000, batch-size 16, and target network update frequency of 200.
4.1 THE PROPOSED TESTER REASONABLY EVALUATES THE AUXILIARY TASKS
To see how well the proposed tester evaluates the auxiliary tasks, we designed good and bad auxiliary tasks in the four-rooms environment. The hand-designed auxiliary task were formulated as subgoal- reaching GVFs with the good and bad hand-designed auxiliary tasks having hallway and comner subgoals respectively. See Figure 2
Note that when learning the auxiliary tasks alongside the main task using the Master-User strategy, the gradient backpropagated from the main task only modifies auxiliary tasks+1 percent of the features. For example, the case of learning the hallway auxiliary tasks, there are 3 auxiliary tasks. There- fore, the gradient backpropagated from the main task only modifies 25% of the features.
The hallway and corner auxiliary tasks improved and hurt learning in terms of learning speed respec- tively as expected (Figure 2 middle graph). The proposed tester evaluated the hallway and corner auxiliary tasks well, assigning higher utility to the hallway auxiliary tasks and clearly indicating the corner tasks are bad.
4.2 THE TASKS
METHOD IMPROVES OVER THE BASELINE OF NO AUXILIARY
Next, we studied the performance of the base learning system when combined with the proposed generate-and-test method. The generate-and-test method uses the combination of the random gen- erator and our proposed tester. The random generator produces subgoal-reaching auxiliary tasks with the subgoals randomly picked from the observation space. More specifically, the gridworld environments, the subgoals are cells in the grid. In the pinball environment, the subgoals are deter- mined by (ar, y) and once the ball within radius 0.035 of subgoal, assumed that the agent has reached the subgoal.
We included four baselines for comparison which included the base learning system with 1) no auxiliary tasks 2) hand-designed good auxiliary tasks 3) hand-designed bad auxiliary tasks 4) fixed random auxiliary tasks. All the auxiliary tasks were in form subgoal- -eagoal--eaching tasks. The subgoals corresponding the hand-designed good and bad auxiliary tasks for all three environments shown in red and blue respectively in Figure 3 For the fixed random auxiliary tasks, the subgoals where randomly picked from the observation space and kept fixed throughout learning.

---

Under review as a conference paper at ICLR 2023
We systematically swept the step-size parameter and report the performance of the best ensure fair comparison. To do so, we ran the baseline with no auxiliary tasks with different values of the step-size for 10 runs. We used the step-size that resulted in the lowest area under the curve and reran the baseline with the best step-size for 30 runs to get the final results. We repeated this process for the baselines with hand-designed auxiliary tasks. For the generate-and-test method, used the same step-size as the baseline with good hand-designed auxiliary tasks. For four-rooms, maze, and pinball the sweep over the step-sizes included (0.000625, 0.0025, 0.01, 0.04), {0.00025, 0.001, 0.004), and {0.0025, 0.005, 0.01}.
The generate generate-and-test and-test method has hyper- pprer-aasmeters of its own: number of auxiliary tasks 2) age threshold 3) replacement cycle 4) replacement ratio. For the gridworld environments, we used 8 auxiliary tasks, age threshold of replacement rate of 1000 steps, and replacement ratio of 0.25. For the pinball environment, we used 5 auxiliary tasks, age threshold of 5000, replacement rate of 5000 steps, and replacement ratio of 0.2.
The proposed generate-and-test method outperformed the baseline with no auxiliary tasks in all three environments (Figure 3). The generate-and-tes method also outperformed the baseline with fixed random auxiliary tasks. This suggests that the subgoals discovered by the generate-and-test are actually better than random subgoals.
Interestingly, the fixed random auxiliary tasks resulted in performance gain the bascline with no auxiliary tasks in all three environments (Figure 3) This is in line with the findings from the literature suggesting that random GVFs can form good auxiliary tasks for reinforcement learning ZZeng et al. 2021).
Four-rooms
Maze
Pinball
A
500
no aux tasks
500
corner tasks
3k
random aux tasks
corner aux tasks
tasks
corner aux tasks
steps
2k
to goal
300
randim aux
300
random aux tasks
(30 runs)
tasks
centre aux tasks
gen-and-test
1k
100
hallway
100
hallway BUX tasks
no aux tasks
aux tasks
gen-and---ee
gen-and-test
50
100
150
200
0
100
200
300
50
100
150
episode
episode
episode
Figure 3: The learning curves for the proposed generate-and and-ttee-ndd-est method (green), the baseline with no auxiliary tasks (orange), the baseline with fixed random auxiliary tasks (black), and the baseline with good and bad hand-designed auxiliary tasks (red and blue). The results are averaged over 30 runs. The proposed The hand-designed good and bad auxiliary tasks respectively improved and hurt the performance. Generate-and-tes also outperformed the baseline with fixed random auxiliary tasks. Fixed random auxiliary tasks also resulted in performance gain over the baseline.
method improved over the baseline with no auxiliary tasks.
4.3 THE AUXILIARY TASKS DISCOVERED BY THE DENERATE-AND-TER METHOD ARE REASONABLY GOOD
We conducted additional experiments analyze how good the auxiliary tasks discovered by the generate-and-tee method are. In the previous subsection, we compared the performance of the base learning system combined with generate-and-test and combined with fixed random auxiliary

---

Under review as a conference paper at ICLR 2023
tasks. Generate-ande and-test outperformed fixed random auxiliary tasks(Figure 3) This suggests that the choice of the auxiliary tasks was important and generate-and-ter discovered and retained useful auxiliary tasks.
The auxiliary tasks discovered and retained by generate-and are shown Figure 4 discovered auxiliary tasks, ran the generate-and-test method for 30 runs and stored the auxiliary tasks that were retained. The green squares correspond to the discovered auxiliary tasks in the gridworld environments. Darker green indicates that the state runs. For the pinball environment, the discovered auxiliary tasks are shown by green circles. In the gridworld environments, the subgoals corresponding to the discovered auxiliary tasks were close to the goal states. In the pinball environment, the discovered auxiliary tasks were more concentrated the central areas reasonable way-points on the path to the goal.
To plot the
chosen
subgoal in many
Four-rooms
Maze
Pinball
Figure 4: Example discovered auxiliary tasks in the three environments. Generate-and-ter discov- ered reasonably good auxiliary tasks: In the gridworld environments, the subgoals corresponding to the discovered auxiliary tasks were close the goal states. the pinball environment, the discov- ered auxiliary tasks were more concentrated in the central areas.
To confirm that the auxiliary tasks discovered by generate-and-test were useful, we stored the aux- iliary tasks discovered and retained, in a pool. We then randomly selected number of auxiliary tasks from the pool and ran the base learning system, learning the main value function from scratch. We kept the auxiliary tasks fixed throughout learning, We repeated this for 30 runs. The discovered auxiliary tasks were useful and substantially improved over the baseline of no auxiliary tasks (Figure
500
500
3k
aux tasks
tasks
steps goal runs)
2k
tasks
300
discovered aux tasks
gen-and-test
1k
100
100
gen-and-te -tetd
discovered aux taskka
discovered BUX Taaks
50
100
150
200
100
200
300
50
100
150
episode
episode
episode
The learning curves corresponding the discovered auxiliary tasks. The discovered auxiliary tasks were kept fixed throughout learning. The discovered auxiliary tasks improved over the baseline of auxiliary tasks.
Figure 5:
4.4 THE REPRESENTATION INDUCED BY THE GENERATEE GENERATE-ANDD-TER METHOD HAS LOWER REDUNDANCY COMPARED THE BASELINE.
Next, we studied the representation that emerged under the generate-and-tes method. There are multitude of metrics for evaluating the emergent representations (Wang et al., [022 Javed White 2019). We used the stable rank of the weight matrix between the input layer and the hidden layer of where a, are the the singular Arora et al. 2018 The stable rank of a matrix A is defined as manya?
values of matrix A. The stable rank provides approximation of the rank of the matrix but it is
8

---

Under review as a conference paper at ICLR 2023
unaffected by the smaller singular values. The stable rank of the weight matrix between the input layer and the hidden layer characterizes the amount of generalization/rcddddany of the network. The larger the stable rank is, the lower the redundancy in the representation is.
The stable rank of the representation learned by generate-and-Ee is larger than the stable rank of the representation learned by the baseline with no auxiliary tasks (Figure This suggests that the auxiliary tasks discovered by generate-and-He and-test resulted a representation with lower redundancy. The lower stable rank of the learned representation together with the better performance encour- aging and suggests that more aggressive pruning by the tester could get a lower stable rank while maintaining good performance.
Four-rooms
Maze
Pinball
12
2.4
40
10
2.2
rep.
8
30
2.0
gen-and-test
stable rank
6-
1.8
(10 runs)
4-
10
1.4
1.6
no aux tasks
2
1.2
100
200
0
100
200
300
100 episode
200
episode
episode
Figure 6: The stable rank of the weight matrix from the input layer to the hidden layer over episodes. The results averaged over 10 runs. The combination of the base learning system with the generate-and-test method resulted case of only having the base learning system. This suggests that the auxiliary tasks discovered by generate-and-test resulted in representation with lower redundancy.
representation with a higher stable rank compared to the
CONCLUSIONS AND FUTURE WORK
In this paper, we proposed a new method for auxiliary task discovery. The proposed method uses generate -enerate-and-test approach. We also introduced a measure of auxiliary task usefulness. Through careful experimentation, we showed that: 1) The proposed tester reasonably evaluates the auxiliary tasks. 2) The generate-and-test method improves over the baseline with auxiliary tasks. 3) The auxiliary tasks discovered by the generate-and and-test method are reasonably good. 4) The representation induced by the auxiliary tasks discovered by generate- generate-and-tes and-test has lower redundancy compared to the baseline with auxiliary tasks.
interesting future work would be investigate different generators and testers for auxiliary tasks. For example, to improve the generator, one idea is to to sample subgoals from the replay buffer in- stead of generating random subgoals. This would similar to the idea of hindsight experience replay (Andrychowicz et al. 2017). Another idea to test for the generator is to generate feature- attainment auxiliary tasks instead of subgoal-reaching auxiliary tasks. For Teature-attainment auxil- iary tasks, the goal is to maximize feature of the state representation (Sutton et al. 2022). In this paper, we took a first step toward designing a functional generate-and andeeee-nn--sss-- method for auxiliary task discovery. However, there is big space of ideas to try for designing the generator and the tester.
Another future direction is to combine the meta-learning approaches with the proposed generate- and-test method. The meta-leaming approaches can be used to refine the auxiliary tasks discovered by generate-and-test. It would be interesting to test the combination of meta-learning with generate- and-test in the three tested environments and see if their combination will result in discovering good auxiliary tasks, outperforming both approaches in isolation.
REFERENCES
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.

---

Under review as a conference paper at ICLR 2023
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:1810.09502, 2018.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference Machine Learning, pp- 263-263. PMLR, 2018.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. arXiv preprint arXiv:2006.02243, 2020.
Shibhansh Dohare, Rupam Mahmood, and Richard Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2 2021.
Yunshu Du, Wojeiech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, arXiv preprint arXiv: 1812.0024, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint
2018
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1.05997, 2016.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Khurram Javed, Martha White, and Rich Sutton. Scalable online recurrent learning using columnar neural networks. arXiv preprint arXiv:2103.05787. 2021.
George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 22, 2009.
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Chunlok Lo, Gabor Mihucz, Adam White, Farzane Aminmansour, and Martha White. Goal-space planning with subgoal models. arXiv preprint arXiv:2206.02902, 2022..
Clare Lyle, Mark Rowland, Georg Ostrovski. and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, pp. 1-9. PMLR, 2021.
Ashique Rupam Mahmood and Richard S Sutton. Representation search through generate and test. In AAAI Workshop: Learning Rich Representations from Low-Level Sensors, volume 10, 2013.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03671 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13255213/ 2013.
Junhyuk Oh, Mattco Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 13:00-0117 2020.
Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self- supervision for reinforcement learning. arXiv preprint arXiv: 1612,0732777 2016.
10

---

Under review as a conference paper at ICLR 2023
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Amold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191-3199. PMLR, 2017.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsu- pervised sensorimotor interaction. The 10th International Conference on Autonomous Agenis and Multiagent Systems- Volame 2, pp. 761-768, 2011.
Richard S Sutton, Steven D Whitehead, al. Online learning with random representations. In Proceedings of the Tenth International Conference on Machine Learning, pp. 314-321, 2014.
Richard S Sutton, Marlos C Machado, G Zacharias Holland, David Szepesvari Finbarr Timbers, Brian Tanner, and Adam White. neward-respecting subtasks for model-based reinforcement learning. arXiv preprint 2tXXi::2022.34666 2022.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 32, 2019.
Han Wang, Erfan Miahi, Martha White, Marlos C Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. arXiv preprint arXiv:2203. 15955, 2022.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):299-292, 1992.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
Zhongwen Xu, Hado P van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 33:15254-1526A 2020.
Zeyu Zheng, Vivek Veeriah, Risto Vuorio, Richard I. Lewis, and Satinder Singh. Learning state rep- resentations from random deep action-conditional predictions. Advances in Neural Information Processing Systems, 34, 2021.


Reviews:
[{"review_id": "X8ubqDN39M", "reviewer": "Reviewer_CKEP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on designing auxiliary tasks that could tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task. Previous work usually designs auxiliary tasks with human knowledge, which are computationally expensive and challenging to tune in practice.\nIn this paper, the authors propose continually generating new auxiliary tasks and preserving only those with high utility. They introduce a new measure of auxiliary tasks’ usefulness based on how useful the features induced by them are.\nExperiments on the Gridworld and the Pinball environments show that their method outperforms random tasks, hand-designed tasks, and learning without auxiliary tasks.\n", "summary_of_the_review": "My biggest concern is the proposed method does not fit more complicated tasks. Using the magnitude of features seems to only work in limited situations. I suggest rejecting this paper unless more evidence is provided.", "strength_and_weaknesses": "### **Strength**\n\n* This paper tackles a very interesting idea. Using auxiliary tasks to augment NN has shown improvement in the supervised learning area. This paper uses this idea to reinforcement learning.\n\n* The writing of this paper is generally good. The idea is neat and easy to understand. Figure 1 helps me understand the framework.\n\n### **Weaknesses**\n\n* **Clarification.** \n    * Title is not concrete. According to Section 2, the paper uses the setting of MDP. I think the title should be more concrete by adding reinforcement learning or q-learning since the authors mention that they use q-learning as the base framework.\n    * How to measure the influence of auxiliary tasks is the key to the entire paper. A high-level idea of this metric (in my understanding, using the magnitude of features for the main task as an indicator) should be mentioned earlier, for example, in the abstract or introduction.\n    * Section 3 can be improved by using subsections to separate contents, for example, generate and test framework, tasks generator, tasks evaluation, etc. The current version put all things together, which makes it hard to find important information.\n\n* **Generalization to other tasks.** At the end of Section 3, the authors discuss the design of the task generator. It seems that the generator just randomly selects sub-goals in the observation space (similar to some curriculum learning algorithms). How to extend this generator to other tasks? Do we still human to design the candidate task?\n\n* **The correctness of the proposed metric.** Following last point, I also suspect that the proposed metric for evaluating the usefulness of tasks only works in limited cases. In the experiments of this paper, the features of different goals may be very different, thus using the magnitude is a good approximation. However, in more complicated situations, such as image input tasks or continuous control tasks, most of the features (representing dynamics or semantic objects) are shared across all tasks, including both good and bad ones. In these environments, I suspect that the magnitude does not make sense. More experiments on complicated environments should be done to investigate this point.\n\n\n* **Questions to experiments.**\n\t* In the middle part of Figure 2, why does the curve of corner aux tasks finally reach the same level as the curve of hallway aux tasks? I expect that the bad aux tasks should harm the learning. I guess one explanation is that adding more goals always enables the agent to solve the sparse reward problem as stated in Hindsight Experience Replay (HER)[1].\n\t* In the middle part of Figure 3, the curve of generate-and-test is much better than the curve of hallway aux tasks. It is a little surprising since hallway aux tasks are reasonable middle points to the goal. \n\t* In Figure 4, it seems that useful tasks are all close to the goal. In long-horizon problems, these tasks still have the sparse reward problem. The intuition behind these tasks is not clear.\n\n\n---\n\n[1] Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O. and Zaremba, W., 2017. Hindsight experience replay. Advances in neural information processing systems, 30.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OWgQdrOfOJn", "reviewer": "Reviewer_Si75", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to generate auxiliary predictions through the generate and test mechanism. The predictions are randomly generated and are then evaluated for their utility by a proxy measure: by looking at the magnitude of the weights between the features and the the main task learner. The authors claim that this approach is somewhere in-between end-to-end learning, where sample efficiency is a concern and hand-specified tasks. The authors report improved sample efficiency on three environments that are smaller in scale. Ablation studies highlight important properties of the algorithm.", "summary_of_the_review": "The paper proposes an interesting way to generate and test random predictions and evaluate the sample efficiency. The experiments are informative, but still leave a bit of doubt in terms of scalability.", "strength_and_weaknesses": "# Strength\n- Clear presentation and motivation\n- Solid and trust-worthy empirical investigation\n- A simple method that seems to do well on the environments tested.\n\n# Weaknesses \n- Concerns in terms of scalability of the method beyond one-hot representations\n- Additional baselines would be a very good addition to the paper\n- A lot of additional hyperparameters", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1bHjF_x0JT", "reviewer": "Reviewer_kAkb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a generate-and-test approach to discovering auxiliary tasks in RL. They form a multi-headed network where each head outputs a (general) value function for each task. At the last layer, the gradient update is only applied to the weights between each head and each dedicated set of features, and the rest of all the weights are used only in forward passes. The authors compute the magnitude of the value that each feature contributes to the value function for the main task and use it as a measure of how helpful that feature is to the main task. They evaluate their approach in the Four rooms, Maze, and Pinball environments.", "summary_of_the_review": "While I appreciate their concise description of the method and the meaningful empirical results, I have multiple concerns. My major concern is about the soundness of the method: each utility is measured by involving fixed random weights and their contributions to the main value function could be undone by some of the main features. I have some suggestions regarding the writing and presentation (especially of the experiments) as well.", "strength_and_weaknesses": "Strengths\n\n- In the used environments and tasks, the proposed method seems to select helpful subgoals in the state space for subgoal-reaching auxiliary tasks.\n- The description of the method is concise and easy to understand. Especially, Alg.1 and Fig.1 are quite helpful for an overview of the proposed approach.\n\nWeaknesses\n\n- I'm mainly concerned about the soundness of the proposed method. If I'm not mistaken, except for the features dedicated to the main task, all the features are connected with the value function for the main task via random weights that are not updated with backpropagation during the training. In my opinion, it means that some auxiliary features might contribute large-scale values to the main value function but with small relevance to the main task, so that some of the main features are trained to \"undo\" (or compensate for) the irrelevant contribution.\n- I think the writing could be improved in some ways. For instance, the measurement of the usefulness of features based on the scales of the contributed values could be summarized and introduced as an overview in the abstract or at least Sec.1 (Introduction). There is a typo regarding the gradient notation in Sec.2. A typo \"auxtasks\" is in Fig.2.\n- I see some issues in the presentation. In Fig.2, the Aux utility plots for \"hallway aux tasks 3\" and \"corner aux tasks 3-4\" are missing, which makes the figure less convincing. Also, presenting and comparing the heat maps of the performance improvements and the measured auxiliary utilities in the state space by testing many more subgoals could be quite informative.\n- Empirically, I don't fully agree that the proposed algorithm \"significantly outperforms\" hand-designed tasks. One issue I see in this regard is that since subgoal-reaching auxiliary tasks are simple and easy to create unless the state space is huge, in the given environments and tasks, it may often be enough to form auxiliary tasks by hand.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject