Know Your Action Set: Learning Action Relations for Reinforcement Learning
Authors: Ayush Jain
Norio Kosaka
Kyung-Min Kim
Joseph J Lim
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
varying action space
relational reasoning

Abstract: Intelligent agents can solve tasks in various ways depending on their available set of actions. However, conventional reinforcement learning (RL) assumes a fixed action set. This work asserts that tasks with varying action sets require reasoning of the relations between the available actions. For instance, taking a nail-action in a repair task is meaningful only if a hammer-action is also available. To learn and utilize such action relations, we propose a novel policy architecture consisting of a graph attention network over the available actions. We show that our model makes informed action decisions by correctly attending to other related actions in both value-based and policy-based RL. Consequently, it outperforms non-relational architectures on applications where the action space often varies, such as recommender systems and physical reasoning with tools and skills. Results and code at https://sites.google.com/view/varyingaction .

Published as a conference paper at ICLR 2022
KNOW YOUR ACTION SET: LEARNING ACTION RELATIONS FOR REINFORCEMENT LEARNING
Ayush Jain"! Norio 2osaka*2 Kyung-Min Kim?
Joseph J. Lim
i University of Southern California (USC), "NAVER CLOVA,
3 Korea Advanced Institute of Science and Technology (KAIST), "NAVER Al Lab
ABSTRACT
Intelligent agents can solve tasks in various ways depending on their available set of actions. However, conventional reinforcement learning (RL) assumes fixed action set. This work asserts that tasks with varying action sets require reasoning of the relations between the available actions. For instance, taking a nail-action in repair task is meaningful only if hammer-action is also available. To learn and utilize such action relations, we propose a novel policy architecture consist- ing of a graph attention network over the available actions. We show that model makes informed action decisions by correctly attending to other related actions in both value-based and policy-based RL. Consequently, it outperforms non-relational architectures on applications where the action space often varies, such as recommender systems and physical reasoning with tools and skills.
INTRODUCTION
Imagine you want to hang picture on the wall. You may start by placing nail the wall and then use a hammer to nail it in. However, if you do not have access to a hammer, you would not use the nail. Instead, you would try alternative approaches such using hook and adhesive-strips or a screw and a drill (Figure[]. In general, we solve tasks by choosing actions that interact with each other to achieve desired outcome in the environment. Therefore. the best action decision depends not only on the environment but also on what other actions are available to use.
This work addresses the setting of varying action space in sequential decision makiin.... Typically reinforcement learning (RL) assumes fixed action space, but recent work has explored variations in action space when adding or removing actions (Boutilier al. 2018 Chandak et al.) [2020a] b) or for generalization to unseen actions (Jain et al.) (0200). These assume that the given actions can be decision-making. But this assumption often does not hold. As the picture hanging example illustrates, the optimality of choosing the nail is dependent on the availability of hammer. Therefore, our goal is to address this problem learning the interdependence of actions.
treated independently
Addressing this problem is vital for many varying action space applications. These include recom- mender systems where articles to recommend vary everyday and physical reasoning where decision- making must be robust to any given set of tools, objects, skills. In this paper, benchmark three such scenarios where learning action interdependence is crucial for solving the task optimally. These (i) shortcut-actions in grid navigation, which can shorten the path to the goal when available, (ii) co-dependent actions in tool reasoning, where tools need other tools to be useful (like nail-hammer), and (iii) list-actions or slate-actions (Sunehag et al. 2015) in simulated and real-data recommender systems, where user response is a collective effect of the recommended list.
There are three key challenges in learning action interdependence for RL with varying action space. First, since all the actions are not known in advance, the policy framework must be flexible to work with action representations. Second, the given action space is an additional variable component, just like state observations in RL. Therefore, the policy framework must incorporate variably sized set of action representations as part of the input. Finally, an agent's decision for each action's 's utility (Q-value or probability) should explicitly model its relationship with other available actions.
"Equal contribution. Corresponding author: ayushj @ usc.edu I Work done while at USC f AT Advisor at NAVER Al Lab
Results::
Code: https:/githua..o////vaaaggggl

---

Published as a conference paper at ICLR 2022
Varying Action Sets
Hammer > Nail Relation
Hook + Tape Relation
Place Nail
Step-1
Apply Tape
H
Hit Hammer
Step-2
Stick Hook
Hang Picture
Step-3
Hang Picture
Task: Picture on Wall
Figure I: Picture hanging task with varying sets of tool-actions. The strategy with each action set depends on all the pairwise action relations. (Left) The agent infers that nail and a hammer are strongly related (bold line). Thus, it takes nail-action in Step-1 because hammer-action is available for Step-2. (Right) With different action set, the nail-action is no longer useful due to the absence of hammer. So, the agent must use an adhesive-tapy tape in Step-1 since its related action of the hook is available for later use. We show that policy with GAT over actions can learn such action relations.
We propose novel policy architecture to address these challenges: AGILE, Action Graph for In- terdependence Learning. AGILE builds on the utility network proposed in Jain et al. (2020) to in- corporate action representations. Its key component is graph attention network (GAT) (Veličkovid et al. 2017) over fully connected graph of actions. This serves two objectives: summarizing the action set input and computing the action utility with relational information of other actions.
Our primary contribution is introducing the problem of learning action interdependence for RL with varying action space. We demonstrate our proposed policy architecture, AGILE, learns meaning- ful action relations. This enables optimal decision-making in varying action space tasks such as simulated and world recommender systems and reasoning with skills and tools.
2 RELATED WORK
2.1 STOCHASTIC ACTION SETS
In recent work, Boutilier al. (2018) provide the theoretical foundations of MDPs with Stochastic Action Sets (DAS-MDPs), where actions are sampled from known base set of actions. They propose solution with Q-learning which Chandak et al. (2020a) extend to policy gradients.
instance of SAS-MDPs is when certain actions become invalid, like in games, they are masked out from the output action probability distribution (Huang & Ontañón 2020 Ye et al. 2020 Kanervisto et al., 2020). However, the assumption of knowing the finite base action set limits the practical applicability of SAS-MDPs. E.g., recommender agents often receive unseen items to recommend and robotic agent does not know beforehand what tools it might encounter in the future. We work with action representations to alleviate this limitation. Furthermore, in SAS-MDPs the action set can vary at any timestep of an episode. Thus, the learned policy will only be optimal on average over all the possible action sets (Qin et al. 2020). Whereas in our setting, the action set only changes at the beginning of task instance and stays constant over the episode. This a more practical setup and raises the challenging problem of solving a task optimally with any given action space.
2.2 ACTION REPRESENTATIONS
In discrete action RL, action representations have enabled learning large action spaces (Dulac- Arnold et al. 2015 Chandak et al. 2019). transfer learning to different action space (Chen et al. 2019b), and efficient exploration by exploiting the shared structure among actions (He et al. 2015 Tennenholtz & Mannor 2019: Kim et al.) 2019). Recently, CChandak et al.) (220bb) use them to accelerate adaptation when new actions are added to an existing action set. In contrast, our setting requires learning in constantly varying action space where actions can be added, removed, or com- pletely replaced in an episode. Closely related to our work, work,Dain et al. (020)) assume a similar setting unseen actions. Following their motivation,
of varying action space while training to generalize we use action representations to avoid assuming knowledge of the base action set. However, their policy treats each action independently, which we demonstrate leads suboptimal performance.
2

---

Published as a conference paper at ICLR 2022
2.3 LIST-WISE ACTION SPACE
The action space is combinatorial in list size in listwise RL (or slate RL). Commonly it has appli- cations in recommendation systems (Sunehag al. 2015 Zhao et al.) 2017 2018 Ie et al. 19b Gong et al. 2019 Liu et al. 2021 Jiang et al. 2018 Song et al. 2020 In recent work, Chen et al. (2019a) proposed Cascaded DQNs (CDQN) framework, which learns Q-network for every index in the list and trains them all with a shared reward. For our experiments on listwise RL, we utilize CDQN as the algorithm and show its application with AGILE as the policy architecture.
2.4 RELATIONAL. REINFORCEMENT LEARNING
Graph neural networks (Battaglia et al. 2018) have been explored in RL tasks with a rich relational structure, such morphological control (Wang et al. 2018 Sanchez-Gonzalez al. 2018 Pathak et al. 2019). multi-agent RL (Tacchetti et al. 2019), physical construction (Hamrick et al. 2018),
and structured perception in games like StarCraft (Zambaldi et al. 2018). In this paper. we propose that the set of actions possess a relational structure that enables the actions to interact and solve tasks in the environment. Therefore, we leverage graph attention network (Veličković et al. 2017) to learn these action relations and show that it can model meaningful action interactions.
PROBLEM FORMULATION
hallmark of intelligence the ability to be robust in an ever-changing environment. To this end, we consider the setting of RL with varying action space, where agent receives different action set in every task instance. Our key problem is to learn the interdependence actions so the agent can act optimally with any given action set. Figure Tillustrates that for a robot with the task of hanging a picture on the wall, starting with a nail is optimal only if it can access a hammer subsequently.
3.1 REINFOR LEARNING WITH VARYING ACTION SPACE
We consider episodic Markov Decision Processes (MDPs) with discrete action spaces, supplemented with action representations. The MDP is defined by a tuple {8, A, T, R, 7 of states, actions, tran- sition probability, reward function, and a discount factor, respectively. The base set of actions A can be countably infinite. To support infinite base actions, we use D-dimensional action representations ca E R to denote an action a E A. These can be image or text features of a recommendable item, behavior characteristics of a tool, or simply one-hot vectors for a known and finite action set.
In each instance of the MDP, subset of actions A C A. is given to the agent, with associated representations C. Thereafter, at each time step / the episode, the agent receives state observation S from the environment and acts with a₂ E A. This results in state transition to 8t+1 and a reward Tt. The objective the agent to learn a policy (a)ssA A) that maximizes the expected discounted reward over evaluation episodes with potentially unseen actions, IEAC Σ 7 mr]]
3.2 CHALLENGES OF VARYING ACTION SPACE
1. Using Action Representations:
The policy framework should be flexible to take a set of action representations C as input and output corresponding Q-values probability distribution for RL.
2. Action Set as part of State: When the action varies, the original state space S is anymore Markovian. For example, the state of a robot hanging the picture is under-specified without knowing if its toolbox contains a hammer or not. The MDP can be preserved by reformulating the state space as S {s C S,A A C A} to include the representations CA associated with the available actions A (Boutilier et al. 2018). Thus, the policy framework must support the input of s o CA, where AA is a variably sized action set.
3. Interdependence of Actions:
The optimal choice of action o, E AA is dependent the action choices that would be available in the future steps of the episode, ay E A. Recalling Figure I nail should only be picked initially from the toolbox if hammer is accessible later. Thus, an optimal agent must explicitly model the relationships between the characteristics of the currett action Ca, and the possible future actions Ca2 Va, A.
3

---

Published as a conference paper at ICLR 2022
Inputs
Q-Learning
State
AGILE
GAT
π
u
Pool
Action Representations
Action Relations
Action Summary
Action Set
Policy Gradient
Figure 2: Given action set, AGILE builds complete graph where each node is composed of an action representation and the state encoding. A graph attention network (GAT) learns action relations by attending to other relevant actions for the current state. For example, the attention weight between the cannon and the fire is high because fire can activate the cannon. The GAT outputs more informed relational action representations than the original inputs. Finally, a utility network computes each action's 's value selection probability in parallel using its relational action representation, the state, and mean-pooled summary vector of all available actions' relational features.
APPROACH
Our goal is to design a policy framework that is optimal for any given action set by addressing the challenges in Sec. 3.2 We build on the utility network proposed by Jain et al. (2020) that acts in parallel on each action's s representation. Our central insight is to use graph neural networks for summarizing the set of action representations state component and learning action relations.
4.1 AGILE: ACTION GRAPH FOR INTERDEPENDENCE LEARNING
We propose AGILE, a relational framework for learning action interdependence RL with varying action space. Given list of action representations C. AGILE builds a fully-connected action graph. graph attention network (GAT) (Veličković et al., 2017) processes the action graph and learns action relations. The attention weights in the GAT would be high for closely related actions such as a nail and a hammer in Figure [ A utility network uses the GAT's resulting relational action representations, the state, and pooled together action set summary to compute each action's 's Q- value or probability logit for policy gradient methods(Figure (
The input to our policy framework consists of the state and a list C = [Cap .... Ca., of action representations for each action a, E A. We build fully connected action graph G with vertices corresponding each available action. If certain action relations are predefined via domain knowledge, we can reduce some edges to ease training (Appendix B.I.. We note that the action the environment state. For instance, screwdriver is related to screw for furniture repair, but a drill machine is more related when the screw is for a wall. Therefore, join the state and action representations, </ (s, obtain the nodes of the graph. Sec. 6.3.2 validates that learning state-dependent action relations leads to more optimal solutions.
Action Graph:
relations can vary depending
The action graph g is input connected, we choose an attention-based graph network that can learn to focus on the most relevant actions the available action A similar insight was employed by Zambaldi al. (2018) where the entities inferred from the visual observation are assumed to form fully connected graph. To enable propagation of sufficient relational information between actions, we use two graph attention layers with an ELU activation in between (Clevert et al.) 2015). We found residual connection after the second GAT layer was crucial experiments, while multi-headed attention did not help.
Graph Attention Network:
GAT. Since the graph is fully-
Action Set Summary: The output of the GAT is list of relational action representations CR {cff, ...c.t which contain the information about the presence other available actions and their relations. To represent the input action set as part of the state, we compute a compact action set summary by mean-pooling the relational action features, RR R Ek<
4

---

Published as a conference paper at ICLR 2022
Action Utility: To use the relational action representations with RL, follow the utility network architecture Tu from Jain et al. (2020). It takes the relational action representation, the state, and input for each available action in parallel. It outputs utility score T 8. ) for how useful an action a is for the current state and in relation to the other available actions. The utility scores can be used Q-value directly for galue-based RL a softmax function to form probability distribution over the available actions for policy-based RL.
the action set summary
logit fed into
4.2 TRAINING AGILE FRAMEWORK WITH REINFORCEMENT LEARNING
The AGILE architecture can be trained with both policy gradient and value-based RL methods.
Policy Gradient (PPO): For every action decision step, we take the output action utility and use a softmax over all available actions to get a probability distribution. We use PPO (Schulman et al.) 2017) train the policy. PPO requires value function Vss') as baseline. We represent the effective state as the concatenation of the state encoding and the action set summary El inferred from the GAT, s' (s, e²). An important implementation detail to make the training faster and stable was to not share the GAT weights used for the actor i (als', <a)) and the critic V(s') networks.
Value-based RL (DQN) The output action utility can be directly treated as the Q('', a) value of the action a at the current effective state s' with the available action set A. This can be trained using standard Deep Q-learning Bellman backup (Mnih et al. 2015).
Listwise RL (CDQN): For tasks with listwise actions, we follow the Cascaded DQN (CDQN) framework of Chen et al. (2019a). The main challenge is that building the action list all at once is not feasible due to combinatorial number of possible list-actions. Therefore, the key is to build the list incrementally, one action at a time. Thus, each list index can be treated as an individual action decision trained with independent Q-networks. We replace the Q-network of CDQN with AGILE to support a varying action space. Sharing the weights of the cascaded Q-networks led to better performance. Algorithm [ provides complete details CDQN for listwise AGILE.
ENVIRONMENTS
We evaluate AGILE on three varying action set scenarios requiring learning action interdependence: (i) shorteut actions in goal-reaching, which can shorten the optimal path to the goal in 2D Grid World when available, (ii) co-dependent actions in tool reasoning, which require other tools to acti- vate their functionality, and (iii) list-actions in simulated and real-data recommender systems where the cumulative list affects the user response. Figure 3 provides an overview of the tasks, the base and varying action space, and an illustration of the action interdependence. More environment details such tasks, action representations, and data collection are present in Appendix A
5.1 DIG LAVA GRID NAVIGATION
We modify the grid world environment Sntevalier-Boisver et al. 2018 where an agent navigates a 2D maze with two lava rivers to reach a goal. The agent always has access to four direction movements and turn-left skill (Figure There are two additional actions randomly sampled out of four special skills: turn-right, step-forward, dig-orange-lan and dig-pink-lava. If the agent enters the lava, it will die unless it uses the matching dig-lava skill to remove the lava in the immediately next timestep. Thus, when available. dig-lava skills can be used to create shortcut paths to the goal and receive higher reward. We use PPO for all experiments in this environment.
5.2 CHAIN REACTION TOOL ENVIRONMENT: CREATE
The CREATE environment (Jain al. 2020 a challenging physical reasoning benchmark with large variety of tools as actions and also supports evaluation with unseen actions. The objective is to sequentially place tools to help push the red ball towards the green goal. An action is hybrid of discrete tool-selection from varying toolsets and a continuous (z, y) coordinate of tool-placemen on the screen. An auxiliary policy network decides the tool-placement based on the effective state s' as input, following Jain et al. (2020). To emphasize action relations, we augment the environment with special activator tools (e.g., fire) that general tools (e.g., cannon) need in contact for being functional. Thus, general tool can be useful only if its activator is also available. Action representations of general tools encode their physical behavior (Jain et al., 2020), while those of activator tools are one-hot vectors. We train AGILE and the auxiliary policy jointly with PPO.
5

---

Published as a conference paper at ICLR 2022
Dig Grid World (Shortout Actions)
CREATE [Co-dependent Actions)
Recommender Systems (List-Actions)
Recommand
Hybrid Action (x.y) position
8-item List
User
Selection
Click or Not
Actiated on Contact
Usar Preference ? Coherent catogories (shirts] Diverse sub callgories (colons)
Semple 23
Sample 2
Always Given
Semple 2
Sample 20
Skils
Gemeral Tools
Activator
Items
Figure 3: Environment Setup. (Leftt) In Grid World, the red agent must avoid orange and pink lava to reach the green goal. The dig-lava skills enable shortcut paths the goal when available. (Middle) In CREATE, tools are selected and placed sequentially to push the red ball to the green goal. General tools (cannon, fan) require activator tools (fire, electric) start functioning. The choice of general tools depends on what activators are available and vice-versa. (Right) In simulated and real-data recommender systems, the agent selects a list of items. Lists with coherent categories and complementary sub-categories improve user satisfaction (e.g., click likelihood).
5.3 RECOMMENDER SYSTEMS
Recommender system (RecSys) natural application of varying action space RI for instance, news articles or videos to recommend are updated daily. Action interdependence is distinctly ap- parent in the case listwise actions. A recommended list of diverse videos is more likely to get user click than videos about the same thing (Zhou et al. 2010). In this work, we experiment with the listwise metric of Complementary Product Recommendation (CPR) (Hao al. 2020).
Complementary Product Recommendation (CPR) is a scenario where user response for rec- ommended list more favorable if the list is diverse at a low level but coherent a high level. In our experiments, each item has primary category (such as shirt or pant) and subcategory (such Entropy of subcalegury This encourages diversity in as its color). We define the CPR of an item-list as, Entropy category subcategories (colors) and similarity in the main category (all shirts). We maximize CPR (i) implicit in the flick-behavion of simulated users and (ii) explicit the reward computed on real-world data.
5.3. SIMULATED RECOMMENDER SYSTEM: RECSIM
We use RecSim (Ie al./ 2019a) to simulate user interactions and extend it the listwise recom- mendation task. We have base action set of 250 train and 250 test items, and 20 items are sampled as actions for the agent in each episode. The agent recommends list-action of size six at each step. We assume a fully observable environment with the state as the user preference vector and the ac- tion representations as item characteristics. The objective implicitly incorporates CPR by boosting the probability of user clicking any item proportional to the list CPR. The implicit CPR objective exemplifies realistic scenarios where the entire list influences user response. One way to optimize CPR is to identify the most common category in the available action set and recommend most items from that category. Such counting of categories requires relational reasoning over all items available in the action set. We train EDQN-based models to maximize the number of clicks in user session.
5.3.2 REAL-DATI RECOMMENDER SYSTEM
We collect four-week interaction data in a listwise online campaign recommender system. Users are represented by attributes such as age, occupation, and localities. Item attributes include text features, image features, and reward points of campaigns. We train VAE (Kingma & Welling (0133) to learn item representations. We create a representative RL environment by training two click-estimation models using data from the first two weeks for training and the last two weeks for evaluation. The training environment consists of 68,775 users and 57 items, while testing has 82,445 users and 58 items, with overlap of 30 items. The reward function combines the user-click and CPR value of the list. The explicit CPR reward is a representative scenario for when the designer has listwise objectives in addition to user satisfaction. We train with CDQN and report the test reward.

---

Published as a conference paper at ICLR 2022
0g Lava or World
CREATE
RECSIN
PookEee Retrmarde Sprkem
Trann Eslooo clus
16
Traas
20
139
2005
7:83
19.03
1path
120
(Ours)
Utiliy Policy
Mask-input Output
Mask-Output
Simple DQN
CREATE
RECSM
Reel Cataa Rocuenendeer System
Test Success fore 163
Eaload
Enatcoment steps (1M)
2.03 Epoch
75.
1000
2200
Figure 4: We evaluate AGILE against baselines training actions (top) and unseen testing actions (bottom). Generalization is enabled by continuous action representations, except Grid World (Ap- pendix Aj. All architectures share the same RL algorithm (PPO CDQN). The results are averaged over random seeds, and the seed variance is shown with shading. AGILE outperforms all baselines that assume a fixed action set (cannot generalize) or treat actions independently (suboptimal).
EXPERIMENTS
We design experiments to answer the following questions in the context of RL with varying ac- tion space: (1) How effective is AGILE compared to prior works that treat actions independently or assume fixed action set? (2) How effective are AGILE's relational action representations for computing action set summary and action utility score? (3) Does the attention AGILE repre- sent meaningful action relations? (4) Is attention necessary in the graph network of AGILE? (5) Is learning state-dependent action relations important for solving general varying action space tasks?
6.1 EFFECTIVENESS OF AGILE IN VARYING ACTION SPACES
We evaluate AGILE against baselines from prior work in varying action space, which either assume a fixed action set or act independently the other actions. We ablate the importance of relational action features by replacing them with the original action representations and compute the action set summary in different ways. The Appendix details the comparison
(C.T) and visualization (Figure) [7)
of all baselines and ablations, hyperparameter-tume tuning (D.2 D.3) and network designs(
6.1.1 BASELINES
Mask-Output (No action representations, No input action set): Assumes fixed action space output. Q-values or policy probabilities are masked out for unavailable actions. It represents prior work on SAS-MDP: Boutilier al. (0018); Chandak
(2020a): Huang Ontañón (2020).
Must-unput-Oupput (No action representations): Augments Mask- Kusk-Output with an extra input of action set via binary availability vector: having 1s at available action indices and Os otherwise.
Utility-Policy (No input action set): Jain et al. (2020) propose parallel architecture to compute each action's 's utility using action representations. But, it ignores any action interdependence.
Simple DQN (No cascade, No input action set): A non-cascaded DQN baseline for listwise RL that selects top-K items instead of reasoning about the entire list. Thus, it ignores two action interdependences: (i) on other items in the list and (ii) on other available actions.
6.2.2 ABLATIONS
Summary-LSTM: A Bi-LSTM (Huang et al. 2015) encodes the list of action representations.
Summary-Deep Set: deep set architecture (Zaheer et al. 2017) with mean pooling is used to aggregate the available action representation list.
Summary-GAT: The relational action representations output from the GAT in AGILE is used only for the action set summary but not for the utility network. This does not scale to tasks with many diverse inter-action relations because the summary vector has limited modeling capacity.

---

Published as a conference paper at ICLR 2022
0g Lava or World
CREATE
RECSIM
PookEee Retrrarde Sprken
TGA EOLEOOO Crols
Yott Succusa
14
Erstromment
30
staps
10.0
5009
1000
10:03
10/0
-200
170
1.20
300
AGILE (Ours)
Summary
Summany-
Summary Deep Set
Figure 5: Test performance of AGILE against ablations with utility network using only action set summary, but not the relational action representations. The difference is most pronounced in CRE- ATE, where there are several diverse action (tool) relations for each action decision.
6.1.3 RESULTS
Baselines: Figure shows baseline results train and test actions. Grid world has no unseen ac- tions, so we report only train reward. Grid World: all methods learn to reach the goal with perfect success but achieve different rewards due to the length of the path. The methods which do not take action set as input, Mask- Output and Utility Policy, resort to the safe strategy of going around the lava rivers 6(b)). This is because the agent must enter the lava and then dig it to take short- cut. So before taking the move-right action into lava, the agent must know whether that dig-lava skill is available. CREATE: Mask baselines do not support generalization due to fixed action set assumption and train poorly as they do not exploit the structure in action representations. Utility pol- icy outperforms the other baselines since can exploit learning with action representations. Finally, AGILE outperforms all the baselines, demonstrating that relational knowledge of other available actions crucial for an optimal policy. RecSim and Real RecSys: result trends are consistent with CREATE, but less pronounced for Real RecSys. Additionally, DQN is worse than CDQN-based architectures because the top-K greedy list-action ignores intra-list dependence.
Ablations: Figure 5 shows ablation results on test actions. Grid World: all ablations utilize the action set summary as input, aggregated via different mechanisms. Thus, they can identify which dig-lava skills are available and enter lava accordingly to create shortcuts. In such small action spaces with simple action relations, summary-
are on par with AGILE. This trend also holds for RecSim and Real RecSys, where the summary can find the most common category and its items are then selected to maximize CPR (e.g., o() Therefore, we observe only 5 20% gains of further evaluate more RecSim tasks. (i) Direct CPR: the agent receives additional explicit CPR metric reward on top of click/no- click reward (Sec. B.3), and (ii) pairing environment: the task is to recommend pairs of associated items based on predefined pairings (Sec. B.4). We reproduce the trend that AGILE >= ablations. However, the difference most pronounced (30- 50%) in CREATE, where each action decision relies on relations between various tools and activators. In Figure the decision of Spring relates with all other tools that spring can activate, and the decision of trampoline relates with its activator, Spring. In ablations, the action set summary must model all the complex and diverse action relations with a limited representation capacity. In contrast, the relational action representations in AGILE model each action's 's relevant relations, and the summary models the global relations.
AGILE over the ablations. To test the consistency of results,
6.2 DOES THE ATTENTION IN AGILE LEARN MEANINGFUL ACTION RELATIONS?
In Figure we analyze the agent performance qualitatively. (a) In CREATE, at / = O, the selected action spring in AGILE's GAT attends to various other tools, especially the tools that get activated with spring, such as Irampoline. At L = 1, the trampoline tool is selected with strong attention on spring. This shows that for selecting the trampoline, the agent checks for its activator, spring, to ensure that it is possible to place spring before or after the trampoline. (b) In Grid World, we visualize the inter-action attention in SSmmary-GA'V GA7's summarizer. We consider the case where both dig- lava skills are available. The agent goes right, digs the orange lava, and is about to enter the pink lava. At this point, the Right action attends with a large weight to the Dig - Pink skill, checking for its presence before making an irreversible decision of entering the lava. In contrast, the Utility Policy always follows the safe suboptimal path as it is blind to the knowledge of dig- skills before entering lava. (c) In RecSim,
observe that the agent can maximize the CPR score by selecting 5 out of 6 items in the list from the same primary category. In contrast, Utility Policy cannot determine the most common available category and is unable maximize CPR.
8

---

Published as a conference paper at ICLR 2022
1 Place Spring
1+1 Place Trampoline
Candidana Selt Categoer Duaribution
Action:
User CHA
Racemmended
Summar [Dun]
Utittt Pollay
tx2
ADILL [Durs]
Attention Map Spring and Trampotine stlend each other
Map stp-right strence dippttiiine
(a) CREATE
(b) Grid World
[c) RecSim
Figure 6: Qualitative Analysis. (a.b) The attention maps from GAT show the reasoning behind an action decision. The nodes show available actions, and edge widths are proportional to attention weight (thresholded for clarity). (b) Utility Policy learns the same suboptimal solution for any given action set, while Summary-GAT (like AGILE) adapts to the best strategy by exploiting dig- skills. (c) AGILE optimize CPR by identifying the most common item category available, unlike Utility Policy. We provide qualitative video results for Grid World and CREATE on the
project page https:://// /tt::///sitee google com/ view htt//aaaryiingaaooe
De Loan Gold word
ORCATE
TesiCera Technnence Syrter
Tas: Soccee Bor (5)
Ondirnmant sapp [3P)
Enveanment (3M
Spach
AGILE (Ours)
AGILE-GCN
AGILE-Only Action
Figure 7: Additional Analyses. (i) GAT v/s GCN (ii) state-action relations v/s action-only relations.
6.3 ADDITIONAL ANALYSES
6.3.1 IMPORTANCE OF ATTENTION IN THE GRAPHI NETWORK
We validate the choice of using graph attention network the relational architecture. In Figure 7 we compare GAT against graph convolutional network (GCN) (Kipi & Welling 2016) to act over AGILE's action graph. We observe that GCN achieves optimal performance for the grid world and RecSys tasks. GCN can learn simple action relations even though the edge weights are not learned. However, it suffers in CREATE and RecSim-paiin paiiiii-aaiiggg (Figure 16), where the action relations diverse and plenty. Moreover, we believe that the attention in GAT makes the graph sparse to ease RL training, which in contrast, is difficult in fully-connected GCN.
6.3.2 IMPORTANCE OF STATE-DEPENDENT LEARNING OF ACTION RELATIONS
We evaluate a version of AGILE where the GAT only receives action representations as input and no state. Thus, the action relations are inferred independently of the state. Figure shows drop in performance for Grid World and CREATE, where the relevant action relations change based on the state. However, this effect is less apparent on RecSim because CPR requires only knowing the most common category, independent of user state.
7 CONCLUSION
We present AGILE, policy architecture for leveraging action relations for reinforcement learning with varying action spaces. AGILE builds a complete graph of available actions' representations and utilizes a graph attention network to learn the interdependence between actions. We demonstrate that using the knowledge of available actions is crucial for optimal decision decision-making and relational action features ease learning in four environments, including real-data recommendation task.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This project was supported by NAVER AI Lab and USC. The authors are grateful to Youngwoon Lee for help with ideation and guidance. We thank Xingdong Zuo for help with data extraction in real-data RecSys. We appreciate the fruitful discussions with Shao-Hua Sun, Karl Pertsch, Jesse Zhang, Grace Zhang, and Jisoo Lee.
REFERENCES
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint aoXiv::806.000 1806.0/261,, 2018.
Lukas Biewald. Experiment tracking with weights and biases. Software available from wandb. com, pp. 33, 2020.
Craig Boutilier, Alon Cohen, Amit Daniely, Avinatan Hassidim, Yishay Mansour, Ofer Meshi, Mar- tin Mladenov, and Dale Schuurmans. Planning and learning with stochastic action sets. arXiv preprint arXiv:1805.025:: 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learn- ing action representations for reinforcement learning. In International Conference Machine Learning, pp. 941-950. PMLR, 2019.
Yash Chandak, Georgios Theocharous, Blossom Metevier, and Philip Thomas. Reinforcement learn- ing when all actions are not always available. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3381-338333 2020a.
Yash Chandak, Georgios Theocharous, Chris Nota, and Philip Thomas. Lifelong learning with a changing action set. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3373-3380, 2020b.
Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In International Conference on Machine Learning, pp. 1052-1061. PMLR, 2019a.
Yu Chen, Yingfeng Chen, Yu Yang, Ying Li, Jianwei Yin, and Changjie Fan. Learning action- transferable policy with action embedding. arXiv preprint NXXX::1909.02200 2019b.
Heng-Tze Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Maxime Chevalier- Thevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. lattps:/// httt::://githbb. com/ cwt.m//maimmebb. cb/ hte//yymmnan minigridi 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint arXiv:1511.07289 2015.
Gabriel Amold, Richard Evans, Hado Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint arXiv: 15/.00767, 2015.
10

---

Published as a conference paper at ICLR 2022
Yu Gong, Yu Zhu, Lu Duan, Qingwen Liu, Ziyu Guan, Fei Sun, Wenwu Ou, and Kenny Q Zhu. Exact-k recommendation via maximal clique optimization. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 617-626, 2019.
Jessica Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin McKee, Joshua B Tenen- baum, and Peter W Battaglia. Relational inductive bias for physical construction in humans and machines. arXiv preprint arXiv: 0e0..0000....3.. 2018.
Junheng Hao, Tong Zhao, Jin Li, Xin Luna Dong, Christos Faloutsos, Yizhou Sun, and Wei Wang. P-companion: A principled framework for diversified complementary product recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Manage- ment, pp. 2517-2524, 2020.
He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. arXiv preprint arXiv:1511004636, 2015.
Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms. arXiv preprint urXiv:2006. 14/71, 2020.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-erf models for sequence tagging, 2015.
Eugene le, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems. arXiv preprint urXiv:1909004840 2019a.
Eugene Ic, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. Slateq; A tractable decomposition for reinforcement learn- ing with recommendation sets. 2019b.
Ayush Jain, Andrew Szot, and Joseph Lim. Generalization to new actions in reinforcement learn- ing. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4661-4672. PMLR, 13-18 Jul 2020. URL http::/pppt http://prrocediigg...z. mlr wtt.m/l.rp.eeee1/////// Bain20b. html,
Ray Jiang, Sven Gowal, Timothy A Mann, and Danilo J Rezende. Beyond greedy ranking: Slate optimization via list-cvae. arXiv preprint arXiv: 018.0,66222222 2018.
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. Action space shaping in deep reinforce- ment learning. In 2020 IEEE Conference on Games (CoG), pp. 479-486. IEEE, 2020.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo. KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the mlaas platform with a real-world case study. arXiv preprint arXiv: 180009957, 2018.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: Ex- ploration with mutual information. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 3360-3369, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL
m press /v2/s/////. w99/kimillmmm aa.taa.httttt httmmm1....mll
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6996 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 1312.61 14, 2013.
arXiv preprint
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02977 2016.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms.
https://
github. www.eommiieeee http:////..ooootoo/////oooaaaaea
2018.

---

Published as a conference paper at ICLR 2022
Shuchang Liu, Fei Sun, Yingqiang Ge, Changhua Pei, and Yongfeng Zhang. Variation control and evaluation for generative slate recommendations. Proceedings of the Web Conference 2021, Apr 2021. doi: 00.11445744238.34499666 3449864
/RLphtt::/// http:://x.cd.. wtw.bb...og//..111///3333331313.
Andrew L Maas. Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Citeseer, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 ((7554)):529-533 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Pytorch: An imperative style,
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros. Learning to control self-assembling morphologies: A study of generalization via modularity, 2019.
Zhiwei Qin, Xiaocheng Tang, Yan Jiao, Fan Zhang, Zhe Xu, Hongtu Zhu, and Jieping Ye. Ride- hailing order dispatching at didi via reinforcement learning. INFORMS Journal on Applied Ana- lytics, 50(5):272-286, 2020.
Alvaro Sanchez-Gonzales Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470-4479, PMLR, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Junshuai Song, Zhao Li, Chang Zhou, Jinze Bai, Zhenpeng Li, Jian Li, and Jun Gao. Co-displayed items aware list recommendation. IEEE Access, 8:64591-64602, 2020. doi: 10.1109/ACCESS. 2020.2984543.
Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Cop- pin. Deep reinforcement learning with attention for slate markov decision processes with high- dimensional states and actions. arXiv preprint arXiv:1512.01122: 2015.
Andrea Tacchetti, H. Francis Song. Pedro A. M. Mediano, Vinicius Zambaldi, János Kramár, Neil C. Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W. Battaglia. Relational forward mod- els for multi-agent learning. International Conference on Learning Representations, 2019.
URL http:::/ net wtt.:/wwwwweeeo.t..ddddee.... to
Guy Tennenholtz and Shie Mannor. 1rXiv:1902.01119. 2019.
The natural language of actions.
arXiv preprint
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 2017.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representutions, 2018.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang. Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-6677 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc 2017. URL hwwp::ttpp/occtcccctcctt /ttpaaattta//ppeeeeeaiiiio
neurips cc/ f
12

---

Published as a conference paper at ICLR 2022
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.
Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep rein- forcement learning for list-wise recommendations. arXiv preprint arXiv:/801. 00209, 2017.
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep re- inforcement learning for page-wise recommendations. Proceedings of the 12th ACM Confer- ence Recommender Systems, Sep 2018. doi: 10.1145/244033.2240374
URL http:
V/dx.doi. 1tt...ob...rgg111.114/322333333. 322333333333337777
3240374
Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. Solving the apparent diversity-accurace dilemma of recommender systems. Proceedings of the National Academy of Sciences, 107(10):451-44555 2010.
13

---

Published as a conference paper at ICLR 2022
A ENVIRONMENT DETAILS
A.1 DIG LAVA GRID NAVIGATION
The grid world environment, introduced in Sec. 5.1 requires an agent to reach a goal by navigating 2D maze with two lava rivers using variable set of skills.
State: The state space is concatenation of two flattened 9x9 grids:
1. Environment: each grid element is an integer ID representing whether the cell empty, orange-lava, pink-lava, goal, or sub-goal
2. Agent: an empty grid except for 1 at geottccoordinates
Termination: An episode is terminated in success when the agent reaches the goal and in failure when it stays in lava for two consecutive timesteps or after total of 50 timesteps. Note that the agent can leave a lava cell only using the dig lava skill of the corresponding color.
The base action is fixed set of 9 skills and in each episode, set of skills is given to the agent. 5 skills are always available: move-right, move-down, move-left, move-up, and turn-left. The other skills randomly sampled from set of skills: turn-right, step-forward, dig-orange- lava, and dig-pink-lan
Actions:
The agent receives a large goal reward on reaching the goal. There are two subgoal rewards for a successful crossing of each lava column (i.e. column 4 and column 8) for the first time. The goal and subgoal rewards are discounted based on the number of action steps taken to reach that location, thus rewarding shorter paths. To further encourage shorter paths, successful lava digging is rewarded. small exploration reward added whenever the agent visits a new cell in the episode. The exploration reward is accumulated and subtracted when the agent reaches subgoal or goal to ensure that the exploration reward does not hinder learning short paths. Thus,
Reward:
N Neurrent steps
R(s, a)
GGol
RGoal
1
Acical
Rexploration Nsueps from prev subgoal
+
Nmax steps
current steps
Subgoal
Rsubgoal
Subboal
Exploration
stoops from prev subgoal
N Nmax steps
Successful Dig RiDgg
+ New State
RExploration
(I)
where RGoal 100, RSabgoal
= 0.5, RExploration = 0.01, RDig = 0.01,
AGaal = 0.99, ÄSubgoal 0.9, Nmax steps 50
The action representations are 1 I-ddimensional vectors manually defined using mix of one-hot vectors, as shown in Table [ Dimensions 1-5 identify the category of skills (movement, elemental, dig-orange, dig-pink), 6-7 distinguish movement skills (right, down, left, up). 8-9 always 0 (originally meant for diagonal skills), 10-11 are used to distinguish elemental skills (turn-left, turn-right, move-forward).
Action Representations:
Category
Skill
Action Representation
move-right
-1
move-down
0
0
0
0
-1
1
0
Movement
move-left
0
0
0
-1
0
move-up
0
0
0
0
1
0
0
turn-left
Elemental
turn-right
0
0
1
0
0
0
0
0
0
-1
move-forward
0
1
dig-orange dig-pink
0
0
0
0
Digging
1
Table 1: Action representations for the skills used in Dig Lava Grid Navigation environment.
14

---

Published as a conference paper at ICLR 2022
A.2 CHAIN REACTION TOOL ENVIRONMENT (CREATE)
The CREATE environment (Sec. requires an agent to place tools in a physics-based puzzle to make the target ball reach the goal position. We follow all the base settings of the CREATE Push Environment from Jain et al. (2020) We add the functionality of new activator tools and reduce the number of available actions per episode from 50 25.
State: The agent receives the past three gray-scale frames, which are stacked channel-wise to make 84x84x3 input.
Termination: The episode ends success when the agent accomplishes the goal and failure when there no remaining movement in the game after 30 timesteps.
Actions: Each original tool from CREATE is now associated with newly added activator tools as shown in Figure [ Thus, we add 5 activator tools the 2110 general tools, which are variations in angle, size, friction, or elasticity of the tools shown in Figure g Unlike Jain et al. (2020), we remove the No- No-Operation action. Activator tools are pass pass-through tools and only serve the function of activating their corresponding general tools when placed in contact.
We split the general tool space into 1098 tools for training, 507 tools for validation, and 507 tools for testing. All 5 activator tools are available for sampling during training, validation, and testing. In each episode, 23 general tools and 2 activator tools are randomly sampled and made available to the agent. The agent outputs in hybrid action space consisting of (1) the discrete tool selection and (2) (x, 3) coordinates of tool position.
WWw
Spring
Trampoline (TRP)
Bouncy Ball (BBL)
Bouncy Box (BBX)
Bouncy Polygons
Magnet
Ramp (RMP)
Fixed Ball (FBL)
Fixed Box (FBX)
Fixed Polygons
Cannon (CNN)
Hinge Lever (HNG)
Hinge Constrained (HGC)
See-Saw (SS)
Water
Funnel (FNL)
Bucket (BKT)
Electric
Fan (FAN)
Belt (BLT)
Figure 8: CREATE Activator Mapping: Original CREATE tools (right) activated by the
spective newly introduced activator tools (left). E.g., Cannon tool (abbreviated as CNN in attention maps) placed on the environment will only be functional when Fire tool is placed in contact with it. Other objects are not affected by non-functional tools they simply pass through.
We adopt the reward structure from Jain et al. (022).. CREATE Push is a sparse reward environment with rewards for reaching the goal and making the target ball move. There is addi- tional alive reward to continue the episode. Placements outside the scene are penalized. The original CREATE environment penalizes overlapping tools. However, since overlapping activators and gen- eral tools is required for solving tasks, we modify the overlap penalty to apply only when either both general tools or both activator tools. Thus, the agent receives the following reward:
Reward:
15

---

Published as a conference paper at ICLR 2022
R(s,0 a) Ralive
1 Goal
RGaal
Iarget hit Rearget hit + 1 invalid Ranvalid
overlap
Rnverlpp
(2)
where Ralive 0.01, RGoa 10.0, Prarget
= 1, and Rinvald
Rovertap - 0.01.
Action Representations: Each action representation is 134-dimensional vector, concatenation
of two 128-D and 6-D vectors, as shown in Table 2 The 128-D vector corresponds to the learned
characteristics of the tool, and the 6-D vector is binary vector denoting whether the tool is general tool, an activator tool, or no-op tool (no-op is disabled for experiments).
For general tools, Jain encode a set of tool observations into a latent representation. Each tool is made to interact with probing ball launched from various angles, positions, and speeds. The tool characteristics can be inferred from this collection of tool interactions. We utilize these 128-D learned tool representations from Jain et al. 2020)) for the general tools and pad them with a 6-D zero-vector. Generalization to unseen tools is possible because trained agent can utilize the 128-D tool embedding to extract the relevant characteristics of any given tool. The agent must infer which activator general tool is associated with using its tool embedding.
(2020) obtain action representations using a Hierarchical VAE. They
For activator tools, the first 128 dimensions are always zero, and the final 6 dimensions correspond to one-hot vector, each for (no-op, Fire, Water, Electric, Magnet, Spring).
Tool
Action Representation
General Tools
Learned 128-D tool characteristics
0
0
0
0
0
0
No-Op
128-D Zero Vector
1
Fire
128-D Zero Vector
0
0
0
0
0
Water
128-D Zero Vector
0
1
0
Electric
128-D Zero Vector
0
0
0
0
Magnet
128-D Zero Vector
0
0
1
0
Spring
128-D Zero Vector
0
0
0
Table 2: Action representations for the tools in CREATE environment.
A.3 RECSIM
The simulated RecSys environment (RecSim in Sec. 5.3.1p requires agent to select list of items that match the user's interest out variety of recommendable items. We simulate users that have preference over high-CPR lists (..c5). The agent's 's task is infer this preference from user clicks and recommend a list of items to optimize both user interest and CPR.
State:
The state is represented by the user interest embedding (c,, R.. where denotes the number of categories items) categories that transitions over time as the user consumes different items upon click. So, when the user clicks an item with the corresponding item embedding(e; R") then the user interest embedding(e..) will be updated as follows,
(eww) = (-yleeu + y) - (1-),), for y E [0,
6 x- Cu + (ee)) with probability (obbability[eTee.. + 11/2 e, - c. Ale,) with probability| - eTee//2
This essentially pulls the user's preference towards the item that was clicked.
The base action set is set of 500 items (250 for each train and test action set), and in each episode, sampled subset of size 20 is given to the agent. To simulate the varying action space environment, we implemented the most common category sampling method to form the candidate- set. Here, the majority of items are sampled from one common category, and the remaining items are sampled from other categories. In this way, identifying the most common category is crucial to recommend coherent list of items. Note that if we just sampled items uniformly across all categories, then CPR maximization would be less interesting as no single category has the potential to fill the entire list of items (i.e., achieve maximum CPR).
Action:
16

---

Published as a conference paper at ICLR 2022
60
45-
2,1
3.0
32
22
5,
5.0
6.
-00
-4
9.0
(a) Items with main category
(b) Items with main and sub category
Figure 9: t-SNE visualization of synthetically generated items in RecSim.
The base reward is a simulated response (e.g., clicks) from users (the user model (le et al., 2019a) stochastically skips clicks item in the list based on the user interest embedding). To better simulate the realistic scenario of user preference being affected by CPR of a presented list, we implement additional features in the model:
Reward:
scorestem Ouser
(ea; 6s) + Ommetric
Sitem
Pitem
Σ e
Satick _r (kppse(p(sttm))
where, Eay 6j IR" the user and item embedding, respectively, .) is the dot product notation and Ouserr Ometriic m71 E IR where m denotes the list-metric (e.g.. CPR-score). So, given the score scorestem of item, the user model computes the click likelihood through softmax function over all items and predefined skip-item followed by categorical distribution (felick or_ski).. It takes the computed likelihood and outputs either click (reward=1) or skip (reward=0) as user feedback.
Originally, le et al. (2019a) use the discrete representation of items based on the one-hot encoding of item- catecoteo However, this does not support generalization over items (actions). Therefore, we implement continuous item representations sampled from a Gaus- sian Mixture Model (GMM) with centers around each item category. Each item category has two sub-categories for items, which are also clustered. This ensures that the action representation con- tains information about the primary and sub-categories. Fig. 9 shows the t-SNE visualization of action representations for the 500 items, based on a GMMs. There are 10 main categories that each have two sub-categories. Figure 9a shows item representations labeled according to the primary category. Figure shows item representations clustered according to sub-category.
Action Representations:
A.4 REAL-DATA RECOMMENDER SYSTEM
The real-data RecSys environment (Sec. 5.3.2) requires an agent to select a list of items that match the user's interest out of variety of recommendable items. We experiment with the scenario in which domain engineers want an RL agent's list-actions to conform to user preference while op- timizing a listwise metric. Since having a high CPR is correlated with better user response (Hao et al. 2020). we use CPR as the additional listwise metric. Thus, we reward the agent based on (i) user models trained from real data and dataset from two different periods, two weeks in late August in 2021 as the offline period (used for training) and the following two weeks from early September in 2021 as the online period (used for evaluation).
(ii) CPR of the recommended list of items. We collected the
State: The state is a concatenation of a sequence historical user interactions. Concretely, a set of item representations (32 dimensional real vectors) of the three most recent clicked items (i.e., 32 x matrix) are appended individually to the user attributes (137 dimensional real vectors) such as age, occupation, and localities. Therefore, the state is in the form of 507 169 x 3 (32 + 137) x 3. To act optimally, the agent must extract the useful representation the user preference through this historical observation.
17

---

Published as a conference paper at ICLR 2022
Test F1 Score
Train Score
0.235
0.022
10.00000
0.550
01444
0.555
0.29
0.50
0.30
120
253
300
400
182
200
333
400
Fooch
Spoch
Train UM Test UM
Figure 10: Real-Data Recommender System: F1 Score for Training/test user models. (Left) The online user model is trained on online-training data and evaluated both maline-training (green) and online-held-out (orange) data. (Right) The offine user model is trained on offline-training data and evaluated on both offlnne-traiiing (green) and offline-held-ou data (orange). Thus, the disparity between online data training and evaluation curves (left) shows that it is hard to train the online user model. contrast, the offline user model generalizes reasonably well (right).
(a)
(b)
(c)
Figure 11: (a) t-SNE visualization of split of train and test item representations. This shows that the train and test items within the same distribution (b) Visualizations of item representations clus- tered according to the main category. This shows that item representations contain information about the main category, which is necessary maximize CPR. (c) Item distribution labeled according to sub category, which is another information necessary for CPR.
The base action set is a set of 85 items, and in each episode, a sampled subset of size 20 is given the agent. We employed the same sampling methods in Sec. A.3 So, the agent needs to select a list of 6 items given the sampled candidate set of 20 items.
Action:
The base reward is simulated response (click or skip) from the user model trained from real-world data. We add the CPR metric (between 0 to 1) to this click reward (1 or 0). The agent's objective is optimize user preference and the CPR of the list of items it recommends.
Reward:
Training of User models: We follow the two different periods (offline and online) in the data extrac- tion procedure. Thus, we trained two different user models (fuser S A IR) to (i) train the agents offline and (ii) evaluate them with online users. The result of training those user models can be found in Figure 10 (a). Thus, the reward is computed as follows; R
(eusers estem) m where fuser is either the offline user model or the online user model that provides us with the simu- lated response of users (e.g., click or skip). The user model architectures
described Sec. C.3.5
Action Representation: In the previous work CDQN (Chen et al.) 2019a)), the authors found it useful characterize the items by wide and deep features. For example, their movie recommen- dation task considered the text description as the wide feature and the movie category as the deep feature. And they utilized the Wide and Deep Network (Cheng et al. 2016) to get useful repre- sentations of items. Following their work, we used the wide and deep network to pretrain the item embedding given the rich item features in the real-world data. However, we empirically observed that employing VAEs for each wide and deep feature of items leads to the better-segregated repre- sentation items. Therefore, given an item instance, we separate the raw item attributes into the deep attributes (e.g., reward points of campaigns) and the wide attributes (e.g., text description),
18

---

Published as a conference paper at ICLR 2022
which are then fed into the VAE based wide and deep network to get the compressed representation by combining the wide and the deep features together. See Sec Sec|C.3.4]fo more details about the net- work architecture. Thus, each action representation is 32-dimensional vector encoded by VAE based Wide and Deep network. The learned representations of items are visualized based on the train-test split, distribution of the main category, and distribution of sub-category in Figure 11
B FURTHER EXPERIMENTAL RESULTS
B. EFFECT OF USING DOMAIN KNOWLEDGE IN ACTION GRAPH EDGES
To show how incorporate domain knowledge about action relations into the action graph (dis- cussed in Section D.
the Dig Lava Grid World environment. Specifically, use the knowl- edge that directional actions are always available. Thus, the only relevant action relations are the ones with the variable actions, i.e., 2 out of 4 actions (including both dig-lava skills). So, while building the action graph, we only keep connections to and from the variable actions instead of having fully-connected structure. The rest of the AGILE architecture and algorithm stays the same. This reduces the number of bidirectional edges from 72 49 to 29 by removing 20 edges between the always available actions. As shown in Figure T2 the learning speed of AGILE is slightly celerated, and the seed variance has reduced. We expect domain knowledge to help efficiency even more, when the action space is large since the edges scale quadratically.
Dig Lava Grid World
48
Reward
32
16
San
AGILE AGILE Dre-Delteee--rrth
Environment steps (1M)
(a) Training Curve: Slightly faster
(b) Attention Map:
Fewer edges
Figure 12:
Effect of using domain knowledge predefine the possible relations via edges in the action graph of AGILE. We know that in the Grid Navigation task, only the action relations with respect to variable actions are important. Thus, we remove all the other edges. This makes the learning slightly faster and more stable as shown by reduction in seed variance. (5 seeds)
B.2 VALIDATING DESIGN CHOICES FOR VALUE-BASED AGILE
We conducted an exhaustive search on the architecture value-based AGILE from two different per- spectives; (a) parammerrssstrss and (b) Architectures. Note that the same hyper-parameter search procedure of this section was applied to all other methods, and the same trend of the improvement in AGILE was found for other methods. In this section, AGILE used in the main results (Fig. is called AGILE-TTued. The barebone version of AGILE- is called UntuE-d illustrate how each change contributes to improvement in performance.
B.2.1 HYPER-PARAMETER SEARCH IN AGILE
AGILE-Tuned without syne-freq-change: In Mnih al. (2015), the authors used the periodic syncing between the target and the main networks to alleviate the issue of frequently moving Q- value targets. this work, we compare two extreme cases of the syne frequency: 10 depicted by (a) and 500 depicted by AGILE-Tuned.
Sync-frey=10 in Fig. (a)
AGILE-Tuned without graph-dim-change: To understand the difficulty in expressing the tion relations through a compact representation, we compare two hidden dimension sizes. The node-features encoded in 32 (Graph-dim=32) or 6A(AGII.E-Tumee dimensions.
19

---

Published as a conference paper at ICLR 2022
RECSIM
RECSIM
AULLS lnsi
AILE- Funed
Train Episode Clicks
Ginet ditt-12 Synofrope1t
AGLE-Untaned
Test Test Episode Clicks
Grapt 12 Sync/reze10
5000 Epoch
7500
2500
7500
10000
Epoch
(a) Hyperparameter search
RECSIM
RECSIM
AGLL-Tund AGLLE-Uttined
AGIL tured
AG&E tirtnee
Train Episode Clicks
Wich or cammer oer carget qerannge
Test Episode Clicks
With sre-saanrrast
target chhange
2500
5000
7500
10000
Epoch
Epoch
(h) Design choices
Figure 13: Results of value-based AGILE on RecSim CPR task (a) hyperparameter testing and (b) validating architecture design choices, on train actions (left) and test actions (right).
Figure 13 shows the result on both train and test actions. AGILE-TTuned outperformed all the methods. Graph-dim=32 is slightly worse than AGILE-Tumed and Sync-freq=10 fails learn any- thing meaningful. Thus, frequently moving target network harms the agent's performance while sufficient expressiveness in the action graph improves the performance.
B.2.2 DESIGN CHOICES OF AGILE
AGILE-Tuned with pre-summarizer: In AGILE-Tured, the concatenation of action repre- sentation and the state is node feature input to the GAT (Sec C.3.I) Here, we experiment and observe that adding a 2-layer MLP with ReLU over the node features does not help performance.
Chen et al. (2019a) compute the target q-values for training CDQN using the list-action at the next time-step. But, there is another potential target q-value from the next item in the current list-action. Here, we compare these two methods to get the target valuu: (a) Matra-listAGILEE-TTuee across-list (No target-q change): the target q-value is from the next timestep.
AGILE-Tuned without target-q-change
the target v-value is from the next list index. (b)
AGILE-Tuned without wvinGAT-changer In AGILE, the GAT output provides the utility net- work with the relational action representation and the action summary. Here, we compared two options: (a) Sharing GAT (No twin- GAT)): there is single GAT working to provide both of them. (b) Non-sharing GAT (AGILE -Tnee):: this employs two different GATs for each.
the implementation of AGILE, compared two different architectural decisions. (a) No main-target separation: Sep- share
AGILE-Tuned without
arate the list-action encoder in CDQN for main and target Q-networks. (b) the same list-action encoder for main and target Q-networks.
20

---

Published as a conference paper at ICLR 2022
RECSIM: Direct CPR
RECSIM: Pairing
Train Eplsode Reward
Train Episode Clicks
1.8
1.2
0.0
7500
10000
10000
Epoch
Epoch
AGILE (Ours)
Utility Policy
Mask-input-Outpe
Mask Daskktttput
Simple DQN
RECSIM: Direct CPR
RECSIM: Pairing
Test Episode Reward
Test Episode Clicks
12-
1.8
1.2
0.0
7500
10000
Epoch
Epoch
Comparison against the baselines train (top) and test (bottom) actions the Direct CPR (left) and Pairing (right) RecSim environments. Along with Figure 4 these results exhibit the same trend that AGILE consistently outperforms all the baselines both train and test actions.
Figure 14:
RECSIM: Direct CPR
RECSIM: Paining
1.6
Test Epls.ode Reward
Test TestEplsse Episode Ciicks
12-
0.8
4
0.4
25.00
5000
7500
10000
2500
S000
75:00
10000
Epoch
Epoch
(Ours)
Summary-GAT
Summary-LSTM
Summary Set
Figure 15: Comparison against ablations on the Direct CPR (left) and Pairing (right) RecSim envi- ronments. Along with F these results exhibit the same trend that AGILE slightly outperforms the various summary ablations, which do not explicitly utilize relational action features and rely only on the summary vector to encode all the necessary action relations.
Figure (3(b) shows the result on both train and test actions. AGILE-Tuned, With pre-summrizer, and No GAT showed the similar performance which is better than target-q-change. The differ- ence between AGIL-E-Tuned and No target-q-change that that the cascaded network in AGILE- Tuned uses the target q- vavul from the intermediate list constructed. This a more accurate target valul compared to the target valuue from another list from future time-step.
B.3 EFFECT DIRECT V/S INDIRECT REWARD IN RECSIM
In Fig we studied the learning capability of AGILE under the action interdependence on RecSim in which the CPR itself was not directly visible to the agents. Therefore, agents need to indi- rectly understand how well the list-action is through user feedback (i.e.. clicks). In this section, strengthen our results by examining the consistency results. The CPR is made directly visible to all the agents in this setting. So, implemented additional environment in RecSim where the reward is a sum the click reward and the CPR metric. We call this environment as Direct CPR. Figure [ s4ossws the comparison of AGILE against the same baseline agents Fig 4 on the Direct
21

---

Published as a conference paper at ICLR 2022
RECSIM: Direct CPR
RECSIM: Paining
Test Episode Reward
Test Episode Ciicks
1.2
0.0
7500
10000
10000
Epoch
Epoch
AGILE (Ours)
AGILE-GCN
AGILE-Only Action
Figure 16: Analyses of (i) GAT v/s GCN and (ii) state-action graph v/s action-only graphs on (left) RecSim: Direct CPR environment and (right) ReeSim: Pairing environment.
CPR RecSim. And in Fig 15 and Fig we got the consistent result that AGILE slightly outper- forms the ablations. AGILE outperforming the baselines shows that the knowledge of dependence on other available actions is crucial for optimal policy. This result consistent with the indirect CPR optimization setting in Fig So AGILE can perform well in the direct maximization of CPR.
B.4
ENVIRONMENT
In the experiment of Fig. we found that in RecSim, the relation of items is easy to model such that AGILE could not outperform the ablations. In contrast, AGILE outperformed the ablations in CREATE and Grid World by correctly utilizing the action relation in decision-making. We hypoth- esize that these environments require complex relations between actions (e.g., tools and activators in CREATE). To this end, implement the pre-defined pairings among items in RecSim such that clicks can only happen when the correct pairs of items are recommended. Since action relations complex, AGILE is expected to outperform the ablations. Figure 4 shows that AGILE beats the baselines and in Fig 15 AGILE slightly but consistently outperforms the ablations. In Fig 16 AGILE outperforming AGILE- GCN shows that GAT is capable of modeling the action relations correctly. AGILE converges faster than AGILE Only-Action. This shows that the state and the partially constructed list crucial to learning attend the other half in pairing items efficiently.
C APPROACH AND BASELINE DETAILS
C.I DETAILS OF BASELINES AND ABLATIONS
Like AGILE, all baselines and ablations receive the state and action representations input and output Q-value or probability distribution over available actions. Figure [7 describes the ar- chitectures of all the methods. Here, we discuss how each baseline and ablation is different from AGILE and why AGILE is expected to outperform them:
Mask-Output (No representations, No input action set): This baseline assumes fixed-action space that is known in advance. Since it does not use action representations, it cannot generalize to unseen actions exploit the structure in action space. Moreover, it does not take the available action set as input and thus cannot solve tasks where action decisions require knowledge of other actions.
Laaak-IIptt-Ouuppii (No representations): By augmenting a binary availability mask of given actions to the able action the inductive bias of order invariance of action set and cannot learn relations explicitly like graph networks.
state input of Mask-Output this method can utilize the information about the avail-
set. However, the input availability-mass is fed into an MIP, which lacks
Utility-Policy (No input action set): By using action representations, this method structure action space for efficient training and also generalize to unseen actions. However, like Mask-Outpun it does not utilize the available action part of the state and makes each action
the
22

---

Published as a conference paper at ICLR 2022
Observatoon
Observation
Availablity Mask
π
W//
Actions
Actions
(a) Mask-Oulput and Mask- Mask-Input- Input-nuppuuu Baselines
(b) Utility-Policy Baseline
Observation
Observation
GAT
π
GAT LSTM/ Deep
Actions
Action Summary
Actions
Action Summary
(c) Summary-GAY GAT, -LSTM, and -Deep Set Ablations
(d) AGILE
State
Observation
Peld dlation
π
Q/e
Actions
Action Summary
(c) AGILE- Onlly Action Analysis
Figure 17: Architectures of all methods used as baselines (Sec. 6.1.1
ablations (Sec. 6.1.2). and
analyses 6.3.2). (a) Following prior work in SAS-MDPs (Boutilier et al.) 2018 Chandak et al,, 2020a) and invalid action masking (Huang Ontañón 2020: Ye
2020, Kanervisto et al. 2020). the mask-output baseline masks out the unavailable actions assuming a known action set. The mask-input-output baseline additionally augments the state with the action availability mask. (b) Utility-policy (Jain et al., 2020) can generalize over actions by using action representations. However, computes each action utility independent of other available actions. (c) Summary Abla- tions augment the hulltyy-poiccy with an extra action-summary input, which is compressed version of the list of available action representations. This compression can be done by mean-pooling over Bi-LSTM output layer, a deep set, graph network (GAT) processed node features. (d) AGILE (Ours) uses GAT's node features both to compute an action set summary and as relational action features replacing the original action representations. While the action set summary is compact representation of the available actions, it does not sufficiently scale when there are many actions the task requires many action relations for each action decision. (e) AGILE-Only Action is the version of AGILE where the state is not used to compute action relations in GAT. This is a sim- pler architecture learn but is not expected to work for certain tasks where action relations change depending on the state.
23

---

Published as a conference paper at ICLR 2022
decision independent of other actions. Thus, it is expected to be suboptimal in all the tasks consider.
SuMmary-LSTM: This an ablation where we do not utilize the per-action relational features computed by GAT in AGILE. Instead, we use raw action features as input to the utility network. However, we utilize the available action set information by summarizing the set input action representations into vector. The summarization based on Bi-LSTM does not use the order in- variance property of the action set, which makes it less efficient to learn than other summarizers. AGILE is expected to outperform all summary-oni ablations in environments where several dif- ferent action relations need to be modeled for computing different action utilities. This is true for complex environments like CREATE, where the agent needs to consider various tools and activators to make an optimal decision about its action choice. In such environments, learning a shared summary-vector for all action utility computations is inefficient and possibly prohibitive for learning. We note that the summary-ablation are sufficient in simple environments such as Grid World navigation. The agent just needs summarize which of the two dig-lava skills are available. Such summary is enough to compute the utility of each of the 7 available actions.
This ablation utilizes the order invariance of action sets using a deep-set. While it still suffers from the limitations of lack of per-action relational action features, is low-parameter network and thus easy to train.
Summary-Deep Set:
Summary-GAT:
This a variation of AGILE, where raw action features replace the relational ac- tion features. Using GAT to extract the action set summary exploits order invariance. However, being a fully-connected graph network, it is can be slow to train while not offering much more than the deep-set summarizer. This can be useful when certain action relations are predefined using domain knowledge (Section B.I.
Only--Only-Attion: This variation of AGILE does not utilize state input in the GAT. Thus, the summary vector and relational action features are computed independently of state-context. While this is simpler architecture to train, can be insufficient in environments where the action rela- tions vary depending the state.
C.2 DETAILS ON LISTWISE AGILE
AGILE is implemented based on the listwise RI. architecture, CDQN (Chen et al. 2019a). As shown in Algorithm| [ CDQN builds a list-action incrementally by selecting N actions in sequence. For each list-index, a Q-network is used to select the best action. In addition to the usual state input, the partially built list-action also an input to the Q-network. Concretely, the state and current list are encoded into latent embeddings for any list index. These concatenated with each action representation in the remaining action set to build a list of nodes to be input into the action graph of AGILE. Treating it as fully-connected graph, AGILE outputs for all inputs actions. These are mean-pooled into a summary vector representing the entire action set. The representations of state, current-list, summary, and relational action features are used as input by utility network (MLP) which outputs the Q-value of that particular action for the current list-index. The action with the maximum Q-value added to the list, and the algorithm moves to the next list-index.
of relational action features
For training AGILE-based CDQN, we maintain a target Q-network that synchronized periodically with the main Q-network. Suppose tuple of (s, Alie, r, 8) is sampled from the replay buffer. For list index n between and N 1, we compute the target Q-value using the current state s with partial list-action 41n input. However, for the last list index using the next state for its first list index, A mean- arcan-squared error loss is used over all the list indices to train the main --network.
N, compute the target Q-value
C.3 NETWORK ARCHITECTURES
C.3.1 ACTION GRAPH
The action graph takes imput the action representations and the state- iatte-informatin (we also include the list-embedding for Listwise AGILE; See Sec. C.2). Given the concatenation of the input com- ponents above, optional 2-layer MI. with ReI.U, called pre-sunmarizer-mlp. transforms it into
24

---

Published as a conference paper at ICLR 2022
Algorithm Cascaded DQN: Listwise Action RL
1: def listwise_st listwise_action():
2: Parameters: Q-network ø Encoders, GAT and Utility network
3: Inputs: State 5, actions A. representations {Cao..". Cas), ,, list-action length N
4: Initialize: List Action diss = []. Candidate Actions A' = A
5: for n = 1.......A do:
6:
Encode State: 6₂ os(s)
7:
Encode Current List: elist (list(
8:
Build Graph Nodes: V {[es, Clisa, Cal
E A'
9:
Build fully-connected Adjacency Matrix of size | &
10:
Relational Action Features: 'relational ΦGAT (V.
va E A'
11:
Action Set Summary: Csummary = [V] Ea Felational
Q-values: Qn a, A') Qutility (Ee, elist Esummary relational
Ablations: ea
relational Co.
13:
Select action: arg max aEA Qn a, A')
Update: Ojist Atisa U an. A' A'
15:
def setwise_updated
16:
Parameters: Q-network Φ, Target Q-network ØT, discount factor y Inputs: state 8, action Ojist = a1.... ON, reward r, next state 8 for I..... do:
17:
18:
Use listwise listwise_action(o) to get Q-function: on = Qn an, alin-1) Use distwise_actionl to get target Q-value
20:
r + max, Qn+1 A aim) A n< N
Yn + y max, (1 a, A)
N
Optimize Loss = Σn= (qn yn)
the node features for the action graph. Then, the resultant node features are passed to two GAT layers followed by a residual connection. The same architecture of GAT is duplicated with different weights provide separate pathways compute the summary vector and relational action features. Therefore, we have two different sets of node features. The first set node features is mean-pood pooled produce the action-summary vector, which is put through 2-layer ML. with ReLU post- process before being passed on to the utility network. The other set node features (i.e., relational action features) is directly fed into the utility network.
AGILE with GCN: When the GCN is used in the action graph, the same input as above goes into a linear layer compress the size. Subsequently, the resultant node-features message-passing based on the normalized adjacency matrix, with all the diagonal elements made 0. Otherwise, the GCN reduces learning the same edge coefficients for all the nodes. The rest of the following architecture is the same as GAT above.
used in GCN
Summary-G.Ne In this ablation, instead of the relational action features being fed into the utility network, the raw action representations are used. The rest of the architecture remains the same.
C.3.2 SUMMARIZERS: BI-LSTM AND DEEP SET
Bi-LSTM: The raw action representations candidate actions are passed on to the 2-layer MIPP fol- lowed by ReLU. Then, the output of the MLP is processed by a 2-laaer bidirectional LSTM (Huang ct al., 2015). Another 2-layer MLP follows this to create the action set summary to be used in the following utility network.
DeepSet:
Similar to the Bi-LSTM variant of the summarizer, we employed the 2-layer MLP with ReLU followed by the mean pooling over all the candidate actions compress the information. Finally, the 2-yyyer MLP with ReLU provides the resultant action summary with the following utility network described in the next subsection.
25

---

Published as a conference paper at ICLR 2022
C.3.3 UTILITY NETWORK
We implemented two types of utility networks to show the potential of AGILE architecture working with different kinds of RL algorithms and environments. Both the utility networks take as input the same components: the state-inford sate-information. the relational or raw action representations, and the action summary of the action graph (See See C.3.I. The utilityy network is 2-layer MLP applied parallelly on all these inputs corresponding to each action. computes a sealar value for each action.
Policy Gradient(PPO): The action utility values are used as logits of a Categorical distribution, which is then used to train the AGILE architecture with policy gradient algorithm, PPO.
Value-) List Encoder: An intermediate list that contains the currently selected list items at each intra-list time-step is passed on to a single layer gated recurrent network (GRU (Cho) et al. 2014)) followed by 2-layer MLP to extract the compact representation of the intermediate list. See Algorithm for details on this.
Q-network: As in Sec C.3.1 the input components for the utility network are the raw action repre- sentation, the state-information, the list-embedding, the node-features, and the action set summary from the action graph - (st, Clists Emodes Esummary ag) in Algorithm [] These are concatenated into single vector and passed on to a 2-layer MLP with ReLU to compute the Q-value of an item.
C.3.4 ACTION REPRESENTATION NETWORK
Hierarchical VAE (CREATE): The CREATE action representations are borrowed from Jain et al. (2020) directly. The Hierarchical VAE network takes as input a list of behavioral trajectories rep- resenting each action (tool) and reconstructs with a two-layer hicrarchy of VAEs. We refer the reader to Appendix 1 of Jain et al. (2020) for complete details of the Hierarchical VAE network.
VAE in Deep and Wide architecture (Real World recommender system): In a VAE, the en- coder is implemented with a 5-layer MLP with Batch Normalization layer (Maas et al. 2013) and LeakyReLU (Maas et al. 2013). On the other hand, the decoder is implemented with a single MLP to process the input, followed by the same architecture of MLPs used in the encoder. Finally, 2-layer MLP with Batch Norm and Leaky ReLU and the hyperbolic tangent activation function is used to reconstruct the input instance.
C.3.5 REWARD INFERENCE NETWORK (USER MODEL IN REAL WORLD RECSYS)
The user model takes as input the user informationii.e. a concatenation of user attributes and a sequence of the user interactions) and a set of item embeddings in the list. The user information is passed on to a single layer gated recurrent network (Cho et al., 2014)) followed by a 2- layer MLP to extract the compact representation of the state. The same GRU network architecture (with different weights) processes the set of item embeddings into list-embedding. Finally, 2-layer MLP takes as input the concatenation of those two embeddings(i.e. state-embedding and list-embedding) and provides the scores of items in the list followed by the sigmoid function to transform to the individual click likelihood.
D EXPERIMENT DETAILS
D.1 IMPLEMENTATION DETAILS
We used Porch (Paszke et al. 2019) for our implementation, and the experiments were primarily conducted on workstations with either NVIDIA GeForee RTX 2080 Ti, P40. or V100 GPUs on Naver Smart Machine Learning platform (NSML) (Kim et al. 2018). Each experiment seed takes about hours for Grid Navigation, 60 hours for CREATE, hours for RecSim, and 15 hours for Real-Data Recommender Systems, to converge. We use the Weights & Biases tool (Biewald 2020) for logging and tracking experiments. All the environments were developed using the OpenAl Gym interface (Brockman et al. (0016). For training Grid Navigation and CREATE environments, the PPO (Schulman et al.; 2017) implementation based on Kostrikov (0188) For the recommender use DQN (Mnih et al.) 2015). We use the Adam optimizer (Kingma & Ba 2014) throughout. We attach the code with details to reproduce all the experiments, except the real- recommender system.
use
system environments,
26

---

Published as a conference paper at ICLR 2022
D.2 HYPERPARAMETERS
We build on hyperparameters used in prior work on CDQN (Chen et al. 2019a) and utility pol- icy (Jain et al. 2020). The hyperparameters for the additional components introduced in AGILE, baselines, and ablations are shown in Table The environment-specifi and RL algorithm hyperpa-
rameters are described in Table 4
Hyperparameter
Value
AGILE
number of GAT layers
2
number of attention heads
number of message passing steps leakyReLU alpha
0.2
graph hidden dimension
64
Residual Connection
True
Mask-Input-Output
Availability mask MLP hidden size
64
Ablations
Summary-LSTM hidden size
64
Summary-LST directions
Summary-LSTM number of layers
Summary-Deep-St hidden size
Summary-GAT hidden size
64
Summarizer pooling operation
mean
Table 3: AGILE, baseline and ablations: Hyperparameters for additional components
D.3 HYPERPARAMETER TUNING
Initial hyperparameters are inherited from the prior works (PPO: Jain al.] (2020); CDQN: Chen al. (2019a)). To ensure fairness across all baselines and our methods, we use the original hyperpa- rameters for the RL algorithms, such as learning rate, entropy coefficient, etc. (Table We choose sufficiently large number for total epochs (DQN) and total environment steps (PPO) ensure all the methods converge.
Our main contribution is the AGILE architecture. Thus, to correctly validate against baseline and ablation architectures, we search over shared parameters together and generally observe the same trend across all methods. This was expected because of the shared underlying implementation of the RL algorithm. Specifically, we searched over the following shared parameters, usually over 5 seeds and sometimes seeds:
Hidden dimension in DQN networks: We searched over {32, 64, 128} and found that 64 RJ 128 32. We choose 64 as the base hidden dimension across all encoders, summarizers, and GATs,
Target network sync frequency: This was sensitive parameter for DQN training. We searched over (10, 20, 100, 500, 1000) and found 500 be the best performing across all methods.
Weight sharing in CDQN: In Chen et al. 2019a),, the authors did not share the weights of Q- nets in their Cascading architecture, but empirically observed that sharing the weights of them improved the performance. This is likely due to the nature of RecSim tasks requiring similar items be recommended across the slate to maximize CPR.
Training batch size (DQN): We searched over {32, 64, 128, 256} on RecSim and chose 128 as, beyond that, we observed diminishing returns. For Real RecSys. we used the same batch size.
Training batch size (PPO): We searched over {4096, 8192) for Grid World and (3072, 4608) for CREATE and observed that larger batch sizes are better. We did not increase the batch size further due limits GPU memory size.
27

---

Published as a conference paper at ICLR 2022
Hyperparameter
Grid world
CREATE
RecSim
Real RecSys
Environment
action representation size observation space candidate actions per episode max. episode length
11
134
20
32
162
84
84
10
507
7
25
20
20
50
30
15
RL Training
training batch size parallel processes discount factor learning rate hidden layer size
8192
4608
128
128
64
48
16
16
0.99
0.99
0.99
0.99
le-3
le-3
le-4
le-4
64
64
64
64
PPO
entropy coefficient continuous action entropy coefficient
0.05
0.005
0.0005
total environment steps value loss coefficient
8 106
10a
0.5
0.5
PPO epochs
4
4
PPO clip parameter
0.1
0.1
DQN
epochs
10000
5000
target network sync frequency
500
500
Q-network hidden dimension
256 X 64
256 x 64
list encoder dimension
64
64
action encoder dimension
32
list size
6
6
replay buffer size initialize replay buffer size
106
106
5000
100
epsilon decay
1 + 0.1
1 0.1
epsilon decay last epoch
500
250
Table 4: Environmenn/Poliey-sccciii Hyperparameters
Below, detail the hyperparameters and network variations searched specifically for the graph attention network (GAT) used in AGILE and variants.
Skip Connection: Adding skip connection was crucial to making AGILE learn well.
Number of graph attention layers: We searched
21,2,3) layers and found layers to be the best performing for both PPO and CDQN based policies.
Number of attention heads: We compared having {1, 2} attention heads and found no meaning-
ful learning for 2 attention heads.
Number of message passing steps: Adding more message-passing steps than 1 did not improve AGILE performance.
Other relevant design choices described and validated in Section B.2
28


Reviews:
[{"review_id": "uCVWdn2rpd7", "reviewer": "Reviewer_FMiS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nLearning action relations and dependence is an important problem in Rl, especially when the action set is varying according to the environment's constraints, as this is often the case in real-world scenarios and applications. Authors address a subset of this broader problem, namely when the action set only changes at the beginning of a task instance. \n\nThe main contribution and the novelty of this paper lie in the utilization of graph attention networks to learn the dependencies of actions using a fully connected action graph. The formulation of the fully connected action graph and the graph attention network is sound, though there are some questions regarding graph formulation which I ask below. \n\nThe authors demonstrate the generalizability of the policy architecture through training both value-based policy-based RL agents. Specific implementation details of how AGILE can be used in widely used RL methods (PPO, DQN) are also given.\n\nAnother strength of the paper lies in its evaluation, where experiments are done both in benchmarks and in applications, the latter with simulated and real-world data. The selection of benchmark environments is suitable for the evaluation. I particularly like the inclusion of the application based evaluation. There are some concerns with the method selection for evaluation which I share below.\n\nThe authors also give further insights into how the attention of the graph network is performed by qualitatively examining the method’s performance in different domains that were evaluated. I encourage the authors to have a more in-depth discussion on this as in its current form it is sparse. For example, it would be useful for the reader if contrastive cases are given for the presented examples.\n\nThe paper is also well written and easy to understand. The formulation of the architecture is clear, though some network descriptions and parameters are missing.\n\nWeaknesses:\n\nThe main weakness I see in this paper is the lack of positioning the work relative to action varying RL methods. The authors do not compare the work of (Chandak et al, ,2020b) to the proposed method (though they have briefly discussed it in the related work section) computationally. While authors do use masked action sets and utility policy as baselines, other action varying baselines are needed for a comprehensive evaluation.\n\nAnother minor weakness in the paper is the lack of clarity in the action graph formulation. Authors mention that domain knowledge can be used if action relations are predefined and known beforehand, but does not give much detail on how and to what extent this can affect the performance.\n\n\n----------------------------------------------------------------------\n\nAfter the rebuttal\n\nI appreciate the clarifications made about the selection of the benchmarks and I agree with the authors here. I especially commend authors for doing an in depth qualitative analysis and adding contrastive cases, this makes a stronger case for the paper's contribution. I have updated my score accordingly. \n", "summary_of_the_paper": "This paper introduces a novel policy architecture (AGILE) for RL agents that learn action interdependence from a varying action space. A graph attention network is used to calculate the action utility and to summarize the action set input. Authors argue that this architecture allows the RL agents’ to learn action relations that lead to optimal behaviour in a changing action space environment. This architecture is then evaluated in three benchmark domains. Further evaluations are done in the recommender systems context, with both simulated systems and using real-world data. Results seem to indicate that the proposed method outperforms non-relational RL methods in most cases.", "summary_of_the_review": "This paper presents an approach that addresses an important problem in RL, namely how to act optimally in an environment with varying actions. The proposed method seems to perform well against the compared methods. Though more baselines are needed, especially from closely related methods to draw conclusions. Evaluation is done using a selection of benchmarks and application domains which I view as a positive. The architecture is described in sufficient detail, though some sections need further clarifications. \n\n\n[1] Huang, S., & Ontañón, S. (2020). A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bJBjaPGL4g", "reviewer": "Reviewer_mRLy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe motivation is solid and intuitive, the summaries of related work are insightful, the environments involve variable sets of actions for which the pairwise relations are clearly important, and the discussion of experimental results is enlightening. AGILE’s benefits are most clear for the CREATE environment.\n\nWeaknesses\n\nAGILE’s margin of improvement over baselines is not large enough to outweigh the spurious effects of inadequate hyperparameter tuning (a well-known issue in deep RL). Unfortunately, the paper makes no mention of hyperparameter tuning. GAT hyperparameters are listed in Table 1, but hyperparameters for baseline architectures (like LSTM and Deep Set) are missing, and standard learning hyperparameters (like learning rate) are not mentioned at all. We are left to wonder whether AGILE’s moderate gains would be maintained if the hyperparameters had been more thoroughly tuned for the baselines. This seriously undercuts the conclusions we can draw from the experimental results. \n", "summary_of_the_paper": "This paper proposes an architecture (AGILE) for handling variable action sets through pairwise attention between the currently available actions, where the action set is free to change between episodes, and each action is described by an associated feature vector. In evaluations against baseline architectures and ablations on a diverse set of suitable environments, AGILE performs the best overall.", "summary_of_the_review": "This work has much to admire, and could make an important contribution if the uncertainty regarding its hyperparameter tuning didn’t cast such doubt on the experimental results.\n\n--- POST-REBUTTAL UPDATE ---\n\nThe authors have provided much more information about their hyperparameter tuning, and have performed additional experiments which provide additional insights. These changes address my concerns, and I have revised my evaluation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U7KKjXtW94x", "reviewer": "Reviewer_RWVF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work frames itself as an extension of Jain et al. 2020 to consider the relations between actions rather than evaluating their utility independently. In this regard it is a moderately incremental development, so the bar for the quality of the study should be reasonably high.\n\nThe motivation for the work is sound and reasonably well-argued, and the positioning with respect to related work is quite clear.\n\nThe empirical study is interesting, and covers a good variety of benchmarks with quantitative as well as some qualitative analysis.\n\nI find there is some room for improvement in the exposition of certain details, and in aligning the discussion more closely with the empirical findings.\nMy main concern is about the description of the ablations in S6.1.2 and S6.3.1, especially because some of the ablations outperform the proposed architecture, at least in the recommender systems experiments.\nSummary-LSTM and Summary-DeepSet clearly use different architectures to produce the action set summary. But it’s unclear to me how they condition on the action. Do they use the raw action representations like Summary-GAT?\nI’m also unsure about the authors’ assertion that Summary-GAT can clearly show the importance of using action interdependence: the utility function can still compare the action representation to the summary of available actions (which can, for example, use different parts of its representation to correspond to different specific actions). The specific definition of interdependence referred to here should be made more clear.\nThere is also no comparison of the total number of parameters or computation used by the different models, which makes it harder to understand the importance of the structural choices.\nIn S6.3.1, GCN is not defined (I assume this is Graph Convolutional Network, but there are many missing details).\nI would also like more detail on the qualitative analysis: it’s unclear how the figures were produced. Do arrow widths correspond to attention weights? Are they left off the figure below a certain threshold? I also find Fig 6c (labelled 6b) quite hard to parse.\n\nClarifying details aside, I don’t feel the discussion aligns well with the full set of empirical results. The authors suggest that noisy/sparse rewards might make a GAT harder to train, but don’t show any particular evidence that this is the defining difference in the RecSim case. A potential followup would be to make the rewards sparser or noisier in the other tasks, to see if the GAT-based approach once again falls behind.\nOverall, it feels that a more thorough discussion of the difference between the RecSim experiments and the other environments needs some more attention. One question might be how relevant the quality of the raw action representations is?\nI also find the concluding remarks somewhat misleading. In S6.3.2 it says that Fig 7 shows a drop in performance across all environments when using AGILE-Only Action, but there is an increase in performance on RecSim, as far as I can see.\nThis claim is then echoed in the final sentence, that relational knowledge is crucial for optimal decision-making (while the ablations seem to suggest that relational knowledge is sometimes less important than affordance knowledge).\n\nI would appreciate it if the authors could clarify the role of these ablations, and which conclusions may be drawn from them.\n\nMinor comments:\n - The notation for the action set summary is a bit weird (S normally being reserved for states). Maybe \\bar{c}^R would better indicate the average over action features?\n - The equation for the action set summary should have c^R rather than c?\n - Last sentence about Action Utility should say the score can be used as a logit fed into a softmax, rather than it can be used as a prob distribution\n - pi(a|s’) should also condition on c^R?\n - I appreciate the inclusion of some more anecdotal comments about which implementation decisions were important (residual connections, parameter sharing)\n\n----------------------\n\nThe authors have comprehensively addressed my concerns, both by improving their algorithm's performance and reflecting appropriate changes in both the description and discussion of ablations (and qualitative analysis); I am raising my score as a result.\n", "summary_of_the_paper": "This paper tackles an RL problem setting in which the actions available to an agent vary from episode to episode, and the optimal action in some states depends on the other actions that are available.\nThe authors’ approach to this setting is to use a graph neural network to process all available actions, both to summarise the available set and to produce a relationally-informed representation for each action. These are then fed, with the state, into a utility function to give an action value or logit.\nExperiments in a number of benchmarks show the value of including information about the available actions.", "summary_of_the_review": "The work is reasonably sound overall (if somewhat incremental) but is let down by the slightly unclear set of ablations and discussion of the empirical results. I would consider raising my score if the authors can compellingly clarify what conclusions can be drawn from their study.\n\n----------------------\n\nI believe the authors have addressed all of my concerns in their responses and updates to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OiwdreT1qU", "reviewer": "Reviewer_3F2U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The central point of the paper is that having variable action spaces requires learning interdependence relations between possible actions. This point is illustrated well in the paper; AGILE outperforms benchmarks that ignore action interdependence (in particular, Jain et al.) and the qualitative results in Section 6.2 show that attentions learned by the GAT are reasonable.\n\nMy main concern with the paper is that a lot of important details are missing in the exposition. In particular, Section 4 relies too much on previous work in order to explain the architecture in Figure 2 without contextualizing that previous work well within the problem setting of AGILE. For instance,\n* How are the nodes of the GAT determined? As far as I understood, when a finite set of base actions are given, each node corresponds to one of the actions. Then, what happens when a set of base actions is not known (which is certainly a setting AGILE aims to address)?\n* In the case of listwise RL, I understand how the number of possible lists is combinatorial and this is problematic for AGILE. However, I do not understand how this issue was resolved; the only explanation given is that \"the lists were built incrementally, one action at a time.\" What does this precisely mean?\n* In Section 5.3.2, the authors mention that \"they collected interaction data\" for the real-world recommender system environment. As far as I understood, this data appears for the first time in this paper however there is not enough details regarding how the data was collected, how it is structured, how the reward models were trained, etc.\n* I think how the experiments with train vs. test actions are conducted need to be clearer. As far as I understood, each model was trained using train actions only but for the bottom results in Figure 4, they were evaluated when test actions were also available. Then, how are the actions in CREATE are represented? I assume not all actions are one-hot encoded in this environment; otherwise, how would it be possible to generalize to actions that are never seen before?\n\nSome minor comments:\n* How baselines in Section 6.1.1 were introduced is very helpful in understanding the related work. It is very clear there in what aspects AGILE differ from the existing work.\n* Needing a hammer to be able to use a nail is used as the main example of how action interdependences come into play. At first, I was confused about how attentions in a graph is able to capture a directional dependence like this but Section 6.2 explains it perfectly. Considering how central the nail-hammer example is to the exposition, maybe how attention weights relate to it can be explained even earlier.\n\n\n\n\n\n", "summary_of_the_paper": "This paper considers the reinforcement learning problem where the action space is fixed. Having a variable action space makes it necessary to learn interdependence between different actions (use of some actions might depend on the existence of other complementary actions), which is modelled via a graph attention network.", "summary_of_the_review": "The key idea of the paper is illustrated well but many details regarding the approach and the experiments are left out.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of how to train an agent to understand relationships and dependencies among available (and potentially changing) actions in an RL environment to more efficiently solve a task. For instance, in the absence of a hammer for the task of putting up a painting on a wall, the agent could use an alternative tool like adhesive strips if available. The paper's main technical contribution is to use train a graph attention network to learn action space relationships under a given action representation. The paper demonstrates the effectiveness of this strategy on a range of environment benchmarks.

The reviewers initially brought up several lacunae in their assessment of the paper. These included the opaqueness in the explanation of the graph network structure, incremental nature of the improvement over the paper of Jain et al 2020, the lack of clear ablation studies and their message, comparisons with baselines drawn from other existing approaches potentially relevant to the setting, and the role of hyperparameters and their tuning.

In response, the author(s) provided detailed clarifications and additional experimental results. Namely, they clarified the details of the graph attention network, added ablation studies to help understand the role of this component, discussed the relevant and (in)applicability of other existing work, and supplied details about hyperparameter tuning. The author response was adequate to convince the reviewers to arrive at a consensus reflecting the positive impression of the paper.

In view of the unanimous opinion of the reviwers, I recommend acceptance of the paper.
Decision: Accept (Poster)