Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles an RL problem setting in which the actions available to an agent vary from episode to episode, and the optimal action in some states depends on the other actions that are available.
The authors’ approach to this setting is to use a graph neural network to process all available actions, both to summarise the available set and to produce a relationally-informed representation for each action. These are then fed, with the state, into a utility function to give an action value or logit.
Experiments in a number of benchmarks show the value of including information about the available actions.
main_review: This work frames itself as an extension of Jain et al. 2020 to consider the relations between actions rather than evaluating their utility independently. In this regard it is a moderately incremental development, so the bar for the quality of the study should be reasonably high.

The motivation for the work is sound and reasonably well-argued, and the positioning with respect to related work is quite clear.

The empirical study is interesting, and covers a good variety of benchmarks with quantitative as well as some qualitative analysis.

I find there is some room for improvement in the exposition of certain details, and in aligning the discussion more closely with the empirical findings.
My main concern is about the description of the ablations in S6.1.2 and S6.3.1, especially because some of the ablations outperform the proposed architecture, at least in the recommender systems experiments.
Summary-LSTM and Summary-DeepSet clearly use different architectures to produce the action set summary. But it’s unclear to me how they condition on the action. Do they use the raw action representations like Summary-GAT?
I’m also unsure about the authors’ assertion that Summary-GAT can clearly show the importance of using action interdependence: the utility function can still compare the action representation to the summary of available actions (which can, for example, use different parts of its representation to correspond to different specific actions). The specific definition of interdependence referred to here should be made more clear.
There is also no comparison of the total number of parameters or computation used by the different models, which makes it harder to understand the importance of the structural choices.
In S6.3.1, GCN is not defined (I assume this is Graph Convolutional Network, but there are many missing details).
I would also like more detail on the qualitative analysis: it’s unclear how the figures were produced. Do arrow widths correspond to attention weights? Are they left off the figure below a certain threshold? I also find Fig 6c (labelled 6b) quite hard to parse.

Clarifying details aside, I don’t feel the discussion aligns well with the full set of empirical results. The authors suggest that noisy/sparse rewards might make a GAT harder to train, but don’t show any particular evidence that this is the defining difference in the RecSim case. A potential followup would be to make the rewards sparser or noisier in the other tasks, to see if the GAT-based approach once again falls behind.
Overall, it feels that a more thorough discussion of the difference between the RecSim experiments and the other environments needs some more attention. One question might be how relevant the quality of the raw action representations is?
I also find the concluding remarks somewhat misleading. In S6.3.2 it says that Fig 7 shows a drop in performance across all environments when using AGILE-Only Action, but there is an increase in performance on RecSim, as far as I can see.
This claim is then echoed in the final sentence, that relational knowledge is crucial for optimal decision-making (while the ablations seem to suggest that relational knowledge is sometimes less important than affordance knowledge).

I would appreciate it if the authors could clarify the role of these ablations, and which conclusions may be drawn from them.

Minor comments:
 - The notation for the action set summary is a bit weird (S normally being reserved for states). Maybe \bar{c}^R would better indicate the average over action features?
 - The equation for the action set summary should have c^R rather than c?
 - Last sentence about Action Utility should say the score can be used as a logit fed into a softmax, rather than it can be used as a prob distribution
 - pi(a|s’) should also condition on c^R?
 - I appreciate the inclusion of some more anecdotal comments about which implementation decisions were important (residual connections, parameter sharing)

----------------------

The authors have comprehensively addressed my concerns, both by improving their algorithm's performance and reflecting appropriate changes in both the description and discussion of ablations (and qualitative analysis); I am raising my score as a result.

summary_of_the_review: The work is reasonably sound overall (if somewhat incremental) but is let down by the slightly unclear set of ablations and discussion of the empirical results. I would consider raising my score if the authors can compellingly clarify what conclusions can be drawn from their study.

----------------------

I believe the authors have addressed all of my concerns in their responses and updates to the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a novel policy architecture (AGILE) for RL agents that learn action interdependence from a varying action space. A graph attention network is used to calculate the action utility and to summarize the action set input. Authors argue that this architecture allows the RL agents’ to learn action relations that lead to optimal behaviour in a changing action space environment. This architecture is then evaluated in three benchmark domains. Further evaluations are done in the recommender systems context, with both simulated systems and using real-world data. Results seem to indicate that the proposed method outperforms non-relational RL methods in most cases.
main_review: Strengths:

Learning action relations and dependence is an important problem in Rl, especially when the action set is varying according to the environment's constraints, as this is often the case in real-world scenarios and applications. Authors address a subset of this broader problem, namely when the action set only changes at the beginning of a task instance. 

The main contribution and the novelty of this paper lie in the utilization of graph attention networks to learn the dependencies of actions using a fully connected action graph. The formulation of the fully connected action graph and the graph attention network is sound, though there are some questions regarding graph formulation which I ask below. 

The authors demonstrate the generalizability of the policy architecture through training both value-based policy-based RL agents. Specific implementation details of how AGILE can be used in widely used RL methods (PPO, DQN) are also given.

Another strength of the paper lies in its evaluation, where experiments are done both in benchmarks and in applications, the latter with simulated and real-world data. The selection of benchmark environments is suitable for the evaluation. I particularly like the inclusion of the application based evaluation. There are some concerns with the method selection for evaluation which I share below.

The authors also give further insights into how the attention of the graph network is performed by qualitatively examining the method’s performance in different domains that were evaluated. I encourage the authors to have a more in-depth discussion on this as in its current form it is sparse. For example, it would be useful for the reader if contrastive cases are given for the presented examples.

The paper is also well written and easy to understand. The formulation of the architecture is clear, though some network descriptions and parameters are missing.

Weaknesses:

The main weakness I see in this paper is the lack of positioning the work relative to action varying RL methods. The authors do not compare the work of (Chandak et al, ,2020b) to the proposed method (though they have briefly discussed it in the related work section) computationally. While authors do use masked action sets and utility policy as baselines, other action varying baselines are needed for a comprehensive evaluation.

Another minor weakness in the paper is the lack of clarity in the action graph formulation. Authors mention that domain knowledge can be used if action relations are predefined and known beforehand, but does not give much detail on how and to what extent this can affect the performance.


----------------------------------------------------------------------

After the rebuttal

I appreciate the clarifications made about the selection of the benchmarks and I agree with the authors here. I especially commend authors for doing an in depth qualitative analysis and adding contrastive cases, this makes a stronger case for the paper's contribution. I have updated my score accordingly. 

summary_of_the_review: This paper presents an approach that addresses an important problem in RL, namely how to act optimally in an environment with varying actions. The proposed method seems to perform well against the compared methods. Though more baselines are needed, especially from closely related methods to draw conclusions. Evaluation is done using a selection of benchmarks and application domains which I view as a positive. The architecture is described in sufficient detail, though some sections need further clarifications. 


[1] Huang, S., & Ontañón, S. (2020). A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the reinforcement learning problem where the action space is fixed. Having a variable action space makes it necessary to learn interdependence between different actions (use of some actions might depend on the existence of other complementary actions), which is modelled via a graph attention network.
main_review: The central point of the paper is that having variable action spaces requires learning interdependence relations between possible actions. This point is illustrated well in the paper; AGILE outperforms benchmarks that ignore action interdependence (in particular, Jain et al.) and the qualitative results in Section 6.2 show that attentions learned by the GAT are reasonable.

My main concern with the paper is that a lot of important details are missing in the exposition. In particular, Section 4 relies too much on previous work in order to explain the architecture in Figure 2 without contextualizing that previous work well within the problem setting of AGILE. For instance,
* How are the nodes of the GAT determined? As far as I understood, when a finite set of base actions are given, each node corresponds to one of the actions. Then, what happens when a set of base actions is not known (which is certainly a setting AGILE aims to address)?
* In the case of listwise RL, I understand how the number of possible lists is combinatorial and this is problematic for AGILE. However, I do not understand how this issue was resolved; the only explanation given is that "the lists were built incrementally, one action at a time." What does this precisely mean?
* In Section 5.3.2, the authors mention that "they collected interaction data" for the real-world recommender system environment. As far as I understood, this data appears for the first time in this paper however there is not enough details regarding how the data was collected, how it is structured, how the reward models were trained, etc.
* I think how the experiments with train vs. test actions are conducted need to be clearer. As far as I understood, each model was trained using train actions only but for the bottom results in Figure 4, they were evaluated when test actions were also available. Then, how are the actions in CREATE are represented? I assume not all actions are one-hot encoded in this environment; otherwise, how would it be possible to generalize to actions that are never seen before?

Some minor comments:
* How baselines in Section 6.1.1 were introduced is very helpful in understanding the related work. It is very clear there in what aspects AGILE differ from the existing work.
* Needing a hammer to be able to use a nail is used as the main example of how action interdependences come into play. At first, I was confused about how attentions in a graph is able to capture a directional dependence like this but Section 6.2 explains it perfectly. Considering how central the nail-hammer example is to the exposition, maybe how attention weights relate to it can be explained even earlier.






summary_of_the_review: The key idea of the paper is illustrated well but many details regarding the approach and the experiments are left out.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an architecture (AGILE) for handling variable action sets through pairwise attention between the currently available actions, where the action set is free to change between episodes, and each action is described by an associated feature vector. In evaluations against baseline architectures and ablations on a diverse set of suitable environments, AGILE performs the best overall.
main_review: Strengths

The motivation is solid and intuitive, the summaries of related work are insightful, the environments involve variable sets of actions for which the pairwise relations are clearly important, and the discussion of experimental results is enlightening. AGILE’s benefits are most clear for the CREATE environment.

Weaknesses

AGILE’s margin of improvement over baselines is not large enough to outweigh the spurious effects of inadequate hyperparameter tuning (a well-known issue in deep RL). Unfortunately, the paper makes no mention of hyperparameter tuning. GAT hyperparameters are listed in Table 1, but hyperparameters for baseline architectures (like LSTM and Deep Set) are missing, and standard learning hyperparameters (like learning rate) are not mentioned at all. We are left to wonder whether AGILE’s moderate gains would be maintained if the hyperparameters had been more thoroughly tuned for the baselines. This seriously undercuts the conclusions we can draw from the experimental results. 

summary_of_the_review: This work has much to admire, and could make an important contribution if the uncertainty regarding its hyperparameter tuning didn’t cast such doubt on the experimental results.

--- POST-REBUTTAL UPDATE ---

The authors have provided much more information about their hyperparameter tuning, and have performed additional experiments which provide additional insights. These changes address my concerns, and I have revised my evaluation.
review_title: null

Meta-review: This paper studies the problem of how to train an agent to understand relationships and dependencies among available (and potentially changing) actions in an RL environment to more efficiently solve a task. For instance, in the absence of a hammer for the task of putting up a painting on a wall, the agent could use an alternative tool like adhesive strips if available. The paper's main technical contribution is to use train a graph attention network to learn action space relationships under a given action representation. The paper demonstrates the effectiveness of this strategy on a range of environment benchmarks.

The reviewers initially brought up several lacunae in their assessment of the paper. These included the opaqueness in the explanation of the graph network structure, incremental nature of the improvement over the paper of Jain et al 2020, the lack of clear ablation studies and their message, comparisons with baselines drawn from other existing approaches potentially relevant to the setting, and the role of hyperparameters and their tuning.

In response, the author(s) provided detailed clarifications and additional experimental results. Namely, they clarified the details of the graph attention network, added ablation studies to help understand the role of this component, discussed the relevant and (in)applicability of other existing work, and supplied details about hyperparameter tuning. The author response was adequate to convince the reviewers to arrive at a consensus reflecting the positive impression of the paper.

In view of the unanimous opinion of the reviwers, I recommend acceptance of the paper.
Decision: Accept (Poster)