Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: EDIT: The authors clarified the presentation, gave a nuanced response to my concerns about the relative scalability, and promised to discuss the relationship to the actions-in architecture in the final version (it's not as central as I first thought given the limited role played by universal mixers). Solid work in its current state.

The authors propose a method that incorporates multi-way relationships between action dimensions in the estimate of action-value functions. This provides a strong inductive bias for reinforcement learning problems with a small number of action dimensions, each  taking several values. This is an interesting approach, but there are presentation and experimental issues preventing me from endorsing its publication.

Currently the presentation obscures some of the limitations of the method. Not until the experiments is it revealed that the method (in its current state) is only applicable to tasks with finite action spaces. Most tasks typically considered as having multiple action dimensions are continuous control tasks, so this is unexpected given the motivation. Not knowing this up front also hurts the clarity of the paper e.g. Figure 1b mentions outputting all possible actions, which is only coherent in the context of this finite cardinality restriction. Now, I'm not saying that not handling continuous action spaces is necessary (your results on discretized continuous action tasks are impressive), just that you should be more up front with this fact.

A second hidden limitation has to do with your method's scaling. If I understand your method correctly, to compute the maximum action-value you have execute the hyper-graph and mixer repeatedly, once for each possible action. This is in stark contrast to the standard actions-out architecture popularized by DQN -- a single forward pass compute all action-values. You (correctly) point out that both your method and DQN grow exponentially in the number of action dimensions, but in practice the latter will scale much better as it just amounts to a larger dimension matmul. Again, this limitation need not be overcome, but it should be mentioned.

On the experimental front, there is a serious baseline missing. Prior to DQN, many RL (with function approximation) papers treated the action as an input. Indeed, the only reason this is less common these days is due to the run-time issue discussed above. Treating the action as an input corresponds to only using the highest order hyper-graph (since the action contains all action dimensions), making that special case not a contribution of this paper. Thus, comparing against this case would be necessary to show that this method improves upon prior work. Additionally comparing against more recent approaches (e.g. auto-regressive processing of action dimensions) would further strengthen the paper, but this is of less importance in my opinion.
review_title: Interesting paper that raises interesting questions.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper incorporates a concept called hypergraph network into reinforcement learning. The idea of hypergraph is to extend edge to hyperedge where a set of vertices can be considered at the same time. This seems natural for scenarios like continuous action control with multi-dimension action space. From experimental results this proposed action hypergraph networks outperform several existing baselines.

Pros:
This paper has relatively high novelty. The idea of hypergraph may be useful for broad range of applications.

Cons:
1. HGQN vs. DQN is this a good comparison? DQN is a good baseline but no longer SOTA for Atari games. I wonder how HGQN vs. rainbow or DDPG for Figure 4 looks like.
2. It will be better if authors provide error bands for Figure 5. It's unclear how significant the proposed method is better.
3. When number of actions gets large, this hypergraph idea will have exponential complexity. How to find a best way to partition the graph (or generalize to 3+ hyperedges)?
review_title: Kind of novel but not sure about scalability and how significant improvement is

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work focuses on learning action representations for problems involving high-dimensional action spaces. The aim is to build a flexible and general methodology for learning representations of multidimensional actions that can be combined with existing architectures (which mostly focus on learning state representations). The ideas initially suggested in Sharma et al (2017) have been generalized and refined in this work leading to impressive empirical results.

Strengths:
* The hypergraph formulation of the action space is well motivated and well grounded in literature.
* The aims set out in the introduction of the paper were duly justified -- the proposed framework is indeed very flexible and general as can be seen from the application to multiple problem types.
* The choice for Mixing function and Hypergraph specification are practical which makes them broadly applicable.
* The empirical results clearly illustrate the benefits achieved due to action representation.
* The paper is clearly written and well-organized.

Weaknesses:
* The experiments with three dimensional actions (Sec 4, 5) demonstrate clear benefits of the method, however, the action dimensionality is small enough to include all possible hyperedges. Sec 6 conducts experiments using five-dimensional actions where we notice tradeoffs between using hypergraphs of different ranks. From Fig 6 & 7, it is unclear what are the benefits of using hyperedges of higher order.
* It would be informative to conduct experiments on problems with higher dimensional action spaces such as Hunters & Rabbits. 
* The experiments lack comparison against other methods that decompose actions such as Metz et al (2017).

Update after author response:
I thank the authors for their detailed rebuttals as well as for engaging with the reviewers. It'd be worth adding a small section on how to scale this method for higher dimensional action spaces (bullet 2 in author response). Comparison with Metz et al (2017) is valid and appreciated. Based on this, I am happy to increase my rating from 7 to 8.
review_title: A novel and general approach to decomposing action-value function

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work considers the idea of adding a representation for action space. This is on top of the usual representation of state-space. Their approach is based on hypergraph representation and shows its merits in the experimental results. But in general, I find the technical contributions (borderline) incremental.
review_title: Considering a representation for action space (on top of state space) is a good idea but technics are incremental.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper considers the problem of representation learning of actions, i.e., learning a decomposition of action-value function in multidimensional action spaces using hypergraphs. The key idea is to represent the actions as a hypergraph and learn a representation for each hyperedge in an arbitrary hypergraph. Consequently, the architecture conduits of a neural network for each hyperedge that are then combined using a mixing function. The choice of mixing function appears to be dependent on the problem -- for instance, it could be an universal mixer such as a neural  network or it could be a simple summation. The key impact of this architecture could be in the bandit problem setting with very large action spaces. Empirical results on a few atari games (29) and simulated physical control benchmarks clearly demonstrate the superiority of the approach.

The paper is written well. The problem is well motivated and sufficient details are provided. Experimental results appear convincing. The use of hypergraphs themselves are not necessarily novel but the implications in large multi-agent systems makes the paper compelling. Personally, i feel that the paper can have a good impact. 

I do have a few comments:

First, I think the term “relational inductive bias” is quite a stretch for here. It really is much more than a simple inductive bias. You are constructing the full architecture. Granted that the structure serves as a bias but this is much more than an inductive bias. It really can be interpreted as a search bias as well. 
Moreover, relational is also misused a lot (not just in this paper). IF you claim relational, then you should demonstrate generalization. Without that one cannot claim relational. I dont think your method is generalizable to varying number of objects in the domain. I suggest calling these as structured problems more than relational domains. The atari games for instance are not even close to being relational. 
I am not sure I understand why a 3-hyperedge will have higher variance. Could you kindly explain this? Does this mean that you are overfitting in terms of creating newer edges that dont exist? Is it too fine-grained of a representation? Some speculation could certainly improve the paper. 

review_title: Good paper but some clarifications are needed!

Meta-review: After reading the reviews and rebuttal and looking over the paper, I feel that the results are indeed strong, and the paper could have an impact in terms of exploiting the relationship among action dimensions. Maybe the only detail that I would add is that going through the example given in Fig 1 completely could be useful as it might not be perfectly obvious how (e.g. considering a simple mixing function like summation) one retrieves the q values for someone not familiar with this particular topic. 
Decision: Accept (Poster)