Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: This paper proposes a method to progressively explore the action space for RL. The proposed method is called “growing action spaces”. The basic idea is that actions can usually be grouped by a hierarchical structure: the lowest level is the coarsest and higher levels gradually refine the action partition. This method effectively captures many RL settings, including multi-agent learning. One effective approach is to apply the action hierarchy. 

Then the paper performs experiments on both simple toy examples and a more complicated one, the Starcraft game. The simple toy problems is “Acrobot” and “Mountain Car” with discretized spaces. A hierarchy of level 3 is considered. The experiments clearly demonstrate the effectiveness of the proposed methods. Also, the behavior of each level of the hierarchy is shown: coarse level of actions help exploration but cannot reach the highest value. 

The paper then demonstrates extensive experiments on the game StarCraft. 50-100 units on each side of the game are used — which is much larger than previous papers. Further difficulties are introduced by randomizing the initializing location, and scripted logic controlled opponents. A 2-level hierarchy are tested and obtain consistently good results in all experiments. Further ablations are tested and explanations are provided. 

Evaluation: Overall, I like this paper. This paper is well-written, everything is clearly explained. The proposed method is novel and effective. 

Some minor comments:
	• What is the epsilon in Figure 1? 
	• Using 3-level of hierarchy in the StarCraft game does not work well. You said the possible reason might be that the high level is pushing the limit of CNN. Did you try a different architecture that has a higher resolution?
	• The training curriculum is still mysterious. Why you pick a random l to start training? Maybe training level by level is better? By starting from the lowest level, you may be able to stop at any level. So you will have l different policies. In practice, you may find the best l this way.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Based on the intuition that smaller action space should be easier to learn, the author proposes a curriculum learning approach which learns by gradually growing the action space. An agent using simpler action space can generate experiences to be used by all the agents with larger action spaces. The author presents experiments on simple domains and also on a challenging video game. In general, it is an interesting research work. I think the author can improve the paper in the following aspect. 

1. Motivation. Curriculum learning is based on the idea that tasks can be arranged in a sequential manner and those tasks learned earlier can be somehow helpful for subsequent tasks. Although it is clear that small action space should be easy to learn, it is unclear why those off-policy samples can be helpful for more complex action space. Smaller action space can correspond to a completely different optimal policy. Imagine that in a tabular environment, two actions A, B are available, and the optimal action is to always take B. Then if the agent with full action space uses the experiences generated by the agent with action space {A} may get completely wrong action values. There must be some constraint of the underlying MDP. The author may provide some experiments in tabular case to illuminate the issue.

2. Relevant works. I think the author should include some discussions regarding large action spaces, since one goal of the proposed method is to handle such situation. There are several works should be discussed. For example, Deep Reinforcement Learning in Large Discrete Action Spaces, Function-valued action space for PDE control. The former handle the large discrete action space by learning an action embedding; while the latter attempts to leverage the regularity in the action space by introducing a convolutional structure for the output of the actor network and hence the proposed method can scale to arbitrary action dimensions. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper presents a method of scaling up towards action spaces, that exhibit natural hierarchies (such as a controllable resolution of actions), throughout joint training of Q-functions over these. Authors notice, and exploit a few interesting properties, such as inequalities that emerge when action spaces form strict subsets that lead to nice parametrisation of policies in a differential way. The evaluation is performed in simple toy-ish tasks, and in micro-management problem in 5 scenarios in the game of SC2.

On a fundamental level, proposed method resembles that of Mix & Match, that authors discuss in the paper. In the M&M paper authors use the matching (distillation) of the policies, to ensure knowledge transfer, while in GAS, authors share information through said differential reparametrisation. Ablations provided imply that this part is indeed crucial (as with independent Qs, called "Sep-Q" learning flat-lines). The ablation testing the off-policy modification, seems a bit less conclusive, despite authors claiming that "This ablation performs slightly, or considerably, worse in each scenario. " We can see in Figure 3 that there were 5 experiments:
- in one [95z vs 50m] GAS works much better
- in two [80m v 80m, 80m v 85m] there seems to be no difference (in terms of longer term performance)
- in one [60m v 65m] GAS works slightly better
- in one [50h v 50h] GAS works slightly worse
This mixed bag of results would rather suggest that the offpolicy part is not the main contributing factor, and might require closer investigation to really understand which part of the system proposed brings the benefits. Could this ablation be also done on the toy-ish tasks from experiment 1? Given its simplicity it should be cheap enough to run these extra experiments (I am assuming the SC2 ones are quite expensive?)

Reviewer finds it hard to understand, given current description, how was M&M baseline adapted to the Q-learning setup? Was the distillation loss replaced with L2 one? Was the exact same architecture used for these experiments? How were the missing parent actions handled? Or did M&M experiments use an actor critic learning instead (which would make the comparison more about Q-learning vs Actor-Critic learning, than about methods of action space scaling). Methods section (mentioning entropy loss) looks like M&M was indeed trained with actor-critic, which would make the baseline hardly comparable. Adapting M&M strategy to Q-learning (or picking other baselines, that work on the same RL setup), in reviewer's opinion, is crucial for actual evaluation of author's contributions (given that this is the only baseline).

On a minor point - given how unique SC2 environment (and problem of unit micromanagement) is, would it be possible to provide baselines results also for the toy-ish experiment 1? 

Overall, I recommend a weak rejection of the paper, and I am happy to revisit this evaluation given that authors address the above comments.
review_title: Official Blind Review #1

Meta-review: This paper presents a novel approach to learning in problems which have large action spaces with natural hierarchies. The proposed approach involves learning from a curriculum of increasingly larger action spaces to accelerate learning. The method is demonstrated on both small continuous action domains, as well as a Starcraft domain.

While this is indeed an interesting paper, there were two major concerns expressed by the reviewers. The first concerns the choice of baselines for comparison, and the second involves improving the discussion and intuition for why the hierarchical approach to growing action spaces will not lead to the agent missing viable solutions. The reviewers felt that neither of these were adequately addressed in the rebuttal, and as such it is to be rejected in its current form.
Decision: Reject