Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces an approach for learning policies across multiple MDPs and using those policies to improve learning performance on the task that the agent designer cares about. The approach assumes that a set of MDPs are provided to the learning agent, and that all of the MDPs have the same underlying task but with different reward densities (i.e., some of these MDPs have shaped rewards, and thus are faster to learn from). The approach operates by training the main agent to imitate the actions chosen by the other agents that are trained on the MDPs with shaped reward functions.

Overall, the approach is interesting and can be applicable to multi-task learning benchmarks, even though in its current presentation the authors do not focus on those settings.

Pros:
The paper is well-written.
The presentation of the idea is clear.
The experiment section and the results are easy to understand.

Cons:
It seems like the overall contribution seems small, as the approach assumes access to many MDPs with different reward densities. Given such an assumption, it is natural to understand why the current approach works. 
The scope of the current work is limited to RTS domains. 
A concise description of the algorithm seems to be missing, making it difficult to understand the overall algorithm.

Questions: 
1.The approach relies on having access to many MDPs. It would be useful to describe the MDPs that the authors have considered for their experiments?
2. It seems like the different policies can be different only if their corresponding reward functions are different. Is that right? If so, in many domains, the approach relies on careful design of different MDPs to get the approach to work. The authors need to discuss these assumptions made by their approach.
3. From the experiments, it seems like the baseline agent with shaped rewards produces the same asymptotic performance as that of the agent with action guidance. If so, then the authors need to justify the use of additional computation for learning the many policies for action guidance.

review_title: Interesting idea. Needs to discuss the many assumptions made by the approach.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces an approach called action guidance, made to address issues in more standard applications of reward shaping. The main idea of their approach is that there are two different kinds of agents, one (auxiliary agents) that learn from shaped reward functions alone and the other (main agent(s)) that learn only from the actual sparse rewards. The authors made use of a simplified RTS domain and demonstrated that their approach outperformed a more naive shaped reward approach. In addition they demonstrated an ablation study on positive learning optimization. 

The basic idea of this paper is simple and elegant, and the paper is well-written. The results are somewhat messy, but overall represent a positive signal for this approach. The paper is also well-written and the video examples are effective at conveying the results. 

I have a number of concerns with this current paper draft. First, it’s unclear to me why the authors chose this RTS environment or why the authors didn’t try several different environments to show the generality of this approach. Further, the evaluation results seem somewhat inconsistent. The action guidance agents overall do quite well but the shaped reward baseline has comparable performance on the first two tasks. This further leads me to believe that this may not be the best environment to test this approach. It’s also somewhat disappointing that the PLO results are inconclusive, this indicates that the work is perhaps still somewhat immature. 

Overall I lean slightly towards acceptance. I think that the basic idea here is potentially very impactful. However, my concerns listed above hold me back from stronger support for the paper in its current state. 

Questions for the authors: 
1. Why did the authors choose this RTS environment?
2. How did the authors account for the performance of the shaped reward baseline for the first two tasks?
3. Do the authors not have a clearer sense of why the PLO results were inconclusive? Or what experiments could be run to delve into this further?

review_title: Interesting approach but somewhat immature

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposed an algorithm called action guidance that trains the agent to eventually optimize over sparse rewards while maintaining most of the sample efficiency that comes with reward shaping. The authors examine three sparse reward tasks with a range of difficulties to prove the effectiveness of action guidance.  However, I don't think this paper is fully prepared for submission as the method is not novel enough and there exist some possible issues that need to be discussed and resolved. 

Below are the detail comments. 

About the method. The key idea behind action guidance is to create a main agent that trains on the sparse rewards, and creating some auxiliary agents that are trained on shaped rewards. And the main agent follows the instructions of auxiliary agents in the initial stage and the probability of it decreases during the following training. A concern is that if there exist several auxiliary agents, how do you arrange the shaped rewards to each auxiliary agent? If there is a conflict between the shaped rewards for the training and guidance of the agent, will the main agent still be trained well? Besides，the method itself is like using imitation learning to obtain initial policy parameters and continues to optimize using sparse reward, the novelty of the method is not sufficient enough.

About the experiments. The baselines use PPO to train agents with sparse rewards or shaped rewards respectively and there are no other SOTA methods designed for sparse rewards compared in the experiments, which is not convinced. Besides, in the environment ProduceCombatUnits, the shaped rewards include the reward for each combat unit the agent produces, which is exactly the sparse reward. Is it means that the agent using shaped rewards has the same optimization direction as the one using sparse rewards? I'm not sure if this is fair enough as the effectiveness of action guidance is not clear in this setting. Lastly, the random opponents in the experiments are not strong, I'm wondering about the agent's performance in a harder setting.

About the writing. The paper is well-written and self-contained. However, the figures to show the typical learned behavior of agents are not clear enough. For example, it's a little bit hard to recognize the enemy units as the blue borders are too thin.

Overall, I vote for a  rejection. 



review_title: This work proposed an algorithm called action guidance to solve the sparse reward problem. However, I don't think this paper is fully prepared for submission as the method is not novel enough and there exist some possible issues that need to be discussed and resolved. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: -----------------------------------------
Post-Rebuttal
-----------------------------------------
I believe both the strong points and the weaknesses I pointed out in my review remain valid. Therefore I am keeping my score unchanged.
In case of acceptance, I suggest to the authors to carefully address all the comments from my review and the other ones in the camera-ready version (especially the issues related to clarity.).

----------------------------------------
The authors propose a reward shaping method to learn faster a reinforcement learning task. Their method consists of learning multiple policies, one optimized on the real reward, while each of the other policies is optimized on a different reward shaping function.

---------------------
# Pros

- relevant research topic
- paper is well-written and generally easy to understand
- novel approach - I can't remember another approach that learns multiple policies from multiple shaping rewards and balance between them
- Experimental results seem good
- Codification freely available

--------------------------------------
# Cons

- Could have included more related works in the experimental evaluation


----------------------------
# Further comments

- I wouldn't say that you train multiple "agents", one from each reward shaping function. A more appropriate name would be saying that you train multiple "policies", each of them maximizing each of the "reward signals". Then, the connection with Probabilistic Policy Reuse would become more obvious (btw, you should discuss the relation of your work with it)

Fernández, Fernando, and Manuela Veloso. "Probabilistic policy reuse in a reinforcement learning agent." Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems. 2006.

Usually, when you are trying another "agent" you are implying that this agent will explore by itself, which is not true for your method. You should read more about that (and discuss the relation with it) in the survey below:

Silva, Felipe Leno, and Anna Helena Reali Costa. "A survey on transfer learning for multiagent reinforcement learning systems." Journal of Artificial Intelligence Research 64 (2019): 645-703.

- Section 1 could be improved by more explicitly discussing the difference between the current paper and each category of related works. In special, I missed a more comprehensive discussion of reward shaping approaches and the inclusion of state-of-the-art reward shaping algorithms in the experimental evaluation.

- Why did you include a graph in Figure 2 in which "shaped reward" performed better than all others? You should perhaps also included the number of time steps used for each approach to converge to the best performance because sample complexity is as important as asymptotic performance.

- It is not very clear to me exactly how the agent selects which policy to follow at every step. Maybe it would be better to have an algorithm in the manuscript.
review_title: Solid paper, with some points to improve.

Meta-review: Summary:
This paper proposes an interesting idea where additional auxiliary tasks allow an agent to more quickly learn in a sparse reward task.

Comments: 
* The authors may want to look at "Parallel Multi-Environment Shaping Algorithm for Complex Multi-step Task"
https://www.sciencedirect.com/science/article/abs/pii/S092523122030655X
as it has a somewhat related idea and is also in RTS games.
* It wasn't clear to me how the auxiliary tasks were generated/selected, or how the algo would work if a poor auxiliary task was used.
* I wasn't sure why SAC-X wasn't empirically compared to in a domain where it and this method could both apply.

Discussion:
The reviewers agreed that this paper could be significantly improved in multiple dimensions.

Recommendation:
I recommend we reject this paper. However, I encourage the authors to work to improve it as I'd really like to understand where and why this method is successful --- I would eventually like to incorporate it into my own work.
Decision: Reject