Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for developing altruistic agents in a multi-agent RL (MARL) setting. The core idea is that an altruistic agent, in the absence of any further reward or goal information, may try to increase the “choices” for the agent it is cooperating with as a proxy. The paper argues that this is a suitable proxy for the unknown true reward function in many environments, because optimal policies tend to choose actions which lead to greater choice (larger coverage of state visitation) in the future, using analysis of instrumental convergence. The method is evaluated on discrete environments where the altruistic agent has to help open a gate, a level-based foraging environment, and a continuous state space tag environment, and the paper shows that the method can lead to altruistic behavior and improved rewards obtained by the leader agent in these settings.

main_review: Strong points of the paper:

The main idea is quite conceptually simple and an interesting approach to develop altruism for MARL.

The experimental section of the paper is well executed, and the analysis of the results is thorough. Particularly the choice estimate analysis for each environment is helpful for understanding why the method may help. The analysis of the failure case is also quite insightful.
 
Weak points of the paper:

The exposition in Section 3.2 about the theoretical support to the objective is rather nonspecific -- I feel that it would be helpful to introduce more specific, technical claims which follow from the arguments in Turner (2019)

While the results on both the discrete gate and the continuous environment are strong, they seem less surprising because they are both navigation-type environments where the value of a state should correlate well with the “choice” afforded by being at that state. The analysis in the results of the hide-and-seek environment mention that it may have outperformed the supervised baseline because it provided a denser reward than the sparse supervised “catching” signal. So, these are environments where it’s unsurprising that the choice heuristic would work well. Because the contribution of this paper is quite dependent on the empirical performance of the method, I think that it needs to be evaluated on additional settings where it is less clear that “choice” is directly correlated with the reward, to be convincing as a generally useful metric when the true environment reward is unavailable.
 
Questions:

In Figure 3, for the LBF environment, I can see how the leader agent should have low IC when waiting at the apple at the top and lower IC when it can choose either of the two apples at the bottom. However, to me this seems to contradict the point that IC is a good proxy for the leader agent receiving high rewards, because the altruistic agent needs to help the leader to harvest the apple regardless, and the altruistic agent is closer to the apple at the top?

What do the scores in Table 1 indicate? It is not described in the caption or in the text. Why is the score for LBF in percentages but it is not for Tag?

summary_of_the_review: The idea of this paper is conceptually simple and I quite like its application to this challenging setting of altruism without true rewards, as well as the analysis that the paper presents. However, mostly because of the choice of environments in the empirical evaluation, I feel that the paper hasn’t provided enough evidence for the claim of the applicability of this altruistic objective, and I think this is a borderline case.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unsupervised learning method for training an agent to assist another agent (called the leader) in solve its task (thus displaying a type of altruistic behavior) without access to the other agent's reward function or policy. The authors propose to use the notion of maximizing the leader's choice which is formalized as maximizing the number of different states the leader can reach at any point (within a given number of steps from its current state). The authors propose three variants of this method and evaluate them on three different domains, while also comparing them with other approaches. 
main_review: ## Strengths

1) I think the paper aims to tackle an important problem, namely that of inducing more altruistic behavior in artificial agents acting in the same environment with others and assisting others in achieving their goals without knowing what those are. 

2) I also like the proposed approach because it seems quite general and doesn't assume access to the leader's reward function, state goal, policy, or trajectories (unlike other methods in this space). 

3) The method is new as far as I know, although some of its elements have been used in other contexts, but I think the authors do a good job at explaining the connections to related work. 

4) I also found the analysis of the different choice formulations from Figure 1 to be quite insightful. 


## Weaknesses

### Baselines 
1) In the paper, you write that for the AvE baseline, you used the hyperparameters suggested by the authors, but this doesn't seem fair since typically these methods need to be fine-tuned for each task / domain as it may require very different hyperparameters than the ones used in the original paper. Could you do a hyperparameter search for AvE and present the results with the best HPs found on the tasks used for evaluation?

2) It wasn't very clear to me why you are not using the AvE baseline for the LBF and Tag domains and also not using the Supervised baseline for the Gridworld tasks. Could you please add these for completion or explain in more detail why they are not used for comparison?

3) I think it would be useful to compare the methods with an oracle which would be the optimal policy for assisting the leader agent. This could provide insight into how far the current method is from such a policy and whether there is still potential for improvement on these tasks by future work. 


### Clarity

1) One of my biggest concerns is that it seems like there might be some significant modes and implicit assumptions made by the method which are not openly discussed in the paper. First of all, it seems like the method assumes that the leader can solve the task during its training stage, while the altruistic agent is taking random actions. This implies that the altruistic agent doesn't need to learn a very complex behavior and is not necessary for the leader's success (even if it might help the leader achieve the goal sooner). A more realistic setting would be one in which the two are trained at the same time or at least, there are multiple training stages for both of them (that alternate). 

Aren't there cases where the altruistic agent can hurt the leader's performance? For example, there might be dead end states in the environment which can be activate by the altruistic agent's actions but which would be better avoided by the leader. Given that the two agents are not training at the same time, the leader's policy may not be robust to such changes in the environment / new states, so they may end up in them if the altruistic agent aims to increase the number of reachable states. Would it be possible to bias the set of states enabled by the altruistic agent towards the set of states which are desirable for the leader to reach?

Could the authors confirm whether this understanding is correct and explain why they decided to not tackle this setting / train in this way? I think these issues should at least be discussed in more depth in the paper. It would also be great if the authors can train the method on a similar scenario (where the approach isn't necessarily expected to do well) to better understand the limitations of this approach and when it can be expected to be effective.

2) Another thing which is not openly discussed in the paper is the fact that it seems like your approach may actually require more training since you have two separate training stages (i.e. first you train the leader on the task and then you train the altruistic agent to assist the leader). Can you comment on the trade-off between final performance and learning efficiency and make this fact more transparent in the paper? It would be great to include a graph with performance as a function of the number of samples used for training for the entire training process, with a breakdown for the leader and altruistic agent.

3) At the beginning of the paper, it is not very clear what are the metrics you are looking to improve, so I suggest mentioning that in the introduction. Initially, it is not clear whether the altruistic agent is supposed to help with a) percentage of times the leader can solve the goal, b) number of steps needed to solve the goal, c) leader's sample / computational efficiency, or something else.  


### Limitations

1) Could you extend this algorithm to non-deterministic environments? Would you just need to replace the unsupervised learning objective with empowerement?

2) Along similar lines, is it possible to extend the algorithm so that the two agents learn at the same time? This would involve dealing with the non-stationarity of the leader's policy. This seems like it would be a more general setting and might be in a better position to handle more challenging tasks, so it would be great to at least discuss it in the conclusion section. 
summary_of_the_review: I think the paper tackles an important and rather neglected problem, proposes a novel and quite general method for this setting, and demonstrate significant empirical gains on multiple tasks. 

My main concern is that some of the potential failure modes of this approach are not discussed in great detail. Because the method is so general, there are many settings where it may not be the most effective approach (which, from my perspective, is not a problem in regards to publishing the paper as long as this point is clear to the readers). I would really like to see an experiment (or more) that aims to tackle a case where this method isn't expected to shine to 1) either openly illustrate the method's failure modes and where it shouldn't be used or perhaps 2) show that it still works decently well even in such settings. Another big concern is the comparisons with other baselines, including the tuning of AvE, and the lack of comparisons with AvE and the Supervised baseline for some of the tasks.

In conclusion, I am leaning towards rejecting this paper, but I am willing to increase my score if the authors can successfully answer the questions above and address my concerns (especially the ones mentioned in the above paragraph).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper posits a method for agents that learns altruistic behaviour from observations and interactions with other agents without knowledge of their objective fucntion by estimating the sub-goals of the agent and actively maximizing their choice.

The authors define 3 different task agnostic method to estimate the choices made by the agents. 

The experimental evaluation is extensive with testing the agents in 3 different environments while reporting the results in comparison with three baselines - Discrete Choice, Entropic Choice, and Assistance via Empowerment.

main_review: The paper is technically sound but very hard to follow through. For instance, I found it very hard to find which baselines they had implemented until I referred to appendix in table 3. 

I still don't understand what does "choice" of the agent mean and how it is different from sub-goals of the agent. I read it multiple times but its very confusing. 

For the model free estimation of choice from observations, it is unclear how the altruistic agent estimates the leader agent’s policy entropy from observations. Can you provide the details such as the policy feature vector? FC layer? and a softmax layer? Additionally, what was the loss function used. 

In the Unsupervised baseline (MaxEnt) implementation, it would be helpful to provide the entropy index since you are trying to maximize this entropy value over the state distribution of trajectory rollouts.

summary_of_the_review: The authors are trying too many different(existing) methods to prove the central concept around how one agent can be altruistic to another by maximizing their reward. The paper is very hard to follow through. Although, the experiments are extensive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides an alternative to empowerment for goal agnostic assistance RL policy training -- choice-based optimization. The core idea is to model possible subgoals as the choices of others and train policy to maximize the choices of other agents for achieving goal agnostic assistance. Evaluation against standard deep RL and the recent assistance via empowerment (AvE) method shows that the choice-based approach can be effective in multiple tasks.
main_review: =====Strengths=====

1. It is an interesting and novel way to train goal agnostic helping policy by maximizing the choice of other agents. The choice-based modeling is clearly motivated and defined, well situated in prior work, sufficiently compared against the similar yet different framework, empowerment. 

2. The writing is very clear and the core ideas are conveyed successfully.

3. The experiments and discussions are thorough and informative. The results are promising.

4. The implementation was well documented and the details seem to be sufficient for reproducing the results.

=====Weaknesses=====

While the overall writing is clear and the experiments are comprehensive, I do however still have confusion/concerns about the comparison between this approach and the prior work, specifically i) empowerment and ii) MADDPG-like approach that learns other agents' policies. It seems to me that choice is similar to empowerment, with the addition of using agent-specific policy instead of arbitrary probing policy. If I understand this correctly, this means that empowerment only depends on the environment and is agent agnostic, while choice is agent dependent -- the definition of choice depends on a specific agent's policy. If this understanding is correct, then I have two questions:

Q1: Can you train a policy using a MADDPG-like method that jointly learns the policy of other agents and the policy of its own? Crucially, this is closer to your approach than the vanilla single agent DRL is, since it also explicitly depends on the policies of others.

Q2: Empowerment is agent agnostic, so in theory, it generalizes better to other situations where agents will have different policies than the ones seen during training. Could you comment on the generalization?

summary_of_the_review: I enjoyed reading this paper and I think the core idea is a valuable contribution to the multi-agent RL research. My main concern is about its comparison with empowerment and other related multi-agent RL work, which is detailed in my main review. I would appreciate some clarification from the authors and would be willing to increase my rating if my concern is addressed in the rebuttal and the revision.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is the first try to train agent behave altruistically towards others without knowledge of their objective or any external supervision. 
The main idea is to train altruist agent giving the leader agents more choice and thereby allowing them to better achieve their goals.  They  introduce three multi-agent environments of increasing complexity to evaluate the proposed method. The results show that it can, in some cases, outperform the supervised baselines.
 
main_review: I am not expert in the specified feild, but in my opnion:
1. The writing of this paper is generally well-organized and of good quality. 
2. This paper, as far as I know, is the first to try to address this important problem.
3 This paper can serve as a baseline and also propose three testbeds for future research. 
summary_of_the_review: A novelty work, worth accept.
review_title: null

Meta-review: This work proposed a method for encouraging an agent showing altruistic behaviour towards another agent (leader) without having access to the leader's reward function. The basic idea is based on the hypothesis that having the ability to reach many future states (i.e., called choice) is useful for the leader agent, no matter what it reward function is. The altruistic agent learns a policy that maximizes the choice of the leader agent. The paper defines three notions of choice, and evaluates them on four environments.

The reviewers believe that this work attempts to solve an important problem, proposes a novel approach, and performs reasonably good experiments. The reviewers are all on the positive side at the end of the discussion phase. Therefore, I recommend acceptance of the paper. I also suggest a spotlight presentation for this work because of the novelty of the problem, which might be of interest to other researchers.

The authors have already done some revisions to their paper (including adding a new environment). I encourage them to consider any remaining comments from reviewers in their final version.
Decision: Accept (Spotlight)