Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles the problem that other agents adapt/change their strategies/behaviors when interacting, resulting in a non-stationary policy. The authors propose to learn a policy that influences the other agents to not change their policy (i.e., to influence the other agents' strategy to stabilize) by penalizing the learning agent with a penalty function of the other agent's distance between the previous and current latent strategies. Hence, the main contributions of this work are:
1 - A stability reward function influencing the ego-agent to learn a policy minimizing the other agent's changes in their policy/behavior
2 - A strategy allowing to learn the stability reward function in an unsupervised manner

Moreover, this work presents simulation results demonstrating the claimed contributions.
strengths_and_weaknesses: In general, this paper is very well-written and tackles an important problem to the research community. The contributions and proposed work's motivation are clearly written, the problem statement is well-defined, and the presented results support the work claims.

The main strengths of this paper are:
1 - The idea of rewarding the learning agent to stabilize the other's strategies when interacting.
2 - The proposed method allowing to learn the stability reward in an unsupervised manner.

The main weaknesses of this work are:
1 - No video displaying the qualitative results of the proposed method. Adding visual qualitative results can improve the significance of this work and can be important to make a good qualitative evaluation of the method. The results presented in the appendix do not a whole episode.
2 - The code used for the experiments is not provided. Unfortunately, it is more and more important to release the code used to generate the presented results and allow others to reproduce presented results to avoid people getting articles with false claims published. Open-sourcing the code used for the proposed method will strengthen the contribution of this work and make future works cite it as a reliable reference.
3 - The experiments are still very simple and far from a real robotic application scenario.
summary_of_recommendation: To summarize, I think this paper tackles an important problem and the proposed method has some novelty. Moreover, the paper is very well-written. However, adding more complex experiments and qualitative results showing the learned policy in action would definitely increase this work contribution. Still, as a conference paper, I think it is in a good state to be accepted.
issues: Clarity issues:
1 - How is the opponent strategy defined? Is z^j a sequence of actions over the rollout j?
2 - In line 145, in the definition of fixed-point strategy, how is \epsilon defined? What are the conditions on \epsilon for the definition to hold?

Notation problems:
1 - Line 157, the V function is not defined. What is V?
2 - Line 145 and182, N is not defined. 

The related works section is missing an important reference and it is important to highlight how this work differs from the following references:
1 - Yang, Jiacheng et al. “Learning to Incentivize Other Learning Agents.” ArXiv abs/2006.06051 (2020)

Possible improvements:
1 - Ablation study and discussion on the impact of beta in the learned policy's performance.
2 -  How is the action space defined for each experiment? In some experiments, it is not clear.
3 - Adding qualitative results in the paper or attach a video clearly showing the learned policy in action.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to encourage agents to stabilize other agents thereby rendering the environment stationary and easier to learn. The underlying approach is to use an added reward function that encourages the agent to stabilize the other agent's strategy.
Experiments in various domains (e.g., driving, sawyer-reach, speaker-listener) suggest that the proposed SILI works well. 

strengths_and_weaknesses: On the whole, the paper proposes a simple yet effective idea for stabilizing interactions between agents. To my knowledge, SILI is novel and the results are relevant for the robotics community, especially those in multi-agent learning and human-robot interaction. The paper is well-written on the whole (up to the points below). Extensive experiments in various domains validate the main claims and shows SILI works well against the baselines and unmodified SAC. 

My only major concern is regarding the learnt latent strategy space $\mathcal{Z}$. The entire approach hinges upon a well-structured latent space, but the paper is vague about how exactly $\mathcal{Z}$ is learnt. Specifically, the authors state the approach is unsupervised but how are the data/trajectories/strategies sampled? It seems crucial that $\mathcal{Z}$ is sufficiently "smooth" and that nearby latent representations correspond to similar strategies; this can be challenging since only noisy trajectories are observed? 

In more complex scenarios, it may be very difficult to learn a stabilizing strategy, especially with humans or sophisticated artificial agents. Again, this is contingent upon a good latent space and the inherent complexity of the other agent. While I think this paper is a step in the right direction, it's difficult to see if it will scale up.  

As a minor comment, what stabilizing strategies were learnt in the various domains? I think qualitative descriptions would really help to increase the impact of the work. In particular, how do they differ to the non-stabilizing strategies? I understand that space is an issue but Fig 4 in the appendix should be in the main paper. 


summary_of_recommendation: I lean towards acceptance. This work nicely executes a first-step towards more effective and stable interactions. I hope the authors can address my question above about the learnt latent strategy space.  

### Post Response 
Thanks for the response about the latent strategy space and for adding the qualitative results. Overall, I remain in favor of accepting this work. 
issues: The major issue for me was about how the latent space $\mathcal{Z}$ is learnt.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a novel reinforcement learning framework for multi-robot coordination. In this framework, the authors use a modified reward function that, in addition to maximizing long-term discounted reward, works to stabilize the other agents' policies to facilitate learning in the ego agent. They show through detailed experiments that this augmented reward function improves the agent's ability to coordinate and maximize the task reward.

strengths_and_weaknesses: The paper is well written and tackles a  relevant problem. The authors give a thorough description of their approach as well as experiments validating their claims. In addition, the authors provide clear discussion sections which summarize the results and discuss limitations.

However, the single figure used to display the results of comparing the proposed method across various environments is not descriptive enough to give a complete understanding of the framework's performance. More information on the results of these experiments would be helpful.
summary_of_recommendation: The paper is well written and tackles a well-studied but
still relevant problem. The authors provide a good
literature review and clearly explain their architecture,
presenting some experiments to support their claim.  I believe it is suitable for CoRL.
issues:  The single figure used to display the results of comparing the proposed method across various environments is not descriptive enough to give a complete understanding of the framework's performance. More information on the results of these experiments would be helpful.
 
Minor comments
1) Real-world on page one should be hyphenated
2) Us and our teammate should be our teammate and us on page 2
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In a two-player game, an ego-agent's reward can be augmented to encourage it to take actions that will stabilize the partner's strategy. The stability of the partner's strategy is defined by the change in a latent parameter that the ego-agent learns via unsupervised learning from the partner's previous trajectories.
strengths_and_weaknesses: Stengths:
The idea of stability as presented in the paper is very clear. The implementation of the stability reward is also very intuitive and straightforward.
The authors provide an extensive review of related work in multi-agent learning. The comparison of stability and surprise is especially interesting. 
The authors provide experimental results on several tasks that demonstrates that the SILI approach exceeds state of the performance of surprise-based algorithms.

Weaknesses & Questions:
The authors formalize a two-player game as a Hidden Parameter MDP, and leverage the assumptions of a static opponent strategy transition function. The application of this work to two-player games is poorly motivated.
I'm worried that this approach would have limited applicability in real human-robot interaction, and especially for competitive or adversarial tasks, in which an opponent may adapt ttheir policy or strategy to disguise their actions and win. 
More discussion is needed about types of tasks in which it is impossible to stabilize the opponent's strategy.

Given strategy transition functions that are fixed, we can instead look at trajectories of N>>1 episodes in sequence. This new task may be considered an MDPs since the opponent's $T^z$ is stationary. One valuable baseline would be to apply SAC or standard RL algorithms to episodes that are composed of 10-100 interactions. 

The authors may want to avoid the use of the term "opponent" since no adversarial tasks are used. Even in the Circle Point Mass task, the "opponent" has a stationary strategy and has no ability to adapt it to avoid capture.

For the Sawyer-Reach task, is the opponent's strategy the choice of goal?
 
The Detour Speaker-Listener task is the most interesting because the speaker's policy changes randomly if the ego-agent fails. What I'm surprised by is the inability of the Oracle to match the performance of SILI. This be due to the representation of the communication strategy provided to the agent. 

Were there any challenges with tuning the relative magnitudes of the rewards, especially for not purely cooperative tasks?
Did the authors face any problems of convergence to sub-optimal policies? Seems like the stability reward could discourage exploration. 
summary_of_recommendation: The authors apply the Hidden Parameter MDP formalism to two-player games, which is an extremely restrictive assumption as compared to the space of all two-player games. But given this problem formulation, the proposed algorithm is compelling and may be useful for partially observable control tasks. 
issues: The authors should add a discussion of the limitation of the Hidden Parameter MDP formalism when applied to multi-agent tasks.

The authors should provide more detail about the dynamics of the opponent strategy for the given tasks. 

The authors should consider avoiding the use of the word opponent, since most of the tasks presented are cooperative.
review_title: null

Meta-review: Overall, this work tackles an important problem in multi-agent domain. The idea proposed in this paper is also interesting together with the promising results in experiments. Reviewers raised consistent comments that this paper is well-written and organized, and the novelty also makes sense. However, reviewers also care about a lot, such as releasing the video results and source codes. The experimental setting is a little bit simplified, which sounds like still far away from the real world scenarios. Authors should also explain more details about the learning procedure.

During the rebuttal period, authors have replied most of questions from reviewers. Most of reviewers think this is a nice work and recommend the acceptance. 
Decision: Accept (Oral)