Enhancing Language Emergence through Empathy
Authors: Marie Ossenkopf
Venue:  (2020)
Keywords: multi-agent deep reinforcement learning
emergent communication
auxiliary tasks

Abstract: The emergence of language in multi-agent settings is a promising research direction to ground natural language in simulated agents. If AI would be able to understand the meaning of language through its using it, it could also transfer it to other situations flexibly. That is seen as an important step towards achieving general AI. The scope of emergent communication is so far, however, still limited. It is necessary to enhance the learning possibilities for skills associated with communication to increase the emergable complexity. We took an example from human language acquisition and the importance of the empathic connection in this process. We propose an approach to introduce the notion of empathy to multi-agent deep reinforcement learning. We extend existing approaches on referential games with an auxiliary task for the speaker to predict the listener's mind change improving the learning time. Our experiments show the high potential of this architectural element by doubling the learning speed of the test setup. 

Under review as a conference paper at ICLR 2020
ENHANCING LANGUAGE EMERGENCE THROUGH EMPATHY
Anonymous authors Paper under double-blind review
ABSTRACT
The emergence of language in multi-agen agent settings is promising research direc- tion to ground natural language in simulated agents. If Al would be able to under- stand the meaning of language through its using it, it could also transfer it to other situations flexibly. That is seen as an important step towards achieving general AI. The scope of emergent communication is so far, however, still limited. It is neces- sary to enhance the learning possibilities for skills associated with communication increase the emergable complexity. We took an example from human language acquisition and the importance of the empathic connection in this process. We propose an approach to introduce the notion empathy to multi-agent deep re- inforcement learning. We extend existing approaches on referential games with an auxiliary task for the speaker to predict the listener's mind change improving the learning time. Our experiments show the high potential of this architectural element by doubling the learning speed of the test setup.
INTRODUCTION
Natural language is not as rule-based as researchers in supervised language learning would prefer. There are limitless context-dependen notions to it, and flexible language use considered as necessary aspect of general AI. Originally, natural language emerged through necessity to achieve successful coordination. Hence, general AI would need to understand the functional aspects of language and learn communication through interaction (Wittgenstein, 1958; Wagner et al. 2003). These considerations led to the research field of emergent communication and the attempt to ground natural language through reinforcement learning.
Deep reinforcement learning has achieved some impressive results over the last years (Arulkumaran et al. 2017). One of its principal aspects is the ability to extract features from high dimensional input data without manual preprocessing. This capability is especially useful if the necessary repre- sentation is unknown to the designer.
Classical deep reinforcement learning approaches rely on a large number training examples, mainly because the sparse reward hardly provides enough feedback shape the deep layers. These deep layers are responsible for the embedding of input data into a meaningful representation. There- fore, it takes many training steps before useful representation emerges; if converges at all. Ac- cording to the theory of the predictive mind (Hohwyl (013),, the human brain generates richer feed- back through learning several unsupervised prediction tasks while training the main task. The purpose of these predictions is to produce more and more expressive models and representations of the world.
Oh et al. (2015) achieved a far more expressive representation of their visual inputs by learning an auxiliary prediction task. The sole purpose of the auxiliary net is to predict the change in the visual input given the last movement action. Training this net does not directly affect the original task, but refines the visual representation to reflect the concepts of a 3D world. [errmann et al. (217)) used predictive tasks to ground natural language, but only focused on better understanding an existent language. We transfer the auxiliary prediction to the task of active communication. This goes along with the theory of mind (Premack & Woodruff 1978; Schaafsma et al., 2015) stating that an essential part of intelligence in interaction emerges through predicting the mental state the interaction partner.

---

Under review as a conference paper at ICLR 2020
We let the speaker train an auxiliary net that tries to predict how the speaker's utterance will change the listener's s hidden state. That resembles humans empathetic way of understanding what a message will do to the listener. We assume this leads to a more communication effective representation of the sensory input; in other words, the input encoding becomes more communicatable. The effect is visible in the essential acceleration of learning successes in developing shared language.
Our main contribution is an elegant extension to multi-agent deep reinforcement learning (MADRL) algorithms aiming to emerge communication. It resembles an empathic connection between speaker and listener, which leads to faster convergence to a shared language. We doubled the learn- ing speed of MADRL algorithm playing referential game by introducing this auxiliary prediction task to the speaking agent. We attribute the improvement to the richer gradients in the lower layers of the neural network to embed the input.
BACKGROUND
Reinforcement Learning (RL) An agent in a reinforcement learning setting can fully or partially observe its current state s E S and is able to choose an action a E A through policy T(s) = a. The chosen action will lead to receiving a reward R. The agent's 's goal in its environment maximize the expected reward (Sutton et al. 1998).
((0) = E(R(s, a)]
RL with neural networks (NN) Using neural networks as policy representation for reinforcement learning has the benefit of being able to represent any policy function and the downside of needing a huge number of data samples to learn. In our case, the policy outputs a direct probability for taking each action. Such policies can be updated by using Policy Gradient methods (Sutton et al. 2000). The policy parameters 8, in this case, the parameters of the neural net, updated according their effect on the objective J with a learning rate 3:
(J(
A0
80
Using the REINFORCE algorithm (Williams the following:
1992) the effect on the objective can be estimated as
[o = Vo log (als))((ss
Long Short-Term Memory Network (LSTM) Recurrent neural networks (RNN) can accumulate input in internal representation over time well produce consistent output over several time steps from it. LSTMs are RNNS that are specifically created to remember information over an extended period of steps (Hochreiter & Schmidhuber 1997).
Auxiliary tasks in RL Auxiliary unsupervised tasks were introduced into RL with NNs by Oh al. (2015). They proposed an architecture that predicts the next visual input, given the internal representation of the last visual inputs and the last taken action. The unsupervised task of correctly predicting the next visual input leads to better performance on the main task, which was playing an atari game. They assume that the auxiliary task enforces a more expressive internal representation of the visual input, which then aids the main task. Hermann et al. (2017) transferred this auxiliary task to natural language acquisition by predicting the next word spoken by instructor.
3 RELATED WORK
MacLennan 1990) started the field of learning communication in artificial agents with the aim to research the mechanisms by which language and communication emerge. Werner & Dyer (1991) contributed by using classical genetic scenarios, where "male" agents had to find "female" agents based on signals they emitted. They extended their setting in 1993 to include predator and prey agents and showed that known prey strategies herding emerge if the agents have the possibility to communicate Werner & Dyer 1993). Robbins (1994) achieved the emergence of more robust signals by introducing lying agents (parasites) in this setting.
The successive advances in the field of learning communication can be assigned to the progress of learning algorithms for neural networks for big part. Kasai al. (2008) started using Q-learning
2

---

Under review as a conference paper at ICLR 2020
the pursuit problem, including learned communication, but also follow up work only reached simple information sharing about the prey position (Kasai et al. 2009 Noro et al. 2014).
The field then alleviated to level of complexity by Sukhbaatar et al. (2016) and Foerster ct al. (2016). They transferred the new progress deep learning to multi-agent coordination to emerge even more complex communication patterns. In previous work, we have shown that these [blind].
algorithms can be further improved even to solve tasks that lie outside the communication range
Though many multi-agent learning setups use communication as a matter of success, only some focus the emerging protocols and their properties (Hernandez-Leeal al. 2018). From those focusing on the emergence of communication or language, significant number of publications used referential games (Lewis 2008) testbed (Jorge et al. 20166 Lazaridou et 2017 Havrylov &
Titov 2017
Das et al.) 2017; Evtimova et al. 2017 Lazaridou et al. 2018).
Speaker
Listener
lives legs fluffy
cat
car
cat
bird
sofa
Figure 1: Illustration of the referential game used testbed. The speaker agent takes the concept "cat" as input and forms message to describe it. The listener agent takes the message and several candidate concepts as input, and decides which concept is the target seen by the speaker.
Especially interesting in that context is the work of Lazaridou et al. (217)) as they could vividly show, that this approach to language emergence can lead to a flexible language use which could be understood by humans even when applied to objects unknown to the algorithm, yet.
4 CONTRIBUTIONS
We introduce the idea of auxiliary tasks into the field of language emergence. The speaking agent is equipped with an auxiliary single-layer perceptron, predict the hidden state of the listener agent, after this ultimately encoded the message. The input for this prediction is the hidden state of the speaker right before it starts forming the message. The aim is to achieve high relation between the hidden states of both agents. This signifies the speaker can communicate its means well. We state that in this application, the auxiliary prediction resembles empathy humans, as the speaker tries to predict how its utterance will affect the listener's 's mindset.
The prediction task is unsupervised and can be trained on the same samples and at the same time as the main task. Training the main RL task automatically generates the samples for the unsupervised task. The gradients can be backpropagated into the encoding layers of the speaker, where they are added to the gradients of the RL task and optimized together. With our approach, we further enhance the possibilities language emergence by providing richer feedback to form the internal communicatable representation in the speaking agent. We provide experimental evidence that these extensions can lead to a doubled learning speed when added to an existing approach to language emergence in referential games.
3

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS
To test the potential of the auxiliary prediction task, used the referential game setup proposed by Lazaridou et al. (2018) shown in Fig. [ Out of the existing implementations, we chose this one because the setup has proven converge emergent communication relatively low computational cost.
Dataset We the Visual Attributes for Concepts Dataset (VisA) of [Silberer et al. (2013).
contains attribute annotations for 500 concrete concepts (like cat or sofa), annotated with 636 general attributes (like is black or made of ofod).. The annotations are human-made and therefore carry an inherent structure that can be seen disentangled.
Agent Setup A speaker agent gets shown a target concept that realized a binary vector with as many entries as possible attributes. The speaker then uses policy T to produce a message m out of alphahet of discrete symbols (numbers by listener agent that observes several candidate concepts C at the same time. The listener uses pointing policy & decide, which of the candidate concepts the speaker agent is describing. Both agents receive shared reward R. if the listener correctly identifies the described concept.
100 our case). The message is then interpreted
R(()) {
=f =
else
The speaker agent consists of a single encoding layer to encode the input vector into a dense repre- sentation hg and an LSTM to produce a message out of hs. The listener agent encodes the message with an embedding layer and an LSTM into a dense representation hい- The listener contains an encoding layer as well, which it applies to every candidate concept respectively to generate a set of representations. It calculates the compliance between message and candidate concepts with the dot product between the message representation and the concept representation. The result treated as Gibbs Distribution. Both policies Ty and n output a probability distribution over all possible actions. For the speaker, the possible actions are the elements of the alphabet, once for every symbol over the length of the message. For the listener, the actions consist of choosing each of the candidate concepts in
For more details see Lazaridou et al. (2018).
Learning As part of the reinforcement learning setting, the agents try to maximize the expected reward. They do not share any parameters but try to maximize the probability of their action that resulted in a positive reward, respectively. Therefore together, they maximize the objective function in each training instance:
R(C')
Σ logg, s (m/lmn)) <l hhs) + loggez
(5)
Empathy Extension To generate richer gradients for shaping the deep encoding layers of the speaker, we assign auxiliary unsupervised prediction task to it. We add single layer Multi- Layer-Perceptron (MLP) to the graph, which predicts the activation of the listener's hidden layer hy after hearing the full message. The input is the activation of the speaker's hidden layer hy before starting the utterance. That corresponds to predicting the effect of the to-be-made sentence on the mindset of the listener. We use the mean absolute error the loss function for the prediction task:
loss = alo (ho)) - hzl
(6)
where is a weighting factor that ensures that the unsupervised task does not corrupt the main inforcement learning task. An a close to 1 would mean, that effectively manipulating the listener's mind is important to the speaker communicating the target concept. we is the linear transfor- mation through the MLP with sigmoid activation function a. The gradients of the unsupervised task calculated the same trial and added to the gradients of the reinforcement task. Hence, additional training steps are necessary. The optimization then uses the summed up gradient.
We implemented the setup using the EGG toolkit Kharitonov et al.
4

---

Under review as a conference paper at ICLR 2020
Speaker
Listener
Message
Decoder
Encoder
Auxiliary Prediction (unsupervised)
Encoder
Decoder
Input
Output
Figure 2: Integration of the auxiliary prediction task of the speaker into the neural network architec- ture.
RESULTS
We found that with of around 0.1, i.e. weighting the prediction gradients 10% compared the main task, we can increase the learning speed to double or triple. To be comparable, we used the same initialization and sampling seeds both options. Good bad initialization can make up for half the learning speed, but the relative learning speed improvement through using the prediction task stays consistent over different initializations. and without the prediction task. For game setup with two candidate concepts and maximum message length of two, marks are reached in half the time, For more complex game setup with five candidate concepts and maximum message length of five, some marks are even reached in third of the time, when using the prediction task.
Fig. 3 we compared the learning curves with
0.
0.7
0.6
candidates: sentence length:
whtu
candidates: sentence
5
withou
0-0.1
0.4
0-a1
0.3
0.1
0.4
2000 2000 5000
4000 5000 6000 7000 8000 5000 Training steps
10000 11000 12000
3000
6000
9000
12000
15000
18000 21000 24000 27000 30000
Training steps
Figure 3: Learning curve for the net with and without prediction task
7 CONCLUSION
Using an auxiliary predictive task on communication learning task has proven auspicious. Sample- efficiency is highly desirable when acquiring language, the fact that our auxiliary task doubles the learning speed is of high significance. Our experiments do only feature small partition of the potential of this elegant mechanism. yet. Higher sample-efficiency allows acquiring more complicated language tasks, that were previously impossible to learn a reasonable time. We plan apply algorithm much more challenging tasks in the future. We
computational cost
5

---

Under review as a conference paper at ICLR 2020
did, for example, only test disentangled input due to computational limitations. The mechanism would be even more useful when applied on entangled input because developing an expressive rep- resentation is then of higher importance. For future research, prediction task for the listener to align with the word usage of the speaker, as well. We hope that this simple but powerful mechanism brings the field of language emergence big step forward.
propose the use of auxiliary
REFERENCES
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep rein- forcement learning: A brief' survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. Proceedings of the IEEE International Conference on Computer Vision, pp. 2951-2960, 2017.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent language in multi-modal, multi-step referential game. arXiv preprint arXiv: 1705. 10369, 2017.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson, Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to com- municate with sequences of symbols. In Advances in neural information processing systems, pp. 2149-2159, 2017.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint arXiv: 700:::7066.00000 06551, 2017.
Pablo Hernandez-Leal Bilal Kartal, and Matthew E Taylor. Is multiagent deep reinforcement learn- ing the answer or the question? brief survey, arXiv preprint arXiv:1810.05587, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Jakob Hohwy. The predictive mind. Oxford University Press, 2013.
Emilio Jorge, Mikael Kågebäck, Fredrik D Johansson, and Emil Gustavsson. Learning to play guess who? and inventing grounded language as consequence. arXiv preprint arXiv:1611.033: 032218, 2016.
Tatsuya Kasai, Hiroshi Tenmoto, and Akimoto Kamiya. Learning of communication codes multi- agent reinforcement learning problem. In Soft Computing in Industrial Applications, 2008. SM- Cia' '08. IEEE Conference on, pp. 1-6. 2008
Tatsuya Kasai, Hayato Kobayashi, and Ayumi Shinohara. The size of message set needed for the optimal communication policy. In Proceedings of the 7th european workshop multi-agent systems (EUMAS 2009). Paper ID#, pp. 40, 2009.
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage Games. arXiv preprint 1xXiv19977000822
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representa- lions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017. URL net -ttp:/oorrumm210-H0033335....
htttps://ppeneviit
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguis- tic communication from referential games with symbolic and pixel input. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Conference Track Proceedings, 2018. URL https::: /pppreeeeeeieeeeep///ee net forum? 1d=HJGv/ AW

---

Under review as a conference paper at ICLR 2020
David Lewis. Convention: philosophical study. John Wiley & Sons, 2008.
Bruce MacLennan. Synthetic ethology: An approach the study of commumication. University of Tennessee, Computer Science Department, 1990.
Kozue Noro, Hiroshi Tenmoto, and Akimoto Kamiya. Signal learning with messages by reinforce- ment learning multi-agent pursuit problem. Procedia Computer Science, 35:233-240, 2014.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information process- ing systems, pp. 2863-2871, 2015.
David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 144::515-526, 1978.
Phil Robbins. The effect of parasitism on the evolution of a communication protocol an artificial
life simulation. In Proceedings of the third international conference on Simulation of adaptive behavior: from animals to animats from animals to animats 3, pp. 431 431-437, 1994.
Sara M Schaafsma, Donald W Pfaff, Robert P Spunt, and Ralph Adolphs. Deconstructing and reconstructing theory of mind. Trends in cognitive sciences, 19(2):65-72 72, 2015.
Carina Silberer, Vittorio Ferrari, and Mirella Lapata. Models of semantic representation with vi- sual attributes. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 572-582, 2013.
Sainbayar Sukhbaatar, Rob Fergus, tion. In Advances in Neural Information Processing Systems, pp. 2244-22522 2016
Learning multiagent communication with backpropaga-
Richard Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 2. MIT press Cambridge, 1998.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- tion processing systems, pp. 1057-1063, 2000.
Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69 2003.
G. M. Werner and M. G. Dyer. Evolution of communication in artificial systems. Artificial Life II, Addison- Wesley, Redwood City, CA, pp. 659 659, 1991.
Gregory M. Werner and Michael G. Dyer. Evolution of herding behavior in artificial animals. In From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, volume 2, pp. 393, 1993.
Ronald J Williams. Simple statistical followngg algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.
Ludwig Wittgenstein. Philosophical investigations. 1953. Trans. GEM Anscombe. Oxford: Black-
well, 1958.


Reviews:
[{"review_id": "ryeHqlTZFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes the reference-game setup of Lazaridou et al. (2018), as a means of enabling emergent communication, and adds an auxiliary task to demonstrate that this helps with language emergence. The auxiliary task is to enable the speaker to predict the hidden state of the listener, after the message has been received. This is (not unreasonably) likened to providing the speaker with some empathy, in that it enables the speaker to try and predict what the effect of the message will be on the listener.\n\nThe main result is that the learning exhibits a speed-up, arriving at roughly the same level of overall reward but in fewer training steps.\n\nThe idea of adding an \"empathy\" auxiliary task to the reference-game setup is an interesting one, and the approach is well-motivated and described, including a background section. Unfortunately, however, the contributions of the paper are some way off what would be required for a full ICLR paper. Note that the main experimental results section takes up only 1/3 of a page, and the overall paper has only 5 pages of content. (As far as I know there is no requirement for an ICLR paper to take up the whole 8 pages, but a submission with only 5 pages is quite unusual.) So the overall contribution could be summarised as taking an existing emergent-language setup with the same speaker and listener neural architectures; adding a single MLP to the speaker; and showing two graphs of training reward, varying the number of candidates (2 and 5). I hope that the authors can perhaps see that this submission would be better suited to a dedicated workshop on emergent communication (and even then it would need more experiments and analysis).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1e1TaJRYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper starts with a conceptual claim that incorporating a notion of “empathy” in language emergency would help agents learn faster.  The paper then proposes a learning mechanism for implementing this, and looks at its empirical effect for the case of a Speaker-Listener game.\n\nThe concept at the core of the paper is thought-provoking, somewhat grounded in human communication, and it’s interesting to see how this can be translated into a learning mechanism for the multi-agent setting.   The specific implementation proposed seems reasonable at a high level, however there are many technical details missing which really hamper the paper’s message & potential scientific impact.   The results are limited to a single game, with just a pairwise comparison (with and without “empathy”), and provide a narrow view into the effectiveness of the proposed technique.\n\nMy main problem with the paper is the clarity & organization problems.  Usually I tend to be lenient on this, thinking poor writing is much easier to fix than poor science.  But in this case the problems are large enough that the paper is just too far from the standard for ICLR publication.  It also fits in 5 pages, so the authors had lots of space to write a much better paper.   I encourage them to do this for a future submission, in addition to more extensive results, because I think the ideas are worthwhile.\n\nSpecific comments:\n\nDesign of the empathy mechanism. Can you motivate why it’s reasonable to “achieve a high relation between the hidden states of both agents”? Is this necessary / sufficient for empathy?  What are alternate framings of this?  What are properties and pros/cons of this framing?\n\nSec.4 needs a lot more detail!   Sections Agent setup, Learning and Empathy Extension in Sec.5 should be moved to Sec.4, since they describe the method, rather than the experiments.\n\nSec.5 needs better clarity.\no\tWhat are m^l_t and M^{<l}_t in Eqn 5?  Define how h_S and h_L are parameterized, and how each is trained.\no\tFig.2 gives a high-level view of the approach, but lacks important details.  Do you apply a loss at both the Speaker’s Decoder output, and the Listener’s Decoder output?  Or just the latter?  What is the loss specifically? I assume combination of Eqn 5 and Eqn 6, but not sure.\no\tIf you train just on the loss of the Listener’s Decoder, does this mean this is backpropagated all the way to train the Speaker?  How would this be done in a real system?  It’s a very strong assumption to say that the Listener will share gradients with the Speaker.  It seems more realistic to assume they will each observe a loss and train independently.\n\nResults are very brief.\no\tHow robust are the results to the specification of the \\alpha (the loss weight from Eqn 6)?  How much data goes to finding a good \\alpha?\no\tWhat is the difference between the left and the right plot?  Is one for the Speaker and the other for the Listener?\no\tHow do you measure “learning speed”, which is the main metric discussed in the text of Sec.6?\no\tHow do the results change by number of concepts in the game?\no\tDo you do any pre-training of the encoder/decoder networks?\no\tCan you show confidence intervals on each curve?\no\tCan you show test performance?\no\tAre there other related games to consider?\n\nMany references missing throughout to support statements, e.g.\no\t“Natural language is not as rule-based as…” \no\t“These considerations led to the research field of emergent communication…”\no\tSec.2:  Earlier refs to RL in general (e.g. work Sutton in the 1980’s). Earlier refs to RL with neural networks (e.g. work of G. Tesauro; work of M. Riedmiller).\no\tReferential game in Fig.1 caption.\n\nSome minor language issues, e.g.\no\t“The field was then alleviated” -> Do you mean elevated?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkx7J160Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper aims to take insight from human language acquisition and the importance of empathic connection to learn better models for emergent language. The authors propose an approach to introduce the notion of empathy to multi-agent deep RL by extending existing approaches on referential games with an auxiliary task for the speaker to predict the listener’s empathy/mind. Experiments show that this gives some improvement with faster convergence.\n\nStrengths:\n- The concept is interesting and grounded in human communication.\n\nWeaknesses:\n- I like the motivation of predicting empathy, but the paper vastly oversells this part: I don't see how predicting the listener's hidden state is the same as modeling empathy. Empathy is a complex human state/emotion that should not be reduced to this.\n- This paper is very preliminary. There are multiple typos in the paper. Figures are not professionally created. There are multiple training details not included such as how the agents are modeled and trained. The authors seem to have run out of time given the short length of the paper.\n- The experimental results are not convincing at all. The improvement is too small and it would help to run the experiment multiple times to see the improvement with respect to variance in the model. There should also be experiments testing the effect of \\alpha on performance. There is no analysis of how well the model is able to predict empathy, as well as ablation studies testing for various design decisions. The authors should also add an analysis of the learned communication protocols and whether they are different in a meaningful way.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the idea of "empathy" to improve learning in communication emergence. The reviewers all agree that the idea is interesting and well described. However, this paper clearly falls short on delivering the detailed and sufficient experiments and results to demonstrate whether and how the idea works.

I thank the authors for submitting this research to ICLR and encourage following up on the reviewers' comments and suggestions for future submission. 
Decision: Reject