Entropy Minimization In Emergent Languages

There is a growing interest in studying the languages emerging when neural agents are jointly trained to solve tasks requiring communication through a discrete channel . We investigate here the information-theoretic complexity of such languages , focusing on the basic two-agent , one-exchange setup . We find that , under common training procedures , the emergent languages are subject to an entropy minimization pressure that has also been detected in human language , whereby the mutual information between the communicating agent ’ s inputs and the messages is minimized , within the range afforded by the need for successful communication . This pressure is amplified as we increase communication channel discreteness . Further , we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks . We conclude by discussing the implications of our findings for the study of natural and artificial communication systems . 1 INTRODUCTION . There has recently been much interest in the analysis of the communication systems arising when deep network agents that interact to accomplish a goal are allowed to exchange language-like discrete messages ( Lazaridou et al. , 2016 ; Havrylov & Titov , 2017 ; Choi et al. , 2018 ; Lazaridou et al. , 2018 ) . Understanding the emergent protocol is important if we want to eventually develop agents capable of interacting with each other and with us through language ( Mikolov et al. , 2016 ; Chevalier-Boisvert et al. , 2019 ) . The pursuit might also provide comparative evidence about how core properties of human language itself have evolved ( Kirby , 2002 ; Hurford , 2014 ; Graesser et al. , 2019 ) . While earlier studies reported ways in which deep agent protocols radically depart from human language ( Kottur et al. , 2017 ; Bouchacourt & Baroni , 2018 ; Chaabouni et al. , 2019 ; Lowe et al. , 2019 ) , we show here that emergent communication shares an important property of the latter , namely a tendency towards entropy minimization . Converging evidence indicates that efficiency pressures are at work in language and other biological communication systems ( Ferrer i Cancho et al. , 2013 ; Gibson et al. , 2019 ) . One particular aspect of communicative efficiency , that has been robustly observed across many semantic domains , is a tendency to minimize lexicon entropy , to the extent allowed by the counteracting need for accuracy ( Zaslavsky et al. , 2018 ; 2019 ) . For example , while most languages distinguish grandmothers from grandfathers , very few have separate words for mother- and father-side grandmothers , as the latter distinction would make communication only slightly more accurate at the cost of an increase in lexicon complexity ( Kemp & Regier , 2012 ) . We show here , in two separate games designed to precisely measure such property , that the protocol evolved by interacting deep agents is subject to the same complexity minimization pressure . Entropy minimization in natural language has been connected to the Information Bottleneck principle ( Tishby et al. , 1999 ) . In turn , complexity reduction due to the Information Bottleneck provides a beneficial regularization effect on the learned representations ( Fischer , 2019 ; Alemi et al. , 2016 ; Achille & Soatto , 2018a ; b ) . It is difficult to experimentally verify the presence of such effect in human languages , but we can look for it in our emergent language simulations . We confirm that , when relaxing channel discreteness , the entropy minimization property no longer holds , and the system becomes less robust against overfitting and adversarial noise . 2 GENERAL FRAMEWORK . We establish our results in the context of signaling games ( Lewis , 1969 ) , as introduced to the current language emergence literature by Lazaridou et al . ( 2016 ) and adopted in several later studies ( Havrylov & Titov , 2017 ; Bouchacourt & Baroni , 2018 ; Lazaridou et al. , 2018 ) . There are two agents , Sender and Receiver , provided with individual inputs at the beginning of each episode . Sender sends a single message to Receiver , and Receiver has to perform an action based on its own input and the received message . Importantly , there is no direct supervision on the message protocol . We consider agents that are deterministic functions of their inputs ( after training ) . As an example , consider the task of communicating a n-bit number , sampled uniformly at random from 0 ... 2n − 1 . The full number is shown to Sender , and its k ( 0 ≤ k ≤ n ) least-significant bits are also revealed to Receiver . Receiver has to output the full number , based on the message from Sender and its own input . Would the Sender transmit the entire number through its message ? In this case , the protocol would be “ complex , ” encoding n bits . Alternatively , Sender could only encode the bits that Receiver does not know , and let Receiver fill in the rest by itself . This emergent protocol would be “ simple , ” encoding less information about the number . We find experimentally that , once the agents are successfully trained to jointly solve the task , the emergent protocol minimizes the entropy of the messages or , equivalently in our setup , the mutual information between Sender ’ s input and messages . In other words , the agents consistently approximate the simplest successful protocol ( in the current example , the one transmitting ≈ n− k bits ) . After training , we can connect the entropies of Sender and Receiver inputs is and ir , messages m = S ( is ) , Receiver ’ s output ( the chosen action ) o = R ( m , ir ) , and ground-truth outputs l by using standard inequalities ( Cover & Thomas , 2012 ) : H ( is ) ≥ H ( S ( is ) ) = H ( m ) ≥ H ( m|ir ) ≥ H ( R ( m , ir ) |ir ) = H ( o|ir ) ≈ H ( l|ir ) ( 1 ) ( Note that , since agents are deterministic after training , H ( m ) = I ( is ; m ) . We can then use these quantities interchangeably . ) Our empirical measurements indicate that the entropy of the messages m in the emergent protocol tends to approach the lower bound : H ( m ) → H ( l|ir ) , even if the upper-bound H ( is ) is far . In our experiments , we observe that when the amount of information that Receiver needs is reduced , without changing other parameters , the emergent protocol becomes simpler ( lower entropy ) . In other words , the emergent protocol adapts to minimize the information that passes through it . We will release the code for our experiments upon acceptance . 3 METHODOLOGY . 3.1 GAMES . We study two signaling games . In Guess Number , the agents are trained to recover an integerrepresenting vector with uniform Bernoulli-distributed components . This simple setup gives us full control over the amount of information needed to solve the task . The second game , Image Classification , uses more naturalistic data , as the agents are jointly trained to classify pairs of hand-written MNIST digits ( LeCun et al. , 1998b ) . Guess Number We draw an 8-bit integer z , 0 ≤ z ≤ 255 uniformly at random , by sampling its 8 bits independently from the uniform Bernoulli distribution . All bits are revealed to Sender as a 8-dimensional binary vector is . The last k bits are revealed to Receiver ( 0 ≤ k ≤ 8 ) as its input ir . Sender outputs a single-symbol message m to Receiver . In turn , Receiver outputs a vector o that recovers all the bits of z and should be equal to is . In this game , Sender has a linear layer that maps the input vector is to a hidden representation of size 10 , followed by a leaky ReLU activation . Next is a linear layer followed by a softmax over the vocabulary . Receiver linearly maps both its input ir and the message to 10-dimensional vectors , concatenates them , applies a fully connected layer with output size 20 , followed by a leaky ReLU . Finally , another linear layer and a sigmoid nonlinearity are applied . When training with REINFORCE and the Stochastic Computation graph approach ( see Section 3.2 ) , we increase the hidden layer sizes threefold , as this leads to more robust convergence . Image Classification In this game , the agents are jointly trained to classify 28x56 images of two MNIST digits , stacked side-by-side ( more details in Appendix ) . Unlike Guess Number , Receiver has no side input . Instead , we control the informational complexity of Receiver ’ s task by controlling the size of its output space , i.e. , the number of labels we assign to the images . To do so , we group all two-digit sequences 00 .. 99 into Nl ∈ { 2 , 4 , 10 , 20 , 25 , 50 , 100 } equally-sized classes . In Sender , input images are embedded a LeNet-1 instance ( LeCun et al. , 1990 ) into 400-dimensional vectors . These embedded vectors are passed to a fully connected layer , followed by a softmax selecting a vocabulary symbol . Receiver embeds the received messages into 400-dimensional vectors , passed to a fully connected layer with a softmax activation returning the class probabilities . We report hyperparameter grids in Appendix . In the following experiments , we fix vocabulary to 1024 symbols ( experiments with other vocabulary sizes , multi-symbol messages , and larger architectures are reported in Appendix ) . No parts of the agents are pre-trained or shared . The optimized loss depends on the gradient estimation method used ( see Section 3.2 ) . We denote it L ( o , l ) , and it is a function of Receiver ’ s output o and the ground-truth output l. When training in Guess Number with REINFORCE , we use a 0/1 loss : the agents get 0 only if all bits of z were correctly recovered . When training with Gumbel-Softmax relaxation or the Stochastic Computation Graph approach , we use binary cross-entropy ( Guess Number ) and negative log-likelihood ( Image Classification ) . 3.2 TRAINING WITH DISCRETE CHANNEL . Training to communicate with discrete messages is non-trivial , as we can not back-propagate through the messages . Current language emergence work mostly uses Gumbel-Softmax relaxation ( e.g . ( Havrylov & Titov , 2017 ) ) or REINFORCE ( e.g . ( Lazaridou et al. , 2016 ) ) to get gradient estimates . We also explore the Stochastic Computation Graph optimization approach . We plug the obtained gradient estimates into the Adam optimizer ( Kingma & Ba , 2014 ) . Gumbel-Softmax relaxation Samples from the Gumbel-Softmax ( Maddison et al. , 2016 ; Jang et al. , 2016 ) distribution ( a ) are reperameterizable , hence allow gradient-based training , and ( b ) approximate samples from the corresponding Categorical distribution . To get a sample that approximates an n-dimensional Categorical distribution with probabilities pi , we draw n i.i.d . samples gi from Gumbel ( 0,1 ) and use them to calculate a vector y with components : yi = exp [ ( gi + log pi ) /τ ] ∑ j exp [ ( gj + log pj ) /τ ] ( 2 ) where τ is the temperature hyperparameter . As τ tends to 0 , the samples y get closer to one-hot samples ; as τ → +∞ , the components yi become uniform . During training , we use these relaxed samples as messages from Sender , making the entire Sender/Receiver setup differentiable . REINFORCE by Williams ( 1992 ) is a standard reinforcement learning algorithm . In our setup , it estimates the gradient of the expectation of the loss L ( o , l ) w.r.t . the parameter vector θ as follows : Eis , irEm∼S ( is ) , o∼R ( m , ir ) [ ( L ( o ; l ) − b ) ∇θ logPθ ( m , o ) ] ( 3 ) The expectations are estimated by samplingm from Sender and , after that , sampling o from Receiver . We use the running mean baseline b ( Greensmith et al. , 2004 ; Williams , 1992 ) as a control variate . We adopt the common trick to add an entropy regularization term ( Williams & Peng , 1991 ; Mnih et al. , 2016 ) that favors higher entropy . We impose entropy regularization on the outputs of the agents with coefficients λs ( Sender ) and λr ( Receiver ) . Stochastic Computation Graph In our setup , the gradient estimate approach of Schulman et al . ( 2015 ) reduces to computing the gradient of the following surrogate function : Eis , irEm∼S ( is ) [ L ( o ; l ) + stop_gradient ( L ( o ; l ) − b ) logPθ ( m ) ] ( 4 ) Here , we do not sample Receiver actions : Its parameter gradients are obtained with standard backpropagation ( the first term in Eq . 4 ) . Sender ’ s messages are sampled , and its gradient are calculated akin to REINFORCE ( the second term in Eq . 4 ) . As in REINFORCE , we apply entropy-favoring regularization on Sender ’ s output ( with coefficient λs ) and use the mean baseline b . Role of entropy regularization As we mentioned above , when training with REINFORCE and Stochastic Computation Graph , we include a ( standard ) entropy regularization term in the loss which explicitly maximizes entropy of Sender ’ s output . Clearly , this term is at odds with the entropy minimization effect we observe . In our experiments , we found that high values of λs prevent communication success ; on the other hand , small non-zero λs is crucial for successful training . In Section 4 we investigate the effect of λs on entropy minimization .

Summary: This paper investigates the phenomenon of entropy minimization in emergent languages. The paper studies two simple speaker-listener prediction tasks where the amount of information that’s needed to be encoded in the speaker’s message can be manipulated. They find, in both games and across training methods, that speaker agents generally learn to minimize the amount of information conveyed in their messages (i.e. minimize the entropy), such that they are still able to solve the task. The paper then conducts some experiments measuring the robustness to overfitting, and find that more discrete channels leads to better generalization when training on partially shuffled labels.