Shaped Rewards Bias Emergent Language

1 INTRODUCTION . In emergent language research , the goal is to study language as it emerges from the inherent properties of the environment , language , and agents . One pitfall for such experiments , though , is that the language simply mirrors some of design choices of the environment or experimental setting more generally . there is a risk that the language simply mirrors the design choices of the environment . For example , Bullard et al . ( 2021 ) introduce a method for discovering optimal languages for communication between independently trained agents , yet rather than emerging from basic principles , the learned language is the result of an intentionally designed search algorithm . Reinforcement learning is a common tool in this field for observing the emergence of language out of a reward maximization pressure . One such design choice which can obscure these emergent properties is adding shaped rewards on top of the base reward of the reinforcement learning environment ( Wiewiora , 2010 ) . The base reward of the environment derives directly from succeeding at the task in question . The difficulty with relying solely on the base rewards is that if the task is especially long or complicated , the agent may only receive a reward infrequently which makes for a difficult learning problem . In such a case , base reward is considered sparse . This motivates shaped rewards which are inserted at intermediate steps based on domain knowledge in order to introduce an inductive bias towards good solutions . For example , the base reward in chess would simply be winning or losing the game . A shaped reward could then be taking the opponents material while not losing your own . While this shaped reward is often a good heuristic , it can lead to local optima ; for example , it discourages strategies which would sacrifice individual pieces in order to win the whole game . While local optima present a problem for maximizing reward , the biases introduced by shaped rewards present a unique problem for emergent language which we will highlight in this paper . For emergent language research , the inductive bias which shaped rewards introduce is especially problematic because it exerts a significant influence on the learned language whose emergent properties are the object of study . This influence can comprise 1 ) biasing the semantics of the language , 2 ) changing a property of the whole language ( e.g. , language entropy ) , or 3 ) masking the influence of some other environmental parameter on the language . For example , some emergent language work incorporates shaped rewards into their environment without accounting for the the biases it may introduce ( Mordatch & Abbeel , 2018 ; Brandizzi et al. , 2021 ; Harding Graesser et al. , 2019 ) . From an engineering and design perspective , tweaking the system to achieve a desired result is standard practice , but from a scientific and experimental perspective , these additional shaped rewards serve as potential confounding factors and hinder accurate observations of the emergent phenomena . We study this by introducing a simple navigation game with continuous states and actions where a sender passes a one-word message to the receiver ( illustrated in Figure 1b and 1a ) . Within this environment , our experiments look at the entropy and semantics of the emergent language in the absence and presence of shaped rewards . In the course of these experiments , we find that reinforcement learning algorithm ’ s experience buffer size has a significant impact on the entropy of the learned language and potentially explains our experimental findings . To this end we introduce a mathematical model based on the Chinese restaurant process for understanding the effect of experience buffer size on emergent language more generally . We highlight the following contributions in our paper : • Demonstrating basic ways in which shaped rewards can undesirably influence emergent language experiments • Presenting a mathematical model for understanding the role of experience buffer size in the entropy of emergent language 2 RELATED WORK . 2.1 TAXONOMY . We intentionally design our environment to study shaped rewards and the entropy of language , which requires it to differ from prior art in specific ways . To elucidate this , we create a taxonomy of emergent language environments based on whether or not the environment has multi-step episodes and the presence of a trivially optimal language ( defined below ) . The taxonomy is given in Table 1 and a brief description of each environment is given in Appendix A . Generally speaking , the motivation for shaped rewards in a given environment is sparsity of the base reward which requires a multi-step , multi-utterance environment . Thus , our experiments naturally require a multi-step environment . We consider an environment to have a trivially optimal language if the information which needs to be communicated from sender to receiver can be perfectly represented in the emergent language . Such a language most frequently arises when the communicated information is derived from a small number of categorical variables encoded in the observation . For example , in an environment where the sender must specify an element of the set { red , green , blue } × { square , circle } using messages from the set { r , g , b } × { s , c } , a trivial language is where color maps to a unique letter in the first position and shape maps to a unique letter in the second position . Other environments have no trivially optimal languages . For example , if the sender must communicate an element of { 1 , 2 , ... , 100 } using messages from the set { a , b , c } , there is no trivially optimally language since the sender can at best partition the set of integers to reach an optimal but imperfect solution . Kharitonov et al . ( 2020 ) gives evidence that there is an entropy minimization in pressure inherent in emergent language settings . Building on this , Chaabouni et al . ( 2021 ) explicitly look at a the tradeoff between entropy and reward—higher entropy languages have the potential to convey more information at the expense of being more difficult to learn . This tradeoff disappears if a trivially optimal language is learned since there is no further reward maximization pressure . Language entropy greater than this minimum , then , does not emerge from the the reward maximization pressure . Although such an environment does not preclude studying entropy we choose to use an environment where the information being communicated is fully continuous so an increase in entropy can always translate to a increase in reward . This leads to a smooth tradeoff between entropy and reward which is illustrated in Figure 2 . 2.2 SPECIFIC APPROACHES . The environment and agent configuration of this paper are most closely related to Chaabouni et al . ( 2021 ) who test the balance between entropy and semantic precision in a two-agent color discrimination task . Although the color space comprises 330 distinct colors , the environment facilitates languages which cover a ( learned ) region of nearby colors . In this way , there is no trivially optimal , one-to-one language as the task is inherently fuzzy . Havrylov & Titov ( 2017 ) use a signalling game with natural images which lacks a trivially optimal language , but using natural images results in many uncontrolled variables , lessening the ability of the experiments to make basic , first-principles claims . Studying shaped rewards has been explored previously in emergent language primarily as inductive bias for encouraging some property , such as compositionality , to emerge ( Hazra et al. , 2021 ; Jaques et al. , 2019 ; Eccles et al. , 2019 ) . This paper , instead , focuses on the negative aspects of inductive biases introduced by shaped rewards regarding how they can hinder empirical investigation of emergent languages . Superficially , the environment used in this paper bears resemblance to Mordatch & Abbeel ( 2018 ) and Kajić et al . ( 2020 ) as these both deal with navigation . In both of these environments , the agents communicate about a discrete set of actions or goal locations , whereas the agents in this paper communicate about a continuous action . Neither of the papers specifically investigates the effects of shaped rewards on the languages learned . 3 METHODS . 3.1 ENVIRONMENT . In this paper we use a simple 2-dimensional navigation environment with two closely related tasks . A sender agent observes the position of a receiver agent , sends a message to the receiver , and the receiver takes an action . In the centerward task ( illustrated in Figure 1a ) , the receiver is initialized uniformly at random within a circle and must navigate towards a circular goal region at the center . In the edgeward task , the receiver is initialized uniformly at random within a circle and must navigate to a goal region comprising the entire area outside of the circle . The centerward environment is a more realistic environment . The edgeward environment , on the other hand , admits a greater variety of languages , since moving consistently in any one direction eventually gets you to the edge ; therefore , learning to move in a variety of evenly spaced direction solves the task more quickly but is not strictly necessary . There are no obstacles or walls in the environment . The receiver ’ s location and action are continuous variables stored as floating-point values . If the receiver does not reach the goal region within a certain number of steps , the episode ends with no reward given . 3.2 AGENT ARCHITECTURE . Our architecture comprises two agents , conceptually speaking , but in practice , they are a single neural network . The sender is a disembodied agent which observes the location of the receiver and passes a message in order to guide it towards the goal . The receiver is an agent which receives the message as its only input and takes an action solely based on that message ( i.e. , it is “ blind ” ) . The sender and receiver are randomly initialized at the start of training , trained together , and tested together . The architecture of the agents is illustrated in Figure 1b . The observation of the sender is a pair of floating-point values representing the receiver ’ s location . The sender itself is a 2-layer perceptron with tanh activations . The output of the second layer is passed to a Gumbel-Softmax bottleneck layer ( Maddison et al. , 2017 ; Jang et al. , 2017 ) which enables learning a discrete , one-hot representation , as an information bottleneck ( Tishby et al. , 2000 ) . The activations of this layer can be thought of as the “ words ” or “ lexicon ” of the emergent language . At evaluation time , the bottleneck layer functions deterministically as an argmax layer , emitting onehot vectors . The receiver is a 1-layer perceptron which receives the output of the Gumbel-Softmax layer as input . The output is a pair of floating-point values which determine the action of the agent . The action is clamped to a maximum step size . 3.3 OPTIMIZATION . Optimization is performed using stochastic gradient descent as a part of proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) . Specifically , we use the implementation of PPO provided by Stable Baselines 3 with our neural networks implemented in PyTorch ( Raffin et al. , 2019 ; Paszke et al. , 2019 ) . Using a Gumbel-Softmax bottleneck layer allows for end-to-end backpropagation , making optimization faster and more consistent than using a backpropagation-free method like REINFORCE ( Kharitonov et al. , 2020 ; Williams , 1992 ) . We will give a basic explanation of how PPO ( and related algorithms ) works as it is necessary to connect it to the mathematical model presented in Section 4 . PPO has two stages sampling and optimizing . During the sampling stage , the algorithm runs the agents in the environment and stores the states , actions , and rewards in a experience buffer ( or rollout buffer ) . The optimization stage then comprises performing gradient descent on the agents ( i.e. , neural network policy ) using the data from experience buffer . The next iteration starts with the updated agents and an empty experience buffer .

Summary: The paper examines the effect of reward shaping in the context of emergent communication, and specifically with respect to the Shannon entropy of the emerged language. The paper showed that shaping the rewards can lead to higher entropy than training with just the base reward. We care about the entropy because this relates to how expressive the language is. There are existing works, such as Kharatinov et al 2020, which examine the relationship between task/environment and the resulting language entropy.  The paper uses PPO as its main outer-level optimizer, which it uses because the task is a multi-step task which might be challenging to learn with other algorithms such as REINFORCE.  The paper provides theoretical analysis of the effect of reward shaping on entropy by considering an analogy between PPO learning and a process they call the 'Extended Chinese Restaurant Process' (ECRP), where at each step, the customer can be divided into beta parts, and each of the beta bits of customers (which are now each of size 1/beta) takes a seat. This is presented as being analogous to how PPO first store multiple episodes over a buffer, and then updates the weights of the network, using the results of the whole buffer; then rinses and repeats.  Experimental results are presented which show that making the task harder/longer (by increasing the world radius) results in a fall of the entropy of the resultant emergent language, but not in the presence of reward shaping.  The paper asserts that whilst multi-step tasks, and PPO, are not much used in emergent communication literature currently, but it seems plausible that this will become the case in the future, and this paper is a small exploratory step in that direction.