Communicating via Markov Decision Processes

1 INTRODUCTION . This work introduces a novel problem setting called Markov coding games ( MCGs ) . MCGs are two-player decentralized Markov decision processes ( Oliehoek et al. , 2016 ) that proceed in four steps . In the first step , one agent ( called the sender ) receives a special private observation ( called the message ) , which it is tasked with communicating . In the second step , the sender plays out an episode of a Markov decision process ( MDP ) . In the third , the other agent ( called the receiver ) receives the sender ’ s MDP trajectory as its observation . In the fourth , the receiver estimates the message from the received trajectory . The shared payoff to the sender and receiver is a weighted sum of the cumulative reward yielded by the MDP and an indicator specifying whether or not the receiver correctly decoded the message . Among the reasons that MCGs are of interest is the fact that they generalize other important settings . The first of these is referential games . In a referential game , a sender attempts to communicate a message to a receiver using cheap talk actions—i.e. , communicatory actions that do not have externalities on the transition or reward functions . Referential games have been a subject of academic interest dating back at least as far as Lewis ’ s seminal work Convention ( Lewis , 1969 ) . Since then , various flavors of referential games have been studied in game theory ( Skyrms , 2010 ) , artificial life ( Steels , 2013 ) , evolutionary linguistics ( Smith , 2002 ) , cognitive science ( Spike et al. , 2017 ) , and machine learning ( Lazaridou et al. , 2018 ) . MCGs can be viewed as a generalization of referential games to a setting where we drop the often unrealistic assumption that the sender ’ s actions do not incur costs . A second problem setting generalized by MCGs is source coding ( MacKay , 2002 ) . In source coding ( also known as data compression ) the objective is to construct an injective mapping from a space of messages to the set of sequences of symbols ( for some finite set of symbols ) such that the expected output length is minimized . Source coding has a myriad of real world applications involving the compression of images , video , audio , and genetic data . MCGs can be viewed as a generalization of the source coding problem to a setting where the cost of an encoding may involve complex considerations , rather than simply being equal to the sequence length . Yet another reason to be interested in MCGs is that they isolate an important subproblem of decentralized control . In particular , achieving good performance in an MCG requires the sender ’ s actions to simultaneously perform control in an MDP and communicate information ( i.e. , to communicate implicitly ) . This presents a challenge due to the fact that approximate dynamic programming , the foundation for preeminent approaches to constructing control policies ( Sutton & Barto , 2018 ) , is ill suited to constructing communication protocols because their values depend on counterfactuals . In other words , the information conveyed by an action depends on the policy at other contemporaneous states , violating the locality assumption of approximate dynamic programming approaches . To address MCGs , we propose a theoretically grounded algorithm called greedy minimum entropy coupling ( GME ) . GME leverages a union of maximum entropy reinforcement learning ( MaxEnt RL ) ( Ziebart et al. , 2008 ) and minimum entropy coupling ( MEC ) ( Kovačević et al. , 2015 ) . The key insight is that maximizing the returns of the MDP can be disentangled from learning a good communication protocol by realizing that the entropy of a policy corresponds ( in an informal sense ) to its capacity to communicate . GME leverages this insight in two steps . In the first step , GME constructs a MaxEnt policy for the MDP , balancing between maximizing expected return and maximizing cumulative conditional entropy . In the second step , which occurs at each decision point , GME uses MEC to pair messages with actions in such a way that the sender selects actions with the same probabilities as the MaxEnt RL policy ( thereby guaranteeing the same expected return from the MDP ) and the receiver ’ s uncertainty about the message is greedily reduced as much as possible . To demonstrate the efficacy of GME , we present experiments for MCGs based on a gridworld , Cartpole , and Pong ( Bellemare et al. , 2013 ) , which we call CodeGrid , CodeCart , and CodePong , respectively . For CodeGrid , we show that with a message space in the 10s or 100s , GME significantly is able to outperform a relevant baseline . For CodeCart and CodePong , we use a message space of binary images and a uniform distribution over messages , meaning that a randomly guessing receiver has an astronomically small probability of guessing correctly . Remarkably , we show that GME is able to achieve an optimal expected return in Cartpole and Pong while simultaneously losslessly communicating images to the receiver , demonstrating that GME has the capacity to be scaled to extremely large message spaces and complex control tasks . Moreover , we find that the performance of GME decays gracefully as the amount of actuator noise in the environment increases . 2 RELATED WORK . The works that are most closely related to this one can be taxonomized as coming from literature on referential games , source coding , multi-agent reinforcement learning , and diverse skill learning . Referential Games Among work on referential games , Foerster et al . ( 2016 ) ’ s work is perhaps most similar in that it is concerned with directly optimizing the performance of a communication protocol . They propose DIAL , an algorithm that optimizes the sender ’ s protocol by performing gradient ascent through the parameters of the receiver . Foerster et al . show that DIAL outperforms methods based on independent Q-learning on a variety of communication tasks . However , DIAL-based approaches are not directly applicable to MCGs , as they would require differentiating through trajectories . Coding Another body of related work concerns extensions of the source coding problem . Length limited coding ( Larmore & Hirschberg , 1990 ) considers a problem setting in which the objective is to minimize the expected sequence length ( as before ) , subject to a maximum length constraint . Coding with unequal symbol costs ( Golin et al. , 2002 ; Iwata et al. , 1997 ) considers the problem in which the goal is to minimize the expected cumulative symbol cost of the sequence to which the message is mapped . The cost of a symbol may differ from the cost of other symbols arbitrarily , making it a strictly more general problem setting than standard source coding ( which can also be thought of as minimizing cumulative symbol cost with equally costly symbols ) . Both length limited coding and coding with unequal costs are subsumed by Markov coding games . And while existing algorithms for both standard source coding and the extensions above are well-established and widely commercialized , they are unable to address the more general MCG setting . MCGs are also related to finite state Markov channel settings ( Wang & Moayeri , 1995 ) . In such settings , the fidelity of the channel by which the sender communicates to a receiver is controlled by a Markov process , which , in contrast to our work , transitions independently of the sender ’ s decisions . Another related setting is intersymbol interference , where the sender ’ s previously selected symbols ( i.e. , actions ) may cause interference with subsequently selected symbols , making them less likely to be faithfully transmitted to the receiver ( Lathi , 1998 ) . MCGs differ from both Markov channel and intersymbol interference settings in that the Markov system controls the cost paid by the sender , rather than interfering with the quality of the channel . MCGs are more resemblant of a setting in which the channel is reliable , but subject to natural variation in costs , such as based on weather or third party usage , as well as variation based on the sender ’ s own usage . Multi-Agent Reinforcement Learning A third related area comes from MARL literature . Strouse et al . ( 2018 ) investigate directly embedding a reward for taking actions with high mutual information into policy gradient objectives . They find that this approach can improve expected return in cooperative settings with asymmetric information . The baseline for our CodeGrid experiments loosely resembles Strouse et al. ’ s algorithm . More recently , Bhatt & Buro ( 2021 ) investigate an alternative approach whereby the sender ’ s behavioral policy deterministically selects the action that maximizes the receiver ’ s posterior probability of the correct message , when computed using the target policy . They show that this modification empirically yields significantly improved convergence properties as compared to other variations of independent reinforcement learning . However , this approach is not directly applicable to settings in which a single action must be used for both communication and control . Diverse Skill Learning A fourth area of related research is that of diverse skill learning ( Eysenbach et al. , 2019 ) . Eysenbach et al . ( 2019 ) propose an unsupervised learning method for discovering diverse , identifiable skills . Their objective , called DIAYN , seeks to learn diverse , discriminable skills . This paradigm resembles our work in the sense that skills can be interpreted as messages and discriminability can be interpreted as maximizing the mutual information between the skill and the state . The baseline used in our CodeGrid experiments can also be viewed as an adaptation of an idealized version of DIAYN to the MCG setting . 3 BACKGROUND AND NOTATION . We will require the following background and notation material to introduce Markov coding games and greedy minimum entropy coupling . Markov Decision Processes To represent our task formalism , we use finite Markov decision processes ( MDPs ) . We notate MDPs using tuples 〈S , A , R , T 〉 where S is the set of states , A is the set of actions , R : S × A → R is the reward function , and T : S × A → ∆ ( S ) is the transition function . An agent ’ s interactions with an MDP are dictated by a policy π : S → ∆ ( A ) mapping states to distributions over actions . We focus on episodic MDPs , meaning that after a finite number of transitions have occurred , the MDP will terminate . The history of states and actions is notated using h = ( s0 , a0 , . . . , st ) . We use the notation R ( h ) = ∑j R ( sj , aj ) to denote the amount of reward accumulated over the course of a history . When a history is terminal , we use z to notate it , rather than h. The objective of an MDP is to determine a policy arg maxπEπR ( Z ) yielding a large cumulative reward in expectation . Entropy To help us quantify the idea of uncertainty , we introduce entropy . Symbolically , the entropy of a random variable X is H ( X ) = −E logP ( X ) . Because the logarithm function is concave , the entropy of X is maximized when the mass of PX is spread as evenly as possible and minimized when the mass of PX is concentrated at a single point . In the context of decision-making , entropy can be used to describe the uncertainty regarding which action will be taken by an agent . When a policy spans multiple decision-points , the uncertainty regarding the agent ’ s actions given that the state is known is naturally described by conditional entropy . Conditional entropy is the entropy of a random variable , conditioned upon the fact that the realization of another random variable is known . More formally , conditional entropy is defined by H ( X | Y ) = H ( X , Y ) −H ( Y ) where the joint entropyH ( X , Y ) = −E logP ( X , Y ) is defined as the entropy of ( X , Y ) considered as a random vector . In some contexts , it is desirable for a decision-maker ’ s policy to be highly stochastic . In such cases , an attractive alternative to the expected cumulative reward objective is the maximum entropy RL objective Ziebart et al . ( 2008 ) maxπ Eπ [ ∑ tR ( St , At ) + αH ( At | St ) ] , which trades off between maximizing expected return and pursuing trajectories along which its actions have large cumulative conditional entropy , using the temperature hyperparameter α . Mutual Information A closely related concept to entropy is mutual information . Mutual information describes the strength of the dependence between two random variables . The greater the mutual information between two random variables , the more the outcome of one affects the conditional distribution of the other . Symbolically , mutual information is defined by I ( X ; Y ) = H ( Y ) −H ( Y | X ) = H ( X ) −H ( X | Y ) . From this definition , we see explicitly that the mutual information of two random variables can be interpreted as the amount of uncertainty about one that is eliminated by observing the realization of the other . Mutual information is important for communication because we may only be able to share the realization of an auxiliary random variable , rather than that of the random variable of interest . In such cases , maximizing the amount of communicated information amounts to maximizing the mutual information between the auxiliary random variable and the random variable of interest . The Data Processing Inequality The independence relationships among random variables play an important role in determining their mutual information . If random variables X and Z are conditionally independent given Y ( that is , X ⊥ Z | Y ) , the data processing inequality states that I ( X ; Y ) ≥ I ( X ; Z ) . Less formally , the data processing inequality states that if Z does not provide additional information about X given Y , then the dependence between X and Z can not be stronger than the dependence between X and Y . Minimum Entropy Coupling In some cases , we may wish to maximize the mutual information between two random variables subject to fixed marginals . That is , we are tasked with determining a joint distribution PX , Y that maximizes the mutual information I ( X ; Y ) between X and Y subject to the constraints that PX , Y marginalizes to PX and PY , where PX and PY are given as input . Invoking the relationship between mutual information and joint entropy I ( X ; Y ) = H ( X ) + H ( Y ) − H ( X , Y ) , we see that this problem is equivalent to that of minimizing the joint entropy of X and Y . As a result , this problem is often referred to as the minimum entropy coupling problem . A visual example is shown in Figure 1 . While minimum entropy coupling is NP-hard Kovačević et al . ( 2015 ) , Cicalese et al . ( 2019 ) recently showed that there exists a polynomial time algorithm that is suboptimal by no more than one bit .

Summary: The paper suggests a combination of MaxEnt RL with Minimum Entropy Coupling to construct a communication method via (on the fly) modulation of a stochastic policy execution. The core idea being that communication does not require a separate channel beyond the capability of the receiver to observe the sender's (otherwise goal-driven) behaviour. The paper is supported by several experiments, including a study of external interferences with the established behaviour-based communication channel in the form of action execution uncertainty.