The Value of Information When Deciding What to Learn

1 Introduction . Information acquisition is a fundamental consideration in the design of sequential decision-making agents . Indeed , the literature over recent years has unequivocally demonstrated its theoretical [ Russo and Van Roy , 2016 , 2014 , 2018a ] and empirical [ Osband et al. , 2016a , b , Nikolov et al. , 2018 ] importance in addressing the challenge of exploration in sequential decision-making problems . Despite this significance , a strategy for information acquisition , however efficient , is only as good as the information it uncovers . An agent that displays remarkable efficiency in seeking immaterial information offers little utility to its designer . Similarly , when faced with the rich complexity of real-world tasks , a computationally-bounded agent aspiring to learn that which is well beyond its constrained resources is equally ineffective . As a concrete example of the latter scenario , consider learning the optimal policy for the game of Go [ Silver et al. , 2017 ] solely relying on the hardware of a mobile device . While not an entirely infeasible task , the amount of time needed to complete such an endeavor may force the agent designer into settling for a small fraction of the optimal policy performance in exchange for a substantial reduction in the difficulty of the learning problem . Recent work has developed a connection between rate-distortion theory [ Shannon , 1959 , Berger , 1971 ] and the problem of deciding what an agent should learn [ Arumugam and Van Roy , 2021 ] , optimally balancing between the requisite information needed by any agent to identify a learning target and the corresponding performance shortfall associated with this target . Intuitively , the complexity of a learning problem can be quantified by the bits of information an agent must acquire from the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . environment in order to realize a particular target . Rate-distortion theory offers a principled framework for reasoning about a learning target that comprises the minimum number of bits from the environment while retaining some degree of near-optimality . Arumugam and Van Roy [ 2021 ] operationalize the classic Blahut-Arimoto algorithm [ Blahut , 1972 , Arimoto , 1972 ] in the multi-armed bandit setting to facilitate computationally-tractable agents that are not only capable of computing such an optimal target at the start of learning but also gradually adapt this target over time as the agent ’ s knowledge of the environment accumulates . The resulting Blahut-Arimoto Satisficing Thompson Sampling algorithm ( BLASTS ) then applies the well-known principle of probability matching in order to select actions [ Thompson , 1933 , Russo et al. , 2018 ] . While BLASTS demonstrates the desired ability to accommodate an agent designer in recovering a wide spectrum of policies with varying levels of performance and learning efficiency , it is well known that the underlying probability matching principle of Thompson Sampling constitutes a sub-optimal exploration strategy [ Russo and Van Roy , 2018a ] . In this work , we combine the optimal computation of learning targets via rate-distortion theory with optimal information acquisition . To achieve the latter goal , we generalize the information-directed sampling ( IDS ) principle of Russo and Van Roy [ 2018a ] so that the fundamental information ratio captures expected regret and information gain about a learning target computed via the BlahutArimoto algorithm . As with BLASTS , the resulting algorithms which instantiate this Blahut-Arimoto Information-Directed Sampling principle ( BLAIDS ) retain a single , real-valued hyperparameter through which an agent designer expresses a preference for the balance struck by the resulting learning target . Moreover , just as BLASTS offers a generalization of Thompson Sampling capable of more efficiently computing optimal policies , we find that a particular computationally-tractable instantiation of BLAIDS mirrors this property relative to its IDS counterpart . En route to deriving BLAIDS , we outline the precise details of the Blahut-Arimoto algorithm itself and connect our setting with various results from the information-theory literature . The paper proceeds as follows : in Section 2 we offer a general presentation of information-theoretic learning targets for sequential decision-making problems . We then hone in on the multi-armed bandit setting in Section 3 to carefully study incorporating the value of information when acting in pursuit of an arbitrary learning target . We conclude in Section 4 with a corroborating set of computational experiments that demonstrate the efficacy of our approach . Due to space constraints , we relegate background material on information theory , related work , and all technical proofs to the appendix . 2 Learning Targets . This section focuses on a general treatment of learning targets via rate-distortion theory , moving beyond the assumptions of Arumugam and Van Roy [ 2021 ] and offering further insight into the mathematical details of the Blahut-Arimoto algorithm . 2.1 Problem Formulation . In this section , we follow the general problem formulation of Lu et al . [ 2021 ] that offers a unified treatment of sequential decision-making problems , spanning multi-armed bandits and reinforcementlearning problems [ Lattimore and Szepesvári , 2020 , Sutton and Barto , 1998 ] . We adopt a generic agent-environment interface wherein , at each time period , the agent executes an action At ∈ A within an environment E which results in an associated next observation Ot+1 ∈ O . This sequential interaction between agent and environment results in an associated history at each timestep Ht = ( A0 , O1 , . . . , At−1 , Ot ) ∈ H representing the action-observation sequence available to the agent upon making its selection of At . More formally , we may characterize the overall environment as E = 〈A , O , ρ〉 containing the action set A , observation set O , and the observation function ρ : H×A → ∆ ( O ) that prescribes the distribution over next observations given the current history and action selection : P ( Ot+1|E , Ht , At ) = P ( Ot+1|Ht , At ) Ot+1 ∼ ρ ( ·|Ht , At ) . An agent ’ s policy π : H → ∆ ( A ) encapsulates the relationship between the histories encountered in each timestep Ht and the executed action At . We maintain that actions are independent of the environment conditioned on history , At ⊥ E|Ht such that π ( a|Ht ) = P ( At = a|Ht ) assigns a probability to each action a ∈ A given the history . An agent designer expresses preferences across histories via a reward function r : H×A×O → R such that an agent enjoys a reward Rt+1 = r ( Ht , At , Ot+1 ) at each timestep . Given any finite time horizon T < ∞ , the accumulation of rewards provide a notion of return T−1∑ t=0 Rt+1 . To develop preferences over behaviors , it is natural to associate with each policy π a corresponding expected return or value across the horizon T as V π = E [ T−1∑ t=0 Rt+1 | E ] , where the expectation integrates over the randomness in the policy π as well as the observation function ρ . A trademark across the sequential decision-making literature is the design of agents that strive to achieve the optimal value within the confines of some policy class Π ⊆ { π : H → ∆ ( A ) } , V ? = sup π∈Π V π . Notice that when rewards and the distribution of the next observation Ot+1 depend only on the current observation Ot , rather than the full history Ht , we recover the traditional Markov Decision Process [ Bellman , 1957 , Puterman , 1994 ] studied throughout the reinforcement-learning literature [ Sutton and Barto , 1998 ] . Alternatively , when these quantities rely solely upon the most recent action At , we recover the traditional multi-armed bandit [ Lattimore and Szepesvári , 2020 ] . 2.2 The Curse of Curiosity . While the design of agents in pursuit of optimal policies is a perfectly natural object of study , it can often occur without regard for the complexity of the environment . In particular , the optimal policy is often a deterministic function of the environment such that , if an agent manages to identify the environment , it possesses all requisite information to determine the optimal policy . An agent designer reflects their initial uncertainty about the environment through a prior distribution P ( E ∈ · ) ; as the history unfolds , what can be learned from the agents current knowledge of the environment is represented by conditional probabilities P ( E ∈ ·|Ht ) . Through this prior distribution , a total of H ( E ) bits quantify all of the information needed for identifying the environment . Prior work has demonstrated both theoretically and empirically how it is an agent ’ s incremental resolution of epistemic uncertainty over the environment E that gives rise to the efficient acquisition of these H ( E ) bits [ Chapelle and Li , 2011 , Russo and Van Roy , 2016 , Osband et al. , 2016a , Agrawal and Jia , 2017 , O ’ Donoghue et al. , 2018 , Osband et al. , 2019 ] . For sufficiently rich and complex environments , however , H ( E ) can become prohibitively large or even infinite , making the pursuit of an optimal policy entirely intractable . This issue warrants the construction of a more general mechanism whereby an agent designer may express tractable learning targets , alternatives to the optimal solution that , while potentially incurring some degree of suboptimality , fall within the means of a computationally-bounded agent interacting with a highly-complex environment . Lu et al . [ 2021 ] introduce this notion of a learning target χ as a generic random variable and ( possibly stochastic ) function of the environment E . Their proposed desiderata for the construction of learning target is two-fold : ( 1 ) demanding only a feasible amount of information from the environment I ( E ; χ ) in order to reconcile χ and ( 2 ) incurring a bounded degree of performance shortfall by aiming for χ , E [ V ? − V πχ ] , where πχ is a target policy induced by χ . Observe , however , that this treatment of a learning target assumes its provision to the agent before interaction with the environment has begun . Consequently , the burden of specifying a suitable learning target that best balances between the two desiderata falls upon the agent designer . To remedy this , we begin by extending the formulation of Arumugam and Van Roy [ 2021 ] and emphasizing rate-distortion theory as a general tool for the design of optimal learning targets . It is important to re-iterate and clarify that the role of πχ is not to achieve performance that is on par with the optimal π ? , but rather to strike an appropriate balance between performance shortfall relative to π ? and the ease of learnability ; the latter desideratum is accurately captured mathematically via bits of information . As an agent shifts its focus away from π ? towards a χ that is easier to learn ( in the information-theoretic sense used throughout this paper ) , we anticipate greater data efficiency . That is , an agent should be able to more quickly identify the desired χ relative to the requisite amount of time and data needed to identify π ? . Naturally , in a complex environment , we expect this data efficiency to be critical and to come at some cost : the sub-optimality of the resulting solution ( which we take as the distortion in our rate-distortion framing ) . An agent designer ’ s willingness to tolerate this sub-optimality stems from the complexity of the environment and an acknowledgement of the potentially intractable amount of data needed to otherwise identify and recover π ? . The more performance shortfall one is willing to tolerate relative to optimal behavior , the more efficiently an agent should be able to find such a satisficing solution . Russo and Van Roy [ 2018b ] already highlight one example of this phenomenon in the multi-armed bandit setting , where the presence of multiple arms can make pursuit of an ε-optimal arm far more data efficient than the optimal arm . From the perspective of the agent designer , one should not incorporate a learning target with the expectation of being competitive with the optimal policy , but rather with the goal of being more data efficient in the acquisition of the learning target relative to that of the optimal policy .

Summary: This paper proposes the use of Blahut-Arimoto algorithm to learn an encoder from the source to the target, where the source is the environment and the target the policy to be recovered under a budget of distortion / suboptimality (this budget can be seen as a bounded-rationality). The output of BA algorithm would be a satisficing (good enough) policy, satisfying the budget constraints but also minimizing the mutual information between the policy and the environment. With this paper, the authors propose an improvement over Blahut-Arimoto Thompson Sampling method by using Information Directed Sampling in combination with Blahut-Arimoto algorithm (to compute target action of corresponding source environment). Finally, they provide an additional version based on the variance of the reward as a tractable approximation of the mutual information denominator.