The Value of Information When Deciding What to Learn

1 Introduction . Information acquisition is a fundamental consideration in the design of sequential decision-making agents . Indeed , the literature over recent years has unequivocally demonstrated its theoretical [ Russo and Van Roy , 2016 , 2014 , 2018a ] and empirical [ Osband et al. , 2016a , b , Nikolov et al. , 2018 ] importance in addressing the challenge of exploration in sequential decision-making problems . Despite this significance , a strategy for information acquisition , however efficient , is only as good as the information it uncovers . An agent that displays remarkable efficiency in seeking immaterial information offers little utility to its designer . Similarly , when faced with the rich complexity of real-world tasks , a computationally-bounded agent aspiring to learn that which is well beyond its constrained resources is equally ineffective . As a concrete example of the latter scenario , consider learning the optimal policy for the game of Go [ Silver et al. , 2017 ] solely relying on the hardware of a mobile device . While not an entirely infeasible task , the amount of time needed to complete such an endeavor may force the agent designer into settling for a small fraction of the optimal policy performance in exchange for a substantial reduction in the difficulty of the learning problem . Recent work has developed a connection between rate-distortion theory [ Shannon , 1959 , Berger , 1971 ] and the problem of deciding what an agent should learn [ Arumugam and Van Roy , 2021 ] , optimally balancing between the requisite information needed by any agent to identify a learning target and the corresponding performance shortfall associated with this target . Intuitively , the complexity of a learning problem can be quantified by the bits of information an agent must acquire from the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . environment in order to realize a particular target . Rate-distortion theory offers a principled framework for reasoning about a learning target that comprises the minimum number of bits from the environment while retaining some degree of near-optimality . Arumugam and Van Roy [ 2021 ] operationalize the classic Blahut-Arimoto algorithm [ Blahut , 1972 , Arimoto , 1972 ] in the multi-armed bandit setting to facilitate computationally-tractable agents that are not only capable of computing such an optimal target at the start of learning but also gradually adapt this target over time as the agent ’ s knowledge of the environment accumulates . The resulting Blahut-Arimoto Satisficing Thompson Sampling algorithm ( BLASTS ) then applies the well-known principle of probability matching in order to select actions [ Thompson , 1933 , Russo et al. , 2018 ] . While BLASTS demonstrates the desired ability to accommodate an agent designer in recovering a wide spectrum of policies with varying levels of performance and learning efficiency , it is well known that the underlying probability matching principle of Thompson Sampling constitutes a sub-optimal exploration strategy [ Russo and Van Roy , 2018a ] . In this work , we combine the optimal computation of learning targets via rate-distortion theory with optimal information acquisition . To achieve the latter goal , we generalize the information-directed sampling ( IDS ) principle of Russo and Van Roy [ 2018a ] so that the fundamental information ratio captures expected regret and information gain about a learning target computed via the BlahutArimoto algorithm . As with BLASTS , the resulting algorithms which instantiate this Blahut-Arimoto Information-Directed Sampling principle ( BLAIDS ) retain a single , real-valued hyperparameter through which an agent designer expresses a preference for the balance struck by the resulting learning target . Moreover , just as BLASTS offers a generalization of Thompson Sampling capable of more efficiently computing optimal policies , we find that a particular computationally-tractable instantiation of BLAIDS mirrors this property relative to its IDS counterpart . En route to deriving BLAIDS , we outline the precise details of the Blahut-Arimoto algorithm itself and connect our setting with various results from the information-theory literature . The paper proceeds as follows : in Section 2 we offer a general presentation of information-theoretic learning targets for sequential decision-making problems . We then hone in on the multi-armed bandit setting in Section 3 to carefully study incorporating the value of information when acting in pursuit of an arbitrary learning target . We conclude in Section 4 with a corroborating set of computational experiments that demonstrate the efficacy of our approach . Due to space constraints , we relegate background material on information theory , related work , and all technical proofs to the appendix . 2 Learning Targets . This section focuses on a general treatment of learning targets via rate-distortion theory , moving beyond the assumptions of Arumugam and Van Roy [ 2021 ] and offering further insight into the mathematical details of the Blahut-Arimoto algorithm . 2.1 Problem Formulation . In this section , we follow the general problem formulation of Lu et al . [ 2021 ] that offers a unified treatment of sequential decision-making problems , spanning multi-armed bandits and reinforcementlearning problems [ Lattimore and Szepesvári , 2020 , Sutton and Barto , 1998 ] . We adopt a generic agent-environment interface wherein , at each time period , the agent executes an action At ∈ A within an environment E which results in an associated next observation Ot+1 ∈ O . This sequential interaction between agent and environment results in an associated history at each timestep Ht = ( A0 , O1 , . . . , At−1 , Ot ) ∈ H representing the action-observation sequence available to the agent upon making its selection of At . More formally , we may characterize the overall environment as E = 〈A , O , ρ〉 containing the action set A , observation set O , and the observation function ρ : H×A → ∆ ( O ) that prescribes the distribution over next observations given the current history and action selection : P ( Ot+1|E , Ht , At ) = P ( Ot+1|Ht , At ) Ot+1 ∼ ρ ( ·|Ht , At ) . An agent ’ s policy π : H → ∆ ( A ) encapsulates the relationship between the histories encountered in each timestep Ht and the executed action At . We maintain that actions are independent of the environment conditioned on history , At ⊥ E|Ht such that π ( a|Ht ) = P ( At = a|Ht ) assigns a probability to each action a ∈ A given the history . An agent designer expresses preferences across histories via a reward function r : H×A×O → R such that an agent enjoys a reward Rt+1 = r ( Ht , At , Ot+1 ) at each timestep . Given any finite time horizon T < ∞ , the accumulation of rewards provide a notion of return T−1∑ t=0 Rt+1 . To develop preferences over behaviors , it is natural to associate with each policy π a corresponding expected return or value across the horizon T as V π = E [ T−1∑ t=0 Rt+1 | E ] , where the expectation integrates over the randomness in the policy π as well as the observation function ρ . A trademark across the sequential decision-making literature is the design of agents that strive to achieve the optimal value within the confines of some policy class Π ⊆ { π : H → ∆ ( A ) } , V ? = sup π∈Π V π . Notice that when rewards and the distribution of the next observation Ot+1 depend only on the current observation Ot , rather than the full history Ht , we recover the traditional Markov Decision Process [ Bellman , 1957 , Puterman , 1994 ] studied throughout the reinforcement-learning literature [ Sutton and Barto , 1998 ] . Alternatively , when these quantities rely solely upon the most recent action At , we recover the traditional multi-armed bandit [ Lattimore and Szepesvári , 2020 ] . 2.2 The Curse of Curiosity . While the design of agents in pursuit of optimal policies is a perfectly natural object of study , it can often occur without regard for the complexity of the environment . In particular , the optimal policy is often a deterministic function of the environment such that , if an agent manages to identify the environment , it possesses all requisite information to determine the optimal policy . An agent designer reflects their initial uncertainty about the environment through a prior distribution P ( E ∈ · ) ; as the history unfolds , what can be learned from the agents current knowledge of the environment is represented by conditional probabilities P ( E ∈ ·|Ht ) . Through this prior distribution , a total of H ( E ) bits quantify all of the information needed for identifying the environment . Prior work has demonstrated both theoretically and empirically how it is an agent ’ s incremental resolution of epistemic uncertainty over the environment E that gives rise to the efficient acquisition of these H ( E ) bits [ Chapelle and Li , 2011 , Russo and Van Roy , 2016 , Osband et al. , 2016a , Agrawal and Jia , 2017 , O ’ Donoghue et al. , 2018 , Osband et al. , 2019 ] . For sufficiently rich and complex environments , however , H ( E ) can become prohibitively large or even infinite , making the pursuit of an optimal policy entirely intractable . This issue warrants the construction of a more general mechanism whereby an agent designer may express tractable learning targets , alternatives to the optimal solution that , while potentially incurring some degree of suboptimality , fall within the means of a computationally-bounded agent interacting with a highly-complex environment . Lu et al . [ 2021 ] introduce this notion of a learning target χ as a generic random variable and ( possibly stochastic ) function of the environment E . Their proposed desiderata for the construction of learning target is two-fold : ( 1 ) demanding only a feasible amount of information from the environment I ( E ; χ ) in order to reconcile χ and ( 2 ) incurring a bounded degree of performance shortfall by aiming for χ , E [ V ? − V πχ ] , where πχ is a target policy induced by χ . Observe , however , that this treatment of a learning target assumes its provision to the agent before interaction with the environment has begun . Consequently , the burden of specifying a suitable learning target that best balances between the two desiderata falls upon the agent designer . To remedy this , we begin by extending the formulation of Arumugam and Van Roy [ 2021 ] and emphasizing rate-distortion theory as a general tool for the design of optimal learning targets . It is important to re-iterate and clarify that the role of πχ is not to achieve performance that is on par with the optimal π ? , but rather to strike an appropriate balance between performance shortfall relative to π ? and the ease of learnability ; the latter desideratum is accurately captured mathematically via bits of information . As an agent shifts its focus away from π ? towards a χ that is easier to learn ( in the information-theoretic sense used throughout this paper ) , we anticipate greater data efficiency . That is , an agent should be able to more quickly identify the desired χ relative to the requisite amount of time and data needed to identify π ? . Naturally , in a complex environment , we expect this data efficiency to be critical and to come at some cost : the sub-optimality of the resulting solution ( which we take as the distortion in our rate-distortion framing ) . An agent designer ’ s willingness to tolerate this sub-optimality stems from the complexity of the environment and an acknowledgement of the potentially intractable amount of data needed to otherwise identify and recover π ? . The more performance shortfall one is willing to tolerate relative to optimal behavior , the more efficiently an agent should be able to find such a satisficing solution . Russo and Van Roy [ 2018b ] already highlight one example of this phenomenon in the multi-armed bandit setting , where the presence of multiple arms can make pursuit of an ε-optimal arm far more data efficient than the optimal arm . From the perspective of the agent designer , one should not incorporate a learning target with the expectation of being competitive with the optimal policy , but rather with the goal of being more data efficient in the acquisition of the learning target relative to that of the optimal policy .

Summary: In reinforcement learning, the ideal thing to do would be to learn all of the information there is about the environment, and then use that to execute the optimal policy. However, in practice, it is impossible to learn _all_ of the necessary information in very complicated environments. Thus, we instead would like to define a _learning target_ that only incentivizes learning a relevant subset of information that is useful for acquiring reward. The learning target must be chosen such that (a) it does not require too much information to learn, and (b) the optimal policy given only the information in the learning target has low regret (relative to the optimal policy given arbitrary information).  So far learning targets have been coded by hand. Ideally we would instead select an appropriate learning target automatically, that outperforms targets coded by hand. The core insight is that constraint (a) is simply lossy compression, where constraint (b) can be thought of as a constraint on the _distortion_ on regret from throwing away some information in the lossy compression. We can then apply standard approaches from rate distortion theory to choose a target that best fits both constraints.  Once we have selected an appropriate target, we must also choose actions that provide us information about the target. This can be done through _information-directed sampling_, in which we choose actions that minimize the ratio between (expected) regret incurred and information gained about the learning target.  The authors instantiate this framework in the case of multiarmed bandits, where the learning targets are environment-conditional policies $p(a \mid \mathcal{E})$. In the case of zero distortion, the learning target is the optimal environment-conditional policy (which obtains zero regret). As distortion increases, the learning target becomes more and more “smeared” (e.g. in environments where the best and second-best arms are very close in reward, the learning target might put similar probabilities on both arms), making it easier to learn. They show that the resulting algorithm outperforms other alternatives that don’t use automatically generated learning targets and/or information-directed sampling.