The Value of Information When Deciding What to Learn

1 Introduction . Information acquisition is a fundamental consideration in the design of sequential decision-making agents . Indeed , the literature over recent years has unequivocally demonstrated its theoretical [ Russo and Van Roy , 2016 , 2014 , 2018a ] and empirical [ Osband et al. , 2016a , b , Nikolov et al. , 2018 ] importance in addressing the challenge of exploration in sequential decision-making problems . Despite this significance , a strategy for information acquisition , however efficient , is only as good as the information it uncovers . An agent that displays remarkable efficiency in seeking immaterial information offers little utility to its designer . Similarly , when faced with the rich complexity of real-world tasks , a computationally-bounded agent aspiring to learn that which is well beyond its constrained resources is equally ineffective . As a concrete example of the latter scenario , consider learning the optimal policy for the game of Go [ Silver et al. , 2017 ] solely relying on the hardware of a mobile device . While not an entirely infeasible task , the amount of time needed to complete such an endeavor may force the agent designer into settling for a small fraction of the optimal policy performance in exchange for a substantial reduction in the difficulty of the learning problem . Recent work has developed a connection between rate-distortion theory [ Shannon , 1959 , Berger , 1971 ] and the problem of deciding what an agent should learn [ Arumugam and Van Roy , 2021 ] , optimally balancing between the requisite information needed by any agent to identify a learning target and the corresponding performance shortfall associated with this target . Intuitively , the complexity of a learning problem can be quantified by the bits of information an agent must acquire from the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . environment in order to realize a particular target . Rate-distortion theory offers a principled framework for reasoning about a learning target that comprises the minimum number of bits from the environment while retaining some degree of near-optimality . Arumugam and Van Roy [ 2021 ] operationalize the classic Blahut-Arimoto algorithm [ Blahut , 1972 , Arimoto , 1972 ] in the multi-armed bandit setting to facilitate computationally-tractable agents that are not only capable of computing such an optimal target at the start of learning but also gradually adapt this target over time as the agent ’ s knowledge of the environment accumulates . The resulting Blahut-Arimoto Satisficing Thompson Sampling algorithm ( BLASTS ) then applies the well-known principle of probability matching in order to select actions [ Thompson , 1933 , Russo et al. , 2018 ] . While BLASTS demonstrates the desired ability to accommodate an agent designer in recovering a wide spectrum of policies with varying levels of performance and learning efficiency , it is well known that the underlying probability matching principle of Thompson Sampling constitutes a sub-optimal exploration strategy [ Russo and Van Roy , 2018a ] . In this work , we combine the optimal computation of learning targets via rate-distortion theory with optimal information acquisition . To achieve the latter goal , we generalize the information-directed sampling ( IDS ) principle of Russo and Van Roy [ 2018a ] so that the fundamental information ratio captures expected regret and information gain about a learning target computed via the BlahutArimoto algorithm . As with BLASTS , the resulting algorithms which instantiate this Blahut-Arimoto Information-Directed Sampling principle ( BLAIDS ) retain a single , real-valued hyperparameter through which an agent designer expresses a preference for the balance struck by the resulting learning target . Moreover , just as BLASTS offers a generalization of Thompson Sampling capable of more efficiently computing optimal policies , we find that a particular computationally-tractable instantiation of BLAIDS mirrors this property relative to its IDS counterpart . En route to deriving BLAIDS , we outline the precise details of the Blahut-Arimoto algorithm itself and connect our setting with various results from the information-theory literature . The paper proceeds as follows : in Section 2 we offer a general presentation of information-theoretic learning targets for sequential decision-making problems . We then hone in on the multi-armed bandit setting in Section 3 to carefully study incorporating the value of information when acting in pursuit of an arbitrary learning target . We conclude in Section 4 with a corroborating set of computational experiments that demonstrate the efficacy of our approach . Due to space constraints , we relegate background material on information theory , related work , and all technical proofs to the appendix . 2 Learning Targets . This section focuses on a general treatment of learning targets via rate-distortion theory , moving beyond the assumptions of Arumugam and Van Roy [ 2021 ] and offering further insight into the mathematical details of the Blahut-Arimoto algorithm . 2.1 Problem Formulation . In this section , we follow the general problem formulation of Lu et al . [ 2021 ] that offers a unified treatment of sequential decision-making problems , spanning multi-armed bandits and reinforcementlearning problems [ Lattimore and Szepesvári , 2020 , Sutton and Barto , 1998 ] . We adopt a generic agent-environment interface wherein , at each time period , the agent executes an action At ∈ A within an environment E which results in an associated next observation Ot+1 ∈ O . This sequential interaction between agent and environment results in an associated history at each timestep Ht = ( A0 , O1 , . . . , At−1 , Ot ) ∈ H representing the action-observation sequence available to the agent upon making its selection of At . More formally , we may characterize the overall environment as E = 〈A , O , ρ〉 containing the action set A , observation set O , and the observation function ρ : H×A → ∆ ( O ) that prescribes the distribution over next observations given the current history and action selection : P ( Ot+1|E , Ht , At ) = P ( Ot+1|Ht , At ) Ot+1 ∼ ρ ( ·|Ht , At ) . An agent ’ s policy π : H → ∆ ( A ) encapsulates the relationship between the histories encountered in each timestep Ht and the executed action At . We maintain that actions are independent of the environment conditioned on history , At ⊥ E|Ht such that π ( a|Ht ) = P ( At = a|Ht ) assigns a probability to each action a ∈ A given the history . An agent designer expresses preferences across histories via a reward function r : H×A×O → R such that an agent enjoys a reward Rt+1 = r ( Ht , At , Ot+1 ) at each timestep . Given any finite time horizon T < ∞ , the accumulation of rewards provide a notion of return T−1∑ t=0 Rt+1 . To develop preferences over behaviors , it is natural to associate with each policy π a corresponding expected return or value across the horizon T as V π = E [ T−1∑ t=0 Rt+1 | E ] , where the expectation integrates over the randomness in the policy π as well as the observation function ρ . A trademark across the sequential decision-making literature is the design of agents that strive to achieve the optimal value within the confines of some policy class Π ⊆ { π : H → ∆ ( A ) } , V ? = sup π∈Π V π . Notice that when rewards and the distribution of the next observation Ot+1 depend only on the current observation Ot , rather than the full history Ht , we recover the traditional Markov Decision Process [ Bellman , 1957 , Puterman , 1994 ] studied throughout the reinforcement-learning literature [ Sutton and Barto , 1998 ] . Alternatively , when these quantities rely solely upon the most recent action At , we recover the traditional multi-armed bandit [ Lattimore and Szepesvári , 2020 ] . 2.2 The Curse of Curiosity . While the design of agents in pursuit of optimal policies is a perfectly natural object of study , it can often occur without regard for the complexity of the environment . In particular , the optimal policy is often a deterministic function of the environment such that , if an agent manages to identify the environment , it possesses all requisite information to determine the optimal policy . An agent designer reflects their initial uncertainty about the environment through a prior distribution P ( E ∈ · ) ; as the history unfolds , what can be learned from the agents current knowledge of the environment is represented by conditional probabilities P ( E ∈ ·|Ht ) . Through this prior distribution , a total of H ( E ) bits quantify all of the information needed for identifying the environment . Prior work has demonstrated both theoretically and empirically how it is an agent ’ s incremental resolution of epistemic uncertainty over the environment E that gives rise to the efficient acquisition of these H ( E ) bits [ Chapelle and Li , 2011 , Russo and Van Roy , 2016 , Osband et al. , 2016a , Agrawal and Jia , 2017 , O ’ Donoghue et al. , 2018 , Osband et al. , 2019 ] . For sufficiently rich and complex environments , however , H ( E ) can become prohibitively large or even infinite , making the pursuit of an optimal policy entirely intractable . This issue warrants the construction of a more general mechanism whereby an agent designer may express tractable learning targets , alternatives to the optimal solution that , while potentially incurring some degree of suboptimality , fall within the means of a computationally-bounded agent interacting with a highly-complex environment . Lu et al . [ 2021 ] introduce this notion of a learning target χ as a generic random variable and ( possibly stochastic ) function of the environment E . Their proposed desiderata for the construction of learning target is two-fold : ( 1 ) demanding only a feasible amount of information from the environment I ( E ; χ ) in order to reconcile χ and ( 2 ) incurring a bounded degree of performance shortfall by aiming for χ , E [ V ? − V πχ ] , where πχ is a target policy induced by χ . Observe , however , that this treatment of a learning target assumes its provision to the agent before interaction with the environment has begun . Consequently , the burden of specifying a suitable learning target that best balances between the two desiderata falls upon the agent designer . To remedy this , we begin by extending the formulation of Arumugam and Van Roy [ 2021 ] and emphasizing rate-distortion theory as a general tool for the design of optimal learning targets . It is important to re-iterate and clarify that the role of πχ is not to achieve performance that is on par with the optimal π ? , but rather to strike an appropriate balance between performance shortfall relative to π ? and the ease of learnability ; the latter desideratum is accurately captured mathematically via bits of information . As an agent shifts its focus away from π ? towards a χ that is easier to learn ( in the information-theoretic sense used throughout this paper ) , we anticipate greater data efficiency . That is , an agent should be able to more quickly identify the desired χ relative to the requisite amount of time and data needed to identify π ? . Naturally , in a complex environment , we expect this data efficiency to be critical and to come at some cost : the sub-optimality of the resulting solution ( which we take as the distortion in our rate-distortion framing ) . An agent designer ’ s willingness to tolerate this sub-optimality stems from the complexity of the environment and an acknowledgement of the potentially intractable amount of data needed to otherwise identify and recover π ? . The more performance shortfall one is willing to tolerate relative to optimal behavior , the more efficiently an agent should be able to find such a satisficing solution . Russo and Van Roy [ 2018b ] already highlight one example of this phenomenon in the multi-armed bandit setting , where the presence of multiple arms can make pursuit of an ε-optimal arm far more data efficient than the optimal arm . From the perspective of the agent designer , one should not incorporate a learning target with the expectation of being competitive with the optimal policy , but rather with the goal of being more data efficient in the acquisition of the learning target relative to that of the optimal policy .

Summary: **Update after author response and discussion**: I am happy to see most of my assumptions positively confirmed and clarified by the authors, as well as the criticism by other reviewers being sufficiently addressed. Some of my improvements remain valid, but I do not consider them crucial for publication. I therefore stick with my original score but raise my confidence - I am in favor of accepting the paper.  **High-level summary**  The paper addresses the question of learning to act well in finite-horizon sequential tasks in unknown environments under computational limitations. This requires solving two problems. (1) In principle, full knowledge about the environment allows agents to act optimally. However, actions that lead to large information gain about the environment often come at the expense of incurring low immediate rewards - trading off these two considerations is non-trivial and has been formalized previously via information directed sampling (IDS), which aims to find policies that minimize the ratio between squared expected regret (of the policy compared to the optimal policy) and information gain (ideally leading to small regret and large information gain). (2) Reducing all (epistemic) uncertainty about the environment can incur a considerable amount of exploratory actions, particularly in complex environments. Often, a less informative (lossy) compression of the environment allows for learning policies that perform very well with considerably lower “exploration cost” (which is particularly relevant for computationally bounded agents). Such a compression has been previously introduced under the name ‘learning target’ which is computed via a (stochastic) mapping from the environment to the learning target random variable. A good learning target requires considerably less information (fewer bits) to be extracted from the environment (compared to reducing all uncertainty about the full environment) while at the same time ensuring that policies trained to maximize the learning target achieve good returns in the actual environments. This trade-off between low reduction in performance and high reduction in sample complexity of exploration has recently been formalized via a rate distortion theory, where the distortion is the expected regret between the policy trained on the learning target and the optimal policy (given an observation history), and the rate is the information that needs to be extracted from the environment to reduce uncertainty about the learning target variable sufficiently.  **Concrete summary**  Previous work (BLASTS) derived an improved Thompson sampling scheme based on learning targets obtained by solving the rate-distortion trade-off via Blahut-Arimoto iterations. While BLASTS provides a principled solution to extract good learning targets, the resulting policies are understood to perform poorly in terms of exploration (increasing informativity of the learning target). The current paper improves upon this by combining IDS (which explores well by taking into account potential information gain and overall performance improvements) with automatically learned targets via Blahut-Arimoto iterations (which solve the rate-distortion trade-off). This leads to a method with a hyper-parameter that allows the designer to put more or less emphasis on keeping the required information acquisition sample complexity low (thus potentially incorporating a range of computational budgets) - the latter of course comes at the cost of potential reductions in performance (returns) of the learned policy compared to an optimal policy (in terms of cumulative regret over the course of learning, but potentially also for the converged solutions if the computational budget is too low). Results are shown for two 50-armed bandits (Bernoulli and Gaussian) where the method compares favorably with other methods (including BLASTS), and on par with IDS on Gaussian bandits.  **Contributions:**  1) Combination of IDS with Blahut-Arimoto learning targets. Both parts are theoretically well motivated and the combination makes sense and leads to a principled theoretical method. To obtain a practical implementation an approximation is needed - the approximation is sensible, though perhaps a bit coarse, but it is theoretically well backed up). Significance: results on the Bernoulli bandit look good, but performance gains compared to IDS don’t carry over to the Gaussian case. Since it remains unclear why, the potential impact remains limited (does the method sound good in theory but easily runs into practical problems, or is there a good reason why we would not expect to see a difference in the Gaussian case?).  2) Solid theoretical foundations - given a constraint on the distortion (corresponding to a value of the hyper-parameter \beta) there is a theoretical guarantee that the learning target is optimal in the sense that there exists no other learning signal target that achieves the same distortion but requires lower rate (“bits extracted from the environment”) - or dually, given a computational budget (set by inverse \beta) no other learning target can lead to higher returns. Significance: given the limited empirical results it remains somewhat unclear to which degree the theoretical results carry over in practice - it cannot be ruled out that (slightly) sub-optimal learning targets could lead to better results in particular cases (e.g. leading to more stable learning, easier to find corresponding policies, etc.). This remains to be seen in the future, but having a solid theoretical background is important - even heuristics might benefit from the theoretical insight presented here.  3) Review of (continuous) rate-distortion theory and the necessary background for the method for a general ML/RL audience (non info-theory experts). Significance: important contribution in itself, though the material is still a bit dense and might need further unpacking / illustrations of the key intuitions for e.g. a general RL practitioner audience. Also, the latter target audience might not be convinced to invest time since the method is currently limited to (non-contextual) bandits.