The Value of Information When Deciding What to Learn
Year: 2021

Abstract:
All sequential decision-making agents explore so as to acquire knowledge about a particular target. It is often the responsibility of the agent designer to construct this target which, in rich and complex environments, constitutes a onerous burden; without full knowledge of the environment itself, a designer may forge a suboptimal learning target that poorly balances the amount of information an agent must acquire to identify the target against the target’s associated performance shortfall. While recent work has developed a connection between learning targets and rate-distortion theory to address this challenge and empower agents that decide what to learn in an automated fashion, the proposed algorithm does not optimally tackle the equally important challenge of efficient information acquisition. In this work, building upon the seminal design principle of information-directed sampling [Russo and Van Roy, 2014], we address this shortcoming directly to couple optimal information acquisition with the optimal design of learning targets. Along the way, we offer new insights into learning targets from the literature on rate-distortion theory before turning to empirical results that confirm the value of information when deciding what to learn.

heading: 1 Introduction
text: Information acquisition is a fundamental consideration in the design of sequential decision-making agents. Indeed, the literature over recent years has unequivocally demonstrated its theoretical [Russo and Van Roy, 2016, 2014, 2018a] and empirical [Osband et al., 2016a,b, Nikolov et al., 2018] importance in addressing the challenge of exploration in sequential decision-making problems. Despite this significance, a strategy for information acquisition, however efficient, is only as good as the information it uncovers. An agent that displays remarkable efficiency in seeking immaterial information offers little utility to its designer. Similarly, when faced with the rich complexity of real-world tasks, a computationally-bounded agent aspiring to learn that which is well beyond its constrained resources is equally ineffective. As a concrete example of the latter scenario, consider learning the optimal policy for the game of Go [Silver et al., 2017] solely relying on the hardware of a mobile device. While not an entirely infeasible task, the amount of time needed to complete such an endeavor may force the agent designer into settling for a small fraction of the optimal policy performance in exchange for a substantial reduction in the difficulty of the learning problem.
Recent work has developed a connection between rate-distortion theory [Shannon, 1959, Berger, 1971] and the problem of deciding what an agent should learn [Arumugam and Van Roy, 2021], optimally balancing between the requisite information needed by any agent to identify a learning target and the corresponding performance shortfall associated with this target. Intuitively, the complexity of a learning problem can be quantified by the bits of information an agent must acquire from the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
environment in order to realize a particular target. Rate-distortion theory offers a principled framework for reasoning about a learning target that comprises the minimum number of bits from the environment while retaining some degree of near-optimality. Arumugam and Van Roy [2021] operationalize the classic Blahut-Arimoto algorithm [Blahut, 1972, Arimoto, 1972] in the multi-armed bandit setting to facilitate computationally-tractable agents that are not only capable of computing such an optimal target at the start of learning but also gradually adapt this target over time as the agent’s knowledge of the environment accumulates. The resulting Blahut-Arimoto Satisficing Thompson Sampling algorithm (BLASTS) then applies the well-known principle of probability matching in order to select actions [Thompson, 1933, Russo et al., 2018]. While BLASTS demonstrates the desired ability to accommodate an agent designer in recovering a wide spectrum of policies with varying levels of performance and learning efficiency, it is well known that the underlying probability matching principle of Thompson Sampling constitutes a sub-optimal exploration strategy [Russo and Van Roy, 2018a].
In this work, we combine the optimal computation of learning targets via rate-distortion theory with optimal information acquisition. To achieve the latter goal, we generalize the information-directed sampling (IDS) principle of Russo and Van Roy [2018a] so that the fundamental information ratio captures expected regret and information gain about a learning target computed via the BlahutArimoto algorithm. As with BLASTS, the resulting algorithms which instantiate this Blahut-Arimoto Information-Directed Sampling principle (BLAIDS) retain a single, real-valued hyperparameter through which an agent designer expresses a preference for the balance struck by the resulting learning target. Moreover, just as BLASTS offers a generalization of Thompson Sampling capable of more efficiently computing optimal policies, we find that a particular computationally-tractable instantiation of BLAIDS mirrors this property relative to its IDS counterpart. En route to deriving BLAIDS, we outline the precise details of the Blahut-Arimoto algorithm itself and connect our setting with various results from the information-theory literature.
The paper proceeds as follows: in Section 2 we offer a general presentation of information-theoretic learning targets for sequential decision-making problems. We then hone in on the multi-armed bandit setting in Section 3 to carefully study incorporating the value of information when acting in pursuit of an arbitrary learning target. We conclude in Section 4 with a corroborating set of computational experiments that demonstrate the efficacy of our approach. Due to space constraints, we relegate background material on information theory, related work, and all technical proofs to the appendix.

heading: 2 Learning Targets
text: This section focuses on a general treatment of learning targets via rate-distortion theory, moving beyond the assumptions of Arumugam and Van Roy [2021] and offering further insight into the mathematical details of the Blahut-Arimoto algorithm.

heading: 2.1 Problem Formulation
text: In this section, we follow the general problem formulation of Lu et al. [2021] that offers a unified treatment of sequential decision-making problems, spanning multi-armed bandits and reinforcementlearning problems [Lattimore and Szepesvári, 2020, Sutton and Barto, 1998]. We adopt a generic agent-environment interface wherein, at each time period, the agent executes an action At ∈ A within an environment E which results in an associated next observation Ot+1 ∈ O. This sequential interaction between agent and environment results in an associated history at each timestep Ht = (A0, O1, . . . , At−1, Ot) ∈ H representing the action-observation sequence available to the agent upon making its selection of At. More formally, we may characterize the overall environment as E = 〈A,O, ρ〉 containing the action set A, observation set O, and the observation function ρ : H×A → ∆(O) that prescribes the distribution over next observations given the current history and action selection:
P(Ot+1|E , Ht, At) = P(Ot+1|Ht, At) Ot+1 ∼ ρ(·|Ht, At).
An agent’s policy π : H → ∆(A) encapsulates the relationship between the histories encountered in each timestep Ht and the executed action At. We maintain that actions are independent of the environment conditioned on history, At ⊥ E|Ht such that π(a|Ht) = P(At = a|Ht) assigns a probability to each action a ∈ A given the history.
An agent designer expresses preferences across histories via a reward function r : H×A×O → R such that an agent enjoys a reward Rt+1 = r(Ht, At, Ot+1) at each timestep. Given any finite time
horizon T < ∞, the accumulation of rewards provide a notion of return T−1∑ t=0 Rt+1. To develop preferences over behaviors, it is natural to associate with each policy π a corresponding expected
return or value across the horizon T as V π = E [ T−1∑ t=0 Rt+1 | E ] , where the expectation integrates over the randomness in the policy π as well as the observation function ρ. A trademark across the sequential decision-making literature is the design of agents that strive to achieve the optimal value within the confines of some policy class Π ⊆ {π : H → ∆(A)}, V ? = sup
π∈Π V π . Notice that when
rewards and the distribution of the next observation Ot+1 depend only on the current observation Ot, rather than the full history Ht, we recover the traditional Markov Decision Process [Bellman, 1957, Puterman, 1994] studied throughout the reinforcement-learning literature [Sutton and Barto, 1998]. Alternatively, when these quantities rely solely upon the most recent action At, we recover the traditional multi-armed bandit [Lattimore and Szepesvári, 2020].

heading: 2.2 The Curse of Curiosity
text: While the design of agents in pursuit of optimal policies is a perfectly natural object of study, it can often occur without regard for the complexity of the environment. In particular, the optimal policy is often a deterministic function of the environment such that, if an agent manages to identify the environment, it possesses all requisite information to determine the optimal policy. An agent designer reflects their initial uncertainty about the environment through a prior distribution P(E ∈ ·); as the history unfolds, what can be learned from the agents current knowledge of the environment is represented by conditional probabilities P(E ∈ ·|Ht). Through this prior distribution, a total of H(E) bits quantify all of the information needed for identifying the environment. Prior work has demonstrated both theoretically and empirically how it is an agent’s incremental resolution of epistemic uncertainty over the environment E that gives rise to the efficient acquisition of these H(E) bits [Chapelle and Li, 2011, Russo and Van Roy, 2016, Osband et al., 2016a, Agrawal and Jia, 2017, O’Donoghue et al., 2018, Osband et al., 2019]. For sufficiently rich and complex environments, however, H(E) can become prohibitively large or even infinite, making the pursuit of an optimal policy entirely intractable.
This issue warrants the construction of a more general mechanism whereby an agent designer may express tractable learning targets, alternatives to the optimal solution that, while potentially incurring some degree of suboptimality, fall within the means of a computationally-bounded agent interacting with a highly-complex environment. Lu et al. [2021] introduce this notion of a learning target χ as a generic random variable and (possibly stochastic) function of the environment E . Their proposed desiderata for the construction of learning target is two-fold: (1) demanding only a feasible amount of information from the environment I(E ;χ) in order to reconcile χ and (2) incurring a bounded degree of performance shortfall by aiming for χ, E[V ? − V πχ ], where πχ is a target policy induced by χ. Observe, however, that this treatment of a learning target assumes its provision to the agent before interaction with the environment has begun. Consequently, the burden of specifying a suitable learning target that best balances between the two desiderata falls upon the agent designer. To remedy this, we begin by extending the formulation of Arumugam and Van Roy [2021] and emphasizing rate-distortion theory as a general tool for the design of optimal learning targets.
It is important to re-iterate and clarify that the role of πχ is not to achieve performance that is on par with the optimal π?, but rather to strike an appropriate balance between performance shortfall relative to π? and the ease of learnability; the latter desideratum is accurately captured mathematically via bits of information. As an agent shifts its focus away from π? towards a χ that is easier to learn (in the information-theoretic sense used throughout this paper), we anticipate greater data efficiency. That is, an agent should be able to more quickly identify the desired χ relative to the requisite amount of time and data needed to identify π?. Naturally, in a complex environment, we expect this data efficiency to be critical and to come at some cost: the sub-optimality of the resulting solution (which we take as the distortion in our rate-distortion framing). An agent designer’s willingness to tolerate this sub-optimality stems from the complexity of the environment and an acknowledgement of the potentially intractable amount of data needed to otherwise identify and recover π?. The more
performance shortfall one is willing to tolerate relative to optimal behavior, the more efficiently an agent should be able to find such a satisficing solution. Russo and Van Roy [2018b] already highlight one example of this phenomenon in the multi-armed bandit setting, where the presence of multiple arms can make pursuit of an ε-optimal arm far more data efficient than the optimal arm. From the perspective of the agent designer, one should not incorporate a learning target with the expectation of being competitive with the optimal policy, but rather with the goal of being more data efficient in the acquisition of the learning target relative to that of the optimal policy.

heading: 2.3 The Rate-Distortion Theory of Learning Targets
text: Rate-distortion theory is a sub-area of information theory that encapsulates the foundations of lossy compression [Shannon, 1959, Berger, 1971, Cover and Thomas, 2012]. Abstractly, ratedistortion theory formulates a lossy compression problem as a constrained optimization in the space of probability kernels or channels, given an information source and a tolerable upper threshold on distortion. The natural goal is to identify a channel that preserves the minimum number of bits from the information source while adhering to the specified distortion upper bound.
More formally, let (Ω,F ,P) be a probability space and consider a random variableX : Ω→ X taking values on the measurable space (X ,X) with an associated marginal distribution PX(·) = P(X−1(·)) that represents an information source. Similarly, define the random variable X̂ : Ω → X̂ that takes values on (X̂ , X̂) and corresponds to a channel output. Given a known, measurable distortion function d : X × X̂ 7→ R≥0 and a desired upper bound on distortion D, the rate-distortion function is defined as R(D) = inf
P X,X̂ ∈Λ I(X; X̂) quantifying the minimum number of bits (on average)
that must be communicated from X across a channel in order to adhere to the specified expected distortion threshold D. Here, the infimum is taken over Λ = {PX,X̂ | ∀A ∈ X : PX,X̂(A× X̂ ) = PX(A) and E[d(X, X̂)] ≤ D} representing the set of all joint distributions on (X × X̂,X × X̂) whose corresponding marginal distribution on X matches the original information source PX while also satisfying the constraint on bounded expected distortion. Intuitively, a higher rate corresponds to requiring more bits of information and smaller information loss between X and X̂ , enabling higher-fidelity reconstruction (lower distortion); conversely, lower rates reflect more substantial information loss, potentially exceeding the tolerance on distortion D. The remainder of this section is devoted to making the mathematical details of this problem precise in our specific context of specifying learning targets. We encourage readers to consult the appendix for the precise details of our notation.
At each timestep, an agent maintains its epistemic uncertainty over the environment via a posterior distribution P(E|Ht). Since the derivation is identical for all timesteps, we suppress time and history from the notation for now and let PE denote this distribution at any arbitrary timestep. We will take the environment E and learning target χ random variables to be associated with measurable spaces (Σ,S) and (Ξ,X), respectively. Let PE,χ defined on (Σ × Ξ,S × X) be a joint distribution over environment-target pairs. Furthermore, let d : Σ× Ξ→ R≥0 be a known, measurable function that quantifies a notion of loss or distortion that occurs by using a particular realization of the learning target within a specific realization of the environment. We may define the rate-distortion function as1
R(D) = inf PE,χ∈Λ:E[d(E,χ)]≤D I(PE,χ) , inf PE,χ∈Λ:E[d(E,χ)]≤D I(E ;χ) (1)
Λ = {PE,χ | ∀A ∈ S : PE,χ(A× Ξ) = PE(A) and ∀t ∈ [T ] : χ ⊥ Ht|E} (2)
where the infimum is taken over the constrained set of joint probability measures Λ such that (1) the E marginal of PE,χ matches the given source distribution PE , (2) the expected distortion does not exceed the given threshold D ∈ R>0, and (3) the target is conditionally independent of all histories given the environment. These first two conditions are typical in rate-distortion theory while the final condition simply ensures that no history can offer more information about a learning target than what is already provided by the environment. While the distortion function d will eventually be defined so as to quantify the performance shortfall of our learning target, we proceed with the generic,
1Typically, R(D) is defined in terms of the mutual information, however the (equivalent) parameterization via the joint distribution PE,χ will be useful in subsequent derivations.
unspecified distortion function for now and assume thatR(D)→ 0 as D →∞. Following from this assumption along with the convexity of mutual information and Gibb’s inequality, we may recover the basic facts that R(D) is a non-negative, convex, and monotonically-decreasing function in its argument [Cover and Thomas, 2012].
The details of computing rate-distortion functions as presented by Arumugam and Van Roy [2021] rely on the classic derivation based on calculus and Lagrange multipliers, under the assumption that the information source and channel output are discrete random variables [Blahut, 1972]. In brief and continuing with the first example of this section, the discrete algorithm proceeds by initializing p0(x̂ | x) = |X̂ |−1, ∀x̂ ∈ X̂ and then iterating the following two updates
qk(x̂) = ∑ x∈X p(x)pk−1(x̂ | x) pk+1(x̂ | x) ∝ qk(x̂) exp (−βd(x, x̂)) ,
where p(x) denotes the information source and β is a Lagrange multiplier. While discreteness carries computational conveniences for practical implementation, here we offer a more general presentation of the Blahut-Arimoto algorithm for arbitrary environments and learning targets. It is important to clarify that the subsequent derivations of this section are classic results in the information-theory literature [Csiszár, 1974, Gray, 2011] and do not represent novel theoretical contributions of this work; that said, we do take our contribution to be distilling such key results from the information-theory literature for broader consumption by the sequential decision-making community and, to that end, offer a full presentation for the benefit of readers.
For the purposes of handling this constrained optimization, we introduce the following objective with parameter β ≥ 0: F(β) = inf
PE,χ∈Λ (I(PE,χ) + βE [d(E , χ)]) = inf PE,χ∈Λ (DKL(PE,χ||PE × Pχ) + βE [d(E , χ)])
As noted by Csiszár [1974], F(β) is the largest vertical-axis intercept of a line with slope −β that includes no point above the rate-distortion curve. Moreover, the rate-distortion function can be expressed in terms of this new objective, sweeping over all possible values of β. Lemma 1 (Lemma 1.2 [Csiszár, 1974]).
R(D) = max β≥0 F(β)− βD
The value of β that achieves this maximum is characterized as being associated with the distortion threshold D. Conversely, a joint distribution PE,χ that achieves the infimum of F(β) has an associate rateR(D) = I(PE,χ) where β is associated with D = EPE,χ [d(E , χ)] , ∫ d(E , χ)dPE,χ(E , χ).
Lemma 1 clarifies an important relationship between the parameter β and the rate-distortion curve, accommodating computation of the rate-distortion function upon specification of a particular β values rather than a distortion upper bound D; in particular, each β corresponds to the slope of a tangent line to the rate-distortion curve such that, given a value of β, the associated joint distribution that achieves the infimum in F(β) generates the point (D,R(D)) on the curve. Per Blahut [1972], computing the infimum in F(β) is handled by an alternating optimization procedure that relies on the following functional:
J (PE,χ,Qχ, β) = DKL(PE,χ||PE ×Qχ) + βEPE,χ [d(E , χ)]
which replaces the marginal distribution over χ induced by PE,χ with an alternative distribution Qχ. When J (PE,χ,Qχ, β) is finite, this implies that PE,χ PE ×Qχ; moreover, if EPE,χ [d(E , χ)] is finite, then PE × Qχ({E , χ|d(E , χ) < ∞}) > 0. Taken together, these facts imply that for any realization of the environment
αQχ,β(E) = [∫ exp (−βd(E , χ)) dQχ(χ) ]−1 <∞
holds PE -almost surely. Thus, for any marginal over learning targets Qχ, we may induce a new joint distribution QE,χ whose corresponding Radon-Nikodym derivative with respect to PE ×Qχ is given by
dQE,χ dPE ×Qχ = αQχ,β(E) exp (−βd(E , χ)) (3)
While the corresponding χ marginal of this new joint distribution need not be identical to Qχ, we can show that our original source distribution PE is preserved under the corresponding E marginal. In particular, for any measurable set A ∈ S, we have
QE,χ(A× Ξ) = ∫
F×Ξ
dQE,χ dPE ×Qχ
dPE ×Qχ(E , χ) = ∫
F×Ξ
αQχ,β(E) exp (−βd(E , χ)) dPE ×Qχ(E , χ)
= ∫ A ( αQχ,β(E) ∫ exp (−βd(E , χ)) dQχ(χ) ) ︸ ︷︷ ︸
=1
dPE(E) = PE(A),
where the second line uses the aforementioned integrability conditions to apply Fubini’s Theorem. To recover an alternating optimization procedure for minimizing our objective and computing the rate-distortion function, we may leverage the following relationship between the original and new joint distributions: Lemma 2 (Lemma 1.2 [Csiszár, 1974]). Let Qχ be an arbitrary marginal distribution over χ. Moreover, let PE,χ be an arbitrary joint distribution over E , χ with an associated marginal distribution Pχ and take QE,χ to be the joint distribution as defined in Equation 3. Then,
J (PE,χ,Qχ, β) ≥ J (PE,χ,Pχ, β) J (PE,χ,Qχ, β) ≥ J (QE,χ,Qχ, β)
Lemma 2 prescribes a natural algorithm for the computation of rate-distortion functions. Upon the provision of a source distribution over environments PE and an initial marginal distribution over learning targets Q(1)χ , one may first compute an updated channel distribution Q(1)E,χ per Equation 3. The induced marginal of Q(1)E,χ serves as the next distribution Q (2) χ , giving rise to another updated channel Q(2)E,χ. Naturally, this process may continue iteratively until convergence. By the second inequality of Lemma 2, the updated channel distributions Q(k)E,χ are non-increasing in the objective and, analogously, the induced marginal distribution is also guaranteed to not increase the objective by the first inequality. This procedure is precisely the classic Blahut-Arimoto algorithm [Blahut, 1972, Arimoto, 1972], generalized beyond the assumptions of discrete source and channel output random variables.

heading: 2.4 Target Policies for Reinforcement Learning
text: With the results of the previous section in hand, an agent may, at any timestep, run the Blahut-Arimoto algorithm using its current posterior beliefs over the environment P(E|Ht) as a source distribution in order to compute a learning target that optimally negotiates between the information required to learn and the resulting target performance shortfall. In the context of reinforcement-learning problems, this learned target χ corresponds to a particular target policy πχ and, to ensure the latter criterion holds, a natural notion of distortion is the expected squared regret between the optimal and target policies: d(E , χ|Ht) = E [ (V ? − V πχ)2|E , Ht ] = E
(Eπχ [ T−1∑ t′=t (V ?(Ht′)−Q?(Ht′ , At)) ])2 ∣∣∣E , Ht  .
The final equality follows from a variant of the performance-difference lemma [Kakade and Langford, 2002] and is proven for our setting as Theorem 1 of Lu et al. [2021]. Unfortunately, while the results of the previous section hold for this particular sequence of rate-distortion functions, the choice of χ as a target policy represents an obstacle to practical implementation. In particular, this would imply that the resulting channel of the Blahut-Arimoto algorithm would map between realizations of the environment E to target policies π : H → ∆(A). Moreover, computation of the distortion associated with each possible realization of the target policy πχ would require either an independent policy-evaluation step to compute the value function V
πχ or demand sampling trajectories of each potential πχ from the environment in order to leverage the final performance shortfall decomposition shown above.
One possible resolution to these issues might include leveraging recent progress in unsupervised skill discovery [Eysenbach et al., 2018] such that potential target policies πχ all come from the resulting parameterized policy class. Computing the associated universal value function approximator [Schaul et al., 2015] for this policy class would allow for efficient computation of the distortion function during the rate-distortion optimization. As an alternative, one might further restrict focus to a tabular Markov Decision Process, at which point the target policy in question can be expressed as a finite sequence of action random variables (one per state); it may then be possible to “stitch” πχ from the solutions to each of the state-dependent rate-distortion functions.
For now, we leave the question of how to adaptively compute optimal target policies in a computationally-tractable manner to future work and, instead, turn our attention to the multi-armed bandit setting where we may more feasibly study the coupling of optimal learning targets with optimal information acquisition.

heading: 3 Target Actions & The Value of Information
text: In order to develop a computationally-tractable algorithm for combining learning targets with IDS, this section focuses on multi-armed bandit problems where we take the learning target χ to be a target action the agent aims to identify from the environment.

heading: 3.1 Blahut-Arimoto Satisficing Thompson Sampling
text: We begin by taking a closer look at the design of BLASTS and examine the extent to which learning targets computed via the Blahut-Arimoto algorithm are preferable to hand-crafted learning targets. As a first observation, we recall that while BLASTS does employ the Blahut-Arimoto algorithm as described in the previous section, the function computed by Arumugam and Van Roy [2021] is actually the so-called “plug-in” estimator for the rate-distortion function [Harrison and Kontoyiannis, 2008]; rather than directly using the agent’s representation of environment uncertainty as an information source, BLASTS draws a fixed number of posterior samples z and uses the resulting empirical distribution as an input source to the Blahut-Arimoto algorithm. Such an approach is amenable to scenarios where an agent is only able to sample from its beliefs over the environment but cannot reliably provide estimates of likelihoods [Osband et al., 2016a, Lu and Van Roy, 2017].
Clearly, as z → ∞, the empirical distribution converges in probability to the true distribution and yet, implicit in the design of BLASTS is an assumption that a similar convergence statement holds for the plug-in estimator and the true rate-distortion function R(D). Fortunately, Harrison and Kontoyiannis [2008] have already proven general consistency results for the plug-in estimator of the rate-distortion function in two distinct settings of interest to sequential decision-making problems: (1) under a finite action set and (2) when the action set is a compact, separable metric space such that for each realization of the environment E , d(E , ·) is continuous; naturally, the latter condition ensures consistency even under a continuous action space, although running the Blahut-Arimoto algorithm to optimize such a continuous channel output is non-trivial [Dauwels, 2005].
While the previous result is encouraging, it immediately begs the question of how many posterior samples z are needed for the plug-in estimator R̂(D) to, with high probability, be an ε-accurate approximation of the true rate-distortion functionR(D)? While Palaiyanur and Sahai [2008] offer an answer to this question in their study of the uniform continuity of the rate-distortion function for discrete random variables, their analysis and corresponding sample-complexity bounds depend on a problem-specific parameter D̃ = min
(E,χ):d(E,χ)>0 d(E , χ) denoting the minimum non-zero distortion
achieved by all environment-target pairs. While this constant does make sense in an informationtheoretic context (for instance, under Hamming distortion, D̃ = 1), minimizing over all possible environments under a notion of distortion commensurate with expected squared regret engenders exceedingly small values of D̃, rendering the associated bounds vacuous. To remedy this issue, we consider a minor modification to the statement of Lemma 2 by Palaiyanur and Sahai [2008], replacing their more general result that carries an unfavorable dependence on D̃ with a less general result in terms of
D(PE) = min E:PE(E)>0 min χ:d(E,χ)>0 d(E , χ)
When the distortion function is aligned with the regret between the optimal action and target action, D(PE) corresponds to the worst-case action gap between the best and second-best actions [Farahmand, 2011, Agrawal and Goyal, 2013, Bellemare et al., 2016] based on the agent’s current posterior over the environment. It is important to note that while our adjustment to Lemma 2 (and its corresponding effect on Lemma 5) of Palaiyanur and Sahai [2008] carries meaningful semantics for our sequential decision-making setting, the proof of the modified result still follows exactly as in their original paper2, which leverages the uniform continuity of entropy. Lemma 3 (Lemma 2 - [Palaiyanur and Sahai, 2008]). Let E , χ be discrete random variables. Let PE denote the agent’s current posterior over E and let P̂E be the empirical distribution. If ||PE − P̂E ||1 ≤ D(PE)
4 , then for any D ≥ 0,
|R(D)− R̂(D)| ≤ 7 D(PE) ||PE − P̂E ||1 log
( |Σ||Ξ|
||PE − P̂E ||1
) .
Corollary 1 (Lemma 5 - [Palaiyanur and Sahai, 2008]). For any δ ∈ (0, 1), ε ∈ (0, log(Σ)) let PE ∈ ∆(Σ) be the current posterior over E . Additionally, let φ−1 denote the inverse of the function φ : [0, 12 ]→ R where φ(t) = t log ( |Σ||Ξ| t ) . If
z ≥ 2 φ−1 ( εD(PE)
7
)2 (log(1δ ) + |Σ| log(2) ) , then P(|R(D)− R̂(D)| ≥ ε) ≤ δ.
While the results discussed so far provide insight into the Blahut-Arimoto algorithm itself, it is worth examining the extent to which learning targets computed via rate-distortion theory improve over a hand-crafted learning target. While such an improvement is certainly expected based on the theoretical analyses of [Russo and Van Roy, 2018b, Arumugam and Van Roy, 2021], it has not yet been demonstrated empirically. Due to space constraints, we defer a presentation of such empirical results that elucidate the value of computing learning targets via rate-distortion theory to the appendix.

heading: 3.2 Blahut-Arimoto Information-Directed Sampling
text: While BLASTS is successful in leveraging optimal learning targets, we consider improving upon the exploration technique used for collecting information about those targets through the algorithmic design principle of information-directed sampling [Russo and Van Roy, 2018a]. Information-directed sampling (IDS) is an abstract objective for sequential decision-making agents where, at each time period, an agent computes a policy based on the current history Ht that minimizes the information ratio
πt = min π∈∆(A)
Eπ [ EE [ R(A?)−R(At) | Ht ]]2 I(A?;At, Ot+1|Ht = Ht) .
Here, R(·) denotes the mean reward associated with an input action. In words, the information ratio of a particular policy π ∈ ∆(A) weighs the squared expected regret of action At ∼ π relative to the optimal action A? against the expected information gain about A? by taking At. Note that the information gain term I(A?;At, Ot+1|Ht = Ht) conditions upon the particular realization of the agent’s history at time period t, Ht. Relating this quantity to the more traditional notion of conditional mutual information only requires integrating over the randomness in Ht: E [I(A?;At, Ot+1|Ht = Ht)] = I(A?;At, Ot+1|Ht). For our purposes, we consider a modified information ratio that leverages the output of the Blahut-Arimoto algorithm.
In particular, we define Blahut-Arimoto Information-Directed Sampling (BLAIDS) as a general recipe for efficient exploration with optimal learning targets wherein an agent first uses its current posterior as a source distribution, computing a learning target χ via the Blahut-Arimoto algorithm. Subsequently, the agent employs IDS to balance optimal behavior and information gain about this target by minimizing the following information ratio
Ψt(π) = Eπ [∆t(a)]2
I(χ;At, Ot+1|Ht = Ht) ∆t(a) = EE,χ
[ R(χ)−R(a) | Ht ] ,
2Under a standard assumption that rewards are bounded in [0, 1] and d(E , χ) is a function of regret, the d? term that appears in their original bound is equal to 1.
where the expectation is with respect to the joint distribution computed by the Blahut-Arimoto algorithm. Observe that the distortion function used here is given by d(E , χ | Ht) = d(E , Aχ | Ht) = E[(R(A?)−R(Aχ))2 | E , Ht] where, conditioned on a realization of the environment E , we may compute R(·) [Arumugam and Van Roy, 2021]. One perspective on BLAIDS is that IDS, with its default focus on A?, will spend time acquiring too much information relative to BLAIDS that will adaptively guide IDS towards uncovering the minimum number of bits needed to achieve a desired level of sub-optimality. Naturally, BLAIDS is a generalization of IDS since, as β →∞, the BlahutArimoto algorithm computes the optimal action A? for each realization of the environment from the input source distribution. Just as with IDS, BLAIDS is a design principle whose abstract objective must be made explicit in a computationally-tractable manner in order to yield a corresponding algorithm. One standard choice for IDS is to leverage a lower bound of mutual information by the variance in rewards [Russo and Van Roy, 2018a, Nikolov et al., 2018, Dwaracherla et al., 2020, Lu et al., 2021]. This results in an upper bound on the per-period information ratio that is suitable for minimization:
Ψt(π) ≤ Eπ [∆t(At)]2
Eπ [vt(At)] vt(At) = V
[ E [ R(At)|χ,Ht ] |Ht ] .
To see how this upper bound lends itself to a computationally-tractable instantiation of IDS, observe that with z posterior samples, the Blahut-Arimoto algorithm results in a distribution over learning targets conditioned on each posterior sample of the environment p(χ = ã|E = e). Denoting the induced marginal over learning targets as q(ã), we have
vt(At = a) = ∑ ã∈A q(ã)
( 1
z ∑ e p(ã|e) q(ã) Et [ R(At)|χ = a, E = e ] − 1 z ∑ e Et [ R(At)|E = e
])2 .
Using this lower bound, we recover variance-BLAIDS as a computationally-tractable analogue to variance-IDS. Once again, to see that variance-BLAIDS generalizes variance-IDS note that as β → ∞, recovery of the optimal action by the Blahut-Arimoto algorithm implies that for any realization of the environment and any ã ∈ A, p(ã|E = e) ∈ {0, 1}, identifying if ã is optimal in that realization of the environment. Consequently,
∑ E p(ã|E) = |Ea?=ã| and q(ã) = z−1|Ea?=ã|, where
E?a?=ã denotes a partition of the posterior samples such that a? ∈ A is optimal for all environments e ∈ Ea?=ã. In the next section, we assess the extent to which variance-BLAIDS improves upon standard variance-IDS through the use of optimal learning targets.

heading: 4 Experiments
text: In our experimental setup, we examine independent Bernoulli and Gaussian bandit problems, sweeping over values of β in order to examine the spectrum of policies induced by the various learning targets. Similar to BLASTS, BLAIDS is agnostic to the particular mechanism used for representing an agent’s epistemic uncertainty. For our experiments, we found the use of a linear hypermodel [Dwaracherla et al., 2020] to be sufficient when optimized via Adam [Kingma and Ba, 2014] (learning rate of 0.001) with a noise variance of 0.1, prior variance of 1.0, and batch size of 1024. We adopt all other hyperparameters of [Arumugam and Van Roy, 2021] for running the Blahut-Arimoto algorithm [James et al., 2018]3 in each time period and run experiments via Google Colab with a GPU. In Figure 1, we report the cumulative regret in each time period across Thompson Sampling (TS), BLASTS, variance-IDS (V-IDS), and variance-BLAIDS (V-BLAIDS) where shading denotes 95% confidence intervals averaged over 10 random trials. For clarity, we split the figures in each problem between those values of β which manage to achieve an optimal policy and those which learn a sub-optimal policy.
Our empirical results highlight opportunities for more efficient learning in light of informationdirected exploration. Notably, the gap between BLASTS and variance-BLAIDS widens as β increases, asserting more importance in pursuing a near-optimal learning target. Curiously, while the Bernoulli bandit does show small performance gains relative to variance-IDS when lowering β, such improvements do not appear in the Gaussian setting and present an interesting direction for future work to investigate the nature of information structures where learning an optimal policy can be improved through automated learning targets. That said, it is worth noting that the prior variance used in our experiments already matches the true variance of the environment and so, under a less-calibrated prior, there might be greater opportunity for variance-BLAIDS to shine over variance-IDS.
As in Arumugam and Van Roy [2021], we also entertain the idea of an intimate relationship between β and the (best) information ratio Ψt = min
π∈∆(A) Eπ [∆t(a)]2 I(A?;At,Ot+1|Ht=Ht) . We also find that the use of
an adaptive βt = Ψ −1 t yields the best performance in both problem classes for learning an optimal policy. While the existence of a meaningful correspondence between β and Ψt seems natural given their respective roles and aligned units (bits per squared unit of regret for the former and squared units of regret per bit for the latter), it is unclear if and when this scaling is correct. On the whole, these results serve as simple, fundamental sanity checks that confirm the importance of prudent information acquisition, even when in pursuit of a learning target besides the optimal action.

heading: 5 Conclusion
text: Learning targets are a general mechanism for guiding the information sought out by a sequential decision-making agent. As environments continue to grow in complexity so as to meet the demands of more challenging real-world tasks, so too will the need for learning targets that fall within the means of computationally-bounded agents. This work falls in with a new direction that leverages rate-distortion theory in order to forge such optimal learning targets and asks how such targets couple with existing methods for optimal information acquisition. Our work confirms that the computationally-tractable approaches for computing optimal learning targets are compatible with the well-studied principle of information-directed sampling. While we have found suitable answers for the multi-armed bandit setting, much remains to be understood about the nature of these learning targets themselves as well as how to extend these algorithms to handle target policies and other richer notions of learning objectives.
Acknowledgments and Disclosure of Funding
Financial support from Army Research Office (ARO) grant W911NF2010055 is gratefully acknowledged.
3https://github.com/dit/dit

Review summaries:
**Update after author response and discussion**: I am happy to see most of my assumptions positively confirmed and clarified by the authors, as well as the criticism by other reviewers being sufficiently addressed. Some of my improvements remain valid, but I do not consider them crucial for publication. I therefore stick with my original score but raise my confidence - I am in favor of accepting the paper.  **High-level summary**  The paper addresses the question of learning to act well in finite-horizon sequential tasks in unknown environments under computational limitations. This requires solving two problems. (1) In principle, full knowledge about the environment allows agents to act optimally. However, actions that lead to large information gain about the environment often come at the expense of incurring low immediate rewards - trading off these two considerations is non-trivial and has been formalized previously via information directed sampling (IDS), which aims to find policies that minimize the ratio between squared expected regret (of the policy compared to the optimal policy) and information gain (ideally leading to small regret and large information gain). (2) Reducing all (epistemic) uncertainty about the environment can incur a considerable amount of exploratory actions, particularly in complex environments. Often, a less informative (lossy) compression of the environment allows for learning policies that perform very well with considerably lower “exploration cost” (which is particularly relevant for computationally bounded agents). Such a compression has been previously introduced under the name ‘learning target’ which is computed via a (stochastic) mapping from the environment to the learning target random variable. A good learning target requires considerably less information (fewer bits) to be extracted from the environment (compared to reducing all uncertainty about the full environment) while at the same time ensuring that policies trained to maximize the learning target achieve good returns in the actual environments. This trade-off between low reduction in performance and high reduction in sample complexity of exploration has recently been formalized via a rate distortion theory, where the distortion is the expected regret between the policy trained on the learning target and the optimal policy (given an observation history), and the rate is the information that needs to be extracted from the environment to reduce uncertainty about the learning target variable sufficiently.  **Concrete summary**  Previous work (BLASTS) derived an improved Thompson sampling scheme based on learning targets obtained by solving the rate-distortion trade-off via Blahut-Arimoto iterations. While BLASTS provides a principled solution to extract good learning targets, the resulting policies are understood to perform poorly in terms of exploration (increasing informativity of the learning target). The current paper improves upon this by combining IDS (which explores well by taking into account potential information gain and overall performance improvements) with automatically learned targets via Blahut-Arimoto iterations (which solve the rate-distortion trade-off). This leads to a method with a hyper-parameter that allows the designer to put more or less emphasis on keeping the required information acquisition sample complexity low (thus potentially incorporating a range of computational budgets) - the latter of course comes at the cost of potential reductions in performance (returns) of the learned policy compared to an optimal policy (in terms of cumulative regret over the course of learning, but potentially also for the converged solutions if the computational budget is too low). Results are shown for two 50-armed bandits (Bernoulli and Gaussian) where the method compares favorably with other methods (including BLASTS), and on par with IDS on Gaussian bandits.  **Contributions:**  1) Combination of IDS with Blahut-Arimoto learning targets. Both parts are theoretically well motivated and the combination makes sense and leads to a principled theoretical method. To obtain a practical implementation an approximation is needed - the approximation is sensible, though perhaps a bit coarse, but it is theoretically well backed up). Significance: results on the Bernoulli bandit look good, but performance gains compared to IDS don’t carry over to the Gaussian case. Since it remains unclear why, the potential impact remains limited (does the method sound good in theory but easily runs into practical problems, or is there a good reason why we would not expect to see a difference in the Gaussian case?).  2) Solid theoretical foundations - given a constraint on the distortion (corresponding to a value of the hyper-parameter \beta) there is a theoretical guarantee that the learning target is optimal in the sense that there exists no other learning signal target that achieves the same distortion but requires lower rate (“bits extracted from the environment”) - or dually, given a computational budget (set by inverse \beta) no other learning target can lead to higher returns. Significance: given the limited empirical results it remains somewhat unclear to which degree the theoretical results carry over in practice - it cannot be ruled out that (slightly) sub-optimal learning targets could lead to better results in particular cases (e.g. leading to more stable learning, easier to find corresponding policies, etc.). This remains to be seen in the future, but having a solid theoretical background is important - even heuristics might benefit from the theoretical insight presented here.  3) Review of (continuous) rate-distortion theory and the necessary background for the method for a general ML/RL audience (non info-theory experts). Significance: important contribution in itself, though the material is still a bit dense and might need further unpacking / illustrations of the key intuitions for e.g. a general RL practitioner audience. Also, the latter target audience might not be convinced to invest time since the method is currently limited to (non-contextual) bandits.

In reinforcement learning, the ideal thing to do would be to learn all of the information there is about the environment, and then use that to execute the optimal policy. However, in practice, it is impossible to learn _all_ of the necessary information in very complicated environments. Thus, we instead would like to define a _learning target_ that only incentivizes learning a relevant subset of information that is useful for acquiring reward. The learning target must be chosen such that (a) it does not require too much information to learn, and (b) the optimal policy given only the information in the learning target has low regret (relative to the optimal policy given arbitrary information).  So far learning targets have been coded by hand. Ideally we would instead select an appropriate learning target automatically, that outperforms targets coded by hand. The core insight is that constraint (a) is simply lossy compression, where constraint (b) can be thought of as a constraint on the _distortion_ on regret from throwing away some information in the lossy compression. We can then apply standard approaches from rate distortion theory to choose a target that best fits both constraints.  Once we have selected an appropriate target, we must also choose actions that provide us information about the target. This can be done through _information-directed sampling_, in which we choose actions that minimize the ratio between (expected) regret incurred and information gained about the learning target.  The authors instantiate this framework in the case of multiarmed bandits, where the learning targets are environment-conditional policies $p(a \mid \mathcal{E})$. In the case of zero distortion, the learning target is the optimal environment-conditional policy (which obtains zero regret). As distortion increases, the learning target becomes more and more “smeared” (e.g. in environments where the best and second-best arms are very close in reward, the learning target might put similar probabilities on both arms), making it easier to learn. They show that the resulting algorithm outperforms other alternatives that don’t use automatically generated learning targets and/or information-directed sampling.

This paper proposes the use of Blahut-Arimoto algorithm to learn an encoder from the source to the target, where the source is the environment and the target the policy to be recovered under a budget of distortion / suboptimality (this budget can be seen as a bounded-rationality). The output of BA algorithm would be a satisficing (good enough) policy, satisfying the budget constraints but also minimizing the mutual information between the policy and the environment. With this paper, the authors propose an improvement over Blahut-Arimoto Thompson Sampling method by using Information Directed Sampling in combination with Blahut-Arimoto algorithm (to compute target action of corresponding source environment). Finally, they provide an additional version based on the variance of the reward as a tractable approximation of the mutual information denominator.

This papers studies the fundamental problem in sequential decision making problems related to exploration. Previous work have used Thomson sampling in these cases however, they are suboptimal. This work uses rate distortion theory and in generalizes previous Blahut-Aritomoto and Information-directed sampling  algorithms with a novel BLAIDS approach. I have some suggestions in terms of presentation to improve readability and clarity.