Bayesian Active Meta-Learning under Prior Misspecification

Sabina J. Sloman 1 Ayush Bharti 2 Samuel Kaski 1 2

Abstract
We study a setting in which an active meta-learner
aims to separate the idiosyncracies of a particular
task environment from information that will transfer between task environments. In a Bayesian setting, this is accomplished by leveraging a prior distribution on the amount of transferable and taskspecific information an observation will yield, inducing a large dependency on this prior when datais scarce or environments change frequently. However, a misspecified prior can lead to bias in theinferences made on the basis of the resulting posterior — i.e., to the acquisition of non-transferable
information. For an active meta-learner, this posesa dilemma: should they seek transferable information on the basis of their possibly misspecified prior beliefs, or task-specific information thatenables better identification of the current task
environment? Using the framework of Bayesian
experimental design, we develop a novel diagnostic to detect the risk of non-transferable information acquisition, and leverage this diagnostic to propose an intuitive yet principled way tonavigate the meta-learning dilemma — namely,
seek task-specific information when there is riskof non-transferable information acquisition, andtransferable information otherwise. We provide a
proof-of-concept of our approach in the context
of an experiment with synthetic participants.

1. Introduction
Most theoretical results in machine learning assume learning algorithms have access to substantial amounts of datafrom the environments in which they will be deployed. Yet
the daunting task in the settings faced by many algorithms inpractice is to bootstrap very little data from several different
1
Department of Computer Science, University of Manchester,Manchester, United Kingdom 2 Department of Computer Science,
Aalto University, Helsinki, Finland. Correspondence to: Sabina J.
Sloman <sabina.sloman@manchester.ac.uk>.

Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023, Honolulu, Hawaii, USA. PMLR 202, 2023. Copyright2023 by the author(s).

task environments to make robust predictions in new environments. This is the problem of transfer or meta-learning1 ,a paradigm in which the learner’s goal is to perform wellin multiple different, but related, task environments (Pan &Yang, 2009; Kveton et al., 2021; Simchowitz et al., 2021).
In the specific setting we study, each task environment thelearner will encounter is characterized by both transferable
effects (i.e., parameter values that are constant across task
environments) and task-specific effects (i.e., parameter values that vary between task environments). The goal of themeta-learner is then to disentangle the transferable from thetask-specific effects.
Our work is especially motivated by the paradigm of learning with human feedback: algorithms that rely on expert
feedback, or “human-in-the-loop learning,” are inherently
constrained by limitations in the time, attention and knowledge of a given expert. While bringing multiple humans
into the loop can overcome these limitations, eliciting feedback from more than one expert requires that the learnerdisentangle transferable knowledge shared between experts
from the biases and idiosyncracies of each.
Bayesian learning algorithms, which use prior information
to make robust inferences in data-scarce settings, are a natural paradigm for such problems (Simchowitz et al., 2021).
We analyze the setting of Bayesian meta-learning underprior misspecification: a learner specifies a prior distribution across a parameter space that includes both transferable
and task-specific parameters, and this prior can diverge fromthe true parameter distribution. We leverage the frameworkof Bayesian experimental design (Rainforth et al., 2023;
Valentin et al., 2023) to quantify the amount of information
gained about the value of a transferable parameter. To thebest of our knowledge, we are the first to use the frameworkof Bayesian experimental design to analyze the general setting of Bayesian meta-learning under prior misspecification(although related work has studied it for special cases; see
Section 4). As we show in Section 1, misspecified priorinformation about the task-specific characteristics of a giventask environment can affect the degree of transferable information (information that will transfer to new environments)
1

We use the convention that definitions in italics refer to terms
that exist in the literature, while definitions in bold refer to novelterms introduced by this work.

Bayesian Active Meta-Learning under Prior Misspecification

ψ1

ψ2

θ

Source task

2

Target task

Figure 1. Problem setup. The value θ of the transferable parameteris shared across tasks, while the value of the task-specific parameterψ differs between tasks. The goal of the active meta-learner is tolearn to take effective actions in the target task by actively eliciting
data from the source task in order to maximize the information
gained about the value of θ.

a learner can acquire. In some cases, misspecified priors caneven lead to non-transferable information acquisition, or
to evidence in favor of the wrong transferable parametervalue.
Contributions. In this paper, we leverage a Bayesian
experimental design framework to analyze the setting ofBayesian meta-learning under prior misspecification. We
introduce the meta-learning dilemma: should an active metalearner seek transferable information on the basis of their
possibly misspecified prior beliefs, or task-specific information that enables better identification of the current task environment? We introduce a credible lower bound on a measure of expected transferable information gain (CLBETIG ),and demonstrate its use as a diagnostic of the risk of nontransferable information acquisition. We then demonstrate
an application of CLBETIG in the design of a Bayesian
active learning algorithm for navigating the meta-learning
dilemma.
Setting. Our setup is summarized in Figure 1. The metalearner can take actions x ∈ X. Each action results inan outcome that can take values y ∈ Y , which follows a
distribution determined by x and the value ω ∈ Ω of a setof unobservable parameters. Ω is partitioned into a set oftransferable parameters θ ∈ Θ and a set of task-specificparameters ψ ∈ Ψ. The values of the transferable parameters are stable across task environments, while the valuesof the task-specific parameters differ in each environment
the learner will encounter. The learner’s goal is to identify
the true value of θ, which we denote θ∗ . In each task thatthe learner encounters, a value of ψ is drawn independently
at random from a true distribution that characterizes the
population of tasks. Together, a value (x, θ, ψ) induces a
distribution over outcomes p(y | x, θ, ψ). This conditional
distribution is assumed to be known to the learner.
However, the learner does not have access to the value θ∗or to the true distribution of ψ, and so assigns to it a priordistribution p(θ, ψ). We will hereafter use q to refer toprobability density functions induced by the true population
distribution of ψ, which is unavailable to the learner. We

Figure 2. A misspecified prior can lead to non-transferable information acquisition. Under the learner’s prior (dashed line), valuesof ψ are concentrated in the upper half of the y-axis, and large
values of y occur most frequently when θ is large. Under the true
distribution (solid line), values of ψ are concentrated in the lower
half of the y-axis. If the learner had access to this prior, they would
consider observations of large values of y as indicative of a smallvalue of θ.

reserve p to refer to density functions used internally by
the learning algorithm. The true distribution over outcomes
induced by marginalizing
R over the parameter space can be
written as q(y | x) := Ψ p(y | x, θ∗ , ψ) q(ψ | θ∗ ). The
learner’s expectationsof the distribution over outcomes is
R R
p(y | x) := Θ Ψ p(y | x, θ, ψ) p(ψ | θ) p(θ).
After observing an action–outcome pair (x, y), the learner
updates the probability distribution p(θ, ψ) according top(θ,ψ)
Bayes’ rule, i.e., p(θ, ψ | y, x) = p(y | x,θ,ψ).
p(y | x)Meta-learning dilemma. To see how misspecification of
the prior over ψ can lead to non-transferable informationacquisition, consider the toy example shown in Figure 2.
The shading shows the probability y = 1 | x under different
values of θ and ψ. The dashed line shows a particular priordistribution one could place on ψ. Under this prior, if thelearner observes y = 1 | x, this is considered evidence for a
relatively high value of θ. Conversely, if the learner observes
y = 0 | x, this is considered evidence for a relatively low
value of θ. However, if the true distribution over ψ is in fact
characterized by the solid line, these observations would
instead be indicative of a relatively low (high) value of θ. In
other words, the effect of misspecification over ψ is to leadto initial convergence on a non-transferable value of θ.
This example highlights a trade-off between learning taskspecific and transferable information. Failing to invest resources in learning about ψ would lead to wrong inferences
about θ. Yet if the learner invests too many resources inlearning about ψ, they will not learn any information thatwill transfer to a new environment. This presents the learnerBayesian Active Meta-Learning under Prior Misspecification

with a meta-learning dilemma which requires striking abalance between seeking transferable and task-specific information.

2. Sequential optimal experimental design
Sequential optimal experimental design (sOED) is an active
learning paradigm in which the input x is selected to maximize the mutual information between the observations it
induces y | x, and the value of parameters ω. sOED leverages the Bayesian modeling framework to define the mutual
information measure, or expected information gain (EIG)
associated with a possible input:
p(ω | y, x)
EIG(x) = Eω∼p(ω) Ey∼p(y | x,ω) log
p(ω)






p(y | x, ω) = Eω∼p(ω) Ey∼p(y | x,ω) logp(y | x) 


{z
}
|
Information gain

3

tion (2) requires estimation of an implicit likelihood (Foster
et al., 2019).
In the same way that we refer to the information gain of aparticular set of values (y | x, ω), we also refer to the transferable information gain (TIG) of a set of values (y | x, θ).
| y,x)| x,θ)The TIG, or log p(θp(θ)= log p(yp(y | x) , is the amount an
observation y | x increases the learner’s beliefs in the value
θ. The interpretation parallels that of the information gain in
the context of the EIG: TIG is positive when θ is more likely
under the posterior p(θ | y, x) than under the corresponding
prior p(θ).
We define the task-specific information gain and expectedtask-specific information gain analogously, i.e., as theinformation an observation provides about a value ψ andas the learner’s expectation of the task-specific information
gain across their prior, respectively:p(y | x, ψ)
ETSIG(x) = Eψ∼p(ψ) Ey∼p(y | x,ψ) log
p(y | x)




(1)
Notice that the EIG is necessarily a function of the learner’s
prior p(y | x, ω), since the true distribution q(y | x, ω) is
unavailable in practice.
| y,x)We refer to the inner term log p(ωp(ω)| x,ω)
log p(yp(y | x)=as the information gain about ω: it is positive when ω ismore likely under the posterior p(ω | y, x) than under thecorresponding prior p(ω).
2.1. Expected Transferable and Task-specificInformation Gain

The meta-learner’s goal is to maximize information gain
about the transferable parameters θ. The more general setting of targeting inference at a subset of the parameters isreferred to as the estimation of nuisance parameters (Foster
et al., 2019) or of an embedded model (Rainforth et al.,2023).
We here distinguish this case by referring to the expected
information gain measure computed with respect to θ as the
expected transferable information gain (ETIG):p(y | x, θ)ETIG(x) = Eθ∼p(θ) Ey∼p(y | x,θ) log
p(y | x)






= Eθ,ψ∼p(θ,ψ) Ey∼p(y | x,θ,ψ) p(y | x, θ)
log
p(y | x)
|
{z
}





Transferable Information Gain(2)
where p(y | x, θ) = Eψ∼p(ψ | θ) [p(y | x, θ, ψ)]. This is
not usually known in closed form, and so evaluating Equa-= Eθ,ψ∼p(θ,ψ) Ey∼p(y | x,θ,ψ) 

p(y | x, ψ)
log
p(y | x)
|
{z
}






Task-specific Information Gain

(3)

3. Sequential optimal experimental design formeta-learning
We now leverage the sOED framework to make more precise the nature of the meta-learning dilemma and develop
tools to navigate it. Section 3.1 introduces the concept ofactual information gain, which will allow us to formalize theconcepts of transferable vs. non-transferable informationacquisition. Section 3.2 introduces a novel diagnostic for
detecting the risk of non-transferable information acquisition. Section 3.3 introduces an application of this diagnostic
to the construction of an sOED method to trade-off between
seeking transferable and task-specific information in a principled way. Section 3.4 gives details of a method to estimate
the ETIG and ETSIG. Section 5 will then combine thesecomponents to provide a proof-of-concept implementationof our proposed approach to navigating the meta-learning
dilemma in the context of an experiment designed to learntransferable information from the behavior of several synthetic participants.
3.1. Transferable vs. non-transferable information
EIG and ETIG give the expectation of the relevant information gain measure under the learner’s prior. In the caseof prior misspecification, these may diverge from the actual average information gain the learner would achieve in
multiple task environments drawn from the true distribution.

Bayesian Active Meta-Learning under Prior Misspecification

This motivates our measure of actual information gain
(AIG):
p(y | x, ω)
AIG(x) = Eω∼q(ω) Ey∼q(y | x,ω) log
p(y | x)(4)
which differs from the EIG (Equation (1)) in that the outerdistribution q(ω, y | x) is the true distribution.
Equation (4) can be rewritten as
AIG(x) = Eω∼q(ω) [DKL (q(y | x, ω) || p(y | x)) −DKL (q(y | x, ω) || p(y | x, ω))] (5)The derivation of Equation (5) is given in Appendix A.
This can yield insights into when the learner should expect
gains in transferable information even under prior misspecification: in the case where the likelihood is known, theterm DKL (q(y | x, ω) || p(y | x, ω)) is zero, and the AIG reduces to an expectation over a Kullback-Leibler divergenceand thus must be non-negative — indicating acquisition oftransferable information.
Like the EIG, the AIG can be straightforwardly extended tothe meta-learning case. The Actual Transferable Information Gain (ATIG) isp(y | x, θ)ATIG(x) = Eθ∼q(θ) Ey∼q(y | x,θ) log
p(y | x)= Eθ∼q(θ) [DKL (q(y | x, θ) || p(y | x)) −DKL (q(y | x, θ) || p(y | x, θ))] .

(6)

Here, the likelihood p(y | x, θ) can also be misspecified:Eψ ∼q(ψ | θ) [q(y | x, θ, ψ)] = Eψ ∼q(ψ | θ) [p(y | x, θ, ψ)]2
is not necessarily equal to Eψ ∼p(ψ | θ) [p(y | x, θ, ψ)].
Thus, unlike in the standard case, the second term inside
the expectation (here, DKL (q(y | x, θ) || p(y | x, θ))) does
not drop out, yielding no general guarantees about the signof the ATIG. In other words, a learner operating under amisspecified prior may acquire evidence for the wrong valueof the transferable parameter. We refer to this as acquiring
non-transferable (as opposed to transferable) information. The sign of the ATIG can be interpreted as an indicatorof whether the learner is acquiring transferable (positive
ATIG) or non-transferable (negative ATIG) information.

4

3.2. ETIG credible lower boundOf course, the learner does not have access to the true distribution and so cannot compute the ATIG. Here, we introduce
the ETIG credible lower bound (CLBETIG ), a diagnostic
that can be computed without knowledge of the true distribution but which can nevertheless indicate when the learner
may be in danger of acquiring non-transferable information.
Notice that the ETIG (Equation (2)) and ATIG (Equation (6))
are both written as an expectation over the TIG, where the
ETIG is evaluated by computing the expectation with respect to the learner’s prior p, and the ATIG is evaluated bycomputing the expectation with respect to the true distribution q. In other words, both can be estimated by computing
the TIG corresponding to a sufficiently large number of
combinations (θ, y | x) and then taking an average weighted
by the appropriate distribution. Given infinite samples of(θ, y | x) and the corresponding TIG values, one of thesesamples would correspond to the actual (as yet undetermined) future observation y | x and true parameter value θ∗ .
The corresponding TIG value, which we refer to as TIG∗ ,would indicate the amount of transferable information gain
the learner would obtain after collecting their data. Quantifying the risk of non-transferable information acquisition
then amounts to determining how likely it is that TIG∗ isnegative.
The intuition behind our approach is to leverage the empirical distribution of TIG values that the learner computes
to estimate the ETIG in order to additionally compute a
credible lower bound on the value of TIG∗ . The ETIG is
estimated as the mean of this empirical distribution of TIG
values, and the ETIG credible lower bound (CLBETIG )can be interpreted as the lower bound of a credible interval
constructed around that mean. If the value of CLBETIG at aparticular value of x is negative, that indicates that collecting
observations induced by x poses a risk of non-transferable
information acquisition.
More specifically, the CLBETIG is computed as:p(y | x, θ)
CLBETIG (x) = Eθ,y | x∼p(θ,y | x) log
p(y | x)sp(y | x, θ)− β Varθ,y | x∼p(θ,y | x) log(7)p(y | x)Equation (6) also reveals the nature of the dilemma
the meta-learner faces: not only are they tasked
with identifying designs that lead each transferable parameter value to make distinct predictions (maximizing DKL (q(y | x, θ) || p(y | x))), but simultaneously withlearning what those predictions are in the first place (minimizing DKL (q(y | x, θ) || p(y | x, θ))).

The margin β determines the size of the credible interval:
larger margins will lead to lower CLBETIG values, i.e., alower implied risk tolerance for non-transferable information acquisition. Section 3.3 discusses methods for selecting
β in the context of our proposed algorithm.

2
The equality follows since the distribution q(y | x, ω) is, byassumption, available to the learner.

We here briefly discuss how CLBETIG connects to existingmeasures and frameworks for similar problems.

Bayesian Active Meta-Learning under Prior Misspecification

Interpretation as an exploration penalty. Related tothe meta-learning dilemma is the exploration–exploitation
dilemma, where online learners face a trade-off between
exploiting (possibly misspecified) prior information abouthigh-valued actions and exploring to gather observations
that will refine this prior information. In the context ofsequential optimization algorithms (e.g., Bayesian optimization; see Section 4), a common way to navigate theexploration–exploitation dilemma is to define an upperconfidence bound (UCB) measure which is an additive combination of the mean of the learner’s distributional prediction
at a given input and the standard deviation around that mean
(Schulz et al., 2018). A UCB measure can be interpreted as
an optimistic, or plausible upper bound on the, prediction at
a given point.
CLBETIG is computed similarly to a UCB measure, withthe critical difference that a multiplier of the standard deviation is subtracted from, rather than added to, the mean
prediction: unlike UCB, CLBETIG is risk-averse in thesense that an input that induces a high standard deviation in
TIG values is penalized rather than favored. In our framework, the standard deviation in TIG values is interpreted as
the degree of risk of a non-transferable information gain (anegative characteristic of an input) rather than of uncertainty
that can be reduced (a positive characteristic in an exploration framework). While UCB measures can be interpreted
as optimistic estimates of a learner’s predictions, CLBETIG
can instead be thought of as a pessimistic estimate of the
ETIG.
Interpretation as a robustness measure. Robust expected information gain (REIG) is the EIG of the worstcase distribution inside an ambiguity set characterized bya pre-specified radius around the prior distribution (Go &Isaac, 2022). Under certain conditions, CLBETIG can beinterpreted similarly to the REIG, where the user specifies
the parameter β rather than the radius of the ambiguity set;
see Appendix B for additional details. Intuition for this canbe established by realizing that a worst-case measure can bethought of as a pessimistic estimate of the ETIG, and thus
as an exploration penalty.

5

Algorithm 1 Sequential optimal experimental design formeta-learning (sOED-ML)
Input: Candidate designs X, prior distributionp(ω, y | X), margin β, number of Monte Carlo samples
N
Output: Optimal design x∗ ∈ X
Sample{(θ1 , ψ1 , y1 ) . . . (θN , ψN , yN )}p(θ, ψ, y | X)
Compute TIG(θi , yi , X) ∀ (θi , yi )ETIG(X) ← Eθpi ,yi [TIG(θi , yi , X)]
SDTIG(X) ← Varθi ,yi [TIG(θi , yi , X)]
CLBETIG (X) ← ETIG(X) − βSDTIG(X)i∗ ← argmaxi ETIG(X)[i]x∗ ← X[i∗ ]if CLBETIG (x∗ ) < 0 thenCompute TSIG(ψi , yi , X) ∀ (ψi , yi )ETSIG(X) ← Eψi ,yi [TSIG(ψi , yi , X)]i∗ ← argmaxi ETSIG(X)[i]x∗ ← X[i∗ ]end ifreturn x∗∼transferable information acquisition. More specifically, we
consider there to be a risk of non-transferable informationacquisition if CLBETIG is negative at the ETIG maximizer— in other words, if pursuing a standard version of sOED
would lead to a substantial risk of non-transferable information acquisition. If this condition is met, sOED-ML selects
the input that maximizes the ETSIG. Otherwise, it selects
the input that maximizes the ETIG. The algorithm is given
in Algorithm 1.
The margin β is considered a hyperparameter of sOED-ML.
In Section 5, we empirically explore implementations inwhich β decays as the experiment progresses. The implied
higher tolerance for the risk of non-transferable information
indicates both a lower risk of misspecification as additional
data is collected, and the reduced benefit of seeking taskspecific information as the data budget from a given task
becomes exhausted.

3.3. Proposed algorithm

3.4. Estimating ETIG and ETSIGWe here describe an application of CLBETIG as a component of an sOED algorithm to navigate the meta-learning
dilemma, i.e., balance the acquisition of transferable andtask-specific information. We refer to this approach assOED for meta-learning (sOED-ML). sOED-ML can besummarized by a simple heuristic: seek task-specific information when there is risk of non-transferable informationacquisition, and transferable information otherwise.

Estimating the EIG is non-trivial, as it requires computing
either the posterior p(θ, ψ | y, x) or the marginal distributionp(y | x), neither of which are in general known in closed
form (Foster et al., 2019). Estimating the ETIG and ETSIG is even more complicated: as discussed in Section 2.1,
these require an additional approximation to the likelihoodp(y | x, θ) itself (Foster et al., 2019).

sOED-ML relies on the CLBETIG to detect the risk of non-While the TIG and task-specific information gain can becomputed as the log ratio of conditional and marginal likeli-

Bayesian Active Meta-Learning under Prior Misspecification| x,θ)
p(y | x,ψ)hoods (log p(yp(y | x) and log p(y | x) , respectively) rather
than as the log ratio of posterior and prior parameter likelihoods, a representation of the posterior parameter distribution is nevertheless required in the context of sOED, where
the posterior from one experimental trial is used as the prioron the following trial. Thus in our setting, the prior distribution across which the conditional and marginal likelihoods,
and the expectations of the TIG and task-specific information gain themselves, is computed is not usually known inclosed form.

One way to circumvent the intractability of the parameter
distribution is to approximate it with the best-fitting memberof a family of variational distributions (Foster et al., 2019).
A common choice of variational distribution is a multivariate
normal, which is theoretically justified as the asymptotic
parameter distribution in the limit of infinite data (Paninski,2005).
To estimate the ETIG and ETSIG in the context of the simulation experiments reported in Section 5, we approximate
the prior distribution as a multivariate normal and then apply importance weights to correct for biases induced by theapproximation3 . Using a multivariate normal allows us toleverage analytical formulations for conditional parameter
distributions when estimating the likelihood. We will usep̃ to refer to probability density functions induced by themultivariate approximation to the prior.
Of course, the quality of the ETIG and ETSIG estimates
depends on the quality of the variational approximation. If
the approximation is biased (in our case, if the true distribution is not a multivariate normal), this will lead to biased
estimates of the ETIG and ETSIG. We follow previous workand sample from the variational prior, but correct for bias inthis approximation using importance weights (Ryan et al.,2015; Foster et al., 2019; Senarathne et al., 2020).
Approximating the prior with a multivariate normal, as opposed to another variational family, offers two practicaladvantages:
1. Bias in the variational approximation can lead to underrepresentation of parameter values that fall in thetails of the true prior. The multivariate normal includes
the variance of the distribution as an explicit parameter,and so can be straightforwardly rescaled to accommodate better coverage of the tails of the posterior.
Following Ryan et al. (2015), to construct our variational approximation we inflate the covariance of the3
Previous work has used the Laplace method (e.g., Ryan et al.
(2015); Senarathne et al. (2020)) or stochastic gradient descent(Foster et al., 2019) to construct this approximation. Here, we use
a moment-matching approximation in favor of the Laplace method
since we observed that in our setting it leads to higher effective
sample sizes.

6best-fitting distribution, and then correct for bias using
importance weighting.
2. The conditional variational prior is known in closed
form, which allows fast sampling from variational approximations to p(ψ | θ) and p(θ | ψ).
Using this method, the estimates of the ETIG and ETSIG
corresponding to a given input x are computed as follows:[
ETIG(x)
=
N
X
i=1

PM

j=1 w(ψj | θi ) p(yi | x, θi , ψj )w(θi , ψi ) log PM

l=1 w(θl , ψl ) p(yi | x, θl , ψl )(8)\
ETSIG(x)
=
N
X
i=1

PM

j=1 w(θj | ψi ) p(yi | x, θj , ψi )w(θi , ψi ) log PM

l=1 w(θl , ψl ) p(yi | x, θl , ψl ). (9)Samples subscripted by i are drawn from the variational
prior p(y | x, θ, ψ) p̃(θ, ψ)4 . Samples subscripted by l are
drawn from the variational prior p̃(θ, ψ). Samples subscripted by j are drawn from the conditional variational
prior p̃(ψ | θi ) (in Equation (8)) or p̃(θ | ψi ) (in Equation (9)), which can be found in analytical form by conditioning the multivariate normal on θi (Equation (8)) or ψi(Equation (9)).
w indicates the relevant importance weighting function. For
example, w(θi , ψi ) is computed as:w(θi , ψi ) =p(θi , ψi )/p̃(θi , ψi )
,
ZN

(10)

where Z N is a normalizing constant that ensures the N
importance weights sum to 1.

4. Related literature
Our work is closely related to work on the effect of misspecified priors on the performance of Bayesian decision-making
algorithms (Simchowitz et al., 2021). Our setting differs
in that we consider the meta-learning problem as an experimental design problem in which the relevant trade-off is
between actively seeking information about the value oftransferable or task-specific parameters. Although related,
this is somewhat different from the exploration–exploitation
dilemma considered by Simchowitz et al. (2021), where
the learner trades off seeking information about the value
4
Although not shown explicitly in Equation (8) and Equation (9), each set of M inner samples is constrained to include
the corresponding sample (θi , ψi ) to avoid pathological behavior
when a value yi has low prior probability (Foster et al., 2020).

Bayesian Active Meta-Learning under Prior Misspecificationof parameters with reward maximization (see discussion in
Section 3.2). Future work should better understand the connections between these two works and how the theoreticalresults of Simchowitz et al. (2021) can be leveraged in, e.g.,
hyperparameter selection for sOED-ML.
In the context of active learning, some work has provided
theoretical results on the effect of (Cuong et al., 2016) andapproaches to addressing (Go & Isaac, 2022) prior misspecification. Unlike this work, our focus here is on theapplication to the meta-learning context, i.e., to estimationof a subset of the parameters or of an embedded model (seediscussion in Section 3.1). This connects our work to otherproblem formulations which constitute estimation of an embedded model. The remainder of this section discusses theseconnections.
Model selection. Applications of sOED to model selection, i.e., to identification of one of a set of models eachcharacterized by a corresponding parameter distribution,is an example of an embedded model problem where thelearner’s goal is to identify the model indicator without particular regard for the corresponding parameter value (Foster,2021). Sloman et al. (2023) showed that a phenomenon
analogous to non-transferable information acquisition canoccur in this context — misspecified parameter distributions
can lead inference to favor the wrong model in early trials —and discussed the resulting model selection/parameter estimation dilemma, which can be seen as a special case of themeta-learning dilemma5 . Past work has navigated the model
selection/parameter estimation dilemma in the context ofsOED by exclusive reliance on the EIG measure (i.e., computing the information gain measure with respect to bothmodel and parameter values) (Borth, 1975) and alternating
between maximizing information gain about the parametervalues and model indicator (Cavagnaro et al., 2016).
Optimization with Bayesian priors. Bayesian optimization (BO) is an active learning method for maximizing blackbox functions. BO can be seen as a case of sequential
Bayesian active learning on an embedded model where theparameter of interest is the location of the function maximum (Hernández-Lobato et al., 2014; Foster, 2021). Priorwork has shown that the choice of prior affects the performance of BO (Schulz et al., 2016). Solutions to this problem
include online hyperparameter optimization (Berkenkamp
et al., 2019), using prior data from experts to form a better
prior (Wang et al., 2023), and constructing robust acquisition functions that explicitly account for the possibility ofprior misspecification (Bogunovic et al., 2018; Kirschneret al., 2020).
5This connects closely with a body of work on the sensitivity ofmodel selection indices to the choice of prior parameter distribution(Vanpaemel, 2010).

7

Multi-armed bandit problems are a related setting, which are
instead characterized by a discrete action space. Theoreticalresults and bounds on the regret the learner should anticipate under prior misspecification have been establishedhere (Kannan et al., 2018; Kveton et al., 2021; Bogunovic& Krause, 2021; Simchowitz et al., 2021). As mentioned
above, many algorithmic solutions to prior misspecification in this setting promote “exploration” by incentivizing
the learner to sample where they have high prior uncertainty (Kveton et al., 2021; Bogunovic & Krause, 2021;
Simchowitz et al., 2021).

5. Experiments
To explore the behavior of sOED-ML, we ran a set of simulation experiments in an adapted version of the the preference
modeling paradigm from Foster et al. (2019). Details of theparadigm can be found in Appendix D of their paper. Inputs were selected from among 20 scalar values of x evenly
spaced between −80 and 80. These inputs are mapped to
outputs y such that the mean of y | x depends on x and θ(which is unknown to the learner), and the variance of y | x
depends on |x| and σ (which is known to the learner).
In the parameterization given in Foster et al. (2019), θ isthe only parameter whose value is unknown to the learner.
We consider this the transferable parameter, and include an
additional, task-specific parameter ψ that transformed inputs
x → ψx. In other words, tasks characterized by high valuesof ψ produced outputs with more extreme means and larger
variances. Following Foster et al. (2019), we set the priordistribution of θ to N (−20, 20) and the prior distributionof ψ to N (0, 1). In all experiments, θ∗ = −12 and ψ ∗was drawn from a true distribution q(ψ). This distribution
was N (0, 1) in the well-specified case (i.e., matched theprior) and N (2, 1) in the misspecified case. We ran bothexperiments where σ = 1 and where σ = 5.
Figure 3 shows the values of the acquisition functions discussed in Section 2 and Section 3 under the misspecified
prior and σ = 1. In this example, choosing the value ofx that maximizes ETIG would results in non-transferable
information acquisition, as indicated by the negative valueof ATIG at the corresponding value of x.
When computing the values of our acquisition functions, we
multiplied the covariance matrix of the best-fitting multivariate normal approximation to the posterior by a factor of
2. We set N (the number of outer samples) to 10,000, andM (the number of inner samples) to 100 (reflecting√resultsfrom Rainforth et al. (2018) that M is optimally ∝ N ).
Figure 4 shows how the value of CLBETIG , computed
according to different margins β at the ETIG-maximizer,
changes over the course of the experiments. When themargins are fixed (purple lines), the value of β was set atBayesian Active Meta-Learning under Prior Misspecification

Figure 3. Values of the acquisition functions discussed in Section 2
and Section 3 in the context of the modeling paradigm used forour simulation experiments. Here, selecting the value of x thatmaximizes ETIG would result in non-transferable informationacquisition.

(a) Well-specified; σ = 1.

(c) Misspecified; σ = 1.

(a) Well-specified; σ = 5.

(b) Misspecified; σ = 5.

Figure 4. Value of CLBETIG at the ETIG-maximizer on each trialof a sequential experiment, as computed according to the margins
indicated in the legend. Lines indicate means and shaded regions
indicated the corresponding standard errors across 100 simulated
experiments.

the beginning of the experiment and remained unchanged.
When the margins decay, the value of β was set to the indicated value at the beginning of the experiment and decayed
exponentially:

βt+1 =

20βt.
19 + 100βt(11)Figure 4 shows that the fixed margins result in values of
CLBETIG that are relatively insensitive to where one is in
the experiment: they remain persistently negative, indicatingthat they are dominated by inherent variance in the outputs
and do not reflect the information the learner has gained
about the value of ψ. The result is that sOED-ML never actively seeks transferable information. On the other hand, thedecaying margins are negative at the beginning of the experiment, i.e., indicate the risk of non-transferable informationacquisition that is reflected in Figure 3, but then increase andstabilize above 0, resulting in the acquisition of transferable
information once the learner has gained information about
the value of ψ.
Figure 5 shows how sOED-ML compares to sequential maximization of ETIG and EIG in terms of the learner’s ability
to recover θ∗ (measured as log (p(θ∗ )); because of the pat-8(b) Well-specified; σ = 5.

(d) Misspecified; σ = 5.

Figure 5. Degree to which each sOED method is able to recoverθ∗ ; sOED-ML is indicated by the corresponding decaying margin
β. Lines indicate means and shaded regions indicate the corresponding standard errors across 100 simulated experiments.

terns described above, we show only results for sOED-ML
using decaying margins). As expected, when the prior iswell-specified (Figure 5a and Figure 5b), ETIG (orangeline) outperforms all the other methods. However, this is
not the case when the prior is misspecified (Figure 5c andFigure 5d): here, as anticipated, ETIG on average leads tonon-transferable information gain (lower log (p(θ∗ ))) after
the first trial, which is protected against by some versionsof sOED-ML.

6. Conclusion
In this paper, we leverage a Bayesian experimental design
framework to analyze the setting of Bayesian meta-learning
under prior misspecification. We expand upon existing measures of EIG and construct a distribution of information gain
values, and then use this distribution to diagnose the riskof non-transferable information acquisition. As we demonstrated with our proposed algorithm sOED-ML, our work
facilitates the development of active yet aware algorithms
that perform well in varied task environments (such as when
eliciting feedback from multiple experts). Future work canleverage the distribution of information gain values implicit
in EIG measures in other ways, such as by developing acquisition functions for robust or “risk-seeking” experimental
design.

Bayesian Active Meta-Learning under Prior Misspecification

Acknowledgements
The authors thank Vikas Garg for helpful comments regarding the problem formulation, Avirup Das for helpfulcomments regarding the experimental implementation, andtwo anonymous reviewers for their feedback. SJS and SK
were supported by the UKRI Turing AI World-Leading Researcher Fellowship, [EP/W002973/1]. AB was supportedby the Academy of Finland Flagship programme: Finnish
Center for Artificial Intelligence FCAI.

References
Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret
bayesian optimization with unknown hyperparameters.
Journal of Machine Learning Research, 20, 2019.
Bogunovic, I. and Krause, A. Misspecified gaussian process
bandit optimization. In Advances in Neural Information
Processing Systems 34 (NeurIPS 2021), 2021.
Bogunovic, I., Scarlett, J., Jegelka, S., and Cevher, V. Adversarially robust optimization with gaussian processes.
In Advances in Neural Information Processing Systems
31 (NeurIPS 2018), 2018.
Borth, D. M. A total entropy criterion for the dual problem
of model discrimination and parameter estimation. Journal of the Royal Statistical Society: Statistical Methodology, 37, 1975.
Cavagnaro, D. R., Aranovich, G. J., McClure, S. M., Pitt,
M. A., and Myung, J. I. On the functional form of temporal discounting: An optimized adaptive test. Journal of
Risk and Uncertainty, 52, 2016.
Cuong, N. V., Ye, N., and Lee, W. S. Robustness of bayesianpool-based active learning against prior misspecification.
In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 2016.
Foster, A., Jankowiak, M., Bingham, E., Horsfall, P., Teh,
Y. W., Rainforth, T., and Goodman, N. Variational
bayesian optimal experimental design. In Advancesin Neural Information Processing Systems 33 (NeurIPS2019), 2019.
Foster, A., Jankowiak, M., O’Meara, M., Teh, Y. W., andRainforth, T. A unified stochastic gradient approach todesigning bayesian-optimal experiments. In Proceedingsof the Twenty Third International Conference on Artificial
Intelligence and Statistics (AISTATS 2020), 2020.
Foster, A. E. Variational, Monte Carlo and Policy-Based Approaches to Bayesian Experimental Design. PhD thesis,
University of Oxford, 2021.

9Go, J. and Isaac, T. Robust expected information gain for
optimal bayesian experimental design using ambiguity
sets. In Proceedings of the Thirty-Eighth Conference on
Uncertainty in Artificial Intelligence (UAI 2022), 2022.
Hernández-Lobato, J. M., Hoffman, M. W., and Ghahramani, Z. Predictive entropy search for efficient global
optimization of black-box functions. In Advances in
Neural Information Processing Systems 27 (NIPS 2014),
2014.
Kannan, S., Morgenstern, J., Roth, A., Waggoner, B., andWu, Z. S. A smoothed analysis of the greedy algorithm
for the linear contextual bandit problem. In Advancesin Neural Information Processing Systems 32 (NeurIPS2018), 2018.
Kirschner, J., Bogunovic, I., Jegelka, S., and Krause, A.
Distributionally robust bayesian optimization. In Proceedings of the Twenty Third International Conferenceon Artificial Intelligence and Statistics (AISTATS 2020),
2020.
Kveton, B., Konobeev, M., Zaheer, M., Hsu, C.-W., Mladenov, M., Boutilier, C., and Szepesvari, C. Meta-thompson
sampling. In Proceedings of the 38th International Conference on Machine Learning (ICML 2021), 2021.
Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE
Transactions on Knowledge and Data Engineering, 22,2009.
Paninski, L. Asymptotic theory of information-theoretic
experimental design. Neural Computation, 17, 2005.
Rainforth, T., Cornish, R., Yang, H., Warrington, A., andWood, F. On nesting monte carlo estimators. In Proceedings of the 35th International Conference on Machine
Learning (ICML 2018), 2018.
Rainforth, T., Foster, A., Ivanova, D. R., andSmith, F. B. Modern bayesian experimental design.
https://arxiv.org/abs/2302.14545, 2023.
Ryan, E. G., Drovandi, C. C., and Pettitt, A. N. Fully
bayesian experimental design for pharmacokinetic studies. Entropy, 17, 2015.
Schulz, E., Speekenbrink, M., Hernández-Lobato, J. M.,
Ghahramani, Z., and Gershman, S. J. Quantifying mismatch in bayesian optimization. In NIPS Workshop on
Bayesian Optimization, 2016.
Schulz, E., Speekenbrink, M., and Krause, A. A tutorial on
gaussian process regression: Modelling, exploring, andexploiting functions. Journal of Mathematical Psychology, 85, 2018.

Bayesian Active Meta-Learning under Prior Misspecification

Senarathne, S. G. J., Drovandi, C. C., and McGree, J. M.
A laplace-based algorithm for bayesian adaptive design.
Statistics and Computing, 30, 2020.
Simchowitz, M., Tosh, C., Krishnamurthy, A., Hsu, D., Lykouris, T., Dudı́k, M., and Schapire, R. Bayesian decisionmaking under misspecified priors with applications tometa-learning. In Advances in Neural Information Processing Systems 35 (NeurIPS 2021), 2021.
Sloman, S. J., Cavagnaro, D., and Broomell, S. B. Knowing
what to know: Implications of the choice of prior distribution on the behavior of adaptive design optimization.
https://arxiv.org/abs/2303.12683, 2023.
Valentin, S., Kleinegesse, S., Bramley, N. R., Seriès, P.,
Gutmann, M. U., and Lucas, C. G. Designing optimal behavioral experiments using machine learning.
https://arxiv.org/abs/2305.07721, 2023.
Vanpaemel, W. Prior sensitivity in theory testing: An apologia for the bayes factor. Journal of Mathematical Psychology, 54, 2010.
Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Nado,Z., Gilmer, J., Snoek, J., and Ghahramani, Z. Pretrained gaussian processes for bayesian optimization.
https://arxiv.org/abs/2109.08215, 2023.

10

Bayesian Active Meta-Learning under Prior Misspecification

11

A. Derivation of Equation (5)p(y | x, ω)
AIG(x) = Eω∼q(ω) Ey∼q(y | x,ω) log
p(y | x)= Eω∼q(ω) [H (q(y | x, ω) || p(y | x)) − H (q(y | x, ω) || p(y | x, ω))]= Eω∼q(ω) [H (q(y | x, ω)) + DKL (q(y | x, ω) || p(y | x)) − H (q(y | x, ω)) − DKL (q(y | x, ω) || p(y | x, ω))]= Eω∼q(ω) [DKL (q(y | x, ω) || p(y | x)) − DKL (q(y | x, ω) || p(y | x, ω))](12)

B. Connections between CLBETIG and Robust EIG
Go & Isaac (2022) developed a measure of robust expected information gain (REIG), which can be interpreted as the lowest
expected information gain from the set of distributions in an ambiguity set of radius ϵ around the learner’s prior, i.e.,
REIG(x, ϵ) = inf {EIG(x, Q) | D(Q, P ) ≤ ϵ} .
Q

(13)

Notice that the EIG is written as a function of both the input x and the prior distribution Q over which the infimum is taken;
to compute the EIG as a function of a given Q, one replaces the probability measures indicated by p in Equation (1) withthose induced by Q. P indicates some fixed reference prior, and D is a suitable divergence measure.
Notice that computing the REIG requires specifying the radius of the ambiguity set ϵ. CLBETIG admits a similar robustness
interpretation as REIG, with the researcher degree of freedom being the choice of margin β as opposed to the radius of
the ambiguity set ϵ. While Go & Isaac (2022) focus on a robust measure of the EIG, rather than the ETIG, they discuss
extensions to their method for the case of a misspecified likelihood. Our discussion of the connection between REIG and
CLBETIG assumes this extended version of REIG as an information gain measure on the transferable parameter θ.
Estimation of both REIG and CLBETIG requires multiple samples of TIG from the learner’s prior p(θ, ψ, y). In the context
of REIG, changing the value of ϵ can be thought of as shifting the REIG between an average of these samples (ϵ = 0; i.e.,the ETIG) to the worst-case TIG from the sample (ϵ → ∞) (Go & Isaac, 2022). Similarly, in the context of CLBETIG ,changing the value of β can be thought of as moving the CLBETIG from the sample average (β = 0; i.e., the ETIG) to avalue of TIG β standard deviations below this mean. In other words, higher β moves the CLBETIG further into the lefthandtail of an approximate distribution of TIG values. In the case where the true distribution of TIG values is continuous and thelearner has access to an infinite number of samples from their prior, they can expect to recover values of CLBETIG that arecloser and closer to the worst-case TIG as β → ∞. Thus, like increasing ϵ in the context of REIG, increasing β shifts the
CLBETIG from the ETIG to the worst-case TIG value.