Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities
Authors: Andreas Kirsch
Yarin Gal
Venue: Accepted by TMLR ()

Abstract: Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradient lengths to estimate how informative data is for a model’s training. Are these different approaches connected, and if so, how? We revisit the fundamentals of Bayesian optimal experiment design and show that these recently proposed methods can be understood as approximations to information-theoretic quantities: among them, the mutual information between predictions and model parameters, known as expected information gain or BALD in machine learning, and the mutual information between predictions of acquisition candidates and test samples, known as expected predictive information gain. We develop a comprehensive set of approximations using Fisher information and observed information and derive a unified framework that connects seemingly disparate literature. Although Bayesian methods are often seen as separate from non-Bayesian ones, the sometimes fuzzy notion of “informativeness” expressed in various non-Bayesian objectives leads to the same couple of information quantities, which were, in principle, already known by Lindley (1956) and MacKay (1992).

Published in Transactions on Machine Learning Research (11/2022)
Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information- cheoreti Quantities
Andreas Kirsch
andreas. kirschhss..cssoro uk carin.gal@@cson.cco
Yarin Gal
OATML, Department of Computer Science University of Ouford
Reviewed on OpenReview: https: // net/ formm? id= UVDAKQANON
Abstract
Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based gradients, and gradient lengths to estimate how informative data is for model's training. Are these different approaches connected, and if so, how? We revisit the fundamentals of Bayesian optimal experiment design and show that these recently proposed methods can be understood as approximations to Information-theoredr
quantities: among them, the mutual information between predictions and model parameters, known as expected information goin or BALD in machine learning, and the mutual information between predictions of acquisition candidates and test samples, known as expected predictive information gain. We develop comprehensive set of approximation ions using Fisher information and observed information and derive unified framework that connects seemingly disparate literature. Although Bayesian methods are often seen as separate from non- non-Baycsian ones, the sometimes fuzzy notion of "informativeness" expressed in various non-Bayesian objectives leads to the same couple of information quantitics, which were, in principle, already known by Lindley (1956) and MacKay (1992).
1 Introduction
Label and training efficiency are key to a wider deployment of deep learning. Deep learning generally requires a lot of data, much of which must be annotated. This is expensive and time-consuming. Together with semisupervised and unsupervised approaches, active learning (Atlas et al., 1990; Cohn et al., 1994) helps increase label efficiency: given access to unlabeled data, active learning selects the most informative samples label for given model, thus decreasing the number of required annotations reach given level of performance. In addition to label efficiency, training deep learning models is also expensive and Lime-consuming, and active sampling improves training elficiency by filtering the training set to focus on the samples that will be the most informative for the model.
Several new approaches in data subset selection for deep learning have been proposed recently: amongst them, BADGE (Ash et al., 2019), BAIT (Ash et al., 2021), PRISM (Kothawade et al., 2022), SIMILAR! (Kothawade al., 2021), and GraNd (Paul et al., 2021). The acquisition functions used to select informative samples in these approaches can be traced back to informatiion-lheordiie quantities (short: information quantities) that are known from Bayesian optimal experiment design (Lindley, 1956; MacKay, 1992). This connection is the topic of this work.
By examining how Fisher information and second-order posterior approximations (Gaussian approximations can be used for estimating information quantities, we develop a unifying perspective and relate these recent mothods to information quantitics used Bayesian active learning: for active learning, the expected information gain (EIG), also known as BBatch)BBADD (Lindley, 1956; Houlsby et al., 2011; Kirsch al., 2019), the (joint) expected predictive information gain (JEPIG or FPIG, respectively) (Kirsch et al., 2021b;

---

Published in Transactions on Machine Learning Research (11/2022)
MacKay, 1992), and, for active sampling, the information gain (IG) (Sun et al., 2022) and (joint) predictive information gain (JPIG or PIG, respeclioely) (Mindermann et al., 2022).
These connections point towards possible failure modes of current methods and potential extensions in principled ways. We examine approximations that lead to last-layer approaches, find a potential bias when using similarity matrices, compare trace and log determinant approximations in regard to batch acquisition pathologies, and trade off weight- and prediction-space methods in principle.
Outline. To achieve this, we look at second-order posterior approximations (Gaussian approximations) in $3, which we use to revisit. Fisher information, its properties, special cases, and approximations in $4. Our contribution here is to summarize results and provide consistent notation that simplifies reasoning about information quantities, observed information, and Fisher information.
In $5, we approximate the information quantities mentioned above using observed information and Fisher information. We provide a comprehensive overview to understand the differences and similaritics and make it easier to spot applicatiion of these approximations in the literature. We pay special attention to the limitations: for example, will see that some approximations that use the trace of the Fisher information do not take redundancics between samples into account. They exhibit the same pathologics as other methods that, in essence, score points individually, also known as top-k batch acquisition (Kirsch et al., 2021a). In $6, we expand our approach to approximations that use similarity matrices of log-loss gradients. Our contributions are a comprehensive overview of the approximations and the connection to similarity matrices.
In $7, we show that BBALD and EPIG on the one hand; and BADGE, BAIT, PRISM', and SIMILAR' the other hand can be seen as optimizing the same objectives. The difference is that (Batch-)BAL. (Houlsby et al., 2011; Kirsch et al., 2019) and EPIG (Kirsch et al., 2021b) operate in prediction space, while Fisher information-bassee methods operate in weight space: we show that an approximation of EPIG, a transductive active learning objective, using Fisher information, matches the BAIT objective (Ash et al., 2021). Similarly, we show how BADGE (Ash et al., 2019) approximates the EIG, using the connection to similarity matrices. Finally, we find that submodularity- based approaches (Iyer el al., 2021) such as SIMILAR (Kothawade al., 2021) and PRISM (Kothawade et al., 2022), which report their best results using the log determinant of similarity matricos, approximate information quantitics when they perform best. We also show that graadeen-lenngt-bbee be connected to information quantities.
methods like EGL (Settles et al., 2007) and GraNd (Paul et al., 2021) can
Limitations. Although our results employ a hierarchy of approximations, we do not examine the error terms in detail. This is in line with how these approximations are used in deep learning, where the approximations often only provide motivation for useful mechanisms. However, we try to identify where these approximations might break, enumerate their limitations, and raise several (empirical) research questions for future work.
2 Background & Setting
This section introduces the relevant notation, concepts, and probabilistic model that we use in this paper.
Information Theory. We follow the practical notation of Kirsch and Gal (2021). In partieular, for entropy, we use implicit or explicit notation, H[X] or H(p(X)), while H(p || q) denotes the cross-entropy similar to the Kullback-Leibe divergence (K || 9), and H(p(z)) denotes Shannon's 's information content?:
H(p(z)) = log logp(z),
(1)
H[z] == H(p(x)).
(2)
H(p(X) || {(X) = Ep(x) (()()))
(3)
H[x] = Hpp())) = H((((X) || p(X)),
(4)
"ussin log determinant objectives
PIn related litcrature, Shaunon's information content is often written h(p(x)). We use unified notation Shannon's information content is but point-wise entropy, and we can differentiate between the two by whether we compute it for outcome random variable X expectation; see Kirsch and Gal (2021).
2

---

Published in Transactions on Machine Learning Research (11/2022)
where p and q are probabilities distributions, X is random variable, and z is an outcome. Conditional and joint entropies are defined as usual (note that we also take an expectation over y):
H YJ = E p( [y]]
(5)
The mutual information I[ for random variables X and Y is defined as:
I[; = H[X] - H[X Y),
(6)
and is also called expected uncertainty reduction expected information gain (Lindley, 1956) because H[X] quantifies the uncertainty about X, and H[X Y] about X after observing Y (in expectation).
Furthermore, when we mix random variables with outcomes of random variables in (conditional) entropies, for example, H[X, Y= = y| Z], or short H[X, y Z), we expand this to an expectation over the random variables conditional on the outcomes:
H y Z p(m.>//) log p(r, |=]]-
Probabilistic Model. We assume supervised setting: for inputs X, we have Bayesian model with parameters Q that makes predictions Y. What makes the model Bayesian is that the parameters follow a distribution p(w). We use the probabilistic model:
p(y w [)) = ply xr, w) p(w).
(8)
We extend this model to additional data D = {x vi)} as follows:
p (x:)) = p(())) (w) p(w) p(y...... Un x , Tny p(w).
(9)
That is, we examine the common discriminative case where, unlike in the generative case, we do not model p(z). The corresponding marginal prediction of the model is p(y xx) = Ep [P(y x w)].
Transductive Objectives. When an objective uses (additional) data Deval, unlabeled or labeled, to guide acquisitions, we refer to the objective as a transductive objective (Yu et al., 2006; Wang et al., 2020).
Active Learning. To increase label efficiency, instead of labeling data indiscriminately, active learning iteratively selects and acquires labels for the most informalive unlabeled data from a pool set Deool according to some acquisition function. An acquisition function scores the informativeness of an unlabeled candidate sample 2, and the sample that maximizes this score is selected for labeling, After each acquisition step, the model is retrained to take the newly labeled data into account. Labels can be acquired individually or in batches (batch acquisition, see below). The expected information gain (EIG)
I[2; yraq re
(EIG/BALD)
and (joint) expected predictive information gain (JEPIG and EPIG, respectively)
[[(Yevvl)) yacq {zeval), ), _ag
(JEPIG) (EPIG)
[[yrvvll yaog Xovall L
are examples of such acquisition functions. EPIG and JEPIG are transductive acquisition functions they depend on Xeval {Irval), }, respectively.
Active Sampling. To increase training efficiency, instead of training with all samples, active sampling (sometimes also called data pruming) (Paul et al., 2021) selects the most informative sample (2.009, 209) from the training set to train on next. This can be done statically before training the model, in which case this is also referred to as core-sel selection, or dynamically, in which case it is also referred to curriculum learning. The information goin (IG)
1[2; y aca a00]
(IG)
3

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: Taronomy of Information Quantities for Data Subset Selection. In general, information quantities can be split into ones for active sampling or active learning, into non-transductive and transductive ones, and in the transductive case, into taking an expectation or the joint over (additional) evaluation samples. Here, show the information quantities for individual acquisition. For batch acquisition, {Y, (y. (9,,), {x²")) can be substituted.
Active Learning
Active Sampling
Non- Transductive
EIG/BALD
[[;; y [aon]
IG
g00 [2"rg]
Transductivee Expectation (using Desll)
EPIG
Eptamety I[yaval, yaey zoval ga0n)
PIG
g Ilg yon z sate
Joint
JEPIG
[[ Yoes| (ee 2009)
JPIG
[[() }: y:ee (aqual), ) zaaa)
and (joint) predictive information gain (JPIG or PTG, respectively)
"[ y ynrg (xjval) J. x. r[yreall yoca xeeaal x o0
(JPIG)
(= Ep yrall Tly yaca sevall 2009))
(PIG)
are examples of such acquisition functions. PTG and JPIG are transductive acquisition functions.
Batch Acquisition. In batch active learning, the acquisition function is applied to a batch of candidates ). [/= and the batch that maximizes the acquisition function is selected for labeling. Similarly, in batch active sampling, the acquisition function is applied to a batch of training samples (,""", ())= and the batch that maximizes the acquisition function is selected for training. For deep learning, batch acquisition is often the only feasible option in both active learning and active sampling. We extend the above acquisition functions to the batch case by substituting set of samples in the definitions: {2:09) for x=*9 {y, for yeen, and [v for yooa, respectively, and treating the sets as joint random variables (Kirsch et al., 2019).
Submodular Acquisition Functions. Choosing the subset {x(-r) naively is intractable due to the exponential mumber of possible acquisition batches. Instead of maximizing the acquisition function on all possible batches, we can often use submodularity (Nemhauser et al., 1978). set function f is submodular when:
S
(submodular)
An acquisition batch {x]00] can be constructed greedily by selecting the samples that increase the acquisition function the most one-by-one. This greedy algorithm is guaranteed to find a --potimall acquisition batch for monotone submodular acquisition functions.
Although the EIG is (monotone) submodular, leading efficient batch acquisition (Kirsch et al., 2019), the other information quantities (IG, EPIG, JEPIG, PIG) are usually not submodular (Kirsch et al., 2021b). We examine the details of this and compare to the relevant literature in $7.
Taxonomy of Information Quantities. Table shows the information quantities along three dimensions: active learning vs active sampling, non-transductive vs transductive, and taking the expectation vs the joint over evaluation samples for transductive information quantities.
Log Loss. While many active learning and active sampling methods are motivated independently of the underlying loss, we will focus on log losses, such as the common cross-entropy loss squared error loss (Gaussian error), as these log losses can be viewed through an informatiion-hhorcce or probabilistic lens.
3 Second-Ordes Order Posterior Approximation
Laplace approximations are a standard tool in Bayesian statistics and machine learning (Daxberger el al., 2021; Immer et al., 2020). In this section, we review the Laplace approximation and introduce as a special case of more flexible socond-order posterior approximation, a Gaussian approzimation. It central to approximating information quantities using observed information, defined in this section, and Fisher information, defined in $4.
4

---

Published in Transactions on Machine Learning Research (11/2022)
Our goal is to approximate the posterior p(w DD Dirain) using (multivariate) Gaussian distribution, where (()),:)) ui) are additional (new) samples, and we start with p(w (Drrain) as the "prior" distribution we D = ((
will drop Dirain and use p(w) when possible, to shorten the notation.
To begin, we complete the square of a second-order Taylor approximation around the log- pagaamemtter likelihood for fixed w
log p(w) = logp(w") + lo [[w w²) + (w wn) " log [Iw - -w)
(10)
= (w - (w" V
"))" llog p( []( (w" p(o') [log p ((="
D)
(11)
Importantly, we can express this more concisely by exlending the notation of H[-] to its derivatives:
Notation 3.1. We write H[ for the Jacobian and H' for the Hessian of H[:]:
H'H V log p(-), H == - = lo))
(12)
(13)
This notation will be helpful throughout this paper, both observed information and Fisher information can be expressed in terms of the Hessian of the negative log parameter likelihood.
Then, we can write:
H[w Re H[w*] + H']W"((w (w--]( - w²) + j(( w*) H"(w"] (w - w*)
(14)
= (w - (w H" H=" "'"""""""""""""""
""" H"[""] (w - (w- -w- -""W""--"''' + +....
(15)
Comparing this to the information content of multivariate Gaussian distribution:
H[N(w; n. Σ) = 7(( (w 11)² - ( (1) .....
(16)
we obtain the Gaussian approximation, which we will apply throughout this paper:
Proposition 3.2. The Gaussian approximation of the distribution p(w) of Ω around some w is given by:
Ω ズ N -1" H" H=)
(17)
where ["[""] must be desinive-defmile.. If wn is also a (global) minimizer of H[w] (that is, H' ['[w"] = 0), we
obtoin the Laplace approximation:
Ω R N(w" H-
(18)
Approximation Quality. However, this approximation can be arbitrarily bad depending on p(w) and wt. Given enough data, is often argued that p(w) will concentrate around the maximum a posteriori (MAP) estimate, giving rise to the Laplace approximation. In statistics, the Bernstein-von Mises theorem is often used to motivate this, but insufficient data to reach concentration of parameters and multimodality in wver-parameterized models (Long, 2021) can be issue for deep active learning and active sampling.
Flat Minimum Intuition. A positive definite Hossian implics that the information content (point-wisc entropy) is convex around wn and, equivalently, that the (log) posterior is concave around wt. The latter provides an intuition for the Gaussian approximation: the Hessian measures curvature, and the "flatter" the Hessian, c.g., the smaller the largest cigenvalue or the smaller the determinant, the less the loss changes when w* perturbed. This leads to the search for flat minima as way to improve generalization (Hinton and Van Camp, 1993; Hochreiter and Schmidhuber, 1994; Smith and Le, 2017).
Notation 3.3. To further shorten the notation, we write HD ( w*] instead of H"((v)) (s)),
5

---

Published in Transactions on Machine Learning Research (11/2022)
Posterior Approximation of Ω D. While the Laplace approximation is centered (global) minimizer, the Gaussian approximation can be used for (potentially low-quality) posterior approximation in general. We can expand H[w*) [D] using Bayes' theorem and the additivity of the logarithm. That is, we have:
H(w" D) = H [*] H[w"] H(D),
(19)
and then, H[D is independent of w:
H' [w" D = [ ["] H + =0= +
(20)
H" w DD = HD [D wn) + H"(W"]
(21)
Proposition 3.4. The observed information H" Hy [ is additive:
H" [xi),w"] = Σ H" "y = Σ v log x.,w").
(22)
Note that the observed information has the opposite sign compared to other works because it simplifies the exposition.
Uninformative Prior. For a Gaussian prior p(w) ~ Nu, Σ), we have H"[w") = D-1 and [ D = HD
w-] + H-1, For an uninformative prior with "infinite prior variance" E + O, we have H" [[[w"] 0, and H" w D H" [D] w*].
Proposition 3.5. The entropy of the second-order approzimation of p(w) around wt is
H[S] ~ - log det H" Ck:
(23)
where C A/2 log 2 a constont (independent of and w*) and is the number of dimensions of w.
While Proposition 3.5 is straightforward, it is the main result for this section as it will allow us to approximate all the mentioned information quantities in $5 and $6.
4 Fisher Information
Fisher information plays a central role in the approximations of information quantities because, unlike the observed information, is always positive semidefinite, We use Fisher information to unify various acquisition functions in $7. The following section revisits Fisher information, its properties, special cases, and common approximations. All proofs are given in $A.
In particular, look at two special cases with more favorable properties: following Kunstner et al. (2019), when we can write our model as ply 2 = f(xww,, where f(z are the logits, and p(y 2) is a distribution from the exponential family, Fisher information independent. of y, which has useful consequences we shall see; and following Chaudhuri et al. (2015), when have Generalized Linear Model (GLM), observed information also is independent of The results for the GLM are often applied as an approximation known as Generalized Gauss-Newton apprecimation (GGN). Together with numerical approximations such as a diagonal approximation or low-rank factorizations, observed information and Fisher information can then be efficiently approximated for large deep neural networks (Daxberger et al., 2021).
Definition 4.1. The Fisher information H"[Y Ir, wr,w"] is the expectation over observed information using the model's own predictions p④y x, w²) for given x at wt:
H" x, = E (H [H" xx w,w"]].
(24)
This notation of the Fisher information is consistent with the notation for information quantities from $2, introduced in Kirsch and Gal (2021), but extended the observed information: the Fisher is but an expectation over the observed information, and the observed information is the Hessian of the negative log-likelihood.

---

Published in Transactions on Machine Learning Research (11/2022)
Proposition 4.2. Like observed information, Fisher information is additive:
H" [xthw"]= Σ H" HY) [r,,"].
(25)
There are two other equivalent definitions of Fisher information:
Proposition 4.3. Fisher information is equivalent to:
H a, wn] = Ep [y x, ", H' o, [1]w"] = Cov(H [r]""]]
(26)
Exponential Family. Kunstner et al. (2019) show in their appendix that if we split discriminative model into prelogits ((=:w) and predictor p(y 2 f(:;w)) Fisher information does not depend on y when p(y =) is distribution from exponential family (independent of w). This covers a normal distribution for regression parameterized by mean and variance predictions or categorical distribution via the softmax function. The following statements and proofs follow Kunstner et. al. (2019):
Special Case:
Proposition 4.4. The Fisher information HY[Y a,, w") for model p(y = f();==)) is equivalent to:
H [x,w"] Val(=;"")" T Epcale (V Hly 2= [x=====")]
(27)
where V Híy f(=;=") is short for 42 Hly 2= f(aww)))
Proposition 4.5. The Fisher information H" Ir, w"] of model of the form plyl =x(==()) is independent of y, where p(y 2) is distribution from exponential family, i.e., logp(y (2) = =TTT(y) A(2) log h(y):
H" [,," Vwf(x;"*) T (V'A::""""""VV>==>>>"""""""" f(o (uf(:w")
(28)
It is crucial that the exponential distribation not depend on w. This simplifies computing Fisher information: no expectation over ys needed anymore. The full outer product may not be needed explicitly either.
As examples, we will consider two common parameteric distributions from the exponential family:
Gaussian Distribution. When p(y 2) NN; 2, 1), we have H"ly 2 for all y, and thus
H" |,, = Vwf(x="").
(29)
Categorical Distribution. When p(y) 8) softmax(3)y, we have H" diag(m) nτ, with ny. = p(yl 2), and thus:
H x,w"] Vuf(x; (diag(*) T TT)) V (x;w")
(30)
Special Case:
Generalized Linear Models. Chaudhuri al. (2015) require that observed information is independent of y, which will also use later. This holds for Generalized Linear Models:

---

Published in Transactions on Machine Learning Research (11/2022)
Definition 4.6. A generalized linear model (GLM) is a model p(y 2 ((x;w) such that log p(y 8) TT)) A(2) + log h(y) is distribution of the expomential family, independent of w, and fx:;w _ is linear in the parameters w.
Proposition 4.7. The observed information H a w"] of GLM independent of y.
H" E, ". Val( Vi Var
(31)
= =Vul(o;w") V2A(w= x:) Vul(=;w").
(32)
Proposition 4.8. For model such that the observed information H" (y Ir w² is independent of y,
have:
H Ix, [r,,'] = H" Ir, [x,w"]
(33)
for any v", and also trivially:
Y x, w,w"] = Pp(ul=)(H"l w]].
(34)
Note that the expectation is over p(y x) and not pl I, w), and (p(tual(a))) (a Ku) Haz w0]l ([() [=]) is additive then.
Proposition 4.9. For a GLM, when f(x;w): RD - RK where K is the number of classes (outputs), D is the number of input dimensions, w R PxK, and assuming the parameters are flattened into a single vector for the Jacohian, have Vuf(s;w) = Idx of (RKx(K-D), where & denoles the Kronecker product, and:
vullow")) VEAlWTo "VAl""" Vul(m==') = ViA(w x) xx."
(35)
This property is useful for computing the Fisher information of GLM in practice (Ash et al., 2021).
py æ. w) vs pyy æ). Having a GLM solves an important issue we will encounter $5: approximating the EIG requires taking an expectation over p(y x) and not p(y w.,"). One can approximate p(y æ) s p(y w,,"), which can be justified in the limit, but this is probably not a good approximation in the cases interesting for active learning and active sampling. With GLM, this not a problem.
Generalized Gauss-Newoon Approximation. In the case of exponential family but not GLM, the equality in Proposition 4.7 is often used as an approximation for the observed information-we simply use the respective Fisher information as an approximation of observed information (via Proposition 4.5):
H | x, x,,"" ~ HY x. w'1 = Vur(=; VEA(w" x) Vu f=r(x="").
(36)
This is known as Generalized Gauss-Newlon (GGN) ayprocimation (Kunstner et al., 2019; Immer el al., 2020). This approximation has the advantage that it is always positive semidefinite unlike the true Hessian.
Last-Layer Approaches. GLMs are often used in deep active learning (Ash et al., 2019; 2021; Kothawade el al., 2022; 2021). If we split the model into p(y a, w) = ply 2 w" f(x)), where = = f(x) are the embeddings and treat the encoder f(x) as fixed, we obtain a GLM based on the weights of the last layer, which uses the embeddings as input.
Armed with this knowledge, we can now derive approximations for the information quantities of interest using observed information and Fisher information and consider their properties. The GGN approximation and last-layer approaches feature heavily in the literature to make computing these approximations more tractable as they reduce computational requirements and memory usage.
8

---

Published in Transactions on Machine Learning Research (11/2022)
5 Approximating Information Quantities
We now derive approximations and proxy objectives for information quantities. We base them on observed information and Fisher information introduced in the previous sections. These approximations help us connect the information quantities to existing the literature non-Bayesian data subset selection in $7.
In particular, we derive approximations for EIG and EPIG as they show the qualitative differences between non-transductive and transductive objectives, and compare the approximations of the IG and EIG: importantly, there is no difference between the latter when we use GLM or the GGN approximation. This covers two of the three dimensions in Table 1. We cxamine JEPIG and the other quantitics in the appendix in $B. We find that the trace approximations of the EPIG and JEPIG objective matches, suggesting that using the trace approximations might be too loose an approximation to capture important qualities of EPIG (Kirsch et al., 2021b). Additional derivations and details can also be found in $B. All this leads to Figure 1, which relates the different approximations to each other and shows that they follow simple patterns.
5.1 Approximate Expected Information Gain
The expected information gain is popular acquisition function in Bayesian optimal experimental design (Lindley, 1956) and in active learning, where it is also known as BALD (Houlsby et al., 2011; Gal et al., 2017).
We can approximate the EIG I[2: (x,00) approximations:
| {x
of acquisition candidates {2ī using Gaussian
[[;; (Y,00) [x = H[2] H[2 (Y,0)) [x,09]]
(37)
H[[2] E sce H|(=, ( [H[s2 {or (cn) {अ.
(38)
= 1/ log det H" [w] Ep((2 "( - log det H" (w (u.")), [2
(39)
= Ep( [| (log
(x(-))) ), w"1 + H" (w]") H "=Ww1
(40)
Ep(ts." (| (log det H=,0) | (, H ["] + Id)]
(41)
using Proposition 3.5 twice, where the constant Cg cancels out in eq. (39) as we subtract two entropy terms.
Generalized Linear Model. When we have a GLM, we can use Proposition 4.8 obtaïn:
[[2; (Y,00) [2;-4]] = = 2 E Ep((s (((""")(<>""))) flg det (H" ( { (x;"9),4) 1, H"" + Id)]
(42)
log det (H"lyY; (x:9),," H [" Id).
(43)
We can upper- upper-bound bound the log determinant and obtain:
s 1 tr (H" ([Y,"))) (x."0)
} w0] H" -w---- } Σ tr (H"((Y)a))) a0 wil H" (1
(44)
(45)
where we have used the following inequality (proof $B.1): Lemma 5.1. For symmebric, posilive semidefinile matrices A, we have (with equality iff = 0):
log det((A A + Id) S tr(A).
(46)
General Case & Exponential Family. For the general case, need to make a strong approximation:
p((x (- (((,,")) ",""),
(47)
which might hold for a mostly converged posterior but probably not in cases with little data. This turns the approximation into an upper bound. Alternatively, we could use the GGN approximation when we have an exponential family for the same result. (but not an upper bound). See $B.1 for the derivation.
9

---

Published in Transactions on Machine Learning Research (11/2022)
Proposition 5.2 (EIG). The expected information gain can be approzimateky upper bounded via:
i[1 ( (x]"),, Dirokn]
} log det
Σ H1 (yae, x00 wn) H Iw""" DDirattj + Ie
(48)
2
Σ tr (H" xany " Din
(49)
Purthermore, we have the following prozy objective:
arg max 1[9;; (Y:"0)) {zz aoo D Drrain
arg max - H[ (Y,m00), }. (2("0), D Dirain and
(50)
{ HIQ (M (x Diraan
log det
Σ Ha | x0oy w- + H"(w" Ditraing
Ck.
(51)
The second statement follows from Equation (37), since H[S2 Dirain) is constant and provides a proxy objective when we are only interested in optimizing the EIG. In $7, we connect it to the expected gradient length approach in active lcarning and show that an ablation Ash et al. (2021) examines the wrong objective.
Batch Acquisition Pathologies. Importantly, this approximation of the EIG using the trace is additive, whereas the one using the log determinant is not. This means that the trace approximation ignores the dependencies between the samples and can only lead to naive top-k batch acquisition; see Kirsch et al. (2019; 2021a) for details of the pathologies of top-k batch acquisition.
5.2 Approximate Information Gain
Following the same steps, we can also approximate the information gain, which is useful for active sampling:
Proposition 5.3 (IG). The information gain I[: (y {z, (2)00), Diroing H[2 D Dirainj
H
(v,"), ), ( 2om, J, Dirain) can be approzimately upper bounded via:
I[2: (v, ( } {,, (00)) D Drroing Looo det [U"K, {x(")) J, w*1 n"(w" Ditraing 1 + Id)
(52)
(H" aam, H D
(53)
Furthermore, we have the following prozy objective:
arg max 1[22; " (2(00) D traing arg max H[S {vi (09),, (x,"²), Drmaing and
(54)
(x:²)
H[2 13, ) (0 D Droing ve log det det (H" (04) (x109) w11 + H" (Dirain)) C
(55)
Comparison to EIG. Importantly, when have a GLM use the GGN approximation, this approximation
of the IG is equal to the one of the FIG. This tells us that active learning GLM with the EIG approximation will work as well as if we had access to the labels. Equivalently, active sampling via IG with the GGN approximation will not work better than the respective active lcarning approach.
5.3 Approximate (Joint) Expected Predictive Information Gain
In transductive active learning, we have access an (empirical) distribution (( e.g., the pool set, and want to find the (x.²0) that maximize the erpected predictive information gain (EPIG) (Kirsch et al., 2021b). The approximations here will help connect BAIT (Ash et al., 2021) to EPIG. For simplicity, consider the non-batch case here. The batch case can be handled analogously. The EPIG objective is defined as:
arg max evaal aaq
reval x acg]
arg max Ep(ceeal Ivyeval yaca eval 209]
(56)
g004
10

---

Published in Transactions on Machine Learning Research (11/2022)
We expand the objective as follows:
Ipyevvll yueg Xeval [aaa] = I[2;; yeval
[ - I[52; yoval Xeevll yuea, 200))
(57)
where 1[2;; Y X can be removed from the objective because it is independent of 2n04, Thus, optimizing EPIG is equivalent to manimizing I[52; yoval Xeval, yaca, x009]
arg max 1[x [ yucq x evall x x
arg min 1[22; yoval x Xoval yucq x v.
(58)
x000
Following Proposition 5.2, this be approximated by:
[[2; yoval X yacg lboez
1 E Ep(geral lever (eanr jf ((wera)) [log det [g | wn] (H" (1/ | w0] + H"" + Id)]. (59)
2
Generalized Linear Model. For a generalized linear model, we can drop the expectation and obtain:
1[; yeval X yrug, _a
Ep det (H [Yvol noval "w²| (H 2 w + H1 (w")) 1=" +d]]
(60)
< log det (Ep(ormn) (Ep(omm)lH"(yox [H eeval w6'l (H" معن w + H" [w=1) + Id)
(61)
tr (Ep(aemt) [H" [Y zeval will Y وم0 w*1 H "1= 1)-
(62)
where we have used the concavity of the log determinant and Lemma 5.1.
General Case & Exponential Family. To our knowledge, there no rigorous way to obtain a similar result in the general case as the Fisher information for an acquisition candidato now lies within an inverted term. Of course, the GGN approximation can be applied when we have an exponential family, which leads to the GLM result above an approximation. See BB.2 for more details.
Proposition 5.4 (EPIG). For a generalized linear model (or with the GGN approzimation), we have:
arg max Tyrevll (V,a0r) Xeval (x." Drraing arg min T(22; y X (y,ar), (x;00)) Deraing (63) (1:"))
(x
with
T[2; y yeal Xeeaa (Vrace), (xp000) Diroiiy
~ Ep(ane) [log det (H"lY eval aval w'1l (H [[X,, ( H" Id)]
(64)
log det (Ep(zme) (Yeall zeval wI (H" (H" {2; w,"") H [wo D + Id)
(65)
< } tr (Ep(uaar) H ay crat w1] (H" ([x (e ( w + H" Derainn)
(66)
Batch Acquisition Pathologies. Unlike for the EIG, the trace approximation of EPIG is not additive in (n:09), and we cannot conclude that it suffers from batch acquisition pathologies like the trace approximation of the EIG.
Approximations for JEPIG, PIG and JPIG. We can follow the same derivation for JEPIG:
I[2; (Y,eval) (aşvalz, yraca, 2ac9]
(67)
2 log det (Ep(tyrrl) ( aa0c)) (y) } (agval), }, wJ] (Ho a00, w0] + HH" [W"" + Id)
Then, applying the steps after eq. (61). we can devise similar approximations. PIG and JPIG follow the same pattern. Details can be found in $B.3 and $B.4. But how do all these approximations relate to each other?
11

---

Published in Transactions on Machine Learning Research (11/2022)
Information
Observed Information
EIG
labels-
- log det (H" ((,,9)
{ag
Hw H
Id)
2
IG log det (H" Kv!")) (x wi) H [w- Dirris 1 Id)
transductive?
2
EPIG
labels
log det (Become) [H* (yrua = w/ (H" (D) (x.
wn") H" [ Ditmin 1)= Id)
PIG
joint?
1 2
log det (Bpiarrd good) [HH (prrad 2 [w"" (H you " H" [w" DDunia 1) IIa)
JEPIG
labels
log dot (H" ((xero)) } (z7*)) w.] (I1" (0%1
(x] ", w"] H" [w0 [Diinin 1)- + Idd
JPIG
1 log dot (I"" (( (zf ww0" (aa (o)")) (x.") H" [w" Datim 1)- + Id)
(a) log det Proxy Objectives
Fisher Information
Observed Information
EIG
w/ labels
(H" ((Y
(a
H"
Drrain
IG (Hu ((v {x(") " w") H" (w" | Dimiin 1-1)
1
transductive?
2
(JJEPIG
labels
tr (Ba(see)) [HI [yeedl a w (HI" (Y (o0 H" [w'' Diraiia
(J)PIG "[[["" [5 ",]
1
(Begorme gemds [II" mool ana
l
[w" Ditain ))-
2
(b) Matrix Trace Proxy Objectives
Figure 1: Compurison of the Approzinations/PPen /Proay Objectives. The difference between active learning and active sampling objectives in using the Fisher information, which is label independent, or the observed information, which uses label information. JEPIG and EPIG have equivalent proxy objectives when using the matrix trace; see $B.4. The notation makes it obvious that in the GLM case, or when the GGN approximation is used, active learning and active sampling approximations match as H" xr,"" (resp. x) H" y a., w"].
5.4 Comparison of the Different Information Quantity Approximations
Figure compares the different information quantity approximations for both the log-determinant and trace approximations. We empirically compare the approximations with prediction-space methods in $E. Importantly, the trace approximations of (E)PIG match those of JEEPIIG up to constant factor (unlike the log-determinant approximations); see BB.4 for details.
Kirsch et al. (2021b) argue that JEPIG converges to BALD in the data limit of the evaluation set-whon there are no outliers in the pool set while EPIG does not. The trace approximation is too strong to preserve this difference. Does this difference matter in practice? We leave this for future work.
Crucially, for GLMs or when using the GGN approximation, the respective active learning and active sampling objectives (EIG and IG, etc.) are equivalent as Fisher information and observed information are the same. contrast, in the general case, the approximations for EPIG and JEPIG do not have principled derivation.
12

---

Published in Transactions on Machine Learning Research (11/2022)
6 Similarity Matrices and One-Sample Approximations of the Fisher Information
Many data subset selection methods (lyer et al., 2021; Kothawade et al., 2022; 2021; Ash et al., 2019) use similarity matrices of the loss Jacobians H' [Ø æ), where y is usually a hypothesized pseudo-label: often the arg max prediction of the model for z. Here, we connect such similarity matrices to the Fisher information and the approximations of information quantities from $5. The proofs are given &C. Together with $5, this section provides unified framework for understanding the approximations of information quantities using Fisher information and lays the foundation for the next section, which will connect the cited works in $1 to the approximations of information quantities.
Connection to Fisher Information. Crucially, given D = {(15, if we let
A [ [=] Hylss wi
(68)
be a "data matrix" of the Jacobians, then 1T w (D) [w] gives the similarity matrix SID w using the Euclidean inner product:
S wly (H'y x, ". w)] H (D] w WIH'(D w
(69)
Sampling {wi} ~ p(( (xre).o w"), the "flipped" product A' (D w I A (D w5) yields a one-sample estimate of the Fisher information H" HY] [ay],ur]:
(" w]] = Σ HY (r, = Ep((v) (a.).r))
Σ H'(ya xy "", H' x, w*]
(70)
(()())))()))arr) M I'DD wi] T H w"].
(71)
Hard Pseulo-Labels. Importantly, using the arg max class for ys. we only obtain a biased estimate (Kunstner et al., 2019, SB).
Connection the Expected Information Gain. When we define an inner product H) (Dirain using the Hessian, we can connect the similarity matrix, which uses this inner product:
S (piisin [D w0]
H D
w H" - w"".
(72)
to our information gain approximations.
Specifically, we apply the matrix-determinant lemma det(AB + M) det M det(Id + BM BM-1A) -A) to obtain:
Proposition 6.1. Given Drruin, {x,0 and (sampled) {v. we have for the EIG:
I[: (Y,00)) (x(00), D train
log det (Str [Dacy w'] + Id)
(73)
2
tr Sh (Dimee Damm wn]
(74)
Proposition 6.2. Assuming uninformation posterior H [w Deruin) XId for X - O, and given D train (I,00,, 1, and (sampled) {v,"), we have for the EIG (before taking A + 0):
[[: (Y:00) (x:00), Dirain
log det (S[Deeg
w+1 + ATd)
D
209
log
(75)
2
As the second term is independent of (x,00) we can use the following prozy objective in the limit:
1
log det (S[Daeg w(l).
(76)
2
13

---

Published in Transactions on Machine Learning Research (11/2022)
Connection to Other Approximate Information Quantities. Interestingly, use the above to obtain approximations of the predictive information gains (EPIG and JEPIG) because the terms that would tend towards -oo cancel out; see $C for details. For EPIG, we have:
Proposition 6.3. Given Deval Dirain {x and (sampled) (ut we have for the EPIG:
I[y (x evaal {-;-)) Drruiny
~ Glog det (SIP'W-/DDomm) (Dti [Deval w"] + Id) lllog det (S (Ducy Deval w"] + +ld)
(77)
+ Glog det (SH" | [Dacy hoy w"] + Id).
(78)
For uninformative prior, we have:
2 log det (S(Deval w")) 1/2 log det (S)Darg, Deval w']) + 1 log det (S[Dacy w'))
(79)
We can drop the terms that only depend on Deoal when we are interested in prozy objectives for optimization.
These results help connect the objectives of PRISM and SIMILAR to the EIG and EPIG in the next section.
7
Connection to Other Acquisition Functions in the Literature
Now, we can connect approaches in non- Ba--BBaysian literature to information quantitics. Additional proofs are given $D.
7.1 BAIT in "Gone Fishing" (Ash et al., 2021), ActiveSetSelect in "Convergence Rates of Active Learning for Maximum Likelihood Estimation" (Chaudhuri et al., 2015), and EPIG (Kirsch et al., 2021b)
Ash al. (2021) introduce the BAIT objective for deep active learning:
arg arg min (I*)(()")) (z w + H" y (
train x train
w + A1)
H I ),"") (BAIT)
where À is a hyperparameter".
BAIT is based on a similar objective for GLMs from Chaudhuri et al. (2015). While Ash et al. (2021) apply this objective to DNNs, they only use the last layer to approximate the Fisher information. The last layer, with appropriate activation functions and losses, constitutes GLM as seen in $4.
Following Proposition 5.4, we immediately see that Ash et al. (2021) perform transductive active learning (using the pool set as evaluation set) and approximate proxy objective for (J)EPIG:
Proposition 7.1. Both Ash al. (2021) and Chaudhuri et al. (2015) perform transductive active learning, approrimaling (J)EPIG (Kirsch el al., 2021b; MacKay, 1992) using last-layer approach (or GLM):
arg max 1[Y cont, (Y:00) X cwal {x [2]]
20rg
arg min tr (H" [[Yee
{If
w") (H" [[Y, (x:000) ",""), w²] H [w" Drrasny)-1).
(80)
with H" H" train]
H" D troin
w+1 + ["'W") and H"] AId.
Proof. This follows immediately for GLM (last-layer approaches) when we expand H" (Drrain]. Chaudhuri et al. (2015) in particular uses an uninformative prior, that is X = 0. Comparing the resulting objectives yields the statement.
3Thhis the BAIT objective computed Algorithm in Ash (2021) and the published implementation wwwpritt_amnnnnnggppe
betps://githhi
14

---

Published in Transactions on Machine Learning Research (11/2022)
Thus, Ash et al. (2021) and Kirsch et al. (2021b) employ the same underlying acquisition function, albeit using very different approaches: Ash et al. (2021) use a last-layer Fisher information matrix, whereas Kirsch et al. (2021b) use approximate BNNs and sample joint predictions.
Research Questions. EPIG is not submodular, and the greedy selection of an acquisition batch does not come with any optimality guarantees. While Kirsch et al. (2021b) ignore this, Ash et al. (2021) propose a heuristic that empirically performs better: They greedily select additional acquisition candidates in forward pass (twice the intended batch acquisition size) and then greedily remove the least informative samples from the batch in backward pass. Would this heuristic also prove beneficial for all the other information quantities that are not submodular?
While Ash et al. (2021) state that they only use the last-layer approach for performance reasons, following $4, it does not seem that this approach translates beyond a last-layer approach for DNNs in a principled fashion (sec $B.2). Is there a principled approach for the general case that goes beyond last-layer active learning when using Fisher information without the GGN approzimation?
Ash et al. (2021) ablate trace and determinantal approaches, similar to comparing eq. (66) and eq. (65), yet they do not include +Id in the log determinant expression, which leads them to examine the EIG in their ablation":
arg min log det (H") | [ J.w"] E (yacs, 100 w H"l"" [Drraiaa)
g00y
arg min log det H [[Y,ova)) {as }. w - log det H [yacc + H"Ww" Deraiu)) = arg max log det Σ Hycc, "" w0] + ["" Diruiu)) + C,
(81)
(82)
and the last term matches the EIG in eq. (65) up constant terms independent of 009 Thus, the ablations in Ash et al. (2021) only compares EIG and EPIG. Could comporing eq. (66) and eq. (65) provide more insightful results about the trade-offs between trace and determinant approzimations?
7.2 BADGE (Ash et al., 2019) and BatchBALD (Kirsch et al., 2019)
BADGE performs batch acquisitio using similarity matrix: Using the concepts of $6, BADGE uses hard pseudo-labels together with last-layer gradient embeddings for the similarity matrix S[Dacq w²). The authors sample from a k-DPP (Kulesza and Taskar, 2011) based on this similarity matrix to select diverse batch of samples for acquisition. However, to further speed up acquisitions, BADGE uses k-MEANS++ (Arthur and Vassilvitskii, 2006; Ostrovsky et al., 2013) instead of a k-DPP: it uses the Jacobians H'j dyly/ x, w*] of the data matrix directly and samples a diverse batch based on the Euclidean distance between these Jacobians. However, sampling from k-DPPs does not pick the most informative batch overall, and the ablations in Ash et al. (2019) show that k-MEANS++ outperforms k-DPP. Finally, the paper only motivates using gradient embeddings with hard pscudo-labels through intuitions: the gradient length captures information about the model's uncertainty, and diverse update directions capture information about the model's diversity (Ash et al., 2019). The paper makes no explicit connection to information theory.
Following Proposition 6.1, since BADGE can be seen as using a last-layer approach for the similarity matrix S[Deon w with hard pseudo-labels, BADGE approximates I[2; (y.09) ((x4)) (274), Dirain] with an uninformative posterior distribution:
Proposition 7.2. BADGE marimizes an approzimation of the EIG with an uninformative prior.
Comparison to BatchBALD. Similarly, BatchBALD (Kirsch et al., 2019) approximates the EIG in the batch acquisition case but by using prediction-space samples. Moreover, BatchBALD uses greedy approach to select batch candidates instead of sampling via k-DPP or K-MEANS++.
Ash al. (2021) accidentally writes arg max instead of arg min in $5.1 in their paper, but c.f. Algorithm with the trace objective. Algorithm & in SB in the appendix use the correct final objective.
15

---

Published in Transactions on Machine Learning Research (11/2022)
As the EIG is submodular, determining the acquisition batch is submodular optimization problem and, therefore, can be solved by greedy selection with 1 - optimality (Nemhauser et al., 1978).
Research Questions. Hard pseudo-labels lead to biased estimates. Would one-sample estimates perform better? And could greedy batch selection work better than sampling via a k-DPP? This would be closer to the batch acquisition strategy followed by BatchBAL.D.
7.3 SIMILAR (Kothawade et al., 2021) and PRISM (Kothawade et al., 2022)
Based on Tyer et al. (2021); Kothawade et al. (2021) and Kothawade et al. (2022) investigate submodular active learning for DNNs: they take an information function f, which non-negative, nontone/non--decrasiii submodular function (and then is also subadditive as consequence):
f(A) 2 0,
(non-negative)
S(A) S S(B) for A C B, f(AUB) f(A) + ( -(AA B),
(monotone)
- f(AU B) f( + f(B)
(submodular)
(subadditive)
for all A, B.C S Drool and define "submodular conditional gain and an "submodular (conditional) mutual
information"
(A = f(AuB) f(B) B) B) J(AUB) - ((A)-((B).
(83)
(84)
For f(fx+)) H[()) {z:)), this simply yields the regular information quantitics. Hence, Kothawade et al.
(2021) and Kothawade et al. (2022) examine other information functions and submodular quantities in the context of active learning: amongst them set covers, graph cuts, facility location, and log determinants (LogDet) of similarity matrices. Like BADGE (Ash et al., 2019), the similarity matrix uses hard pseudo-labels. Like BatchBALD (Kirsch et al., 2019), they use a greedy approach for acquisition (Nemhauser et al., 1978).
Using our results, we immediately see that the LogDet objective, which we can write as log det S(Decq | w"), exactly matches the EIG approximation in $6.2; furthermore, in $D.1, we show that the LogDetMI objective matches an approximation of JEPIG (and similarly, derive the LogDetCMI objective as well):
Proposition 7.3. The LogDet objective log det S(Decy w+] is an approzimation of the EIG and the LogDelMI objective
log det STDang w01 log det (S w'] - SiDaci cooal [ w*] [w']'' Siperal, Dang w=1) (85)
is an approzimation of prozy objective for EPIG, where we use S wt] to denote the (non- symmettic) similarity malrix between Dj and D2.
Notably, the experimental results for the LogDet-based quantities are reported as among the best Kothawade et al. (2021) and Kothawade et al. (2022). As such, since the LogDet quantities approximate Shannon's 's information quantities (which are not explicitly examined in those works), the promising experimental results compared to other submodular information functions support the hypothesis that approximating Shannon's 's information quantities works well in active learning and active sampling.
Research Questions. Similar to BADGE, the scores are biased by using hard pseudo-labels. Could one-sample estimates perform better? Furthermore, as LogDetMI is not submodular, could the approach from BAIT of expanding and shrinking the acquisition batch in a forward and backward pass improve performance here as well?
7.4 Expected Gradient Length
The Expected Gradient Length (EGL) (Settles al., 2007; Settles, 2009) is an acquisition function in active learning and is usually defined for non-Bayesian models. Originally, it was an expectation over the gradient.
16

---

Published in Transactions on Machine Learning Research (11/2022)
norm. In more recent literature (Huang et al., 2016), it is introduced using the squared gradient norm:
| [y ang wacn Sapenau w=]]
(EGL)
Using diagonal approximation of Fisher information, we show in $D.2:
Proposition 7.4. The EIG for a candidate sample xa approzimately lower-bounds the EGL:
x " < (p(
| x "01l"""ll + const.
(86)
7.5
Deep Learning on a Data Diet
In active sampling, Paul et al. (2021) use the gradient length of given labeled samples y (averaged over multiple training runs) as an acquisition function to select the most informative samples from the training set speed up training:
Eq(w) [[ w "
w]]l?
(GraNd)
which they call the gradient norm score (GraNd). The expectation is taken over the model parameters initialization or after training for few epochs as this is not easily expressed using a posterior distribution, we use qww) to denote the distribution.
Proposition 7.5. score (GraNd) at w up a second-order term:
The IG for a candidate sample a0 opprozimately Nower-bounde the gradient norm
V P(U 2 ()) const. pyy x.ww
21[2;; you gog aoo S Eq(w)] H
[y a0r w|ll Eq(w) ftr
(87)
The second term might not be negligible. Hence, GraNd (the first term on the left) might deviate from the information gain. How does the information gain compares to GraNd in practice?
8 Conclusion & Outlook
We have examined Fisher information and Gaussian approximations and have derived weight-space approximations for various information quantities. This has allowed us to connect these information quantities to objectives already used in the literature. Moreover, we can make the following concluding points:
Last-Layer Approaches. Methods that only use last-layer Fisher information or similar perform data subset selection on embeddings only, despite feature learning being arguably the most important strength of deep neural networks. However, these approaches can find great use with large pre-trained models, which are only fine-tuned on new data domains (Tran et al., 2022).
Bias in Hard Pseudo-Labels. Several methods (BADGE, PRISM, SIMILAR) use hard pseudo-labels for computing the similarity matrices, leading to biased estimates. Can onc-sample estimates perform better? Can the equivalent approxinations that do not make use of similarity matrices perform better?
Trace vs log det Approximations. We have presented hierarchy of approximations and bounds. Ablating these approximations along multiple dimensions, including whether to use the trace log det approximation and whether to use a GLM, the GGN approximation or the full Fisher information, could provide interesting insights into what is attainable.
Batch Acquisition Pathologies. Approaches that use the matrix trace instead of the log determinant can end being additive for batch candidates (x:"0] and, therefore, by definition, cannot take redundancies between batch candidates into account, leading to failures detailed in Kirsch et al. (2019; 2021a). This is an issue for trace approximations of the (E)IG. Does the trace approximation of (JE)PIG handle batch
17

---

Published in Transactions on Machine Learning Research (11/2022)
acquisition pathologies better? Similarly, most information quantities are not submodular, yet we use greedy algorithm to select the acquisition batches. Is the heuristic proposed by Ash et al. (2021) generally beneficial?
Weight vs. Prediction Space. BADGE, BATT, and the LogDet objectives of PRISM and SIMILAR (Ash et al., 2019; 2021; Kothawade al., 2021; 2022) approximate information quantities in weight space, while (Batch)BALD, (J)EPIG, and (J)PIG (Houlsby et al., 2011; Kirsch et al., 2019; 2021b; Mindermann et al., 2022) approximate the information quantities in prediction space. Both approaches have their limitations:
Weight-space approaches can suffer from the Gaussian approximation being of low quality: the Laplace approximation only captures the posterior distribution well once it concentrates sufficiently, However, this is unlikely happen low-data regime.
Prediction-space approaches suffer from combinatorial explosion the batch acquisition size increases because prediction configurations have to be enumerated sampled to approximate the information quantities. In addition, many parameter samples might be needed to obtain low-variance estimates.
Importantly, prediction space approaches also require drawing samples from the posterior distribution but do not estimate the information quantities using the posterior distribution, unlike weight space approaches.
Informativeness Scores. Taking step back, we have seen that a Bayesian perspective using information quantities connects scemingly disparate literature. Although Bayesian methods often from (non-Bayesian) active learning and active sampling, the sometimes fuzzy notion of "informativeness" expressed through various different objectives in non-Bayesian settings collapses to the same couple of information quantities, which were, in principle, already known by Lindley (1956) and MacKay (1992).
separate
Acknowledgements
The authors would like to thank their anonymous TMLR reviewers for their kind, constructive and helpful feedback during the review process, which has significantly improved this work. We would also like to thank Freddie Bickford Smith, Tom Rainforth, Lisa Schut, and Hugh Goatcher, as well as the members of OATML in general for their feedback at various stages of the project. AK supported by the UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant reference EP BP/L015897/1)
References
David Arthur and Sergei Vassilvitskii. k-meanst The advantages of careful seeding. Technical report, 2006.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alckh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, 2019.
Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade, Gone fishing: Neural active learning with fisher embeddings, 2021.
Les Atlas, David Cohn, and Richard Ladner. Training Connectionist. Networks with Queries and Selective Sampling, Newral Information Processing Systems, 1990.
Kamalika Chaudhuri, Sham Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation, 2015.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. 15(2):201 221, 1994.
Machine learning,
Thomas M Cover and Thomas. Determinant inequalities via information theory. SIAM journal on Matrir Analysis and Applications, 9(3):384-38 -922, 1988.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Newral Information Processing Systems, 34:20089 20103, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in doep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1183 1192. PMLR, 2017.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the siath annual conference on Compulational learning theory. pages 5 13, 1993.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advonces in neural information processing systems, 7, 1994.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Longyel. Bayesian active learning for classification and preference learning. aixiv preprint urXiv:1112.57/33 2011.
Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates. Active learning for speech recognition: the power of gradients, 2016.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization, 2020.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications machine learning. In Algorithmic Learning Theory, pages 722-754. PMLR, 2021.
Andreas Kirsch and Yarin Gal. A practical & unified notation for informattion--heoreticc quantities in ml. arXiv preprinl arXin:2166 12062, 2021.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neurol Information Processing Systems, pages 7024-7035, 2019.
Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Branchaud-Charron, and Yarin Gal. Stochastic batch acquisition for deep active learning, 2021a.
Andreas Kirsch, Tom Rainforth, and Yarin Gal. Test distribution-aware active learning: A principled approach against distribution shift and outliers, 2021b.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning realistic scenarios. Advances in Neural Information Processing Systems, 34:18685-18697, 2021.
Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Tyer. Prism: A rich class of parameterized submodular information measures for guided data subset selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10238 10246, 2022.
Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In ICML, 2011.
Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approximation for natural gradient descent, 2019.
Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of Mathematioal Statistics, pages 986-1005, 1956.
Quan Long. Multimodal information gain in bayesian design of experiments, 2021.
19

---

Published in Transactions on Machine Learning Research (11/2022)
David JC MacKay. Information-basee objective functions for active data selection. Nearal computation, 4(4): 690-604, 1992.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are Icarnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630-15649. PMLR, 2022.
George Nemhauser, Laurence A Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions i. Mathemalical programming, 14(1):265 294, 1978.
Rafail Ostrovsky, Yuval Rabani, Leonard J Schulman, and Chaitanya Swamy. The effectiveness of lloyd-type methods for the k-means problem. Journal of the ACM (JACM), 59(6):1-22 2013.
Adam Passke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers. http:/papprrs..uiss.oooommm coourpp..ccppppe... hiwpiieeettt---iigg--rrrromaaaa
librrry pdf.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training, 2021.
Burr Settles. Active learning literature survey. 2009,
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. Advances in neural information processing systems, 20, 2007.
Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent, 2017.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. Information online memory selection for continual learning, 2022.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kchang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXie:2207.0741 2022.
Chaoqi Wang, Shengyang Sun, and Roger Grosse. Beyond marginal uncertainty: How accurately can bayesian regression models estimate posterior predictive correlations?, 2020.
Kai Yu, Jinbo Bi, and Volker Tresp. Active learning via transductive experimental design. In Proceedings of the 23rd international conference on Machine learning, pages 1081-1088, 2006.
20

---

Published in Transactions on Machine Learning Research (11/2022)
A Fisher Information: Additional Derivations & Proofs
Proposition 4.2. Like observed information, Fisher information is additive:
H" [x.),=]= Σ H" x. w.].
(25)
Proof. This follows immediately from Y T Y Bis "g for i + j and the additivity of the observed information:
H[ [= Ep(ie: Ep(lve)laaa )ar) [v) [xr],w"]] = BPp(ucHHaaaaeee 1 H" H [y6 x],
(88)
Σ a [M [yv Iy "I]
11 [x),"").
(89)
Proposition 4.3. Fisher information is equivalent to: H xr w]] Fppule, [y x x,,")" H'(y x w"] = [o | wr,""]
(26)
To prove Proposition 4.3, we use the two generally useful lemmas below: Lemma A.1. For the Jacobian H' [y x. w), we have:
H'y x. [x,w"]= - log ply x. [x,"")]= V p(y xx, w²)
(90)
ply w.
and for the Hessian Hy x, w"), we have:
V p(y xr k,w") pyy x,w")
H" [y 20 H'yy lr. ",,"" H'(y/ x.w")]
(91)
Proof. The result follows immediately from the application of the rules of multivariate calculus.
0
Lemma A.2. The following expectations over the model's own predictions vanish:
Eptule, [H'(e) xr, w"]l 0,
(92)
2 p(y Ir, w*) 0. Fr, ,.,"")
E F(ull,,w))
(93)
p(u
Proof. We use the previous equivalences and rewrite the expectations integral; the results follows:
V p(y x w.."")
Ep(gla,-) H'yy I, wI] pp(ol v logp(y x, w.w")l Ep(e(
(94)
p(u x (+
V p(y 1 x,,")) dy = - /
/P(U p(yl x,ww") dy = - - = 0,
(95)
V2 p(y I, w,)"")
pluluo dy Vz / / (plllaw))dg - = 0.
E (p
(96)
plu F, (.ww))
Proof of Proposition 4.3. With the previous lemma, we have the following.
CY x, "I = E x. 1T H'[Y x. [[
[ x., 1. E[H [Y x [,,"]]]
(97)
21

---

Published in Transactions on Machine Learning Research (11/2022)
EY w H |
(98)
For the expectation over the Hessian, we plug Lemma A.1 into Lemma A.2 and obtain:
Vi p(y 2. wn)
Y x w.w"] = Ep [H by x w" = E H'y x w'I H' ar w-]
(99)
p(u xt w*)
p((lle.")) [H ly "" H'y ar "]]""] - = CovlH'lY [r]""]]
(100)
A.1 Special Case: Exponential Family
Proposition 4.4. The Fisher information HY x, w. for model p(y 1 2 = wxx;=")) is equivalent to:
HY [,,"] (=x(== ) = [V
(27)
where Hly f(x;=") short for V H 2= (
Proof. We apply the second equivalence in Proposition 4.3 twice:
Y [x,w"] Cov (Y x, []] Cov Co V H 1 f(=;w=) Vw
(101)
(ul(x;=²) Cov(V: Hly 2 = f(x; ( )]] Vn (xr;w)
(102)
= -ui(x;"') Epcole (V2 Hly = [[] Vul(aw"")
(103)
A.2 Special Case: Generalized Linear Models
Proposition 4.7. The observed information H" [y w of a GLM is independent of y.
H" [y x [] (=f(=;"") VHu 2 Vu Vwf(r;"") VEA(W x) v=f(x;"").
(31)
(32)
Proof.
H xr w"]
(104)
V H' x, w]
(105)
V Hly è f(x;w")) V fx;=")
(106)
V. Hly f(z;w") V f=((=="") + FVl("a""" Hy f(z; V=f(z;"")
(107)
=V
Vuf(z,"") VEA(W x) Vaf(x;"").
(108)
Proposition 4.9. For GLM, when f(x:w) RD. - RK where K is the number of classes (outputs), D is the number of input dimensions, w E R DxK, and assuming the parameters are flattened into single vector for the Jacobian, we have Vuf(a,w) = Idx 8 of E (R))(K-D), where & denoles the Kronecker producl, and:
Vullaw"") "VAa""""""AA"""" Vo. Vul(=="")
ViA(w= (000 xx..
(35)
22

---

Published in Transactions on Machine Learning Research (11/2022)
Proof. We begin with a few statements that lead to the conclusion step by step, where E R?, A RCx, Ge R C
(xx"); I4 z.j (ldc 2 x )odd I; llc d}
(109)
(110)
AG); = =E Gbs Ald Gij
(111)
k,
(A M8xx DDD+iDD+ = Aed Di Tij. (Ide (Udc&x')) A (Idc @ x" )x")))
(112)
Σ
(Ide @ x² D4i Au (Ide @ x² DaDD+
(113)
k,l
Σ k.l
24 - 1lk = c} A& 2j 1f1 d}
(114)
5q Acd Tj
(115)
(A x DeD+idD+s
(116)
Vul(sw") [2) 0rx"" = Vullaw")) ((A(T)) Vallan""\
(117)
Proposition 4.8. Por model such tuat the observed information H" [y x. wn) is indeeeeneett of y, we have:
Y xr, w-) = H"y"" Ir, w")
(33)
for any yf, and also trivially:
Y x.,")] = F [y x, "]]
(34)
Proof. This follows directly from Proposition 4.5. In particular, we have:
H ar w²] Ep(s ww [H" [y |x, wr,w"]] H" ar, w*),
(118)
where we have fixed y to arbitrary value.
B Approximating Information Quantities
B.1 Approximate Expected Information Gain
Lemma 5.1. For symmetric, positive semidefinäte matrices A, we have (with equality iff A = 0):
log det((A + Id) < tr(A).
(46)
Proof. When A is positive semidefinite and symmetric, its eigenvalues (A:); are real and nonnegative. Moreover, A + Id has cigenvalucs (A; + 1);; det(AA + Id) = II,(A, + 1); and tr A = E. A.. These propertics casily follow from the respective eigenvalue decomposition. Thus, we have:
deg S log (A++ 11) Logg(, +1) 5 Σ A =t(AA),
(119)
where we have used log(x + 1) S x iff equality for x = 0.
General Case. In the main text, we only skimmed the general case and mentioned the main assumption.
Here, we look at the general case in detail.
For the general case, we need to make strong approxinations to be able to pursue similar derivation. First, cannot drop the expectation; instead, note that the log determinant. is concave function the
23

---

Published in Transactions on Machine Learning Research (11/2022)
positive semidefinite symmetric cone (Cover and Thomas, 1988), and we can use Jensen's inequality on the log determinant, term from Equation (41) as follows:
E () det (1, (,.-t) w,"" [" Id)]
(120)
I log det "p((o/"<aa""" (K)"")) [H ((, ) [x=]]) H" [-]
+ Id)
(121)
Second, we need use the following approximation:
p((, (2) = p((v ( w*)
(122)
to obtain Fisher information and use its additivity. That is, we obtain:
Epttar (aw) Hv, )) (x J w']] = H } (a (,;")),),,,, ),,",, E H" (yac, | x
(123)
Plugging all of this logether and applying Lemma 5.1, we obtain the same final approximation:
[[ (Y,00) {x] ... ((((o"""" Dog det det (H" ( { w"] H" [w-]- Id)]
(124)
log det
"[(("""")[<"""" [H" [,, } ( (x))ae)] w'Il H --] + Id]
(125)
N log det
((((")""))("""""")"""" ( [ K } {a J, w wIll H [w + 1d)
(126)
2 log det
Σ H'|yacy; i w5 H --- + ld
(127)
tr (Yagy 2 w-] H"
(128)
Unlike in the case of generalized linear models, a stronger assumption was necessary to reach the same result. Alternatively, we could use the GGN approximation, which leads to the same result.
B.2 Approximate Expected Predicted Information Gain
In the main text, only briefly referred to not knowing a principled way to arrive at the same result of Proposition 5.4 for the general case. This is because unlike the expected information gain, the Fisher information for an acquisition candidate now lies within an matrix inversion. Even if we used the fact that degdetdd + XY-1) is concave in X and convex in Y, we would end up with:
(129)
1/ F Fppyrrol ggann pomal gomwa (mma) p(anvat) [log det (H" nevvr ww") [H"(yac 2000 w + (" + Td)] (130)
< 2 Ep(yocq|auea) [log det (Epryerus gorat [H ovvvll
levar wi]l ooo 2009 w01 + H ( Id)]
(131)
2. log det (Eproeol y [H" oval ooval w1 (Ep(y>eq)1 H |ا و w|]l + H" Id)
(132)
log det (Ep(zmai) [H" reval evall wI'l (Ha b0n ~-' + H" [w)]] + Id)
(133)
s for (Ep((orat) [H" eval ww] yacg وعت w') + (
(134)
Note the ... 2, which invalidates the chain. The errors could cancel out, but a principled statement seems hardly possible using this deduction.
B.3 Approximate Predictive Information Gain
Similarly to Proposition 5.3, we can approximate the predictive information gain. We assume that we have access to an (empirical) distribution (va ,yeval::
24

---

Published in Transactions on Machine Learning Research (11/2022)
Proposition B.1. For generalized linear model (or with the GGN approcimation), when take the expectation over D (ax yy yycuu), we have:
arg max [[Yrvul,, :y y X evat an D Dirain] arg min [[2; youd X eval nnn xan Deratn) go0g
(135)
with
1[2; y eval X enal yacg a Direinn
(136)
(p(aatn 1[2 y coal reval yang x 200y Dirusuy
Ep (z yount) gemmt l log det (1 oval x aual w"] wa booo war
"mn ww) + (w"" (Diminn) + Id)]
(137)
< 2 log det (Ep ( yee eval eeval w']l (H) ( boad
| x 2009 wn] + H" | DDrrain)) + Id)
(138)
frr (Ep(asol gronty [H" eval evval wI]l (H wor wi]  [" D
(139)
All of this follows immediately. Only for the second incquality, need to use Jensen's incquality and that the log determinant is the posilive semidefinite symmetric cone (Cover and Thomas, 1988). Like for the information gain, there is no difference between having access to labels or not when we have a GLM or use the GGN approximation.
B.4 Approximate Joint (Expected) Predictive Information Gain
A comparison of EPIG and JEPIG shows that JEPIG does not require an expectation over f(z but uses of evaluation samples (arval). As such, we can easily adapt Proposition 5.4 to JEPIG and obtain:
Proposition B.2 (JEPIG). For generalized linear model (or with the GGN approzimation), we have:
arg max [[(Y,eva)) [Y,00)} {2 (2,0²), D Dinasn = arg min 1[22; [Yrevel) {Işval), (Y,e0), (2,00), Deruing I (x
(x
(140)
with
[[; (Y,eul)) | {2z (Y.ao)) (x Deroin
blog det (H")(Y) (a. w01l (H' ((Y, (,:"))a) wn1 + H"lw" D Id)
(141)
$ Str (H" [[V,eoul] | (or w (H" ((Y,e)) (x,"0), wi + H"("" Diramt)-1)
(142)
Similarly, for JPIG, we obtain without relying on the GGN approximation or GLMs:
Proposition B.3 (JPIG). We have:
arg max [[ 20 {2g (x,00) (,D)),D"nn Dinain = arg min I[2; Eugualy fagaa,, (uto)) (x:00) Drrain (1
(143)
with
T[2: (vlmal) (argual), (Y.er),, (or)), Derain)
flog det Hyraay (agval) w0] (11"1) (00) (1 w + [" | Diroat)) + 1d) $ tr (H"( fo w0 Kye") far0 w1 + [w" D
(144)
(145)
Comparison between (E)PIG and JEE)PIG approximations. As observed information and Fisher information additive, the difference between the approximations when have empirical, that is finite,
25

---

Published in Transactions on Machine Learning Research (11/2022)
evaluation distribution p(x with M samples is factor of E inside the log determinant trace:
Pa (eggeaalggrala govat) (y oval " = E Σ H Hy
| ". w E
1 H" (ut } (a J,w")
(146)
For the log determinant, E log (A + Td) log det( A + Td), but for the trace approximation, that both approximations equal up to constant factor. For example:
i ( [Y | ww'] [H" {at (0 }, wn] (w" (Dtrain))
(147)
uir H1 { ", (H" ((Y, { w,w"] + H (w" (Dtrain))1))
(148)
F.
28 tT("
{ } w6I (H" [ [[ {2?-)) w H [w] D
(149)
C Similarity Matrices and One Sample Approximations of the Fisher Information
Proposition 6.1. Given Dirain {x:²²} and (sampled) (y, we have for the FIG:
[[2; (Y,00) ( Deraby
log det (Wi"ii (Dany ['' + Id)
(73)
trr SHi Iwr [Dramm] [Dam w*]
(74)
Proof.
[[2;; (Y,0C)) I ( Diraiu
A
1 log det
(H" ([Y, (2x ",," H lwo* - + Id)
(150)
(HH [[Y,ea) (ooiie) limm + H"["" Dirain)) H" [w- Diraing
} loog det
(151)
( wo] An Dun wn) + H Dirain)) H" (w-" DDrainn)1
1 log det
(152)
log det [ wn) H" - (Daca (DA' whr + Id) (153)
= blog det (Str)u-)pmmm))) (Ducg w0] + Id),
(154)
where we have used the matrix determinant lemma::
det(AB + M) = det (BM(BM--A4 Id) det M.
(155)
Connection to the Joint (Expected) Predictive Information Gain, Following eq. (57), JEPIG can be decomposed the difference between two EIG terms, which we can further divide into three terms that are only conditioned on Dirain.
[[(roval); ); yrcq (afval) ), acg Dirain]
(156)
= =1[Ω; (Yeeal)) {I,val), Dirain 1[2;; {Y.eval) {agval), yrac,, gaca Dirain
I[2; (Y,eva)) (2r Dirainj I[2: (Y.vvl), yaoq [nrgva)
200 Diraing + I[ yooq 000 Drrain] (157)
Using Proposition 6.1, we can approximate this as:
I[(Y, yacq الس }, x aoq braain
(158)
~ blog det (H - Limmind Deall wn] + Id) - -loog det (Dinaiii DDacq [Dacy, Deval [w0 + (P)
(159)
+ blog det (h |mmlid| Daaq [w*] Id)
(160)
26

---

Published in Transactions on Machine Learning Research (11/2022)
Furthermore, can apply the approximation in Proposition 6.2 and find that the log A terms cancel because [Dacq] + |Duraiu] = 'Dacq U D Taking the limit À -> 0, we obtain:
[[[Y,val) y {2. x a00 Dirain]
(161)
~ I log det (s(peval w"] + XId) - 1 log det (SiDacq, Deval w=] + AId) + 1 log det (S(Dacq | + A/d)
(162)
log det (S(peval
w=)) 2 log det (S(Dacy Deval w=)) + 2 log det (S(Dacq | w w'))
(163)
Finally, the first term is independent of Daca, and if we are interested in approximately maximizing JEPIG, can maximize as proxy objective:
log det (SHr) [Dtramn] (Ducq w-] + Id) - log det (SHr |Drraam [Ducg Deval w-1 + Id).
(164)
log det (S(Dacq [w")) - log det (S(Daca Deval wn)).
(165)
Connection to Other Acquisition Functions in the Literature
D.1 SIMILAR (Kothawade et al., 2021) and PRISM (Kothawade et al., 2022)
Connection to LogDetMM. If we apply the Schur decomposition to log det SyDucg Deval w*] from eq. (165),
we obtain the following:
log det S(Dacq w*] log det S(Daca, Deval [w']
(166)
= log det STDucq wn] log det S(Deval w*]
(167)
- llg (S w') - Sipacy, Deval w w Deval wn] S[Deval, Daca w")),
where S(Dacy; Deval
w*] is the nonsymmetric similarity matrix between Dacq and Deval etc.
Dropping log det S(Deval w*] which is independent of Dacq, can instead maximize:
log det S(parg w'] log de [w"] S(Durs, Deval I w||SSroval |w"] S(pmaal Dang w*).)
(168)
which is exactly the LMlDetMM objective of SIMILAR (Kothawade et al., 2021) and PRISM (Kothawade et al., 2022).
We can further rewrite this objective by extracting S(Dacq w*] from the second term, obtaining:
log det S(Dacq [h) - log det (S(Dacq w²] - S(Dacq Deval w "[[[evll | Dacq wnl)
(169)
- log (Id- - S[Dacq w S(Daca, Deval w w*]S[Deval | S(Deval, Dacq w=1).
(170)
Connection to LogDetCMI. Using informatiin-thhoreed decompositions, it is easy to show that:
[[(Y:vvl); yan | (lvvia) garg, (Y:), {x:), Derain
(171)
[[(Y:oml) y yacn, {K} { ), "0 {2,), D Dirainj - 1[(X; (Y.) {2i), Diraing
(172)
These are JEPIG terms, and using above approximations, including (170), leads to the LogDet objective:
det(Id - S[Dacg w/| - SIDaca; Deval | | [w*] S(Deval, Daog w'1))
log
(173)
det(Id SIDaca, D [4] SIDaca, D; Deval w S [+]] S'Deval, Ducg D w+]l
27

---

Published in Transactions on Machine Learning Research (11/2022)
D.2 Expected Gradient Length
Proposition 7.4. The EIG for a candidate sample x044 upprozimalehy lower-bounds the EGL:
2 21 Y y
x 20 < "0
|| , w']l + const.
(86)
Proof. The EIG is equal to the conditional entropy up to a constant term, via eq. (51) in Proposition 5.2:
[[2 yoo x ( log det (yacy 1 + H" (Dirsin)) + const.
(174)
We apply a diagonal approximation for the Fisher information and Hessian, noting that the determinant of the diagonal matrix upper-bounde the determinant of the full matrix:
S log det (H'gaag [Yaaq ra09 w"] + H'diagl" | Dirsiu)) + const.
(Y Σ log (Hiiaag [Yacq 1=00 w/] + H Haiag,ualu" (Dtaain) const.
(175)
(176)
We use logæ and that Iw Dirain) is constant:
H'lsogg (Y (aag 200 w"] H ding kk lws
Dirraiu) + const.
(177)
Σ H ki Yung 2000 w"] const.
(178)
From Proposition 4.3, we know that the Fisher information is equivalent to the outer product of the Jacobians: [[[[aacq]a | w'] =x) [yace a00g w"] H' aann 2009 w*'",, and we finally obtain for the diagonal elements:
Σ Epppacul w [ 2,00,00 w + const.
(179)
Epp mon ww) H
000 w 11] + const.
(180)
D.3 Deep Learning on Data Diet
Proposition 7.5. The IG for condidate somple a0 approrimately lower-bounds the gradient norm score (GraNd) at w up to second-order term:
V p(y xr (m:w)) const. p(y x.w)
21[2; yoy x 2or Eq(w)
||| 1y0 000, w|]l Eg(w) [tr
(87)
Proof. For any fixed w", the 1G equal the conditional entropy up constant term, via Proposition 5.3:
iΩ;1 Y l < log det (H" ang w"] H"[ [Dtrain)) + const.
(181)
As in the previous proof, we apply a diagonal approximation for the Hessian, noting that the determinant of the diagonal matrix upper-bounds the determinant of the full matrix:
< z log det (Hittag lypcn "00 + H'hsgll" (Dtrain)) + const.
(182)
= 1 Σ log (alii aog [,a0 + H Hi ww" Dirainj) + const.
(183)
Again, we use logx S x and that H" [ Dirain] is constant:
<
1
aoq 2009,w + H" Hi to (w" Dirainn) + const.
(184)
2
28

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2: Spearman Rank Correlation of Prediction-Space and Weight-Spuce Estimates. BALD and EPIG are both strongly positively rank-correlated with each other. The weight-space approximations are strongly rank-correlated with the pppdiction-space approximations, but the weight-space approximations are less accurate than the prodiction-space approximations. Note that we have reversed the ordering for the proxy objectives for JEPIG and EPIG as they are minimized while EPIG is maximized.
BALD
EIG
EIG
EPIG
EPIG
JEPIG
(J)EPI (Trace)
(Prediction)
(LogDet)
(Trace)
(Prediction)
(LogDet)
(LogDet)
BALD (Prediction)
1.000
0.955
0.940
0.984
0.948
0.955
0.927
EPIG (Prediction)
0.984
0.918
0.897
1.000
0.918
0.918
0.903
10a
10a
10
Score
Score
10-2
10-4
10-6
(E)GG (Trace)
200
400
600
800
1000
(E)IG (LogDet)
Pool samples (descending by avg rank)
10-8
BALD (Prediction Space)
EPIG (Prediction Space)
J(E)PIG (LogDet)
200
400
600
800
1000
(E)PIG (LogDet)
[JEPIG (Trace)
Pool samples (descending by avg rank)
(J)EPIG Approrimations The scores match approximations match for small scores (because of qualitatively. Note we have reversed Lemma 5.1). They diverge for large scores. Qualitatively, the ordering for the proxy objectives for the order matches the prediction-space approximation using JEPIG and EPIG as they are minimized BALD with MC dropout.
Figure 3:
Figure 2: EIG Approzimations. Trace and leg (Normalized).
while EPIG maximized.
S 1 Σ H Hal ,00 + const.
(185)
From Lemma A.1, know that the Hessian is equivalent to the outer product of the Jacobians plus second-order term: Hy(ya 1,00,,w" wn] = H' yoon ,=0 w²1 H' (ylya 2 w-|T V p(y ww") and we finally obtain for the diagonal elements:
ply p ,0")
V2 plyan 2arq, ww))
H' ly agn Σ
arg w0 - tr
+ const.
(186)
plyacq pack wi)
V2 plaraoo zaتa w²) p(yacq gaca w²) + const.
H |yy 200 w
1/2 tr |||
(187)
Taking an expectation over w" ~ (ww) yields the statement.
E Preliminary Empirical Comparison of Information Quantity Approximations
The following section describes an initial empirical evaluation of the bounds from $5 on MNIST. We train a model on a subset of MNIST and compare the approximations of EIG and EPIG in weight space to BALD and EPIG computed in prediction space. We use a last-layer approach (GLM) which means that active sampling and active learning approximations are equivalent. We do not attempt to estimate JEPIG in prediction space.
Codee https://github com//laacat//2208005544
29

---

Published in Transactions on Machine Learning Research (11/2022)
Acquisition Function EPIG (Prediction Space)
Acquisition Function (JE)PIG (Trace)
540
0.025
520
0.020
500
Score
0.015
480
460
0.010
440
0.005
420
0.000
Acquisition Function (E)PIG (LogDet)
Acquisition Function J(E)PIG (LogDet)
5585
122
5580
5575
120
Score
5570
118
5565
5560
116
5555
5550
200 400
600
800
1000
200
400
600
800
1000
Pool samples (descending by rank)
Pool samples (descending rank)
Figure 4: (J)EPIG Approzimations. The scores match quantitatively. Note the proxy objectives for JEPIG and EPIG are minimized while EPIG is maximized. The value ranges are off by a lot: the true EPIG score is upper bounded by log K 2.3 nats.
30

---

Published in Transactions on Machine Learning Research (11/2022)
Setup. We train BNN using MC dropout 80 randomly selected training samples from MNIST, achieving 83% accuracy. The model architecture follows the one described in Kirsch et al. (2019). We use 100 Monte- Carlo dropout samples (Gal and Ghahramani, 2016) to compute the prediction-space estimates. For EPIG, we sample the evaluation set from the remaining training set (20000 samples). We randomly select 1000 samples from the training set as pool set. We compute BALD and EPIG in prediction space as described in Gal al. (2017) and Kirsch et al. (2021b). For the weight-space approximations, we use a last-layer approximation we thus have GLM. For the implementation, we use PyTorch (Paszke et al., 2019) and the laplace-torch library (Daxberger al., 2021).
We chose 80 samples and 83% accuracy as the accuracy trajectory of BALD and EPIG is steep at this point, see e.g. Kirsch et al. (2019), and thus we expect a wider range of scores.
Results. In Figure 2, we see a comparison of BALD with the approximations in eq. (48) and eq. (49). Not shown is oq- (51), which performs like eq. (48) (up to a constant). In Figure and Figure 4, we show a comparison of EPIG with the approximations in Equation (65), Equation (66), and Equation (141). Figure 3 shows normalized scores individually as the score ranges very different.
Importantly, while the prodiction-space scores (BALD and EPIG) have valid scores as the EIG EIG/BALD /BALD and EPIG scores of sample are bounded by the log K, the weight-space scores are not valid. As such, they only provide rough estimates of the information quantities.
However, as we see in Table 2, the Spearman rank correlation coefficients between the weight-space and prediction-space scores are very high. Thus, while the scores themselves are not good estimates, their order scems informative, and this is what matters for selecting acquistion samples in data subset selection.
Future Work. The quality of these approximations needs further verification using more complex models and datasets. Comparisons in active learning and active sampling experiments are also necessary to validate the usefulness of these approximations. However, given the connections shown in $7, we expect that the approximations will be useful these settings as well.
31


Reviews:
[{"review_id": "RkZdUPAuuo", "reviewer": "Reviewer_do8M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper has two main strength points:\nS1) A good single exposition of many steps that are usually \"left to the reader\" in the original papers, in the form of a comprehensive chain of prepositions\nS2) Insights in the \"true\" motivation behind model choice in recent literature. As an example they show why GLM and exponential models, despite being restrictive, result in much simpler information measures and easier approximations. This might be well known to experts in the field but sheds some light for someone that just approached it.\n\nThe main weaknesses are:\nW1) Despite the attempt to unify all the approaches, many of these are only a loose fit in the framework and the authors have to fall back to more vacuous statements such as \"approximately lower bounds\". Similarly, many of the original methods were already originally derived starting from an information measure perspective but had to cut corners and implement heuristics to become practically efficient presented   (e.g. BADGE replacing k-dpps with k-means++). This makes fitting them in a clean framework (including the one presented by the authors) more difficult.\n\nW2) Creating a new unified framework can have various benefits. One is simplifying exposition and understanding, which the authors achieve. Another is using the more general formulation to obtain novel insights, which the authors only partially achieve (e.g. last layer and batching pathologies were known, but the authors give a stronger and cleaner understanding). Finally, using the more general framework to explore new algorithmic design space that interpolates between existing special cases, which the authors do not attempt.\n\nW3) The work is narrowly tailored in an information theoretic/bayesian perspective, but many of the comments regarding entropy and information gain approximation can also be tackled from a sub-modular function approximation perspective. It would be important to at least introduce this to reader as a potential separate path when looking for unifying frameworks.\n\nW4) A minor comment would be that on a 12 page paper, the \"meat\" of it (i.e. tying the framework to the actual algorithms) takes place in only the last 2 pages (section 7). This leaves the reader a bit confused trying to backtrack the logic flow of the individual algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contribution-wise, the authors' main claim is the derivation of a taxonomy of DSS approaches based on information theoretic quantities.\n\nMore in details, the authors begin by introducing two fundamental quantities related to the entropy of a posterior: its hessian (for taylor expansion) and the fisher information (sec 2,3,4). Then (sec 5,6) the authors show how some of the most popular and generic active learning acquisition measures (EIG and [J]PEIG) can be approximated using these quantities, and provide some insight on the behavior of these approximation under different assumptions. Finally (sec 7) the authors try to interpret several recent contributions in the active selection literature (BAIT, BADGE, EGL, ...) as different approximation approaches to these information measures, in an effort to unify their presentation and provide a general understanding the choices of each different approach when constructing the approximation.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "W1) This is more on the impact side and future work, i.e. having a better/stronger framework\n\nW2) Again on the impact side, but more immediate. It would be good to consider at least one new intermediate approach in detail, either to show that it might have new strengths, or to motivate a negative result of why it would fail and avoid other people from wasting time on it in the future.\n\nW3) Introduce references to sub-modularity\n\nW4) Easier said than done before trying to write the paper myself. Maybe spread section 7 across Sec. 5/6 to be closer to the logical path, or include a proof graph as a visual aid to navigate the derivation. ", "broader_impact_concerns": "No broader impact needed", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s0p5TdVKnx", "reviewer": "Reviewer_7mfr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the paper delivers on its aims.  It does show us how a wide array of existing acquisition functions can be understood to be approximations to information gain.  The paper also contains a lot of useful mathematical manipulations and lemmas and could serve as a  useful reference for people interested in these sorts of things.\n\nHowever, I feel as though the paper could be greatly improved with some editing.  As it is written, I found the paper very difficult to read.  In general, the paper lacks signposting and hand holding.  It isn't until page 11 that we really get some payoff from what forms the bulk of the paper, which is some rather detailed mathematical manipulations.  And even then, what feels like the primary selling point of the paper, actually demonstrating how a wide class of acquisition functions can be understood through the same lens, feels quite rushed.  \n\nThe math is good and results can be interesting, but at present the paper really stretches the reader's good will.  As a reviewer I had a very strong incentive to read the paper carefully and many times.  Perhaps people who work very closely with acquisition functions, active learning and active sampling would be similarly motivated to work through it, but I can't help but feel as though the paper could attract a much wider audience with some edits to the prose focused on saying more of the quiet parts out loud.\n\nAs a reader, I would appreciate a lot more hand-holding, signposting and direction. Through much of the paper I wasn't quite sure what we were doing, why we were doing it or why I should care.  Then once the result was generated, I felt abandoned when the paper moved on to the next section without helping me to appreciate what we had just accomplished.  Overall it really felt like a lot was left unsaid.\n\nAs a small microcosm of the sort of thing I'm trying to get across, at the bottom of page 2 we have a bolded section in the Background and Setting section that tells us that Transductive Objectives are when an objective uses additional data to guide acquisitions.  Being right at the beginning of the paper, and having a dedicated bolded section, as a reader, I was expecting transductive objectives to play a dominant and immediate role.  However, it isn't until page 9, 7 pages later that we'll see the word transductive appear again, where it is essentially redefined for the reader.  In total it only appears 4 additional times throughout the whole paper.  This strikes me as a mismanagement of the readers attention and valuable real estate on the second page of the paper.\n\nAnother example of the same sort: the active learning and active sampling sections immediately afterwards.  Here, the goal is to define IG, EIG, PIG, EPIG, JPIG, and JEPIG.  After my first reading of the section I was feeling very overwhelmed, as despite all 6 of these quantities being very closely related, my initial read left me only knowing that 6 different things have been defined and I was worried as a reader I would be expected to now be able to maintain the distinctions betwixt them. They seem to, rather simply, span the set of three independent choices: a difference in whether the acquisition labels are observed or inferred (\"expected\"), the difference between being information defined in weight or prediction space (\"predictive\") and the difference between being defined per example or jointly for a batch (\"joint\"). Not being familiar with these different acquisition functions before reading the paper, this was the sort of work I had to do as a reader that I would have instead appreciated the paper did for me.  Section 2 *defines* what these different acquisition functions are, but doesn't help me *understand* what they are and how they relate.\n \nI understand that styles differ but right now, the paper puts all of the cognitive load out front, Section 2 defines many terms before we know why we need them. Section 3 then takes us through the second order approximation of the posterior and associated entropy but prior to any motivation for why we might be interested in such a thing.  It wasn't until my second read through the paper that I appreciated that the second order approximation was crucial for establishing the links to the other acquisition functions.  Then at the same time, after working through Section 3 for the first time, we end with Preposition 3.5 and then we immediately move on to Section 4.  I feel as though with just a few sentences at the beginning of Section 3 telling the reader what we are about to do and why, and then a couple more sentences at the end of Section 3 spelling out for the reader what was just accomplished and why it is interesting, readers on their first passes through the paper, will have a much improved experience.\n\nIn the same sort of spirit, overall, I feel a weakness of the paper is a failure to elucidate its own results.  For example, in Section 5 we work out formulas for EIG, IG, EPIG and in the approximate second order and exponential family approximations the paper explores.  However, I'm not sure what the payoff is.  For example we generate equation 48, but as a reader, I'm not sure why we care.  Is this formula easier to evaluate than other approximations of EIG?  Does this formula make it easier to understand some properties of EIG or reason about the accuracy of the approximation?  Is this something that will be useful later in the paper, or useful later in my life?  Really, I was hoping for a lot more sections like the, well done, \"Batch Acquisition Pathologies\" or the \"Comparison to EIG\" sections.  Here the paper takes a moment to reflect on what has been done and help make connections in the reader's mind.  Please add more sections like those throughout.  Please let the reader stand on your shoulders more.\n\nReaching the end of section 5, we are told that the same kinds of approximations for JEPIG, PIG and JPIG are in the appendices.  This again feels like a bit of a rug pull.  As a reader I'm left wondering whether, either, the JEPIG, PIG and JPIG approximations are important and I've been robbed of them in the main text due to space, or more pessimistically, left to wonder if perhaps the approximations for EIG, IG, and EPIG are also perhaps not so important if their brethren could so easily be disposed of.  Why should we care about these approximations, what do they teach us?\n\nHaving laid all of the groundwork, in Section 7 the paper is able to very quickly establish links between many existing acquisition functions BAIT, ActiveSetSelection, BADGE, SIMILAR, PRISM, EGL, and GraND. We are now capable of seeing how they arise from various second order approximations to various forms of the information gain.  The paper does accomplish its goal of establishing these links, to its credit, and given the extensive space dedicated to the mathematical setup, the associations can be made rather quickly: to its strength.  However, again, I have to admit, as a reader, after my first few reads of the paper, I felt robbed.  \n\nSection 7 goes by very quickly, and being terse it assumes a high level of familiarity with the existing literature, as well as a great deal of familiarity with the previous sections of the paper.  I understand the triumph that comes from finding a unified perspective or unique lens by which you can suddenly make sense of whole swath of existing work and I believe such perspectives should be shared and cherished.  At the same time, as authors you need to realize that this realization likely took a good amount of time and effort on your part to build up familiarity with the new framework before they clicked as being so obvious.  For readers of this paper, the paper itself is the only introduction they will have had to this new framework, and so I think you owe the readers a great deal more explanation, a more detailed exposition on the relationships and power the new framework provides.  \n\nI also would very much appreciate a more thorough overview of what BAIT, ActiveSetSelection, BADGE, SIMILAR, PRISM, EGL, and GraND themselves are or do.  I realize space is always tight, however if the one of the primary goals of this paper is to leave readers understanding how all these various acquisition functions relate, I feel more time spent explaining what these various acquisition functions *are* even if traded for more space for the mathematical details (which can always be relegated to an appendix), the better as far as adding rhetorical punch goes.  Honestly, I'd suggest considering replacing the current background section 2 with a background section that gave an overview of these acquisition functions.  I feel that would do a better job to motivate our task of unifying them.\n\nThe Conclusion, in Section 8 does a good job of letting the readers stand on the authors shoulders.  It helps us understand from a high level the implications of what we have just worked through.  I wish only there was more of it, that these sorts of high level reflections on the work being done in the paper would appear repeatedly and throughout the paper itself.  I'm left with many questions.  Some of these methods approximate information quantities in weight space while others approximate in prediction space, so what does that tell us?  Does that effect their accuracy? Their utility? Their deficiencies? Should we prefer one over the other?  I now understand that these things differ, but again, I'm not sure why I should care, I'm not sure how knowing this helps me.\n\nOverall, I believe the paper does what it purports to do, and that there is an audience for its findings.  At the same time I believe it has a lot of potential to reach a much larger audience and provide a much greater benefit to its readers which some edits to the prose.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Overall, the primary goal of the work appears to be to argue that a wide array of existing acquisition functions used in active learning and active sampling (BAIT, ActiveSetSelection, BADGE, SIMILAR, PRISM, EGL, and GraND) can be understood as various approximations to the more traditional information theoretical acquisition function of information gain (IG, along with its variants EIG, PIG, EPIG, JPIG, JEPIG).  In particular, these existing acquisition functions are motivated as approximations of information gain powered by a second order, laplace-like, approximation of the posterior distribution, which motivates approximating information gain by means of the properties of the Fisher information matrix.\n\nAs a secondary contribution, the paper includes a wide array of useful mathematical summaries and results, including the same fisher powered approximations of classic acquisition functions.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, I hope its clear that my biggest request would be a fairly substantial edit to the prose of the paper.  A lot more signposting, reflection, repetition and handholding would help the reader a lot.  I think it would substantially strengthen the impact of the paper and the size of its audience.  I'm not entirely sure if this is or should be a requirement for my acceptance, but obviously, if taken seriously I would have no hesitation recommending the paper for acceptance.\n\nSome other lesser suggestions:\n\nIn Equation 8, some brackets would help disambiguate the action of the operators.\n$$ \\nabla_\\omega \\log p(\\omega^*) (\\omega - \\omega^*) = \\left( \\nabla_\\omega \\log p(\\omega) \\right) (\\omega - \\omega^*) \\neq \\nabla_\\omega[ \\log p(\\omega^*) (\\omega - \\omega^*)] $$\nor if we want to avoid additional brackets just swap the order of the terms\n$$ (\\omega - \\omega^*) \\nabla_\\omega \\log p(\\omega^*) $$\nor minimally a footnote about your binding convention for $\\nabla_\\omega$.  Its a minor problem and the reader can figure it out, but overall one of the strengths of the paper is its careful and concise notation so the ambiguity here is a bit jarring.\n\nI think there is a spurious extra comma in equation 17 $H(\\mathcal D | , \\omega^*) \\to H(\\mathcal D | \\omega^*)$\n\nThe \"empirical\" fisher note, as worded I think does more harm than good.  The notation is very suggestive of the other \"empirical\" fisher as there isn't any distinction between the model sampled labels ${y_i}$ and observed empirical data points $\\{x_i\\}$ except in the functional form on the far left.  This might be a good time to use hats or something for the xs?  I'm not really sure, but I find the notation in 22 very confusing and it tripped me up on first read.  Perhaps start with definition of the per instance Fisher in 24 and then define the Fisher in 4.1 with the sum in 23?\n\nI think 25 would be aided with the intermediate equality $-\\nabla_\\omega \\log p(y|x,\\omega^*)$.\n\nBut as a broader question why are Lemma 4.4 and 4.5 included in the main body?  Are they useful outside of the proof in appendix A?\n\nProposition 4.7, typo, I think you want to say exponential family distribution not exponential distribution.\nand again on the line below eqn 30.\n\nIn the yellow box in the conclusion, please add the acronyms for the acquisition functions along with the references.\n\nminor nits, where they are defined, the (joint) expected predictive information gain (EPIG and JEPIG respectively), you should change the order in which they appear in the parentheses, no?\n", "broader_impact_concerns": "I don't have any broader impact concerns about the paper, it is very abstract and theoretical in nature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ijJZYKWyZ", "reviewer": "Reviewer_xAk1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Establishes upper bounds on common information theoretic quantities of interest\n- Provides mathematical examples for special cases (GLMs, exponential families)\n- Thorough background section and relates previous works to one another\n\nWeaknesses/Questions:\n- Presentation of the background was not easy for me to read. At times I was unsure of what was new work vs existing work. More room for motivation, comparison of effectiveness of these different methods.\n- No mention of implementation details or whether relevant software packages already exist.\n- Limited analysis of the quality of the bounds. How “good” are these approximations in practice? Further, how good is the bound?\n- The empirical Fisher information matrix has been found to not always give good second-order information (in the Kunster 2019 paper that is referenced). How does this affect the upper bounds?\n- Just above section 5.2:  “The log determinant of the Fisher information version might well capture these dependencies.”\nIs there more to support this conjecture (intuition, theory)? How could we check this? Does this necessarily mean a tighter bound?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper connects a variety of existing methods used in active learning for sample selection and provides detailed background information on existing techniques in the field. It focuses on using methods of approximating the Fisher information and Hessian to come up with new upper bounds for common sampling criteria (EIG, IG, EPIG, JEPIG, PIG, JPIG) and connects previously introduced information quantities to each other.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please address the concerns and questions in the weaknesses section. ", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the acquisition function in active learning and active sampling. It shows how a wide array of existing acquisition functions can be understood as approximations to information gain, emerging from a second-order posterior approximation. Through their paper, the authors contribute to a more unified perspective on these different approaches and to a better understanding of the involved approximation choices. 

The paper's original version was arguably hard to follow, but it significantly improved during the review due to the authors' ongoing efforts and in response to the reviewers' guidance. The authors significantly improved the paper through reorganization, the addition of clarifying figures, and the inclusion of more background material on existing approaches. It will serve as a reference for researchers working on active learning. 
Decision: Accept (poster)