Bayesian Quantification with Black-Box Estimators
Authors: Albert Ziegler
Paweł Czyż
Venue: Accepted by TMLR ()

Abstract: Understanding how different classes are distributed in an unlabeled data set is important for the calibration of probabilistic classifiers and uncertainty quantification. Methods like adjusted classify and count, black-box shift estimators, and invariant ratio estimators use an auxiliary and potentially biased black-box classifier trained on a different data set to estimate the class distribution on the current data set and yield asymptotic guarantees under weak assumptions. We demonstrate that these algorithms are closely related to the inference in a
particular probabilistic graphical model approximating the assumed ground-truth generative process, and we propose a Bayesian estimator. Then, we discuss an efficient Markov chain Monte Carlo sampling scheme for the introduced model and show an asymptotic consistency guarantee in the large-data limit. We compare the introduced model against the established point estimators in a variety of scenarios, and show it is competitive, and in some cases superior, with the non-Bayesian alternatives.

Published in Transactions on Machine Learning Research (06/2024)
Bayesian Quantification with Black-Box Estimators
Albert Ziegler
albert@abow. com com
XBOW, Head of AI Uppsala, Sweden
Pawel Czyž
pawelpiotr.cay ethe.
ETH AI Center and Department of Biosystems Science and Engineering
ETH Zürich
Zürich, Swilzerland
Reviewed on OpenReview: https: // openreview. net/ forum? id- Ft.kHHrowz
Abstract
Understanding how different classes are distributed in an unlabeled data set is important for the calibration of probabilistic classifiers and uncertainty quantification. Methods like adjusted classify and count, black-box shift estimators, and invariant ratio estimators use an auxiliary and potentially biased black- classifier trained on different data set estimate the class distribution on the current data set and yield asymptotic guarantees under weak assumptions. We demonstrate that these algorithuns are closely related to the inference in particular probabilistic graphical model approximating the assumed ground-truth generative process, and we propose Bayesian estimator. Then, we discuss an efficient Markov chain Monte Carlo sampling scheme for the introduced model and show an asymptotic consistency guarantee the large-data limit. We compare the roduced model against the established point estimators in a variety of scenarios, and show it is competitive, and in some cases superior, with the non-Bayesian allernatives.
1 Introduction
Consider medical test predicting illness (classification label Y), such as influenza, based on symptoms (features X). This often can be modeled as an anti-causal problem' (Schölkopf et al., 2012), where Y causally affects X. Under the usual i.i.d. assumption, one can approximate the probabilities P(Y | X) using a large enough training data set.
However, the performance on real-world data may be lower than expected, due data shift: the issue that real-world data comes from different probability distribution than training data. For example, well- calibrated classifiers trained during early stages of the COVID-19 pandemic will underestimate the incidence of the illness at the time of surge in infections.
The paradigmatic case of data shift is prior probability shift, where the context (e.g., training and test phase) influences the distribution of the target label Y, although the generative mechanism generating X from is left unchanged. In other words, Perain (X Y) = Prest(X X Y), although Perain (Y) may differ from Prss (Y). If (Y) is known, then Ptest X) can be calculated by rescaling Pusiia (Y X) according to Bayes' theorem (sec Sacrens et al. (2001, Sec. 2.2) or Schölkopf et al. (2012, Soc. 3.2)). However, Puest is usually unknown and needs to be estimated having access only to finite sample from the distribution (X).
This task is known as guantifteation (González et al., 2017; Forman, 2008).
Although quantification found applications in adjusting classifier predictions, it is an important problem on its own. For example, imagine an inaccurate but cheap COVID-19 test, which can be taken by a
While influenza causes high fever, many medical problems the causal relationships much more complex (Castro al.,
2020).

---

Published in Transactions on Machine Learning Research (06/2024)
Y
(N&
Fix
N
T
(NL
Mtrue
N'
Mapprox
N
M
High-dimensional model Mirue- Dashed arrows are used to create low-dimensional repre- sentations C, and C²2 using given black-box function f. Filled nodes represent observed random variables. The top row represents the labeled data set and the bottom row represents the unlabeled data set. Mid- dle: Tractable approximation Mapprox (Sec. 3). Right: Model Mamall enabling fast inference when is black-box classifier clustering algorithm (Sec. 3.2).
Figure 1: Left:
significant fraction of the population on a weekly basis. While this test may not be sufficient to determine whether a particular person contagious, the estimate of the true mumber of positive cases could be used by epidemiologists to monitor the reproduction number and by the health authorities to inform public policy?
We advocate treating the quantification problem using Bayesian modeling, which provides uncertainty around the Prest ostimate. This uncortainty be used directly if the distribution the whole population is of interest, il can be used to calibrate probabilistic classifier to yield a more informed estimate for the label of a particular observation.
A Bayesian approach was already proposed by Storkey (2009, Sec. 6). However, that proposal relies on a generative model P(X Y), which can be misspecified or computationally intractable in high-dimensional settings. Hence, quantification is usually approached either vía the expectation-maxiizaaiii rithm (Saerens al., 2001) or a family of closely-related algorithms known as invariant ratio estimators (Vaz et al., 2019), black-box shift estimators (Lipton et al., 2018), or adjusted classify and count (Forman, 2008), which replace the generative model P(X Y) with (potentially biased) classifier. Tasche (2017), Lipton et al. (2018), and Vaz et al. (2019) proved that these algorithms are asymptotically consistent (they rediscover Prest(Y) in the limit of infinile data) under weak assumptions and derived asymptotic bounds on the related error.
(FM) algo-
Our contributions are:
We show a connection between the quantification algorithms employing a black-box (and poten- lially biased) classifier with Bayesian inference in a probabilistic graphical model approximating the ground-trut generative process.
2. We present a tractable Bayesian approach, which is well-suited for low-data problems. Established alternatives provide asymptotic estimates on error bounds, but may be far off for small samples (to the point that some the estimates for PY) may negative). The Bayesian approach explicitly quantifies the uncertainty and does not suffer from the negative values problem. Moreover, is possible incorporate expert's knowledge via the choice of the prior distribution.
We prove that the marimum a posteriori inference in the considered model is asymptotically con- sistent under weak assumptions.
2 The quantification problem and existing solutions
Consider a classification problem with y = (1, 21, 2..., L} labels and observed features coming from a measur- able space X. A given object is then represented by two random variables (r.v.): a Y-valued r.v. representing
Note however that testing strategies may be adapted the outbreaks, which in induce correlations between observed data, violating the usual assumption that the data exchangeable. We discuss contraindication in Sec.
2

---

Published in Transactions on Machine Learning Research (06/2024)
the label and X-valued r.v. representing the measured features. We consider anti-causal problem in which there exists a probabilistic mechanism P(X Y), responsible for generating the ures from the label. We focus on two populations sharing the same generative mechanism, assuming that there exist probability distributions Purain every y E V. In the literature this assumption referred to as prior probability shift (Storkey, 2009). We will write Ky for the conditional distribution P(X | Y = y), which is the generative mechanism shared by both populations.
and (X, over X' x y such that ( Y y) = Persin (X Y = y) for
The quantification problem (González et al., 2017) asks whether given finite ii.d samples from the distri- butions Pirain(X, (X and Puest (X) it is possible to determine the distribution Piest (Y). In principle, if the data are abundant, one can use samples from Pirain (X, Y) to determine the conditional distributions Kg and then find all probability vectors (Y))t()) which are compatible with the distribution of the features (X), written as a mixture distribution of generative mechanisms Ky:
Piest (X) = Piea = y) Kÿ-
Piest yEY
(1)
The uniqueness of the vector (Y) follows under strict linear independence assumption of mcasures K (Garg et al., 2020). We review the notion of strict linear independence in Appendix A, but for a finite discrete space X it reduces to the linear independence of probability vectors Ký, which allows constructing the left inverse of the P(X Y) matrix.
In practice, however, it is not possible to fully reconstruct the distributions (X) and Ký from finite samples and principled statistical approaches are needed. To formalize the problem, consider a probabilistic graphical model Murue in Fig. I: probability vectors Peeain (Y) and Piest (Y) are modeled with r.v. and n' valued in the probability simplex A = = E 10, y1 + +... + VL ] and the ground-truth generative processes Ke are modeled via parametric distributions (y(;0) with a parameter vector 0. We observe N pairs of r.v. (X,, Yi) for E (V sampled independently according to the model
Y π ~ Categorical
X; Y,,0 (:;)-
Additionally, we observe N' r.v. X for E (1....,N') N'} sampled independently from the mixture distribution
X3 n 0~ Σ T Ky(:0),
or, if latent variables Yf valued in y are introduced,
Yj I n' ~ Categorical(L. T')
X3 1j,0 0 Kyy(:0).
2.1 Likelihood- based methods
Our work draws on two major groups of quantification methods, with the description of other approaches deferred to Appendix D. The first group proceeds by considering a generative probabilistic model of the data.
Peters& Coberly (1976) assume that each ()(:0) a multivariate normal distribution. Then, they estimate 0 using labeled data {X;, Y.) and find the maximum likelihood solution for T' by an iterative optimization algorithm. Storkey (2009) discusses approaching quantification problems within fully Bayesian estimation, which requires marginalization of f, and notices that such marginalization may not generally be tractable for complex generative models (=):)). Moreover, tractable generative model of high-dimensional data is likely to be misspecified,, which may compromise Bayesian inference (Watson & Holmes, 2016; Lyddon et al., 2018).
Sacrens et al. (2001) observe that specifying high-dimensional distributions (=(:)) may be avoided if one instead has access to oracle probabilistic classifier r: X AL-1 such that each r(x) P(Y X; 2r, R).
3

---

Published in Transactions on Machine Learning Research (06/2024)
Then, they show how to use a candidate value x' to recalibrate r(x) and marginalize latent variables Y (FM) manner, which iteratively optimizes n', targeting the maximum
in the Expeetation-Maxiiiaallle likelihood estimate. In Appendix D.1 we give a detailed treatment of this algorithm, together with two simple extensions: when a Dirichlet prior is used for P(r'), EM targets the mazimum a posteriori (MAP) estimate of the posterior distribution P( (X)):r). Moreover, we describe a Gibbs sampler allowing drawing from the posterior P(x'| (x)),r)
However, this posterior is generally different than P(x|) {X,), (x, Y,)), which requires marginalization over r, and the performance of the EM algorithm depends on the access to the oracle classifier r, which has to be well-calibrated (Garg et al., 2020). As modern classification methods, such as neural networks, are often miscalibrated (Guo et al., 2017), one has to leverage the labeled data set (X;, Y, to improve calibration. Alexandari et al. (2020) introduces calibration methods which can yield the state-of-the-art results using the expecatiin-maxxiizztion algorithm.
Although both approaches are based in sound likelihood framework, Peters & Coberly (1976) require learning high- digensinnni generative models (X;:0) P(X Y v. and Saerens et al. (2001) assume an access to a well-calibrated oracle classifier P(Y, X1, r). Moreover, each iteration of all mentioned approaches requires operations involving all N' variables Xj. This may limit the scalability of either algorithm to large data sets.
2.2 Methods involving an auxiliary black-box classifier
The second group of approaches is based around a modification of Eq. 1 and assumes access to given auxiliary black-box mapping: consider a given measurable space C and a measurable mapping" f: X' + C. For example, f can be a pretrained feature extractor (such as a large language model), clustering algorithm, or generic classifier, trained on large data set with possibly a different set of categories.
Then, one can define new observed r.v. C. = ((x)) and C;= = f(x),, which in Fig. corresponds to the part of the diagram with dashed arrows. Note that the new variables act only summary statistic and do not increase the amount of information available. Namely, given {(X,, Y.) and (X)), the r.v. x' is independent of {C,} and {C),, i.c.: N' TT {C.), [C]] f(x,, Y)), (X}).
However, the prior probability shift assumption implies that the distributions P(C, | Y - v) are equal for an arbitrary label y E y and indices i, j. In particular, Eq. can be used with original features X replaced with the newly introduced representations C = f(X). As they are of lower dimension than X, it may be easier to approximate required probabilities with the available data samples.
v) P IY/=
For example, Vaz et al. (2019) propose invariant ratio estimators, which generalize earlier approaches of adjusted classify and count (Forman, 2008; Tasche, 2017) and its variant introduced by Bella et al. (2010). Namely, for a given mapping f: X' >> RRL11 one constructs
8'= I= N MCC,
C P.y Ssp
ΣG,, where (1, N) Y; y),
SESy
and solves the set of equations given by fr' and कं +... #z = 1. In Appendix D.2 we review the closely-related algorithms employing black-box classifiers and based on matrix inversion (solving a set of lincar equations), including the popular algorithm of Lipton et al. (2018).
Estimators employing black-box classifiers offer four advantages over likelihood-based methods. First, as auxiliary mapping f can produce low-dimensional representations, estimating probabilities appearing in Eq. 1 may be more accurate. Secondly, Peters & Coberly (1976) require training a potentially high-dimensional generalive model and Saerens et al. (2001) require a well-calibrated oracle probabilistic classifier, which may be hard to obtain in practice. Third, each optimization step in a likelihooddbased method requires O(N') operations, Black-box method f has to be applied only once to each X to construct the summary statistic, which is then used for solving a linear set of equations (cf. Eq. 1). Finally, even when P(X Y) is not invariant
3 Although for the clarity of the exposition will use a notation corresponding a measurable function f. the results hold mutatis matandi for an arbitrary Markov kemnl (Klenke, 2014, Sec. 8.3), so that does not need to be deterministic.
4

---

Published in Transactions on Machine Learning Research (06/2024)
(i.e., the prior probability shift assumption does not hold), for appropriate dimension reduction method f it may hold that the distribution of low-dimensional representations, P(C' Y), is invariant (Arjovsky et al., 2019). Lipton et al. (2018) calls invariance of P(C Y) the weak prior probability shift assumption.
However, at the same time methods employing black-box dimension reduction methods f have three un- desirable properties. First, dimension reduction methods may incur loss of information (Fedorov et al., 2009; Harrell, 2015, Sec. 1.3). In particular, even if the ground-truth distributions K; P(X Y = y) are strictly linearly independent, the pushforward distributions P(C Y = y) do not have to be. Secondly, solving Eq. 1 requires approximating probability distributions basing the laws of large mumbers: although these methods have desirable asymptotic behaviour, likelihood-basee methods explicitly work with given finite sample. Finally, solving a linear set of equations is not numerically stable when P(C Y) has large condition number. In the next section we show how to solve the last two issues within our proposed Bayesian framework.
3 Bayesian quantification with black- shift estimators
We work in the setting of Fig. 1 with N labeled examples (X,, Yx) and N' unlabeled examples .i,..., X'x obtained under the prior probability shift assumption. Additionally, we assume that we work with a given dimension reduction mapping f: X + C. A fully Bayesian treatment (Storkey, 2009) relies assumed parametric generative mechanism (X; P(X Y = y, 0) and marginalizes over all possible values of parameter 0 to obtain the values of the latent variables æ and ''. From the graphical structure in Fig. 1 we note that the posterior factorizes as
P( x ((x, YJ), (x))) P(T (Y.)) P(#| I (X)), ((,,, YDJ),
and
P( (X]), ((X, x P(t) III (X: 9) II> x (x,,(
(2)
The posterior P( (Yi)) is analytically tractable when Dirichlet prior P(œ)) is used, the difficulty in quantification relies in finding P(r' {Xf), {(,,, Y.)).. If 0 is of moderate dimension, this distribution can be approximated by using Markov chain Monte Carlo (MCMC) algorithms (Betancourt, 2017) by jointly sampling n' and 0 from the posterior P(r', 0 ((),, (X₃}) and retaining only the x' component. However, in complex problems involving high dighndimensional variables and large sample sizes N and N', MCMC methods become computationally expensive, which may limit their applicability (Betancourt, 2015; Bardenet et al., 2017: Izmailov et al., 2021). Moreover, if parametric kernels (y o) are misspecified, which is arguably often the case in high-dimensional problems, the resulting inference may be compromised (Watson & Holmes, 2016; Lyddon et al., 2018); investigate this issue in Sec. 4.3.
Both tractability and robustness to model misspecification can be simultaneonsly addressed by employing the provided black-box fcature extractor f to replace X, with C, and X; with Cj: Lewis et al. (2021) propose to improve robustness to model misspecification in regression models by conditioning on an insufficient summary statistic, rather than original data. In our case, we consider the conditional distribution
P( I (C), (()) & P(a)) / R (C:
II T ())(())))
(3)
where Ky(p)) are distributions the low-dimensional space C, rather than on nigh-dimensional space X, parameterized by vector y. Although it is possible to take & = and define k(: to be the pushforward measure of k(=0)) by the dimension reduction method f, we generally hope that a low-dimensional distri- bution (c) may require fewer parameters and yp will be of a much lower dimension than 0, making the integral from Eq. 3 more tractable than Eq. 2.
Apart from improved tractability, conditioning on summary statistic may improve the robustness due to casier specification of low-dimensional distributions (c:gg). Finally, even if the prior probability assumption does
5

---

Published in Transactions on Machine Learning Research (06/2024)
not hold, i.e, P(X YY) is not invariant, the distribution of low-dimensional representations P(C Y) may be invariant (Arjovsky et al., 2019), which in notation of Lipton et al. (2018) corresponds to the weak prior probability shift assumption. On the other hand, conditioning on an insufficient statistic loses information: the trivial approximation C = {1} and f(x) = forgets any available information and results in the posterior being the same as the prior, P(r' (C,,)),
P(r'), even in the limit of infinite data.
Although the outlined methodology of approximating the intractable inference with simpler model with given black-box dimension reduction method f is general, below analyse the simplest possible model, where C = {1, ...,, K} and f is given by black-box classifier, clustering algorithm, trained potentially very different data set.
3.1 The discrete model
Consider C = f1.2...... K} and a given black-box function f: X C. For example, f can be miscalibrated classification algorithm trained on entirely different data set (in particular, it is possible that K L) or function assigning points to predefined clusters. If K < L, we find the full Bayesian posterior, rather than providing point estimate
not able to identify (Yest()) basing on
the outputs of f. However, based on matrix inversion, the posterior distribution on Ií' may shrink along specific dimensions, providing accurate estimate of class prevalence for several classes. On the other hand, if K 2 I. and there is a strong enough correlation between outputs of and ground-truth labels, the guarantees on methods employing black-box shift classifiers and matrix inversion (Tasche, 2017: Lipton et al., 2018; Vaz et al., 2019) ensure identifiability of the prevalence vector n' in the large data limit, we will see in Theorem 3.1.
In this case, the model (x):p)) is particularly simple independently of the true data-generating process K as each of (:) distributions is supported on a finite set C, they have be categorical distributions. Namely, o (Pyk) is a matrix modcling the ground-truth probability table P(C k Y = y) and the model will not be misspecified provided that the weak prior probability shift assumption holds and that the prior ip, and n' is positive the simplices A K (for each py:) and A L (for and r').
The approximate model Mapprox takes the form
n, 6 P( Yi n (a m) C Yi Categorical(K, Y n' Categorical(L,**), C2 1/.00 Categorical pyy:).
We do not put specific requirements on the prior P(π, n .4) other than being positive on the probabil- ity simplices: for example, the Dirichlet distribution or the logistic normal distribution can be used. If precise problem-specific information is not available, we recommend using weakly informative prior distri- butions (Gelman et al., 2013, Soc. 2.9), such the uniform prior over each simplex. However, we note that if the Dirichlet priors used, the model conceptually resembles very low-dimensional variant of latent Dirichlet allocation (Pritchard et al., 2000; Blei et al., 2003), with observed r.v. Y: and C; providing information on the matrix. In Section 3.2 we will show how to construct scalable sufficient statistic and perform efficient inference using Hamiltonian Markov chain Monte Carlo methods (Betancourt, 2017).
Although for K < the model is not identifiable, for K 2 I. shares asymptotic properties similar to the alternatives based on matrix inversion. In Appendix C we prove the following result:
Theorem 3.1. Assume that:
1. The weak prior probability shift assumption holds, i.e., Ptrain(C Y) Pteat(C Y) is invoriant between the populations.
Y =v)) yk is of rank L and all entries are strictly positive. 2. The ground- ground-truth truth matrix D- = (P(C = x
The ground- truth prevalence vectors n² Peraso(Y) and T" PY)ss(Y) have only strictly positive entries.

---

Published in Transactions on Machine Learning Research (06/2024)
4. The prior P(T, n², () is continuous and strictly positive on the whole space A x (AK-1, x ... x A -
A A
Then, for every and > 0, there exist N and N' large enough such that with probability at least 1-6 the maximum a posteriori estimate R,, is in the e-neighborhood of the true porameter values ,," ,,'""""
Compared to the traditional approaches, do not explicitly invert the matrix P(C Y) (modeled with 4), as any degeneracy is simply reflected in the posterior, showing that we did not learn anything new about the prevalence of some classes. However, if the full-rank condition holds, the mazimum a posteriori estimate asymptolically recovers the true parameters. This result is similar to the standard results on methods employing matrix inversion, which typically assume conditions 1-3. As an additional condition, we require that the prior positive on the whole space, which ensures that the ground-truth parameters lic in the region with positive densily. This result in conceptually similar to the classical Berustein von Mises theorem linking Bayesian and frequentist inference in the large data limit, and in particular depends on the assumption that the model is not misspecified. However, we stress that in Bayesian statistics one should use the whole posterior distribution, rather than relying on single point estimate. In Appendix C.1 we provide a further discussion of this point.
3.2 Fast inference in the discrete model
One advantage of methods employing black-box classifiers and matrix inversion over likelihood-based methods is that they require O(N + NN) computation time to preprocess the data, and then estimate is generated by matrix inversion, which is polynomial in K and L. Likelihood-based methods, however, require generally at least O(N') operations per each likelihood evaluation. As such, these methods may not be scalable enough to large data sets when extensive resampling methods, such bootstrap (Efron, 1979; Tasche, 2019), are roquired.
For the discrete Bayesian model, however, it is possible to preprocess the data in O(N + N') time and then evaluate the likelihood and its gradient in polynomial time in terms of K and L, without any further dependence on N or N'. In this section we show how to construct a sufficient statistic for τ, T', and p, whose size independent on N and N' and allows one to perform efficient inference with existing sampling methods.
Define K-tuple (Ee) of r.v. summarizing the unlabeled data set by N₂ = [ E (1....., N') C. =)} which can be constructed in O(K) memory and O(N') time. Then, for each y E y, we define a K-tuple of r.v. (Fot)kEe, such that Fyg- = Hi E (I....,N} Y, = y and C, = k3/, which requires O(LK) memory and O(N) time. Finally, we define an L-tuple of r.v. (Ny)vey by N₂ = Fyt +... Fyr.
is proportional to the likelihood In Appendix we prove that the likelihood P((Y, C.), C.)(C))) k, ,,, P((N)) (NK), (Fys) ,, s) in a smaller model, Mamall:
(N₂) π Multinomial(N, TT,
(Fy.) N2 Multinomial(N,, Pp:))
(N%) ,' Multinomial("".pP p"T" m²)
Hence, by the factorization theorem of Halmos & Savage (1949), we constructed a sufficient statistic for the inference of π, n², 9,, whose size is independent of N and N'. In turn, (rather than Mapprox) to sample n, n' and yp from the posterior, allowing us to perform each likelihood evaluation in O(LL), rather than O(N + N'), time. Moreover, the gradient of the likelihood is available, so we can use any of the existing Hamiltonian Markov chain Monte Carlo algorithms (Betancourt, 2017: Hoffman & Gelman, 2014).
use the likelihood of Masall
4 Experimental results
We evaluate the proposed method in four aspects. In Sec. 4.1 we analyze the benefits of using Bayesian approach, rather than matrix inversion, in problems which are identifiable, but where the matrix P(C YY)
7

---

Published in Transactions on Machine Learning Research (06/2024)
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
2
2 Class
2
Class
Class
Class
Figure 2: Uncertainty of the prevalence estimates in the nearly non-identifiable model. Samples from the proposed Bayesian posterior quantify uncertainty better than bootstrapping point estimators based on matrix inversion.
has large condition number, requiring large number of samples be estimated accurately. In Sec. 4.2 we compare how the posterior mean compares with the existing point prediction methods employing black-box classifiers using extensive simulated data sets. In Sec. 4.3 investigate how the posterior in approximate model, P(x'| {Y, C.), (C))), compares to the posterior P(r') (Y, );), (Xf1) with properly specified, as well as misspecified, generative models. Finally, in Sec. 4.4 we present an application of quantification mel hods to the problem of estimating cell type prevalence from single-cell RNA sequencing data. all settings, we assume no specific knowledge of the problem (which could be used by principled prior clicitation) and use the uniform distributions as the priors for n, n², and Sy: vectors.
4.1 Tighter estimation for hard-to-identify model
The Bayesian inference does not require explicit inversion of the P(C Y) matrix. We therefore hy- pothesise that it may be preferable in cases where P(C Y) has large condition mumber. Hence, in this section we consider a case with I, = K = 3 and a given black-box classifter which can only weakly distinguish between classes and 3, i.e, the ground-truth matrix P(C Y) given by
0.96
0.02
0.02
80" = (p(())
0.02
0.50
0.48
0.02
0.48
0.50
Although the matrix is full-rank (and asymptotic identifiability for all methods employing black-box classifiers holds), having an access to finite sample may limit practical ability accurately estimate the prevalence.
We simulated data set with N N' 1000 data points and ground-truth prevalence vectors л" (1/3, 1/3) and x'* = (0.5, 0.35, 0.15). In Fig. 2 we plot posterior samples from the proposed Bayesian model together with the bootstrapped (Efron, 1979) predictions of three methods employing black-box classifiers and performing explicit inversion: restricted and unrestricted invariant ratio estimators (RIR and UTR respectively; Vaz et al. (2019)) and black-box shift estimator Lipton et al. (2018) (BBS). We used 100 bootstrap samples using the stratified bootstrapping procedure introduced for quantification problems by Tasche (2019); in Appendix E additionally reproduce this experiment varying the sampled data sets.
We see that the Bayesian approach identifies the component T/, Icaving large uncertainty entries wh and zz. On the other hand, all bootstrapped methods struggle with estimating any of the components. Moreover, UIR and BBS often result estimates with negative entries. As we further illustrate in Appendix E, this bchaviour is typical for low sample sizes, and bootstrapped predictions become stable for N = N' 10 samples. However, the proposed Bayesian approach does not suffer from these low-data issues, appropriately quantifying uncertainty.
8

---

Published in Transactions on Machine Learning Research (06/2024)
b.
BAY
MAE
0.1
BBS
CC
0.0
RIR
0.6
0.8
102
10*
0.6
0.8
Prevalence
n''
Unlabeled sample size N'
Classifier quality
d.
MAE
0.1
0.0
5 7
0.6 0.8 Misspecified quality q'
Classifier outputs K Number of labels L K
Figure 3: Mean absolute error for quantification using simulated categorical black-bo classifiers under different scenarios, the proposed Bayesian approach (BAY) shown in bluc.
4.2 Simulations employing the discrete model
Although we advocate for quantifying uncertainty around the prediction of x', quantification typically posed as a point cstimation problem. We therefore compare the point predictions three matrix-inversion methods mentioned before (RTR, UTR, and BRS) with the posterior mean the Bayesian model (BAY) and a simple baseline known as "classify and count" (CC; Tasche (2017) proves that this method may not converge the ground-truth value even in the high-data limit).
Experimental design In this paragraph we describe the experimental design with the default parameter values. They are changed one at time, as described in further sections, and for each setting we simulate labeled and unlabeled data sets $ = 50 times and, for cach method, we record the mean absolute error (MAE) between the ground-trute value x'* and the point estimate a'. Using root mean squared error (RMSE) does not qualitatively change the results (see Appendix E.2).
We fix the data set sizes N = 100 and N' 500 and use - K = 5 default setting. The ground-truth prevalence vectors are paramctrized as π = (1/L, 1/L) and x'((r) = 1-r I- 1.=1) By default, we use r = 0.7. The ground-truth matrix P(C YY) is paramcterized as b = q and Pyk = (1 g)/(K - 1) for + y and K 2 L, with the default value q = 0.85. Whenever < we use o 1/K for y E (T. 1,, 2,... K} to obtain valid probability vector.
Changing prevalence We investigate the impact of increasing the prior probability shift (the difference between and T²) by changing πं E (0.5, 0.6, 0.7, 0.8, 0.9} and summarize the results in Fig. 3a. CC is adversely impacted by strong data shift. The other estimators all perform similar to each other.
Changing data set size We investigate whether the algorithms converge to the ground-truth value in the large data limit. We vary N' e {10, 50, 100, 500, 10?, 10*3. As shown in Fig. 3b, the large data limit appears very similar (except for CC), agreeing with asymptotic identifiability guarantees for BBS, RIR and our MAP estimates, although BBS appears slightly less accurate than the others in a low data regime.
Changing classifier quality We investigate the impact of classifier quality by changing it in range g e (0.55, 0.65, 0.75, 0.85, 0.95] and show the results in Fig. 3c. All considered method converge to zero error for high quality, but the convergence of CC is much slower than for the other algorithms.
Changing the classification granularity We change K E {2, 3, 5, 7,9}, creating a setting when a given black-box classifier, trained on different data distribution, still informative about some of the classes, but provides different information. In particular, the CC estimator cannot be used for K + Although the original formulation of BBSE and IR assumes K L,, we proceed with least square error solution. Our choice

---

We see that the performance of BBB, IR and BAY estimates deteriorates for large discrepancies between q and q'. However, for q - - 0.05, the median error of BBS, IR and BAY is still arguably tame, so we hope that these methods can be employed even if the prior probability shift assumption is only approximately correct. Published in Transactions on Machine Learning Research (06/2024)
P(X M
Prrin(X (X) and (X)
Posterior samples
Coverage
1.00
Y-0
Train Test
Gaussian
Y-1
Student
0.50
Discrete (5 bins)
0.25
Discrete (10 bins)
0.00
0.25
0.75
Pses(Y = 11
Expected
Figure 4: First two plots: conditional Student distributions P(X Y) and the training and test populations. Third plot: posterior samples under four different. Bayesian models. Fourth plot: coverage of credible intervals.
of 4" given above guarantees that the classifier for K > L = 5 will contain at least as much information as a classifier with a smaller number of classes. Conversely for K < L, the information about some of the classes is not possible for the matrix P(C) | Y) to have rank
will be insufficient even in the large data regime I, and asymptotic consistency does not generally hold.
The results shown in Fig. 3d. While all methods considered (apart from CC) suffer little error for K 2 L, note that our model-based approach can still learn something about the classes for which the classifier is informative enough, while the techniques based on matrix inversion are less effective. Additionally, we should stress that the Bayesian approach gives the whole posterior distribution on n' (which can still appropriately shrink along recl-recoverable dimensions).
Changing the number of classes Finally, we jointly change I, = K (2,3, 7,9). We plot the results in Fig. 3e. Again, classify and count obtains markedly worse results, with smaller differences between the other methods.
Model misspecification Finally, we study whether the considered approaches are robust to breaking the weak prior probability shift assumption: the unlabeled data are sampled according to a different P(C' Y) distribution, parameterized by o' The weak prior probability shift assumption corresponds to the set q' = q, which marked with a dashed black line in Fig. 3f. Although in this case asymptotic identifiability guarantces do not hold, we believe this be an important case which may occur in practice (when additional distributional shifts are present).
Note that in the case when g' > q, (i.e., the classifier has better predictive accuracy on the unlabeled data set than on the labeled data set, which we think rarely occurs in practice), CC outperforms other methods.
4.3 Uncertainty assessment in misspecified model of mixture of Student distributions
As mentioned in Sec. 3.1, using black-box function f: X - C to reduce the dimensionality to set (1,,,..., K) not only improves the tractability of the problem, but also has the potential to make the model more robust to misspecification by replacing the prior probability shift assumption (invariance of P(X Y) with its weak version (invariance of P(C Y)) and by learning the parameters of a categorical discrete distribution, rather than parameters of a potentially misspecified distribution K, (X; On the other hand, Mapprox loses information from the problem, so we do not expect to be as appropriate as a properly specified generative model for P(X Y).
To investigate properies of the Mapprox approach generated low-dimensional data, so that Bayesian inference in M is still tractable: we consider mixture of two Student t-distributions presented in Gaussian mixture model
Fig. 4 from which we sample N = N' = 10x points. We implemented NumPyro (Phan et al., 2019), setting weakly-informative priors
and a Student mixture distribution on their parameters (see Appendix E.3). For the Mapprox we partitioned the real axis into K bins: (-oo, -4), [-4, a1), a2)..)..... .ao......a..s.... [ak-s, 4), [4, oo), with all intervals (apart from the first and the last one) of equal length.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Samples
Cell types
All cell types
Aggregated cell types
Proportion
0.50
Proportion
0.50
Observed
BBS
0.25
0.25
EM
0.00
0.00
(soft)
Astrocyte
Neuron OPC
Olgo. Vascular
Myeloid Malignant OPC
Vascular
Astrocyte
Oligo.
Neuron
Myeloid
Malignant
OPC
Other
Astrocyte Oligo.
Malionant
Myeloid
Figure 5: First two panels: principal components of the feature vectors X colored by the biopsy sample and cell types. Third pancl: inferred cell type proportions. Fourth panel: inferred cell type proportions with vascular cells and neurons merged into one type.
In Fig. 4 we see that using 10 bins yields very similar posterior to the one of a well-specified model and that using 5 bins yields wider posterior, which agrees with the perspective that discretization resulted in information loss. However, misspecified Gaussian mixture model concentrates around wrong value.
We repeated the simulation S = 200 times, excluded the runs with convergence issues (see Appendix E.3), and checked the frequentist coverage of the highest-density credible intervals. The coverage of the discretized models as well as the properly specified Student mixture agrees well with the expected value. However, the credible intervals from the misspocified Gaussian mixture are systematically too narrow. As we demonstrato in Appendix E.3, misspecification less problematic for lower sample sizes and more problematic for large sample sizes.
We conclude that, in the considered setting, conditioning on an insufficient statistic, as proposed by Lewis et al. (2021) for regression problems, is a viable solution also for quantification. However, it is not as data-efficient as a properly-specffied generative model if one is available. We did not compare obtained posteriors in high-dimensional problems due to high computational costs associated with running MCMC on high-dimensional generative models.
4.4 Prevalence estimation in real world data
As a more practical application of the proposed quantification method, we consider single-cell RNA se- quencing data. Darmanis et al. (2017) collected biopsy specimens from four glioblastoma multiforme tumors corresponding to four different populations cells. Each cell belongs to one of six healthy types (astrocyle, neuron, oligodendrocyte, OPC, myeloid or vascular) or a malignant cancer cell, yielding in total = distinet cell types.
Each cell is sequenced to yield gene expression vector X with 23,368 entries. Basing gene expression vectors and known marker genes, the cells have been assigned to the considered cell types we treat provided annotations as ground-truth labels. In Fig. 5 we plotted the first two principal components of the whole data set to visualise the distribution of features within each sample and with relation to the cell type. We note that although Darmanis et al. (2017) used TPM normalization (Zhao al., 2021) to normalize the features X and the cells seem roughly cluster within cell types, the samplc-specific effects are still visible (sce also Appendix E.4), so that the prior probability shift assumption, of invariant P(X Y), is violated.
We consider semi-realistic scenario in which one wants to estimate cell prevalence in automated fashion employing a given black-box cell type classifier. We treat the first two samples as an auxiliary cell atlas on which a generic black-box cell lype classifier was trained (we use a random forest), the third sample as an available labeled data set, (XX,,)),, and the fourth sample as an unlabeled data set, (X}), for the quantification problem. Although the prior probability shift is violated, we can hope that the labels predicted by the random forest are more invariant and that methods working under the weak prior probability shift assumption may still yield reasonable estimates.
11

---

Published in Transactions on Machine Learning Research (06/2024)
We consider quantification method using predicted labels (posterior mean in the proposed model, BBS, RIR, and CC) as well as two methods accepting probabilities, rather than labels: Expectation-Maximialll (EM) and a soft variant of the restricted invariant ratio estimator (RIR (soft)). We do not use recalibration tochniques proposed by Alexandari et al. (2020), using the vanilla probabilitics provided by the random lorest.
Generally, the posterior mean captures well the true cell types prevalence, showing large uncertainty around the vascular cells and neurons. Similar performance is obtained by the simple CC baseline, owing to the good performance of the classifier. Interestingly, the methods using estimated probabilities (RIR (soft) and EM) obtained the worst performance. We hypothesise that this is due to the violated prior probability assumption and that using discrete labels, rather than continuous probabilities, improves the robustness.
As methods employing matrix inversion and a black-box classifier (BBS and RIR) failed due to non- invertibility of the estimated matrices, we then decided to merge two least prevalent cell types occurring in the data set (vascular cells and neurons) into single "Other" class. In that case, RIR and BBS obtain performance on par with posterior mean and CC.
5 Discussion
The presented approach generalizes point estimates provided by black-box shift estimators and invariant ratio estimators to the Bayesian inference setting. This allows one to quantify uncertainty and use existing knowledgee about. the problem by prior specification. Moreover, by the construction of the sufficient statistic our approach is tractable even in large-data limit (for either data set considered). In all our experiments, the suggested estimator obtained at least as good performance as the existing methods, outperforming them in the K < L case where the number of modeled classes differs from the "true" number of classes. Compared to point estimates with asymptotic guarantees, our approach "knows what it does not know", mcaning that the posterior is meaningful even if the matrix P(C Y) is not (left-) feett)iivertibbe, and it is specific for the prevalence values of those classes for which the feature extractor f is sufficiently informative.
Moreover, the proposed approach aligns with the approaches employing black-box feature extractors f, which can be trained on different data set and may be not calibrated properly. This is particularly useful when a hard, fully black-box classifier is given without the possibility of retraining it, which is an increasingly common theme with modern Al applications, which are often huge assets doing sophisticated processing, and also often proprictary and only available through APIs.
However, the method we introduce not free from challenges. As in all Bayesian inferences, care required regarding modeling assumptions: whether the discrete model is applicable and what prior should be used. In particular, even the weak prior probability shift assumption may not hold (e.g., if the labeled and unlabeled data sets were collected under radically different. conditions or the labeled and unlabeled data sets have different classes y). Additionally, Bayesian inference often carries a model choice problem, and different choices for K or the discretization method ʃ may yield different posteriors on the prevalence vector n', especially in the low data regime. If the generative model P(X Y,0) is well-specified and tractable, we suggest to use this instead of an approximation P(C' Y, If it is not tractable, we suggest to use the available classifier with K classes, observing the quality of & = P(C Y) matrix, and perhaps training one's own classifier on some hold-out data set. The experiments performed in Section 4.2 and Theorem 3 of Lipton et al. (2018) suggest that improving the classification accuracy results in more accurate prevalence quantification. Finally, we rely on the Markov chain Monte Carlo sampling schemes, which explore an O(K Ldddinnnsiol parameter space. Although in this manuscript we focus on problems with a moderate number of classes, there exist complex data sets, such as ImageNet (Deng et al., 2009), with thousands of categorics. We note that this can pose additional computational challenges for the MCMC samplers (Betancourt, 2015; Bardenet et al., 2017: Izmailov et al., 2021), which we do not investigate this work.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Statement of broader impact
This article discusses a Bayesian method of quantifying the prevalence of different classes in an unlabeled data set. We note that. in general the parameler posterior conditioned on the full data view can be different from the posterior conditioned on some representation C'= f(X) in cases where a reliable model P(X Y) is available and the inference is tractable, we suggest to use this instead of our discretized method. Secondly, the model need not apply - perhaps prior probability shift is not the only distribution shift occurring in the problem or the data may not be exchangeable. In epidemiology, for example, outbreaks induce correlations between the healthiness of different people that can easily extend to sampling. Finally, even if the assumptions hold, recalibrating a probabilistic classifier with quantification may have undesirable consequences regarding fairness (Plecko & Bareinboim, 2022).
Code availability and reproducibility
We ensured reproducibility by designing all experiments as Snakemake workflows (Mölder et al., 2021). The code and workflows used to run the experiments and generate the figures are available in the https: / //ggthhu.ccom/p/en wtt.tt/...oooppaeelaaaaa//aaaaat repository.
Acknowledgments
We would like to thank Ian Wright for valuable comments on the manuscript. This publication was supported by GitHub, Inc. and ETH AI Center. We would like to thank both institutions.
13

---

Published in Transactions on Machine Learning Research (06/2024)
References
Amr M. Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In Proceedings of the 37th International Conference on Machine Learning, ICML'20 JMLR.org, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant Risk Minimization. aiv e-prints, art. 1.XXv::197700883,, Jul 2019.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In 7th International Conference on Learning Representations, TCL.R 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
net/
forum?id=::10r3mm
Rémi Bardenet, Arnaud Doucet, and Chris Holmes. On Markov chain Monte Carlo methods for tall data. Journal of Machine Learning Research, 18(47):1-43, 2017. URL http://jmlr http://paape///////5----------
html.
Antonio Bella, Cesar Ferri, José Srrnández-Oralloe and María José Ramfrez- Quintana. Quantification via probability estimators. In 2010 IEEE International Conference on Data Mining, pp. 737-742, 2010. doi:
10.1009/ICDDM.2000.7.5
J.M. Bernardo and A.F.M F.M. Smith. Bayesian Theory. Wiley Series in Probability and Statistics, Wiley, 1994. ISBN 9780471494645.
Michael Betancourt. The fundamental incompatibility of scalable Hamiltonian Monte Carlo and naive data subsampling. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 133-540, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceeddin wtw.eeediiinnmm.rpeeeeeeeeeeeeee
mlr, prees//33//betancoutt1 htmll..
Michael Betancourt. A conceptual introduction to Hamiltonian Monte Carlo, 2017. URL https://arxi::
org/abs 1atpb///110.0200 02434.
David M. Blei, Andrew Y. Ng, and Michael Jordan. Latent Dirichlet allocation. J. Mach. Learn. Res., 3 (null):9933 99221,, mar 2003. ISSN 1532-4435.
Mathieu Blondel, Akinori Fujino, and Naonori Ueda. Large-scale multiclass support vector machine training via euclidean projection onto the simplex. In 2014 22nd International Conference on Pattern Recognition, pp. 1289 1294, 2014. doi: T0.1109/CCPR.20142233
Daniel C. Castro, lan Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communi- cations, 11:3673ff. 7 2020.
Spyros Darmanis, Steven A. Sloan, Derck Croote, Marco Mignardi, Sophia Chernikova, Peyman Samghababi, Ye Zhang, Norma NefT, Mark Kowarsky, Christine Caneda, Gordon Li, Steven D. Chang, Tan David Connolly, Yingmei Li, Ben A. Barres, Melanie Hayden Gephart, and Stephen R. Quake. Single-cell raa- seq analysis of infiltrating neoplastic cells at the migrating front of human glioblastoma. Cell Reports, (1(5):1399-1410, 2017. ISSN 2211-1247. doi: https://doi.orr//0000 1ww.o//////..../0///111111 wwwp::/.e...eee....0. celrep.2017.100000 10.030. URL https: // www.ssciencedrrect. wcw./science//arttileeeee http://caweaereeeeee/////////11/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 1882555. Teee, 2009.
B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1 26, 1979. doi: 10.1214/aos/117663A1 URL tttps://doi... ort.:/...rr.....0 1tt///..1224/2//1177/556533
Valerii Fodorov, Frank Mannino, and Rongmei Zhang. Consequences of dichotomization. Pharmaceutical Statistics, 8(1):50-61, 2009. doi: https://doo.org/10100////ps.... p0..002/pp...31.. URL W http:///nnnnnnlllle wiley. /t0p///0/aabs/1..10000/// 3311.
14

---

Published in Transactions on Machine Learning Research (06/2024)
George Forman. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery, 17:
164-206, October 2008.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3290-3300. Curran Associates, Inc., 2020.
Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin. Bayesian Data Analysis, Third Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013. ISBN 9781439840955.
URL httpp::/tbooooo groook...goole...... google. pl/ /l//bolss?.d-221LAAAAAQBA
Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Saul-Christian Bürkner, and Martin Modrák. Bayesian workflow, 2020.
Pablo González, Alberto Castaño, Nitesh Chawla, and Juan del Coz. A review quantification learning. ACM Computing Surveys, 50:1-40, 09 2017. doi:
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1321-1330. JMLR.org, 2017.
Paul R. Halmos and L. J. Savage. Application of the Radon-Nikodym Theorem to the Theory of Sulficient Statisties. The Annals of Mothematical Stotistics, 20(2):225 241, 1949, doi: 1a.221//aoms117/// URL https:/ //o: http//dii.rgg////// org/10.1 1211/ //////mm///////////////
F.E. Harrell. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Springer Series in Statistics. Springer International Publishing, 2015.
ISBN 9783319194257. URL https://boot httpoooos.oooo..//oooooooooe.... htt..ppppooooos..dddgggg.......
Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(47):1593- 1623, 2014. URL http: html.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson What are Bayesian neural network posteriors really like? In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 4629 4640. PMLR, 18-24 Jul 2021. URL https: attps://proceedings. .mlr. htl..ppres/vv1///// izmailov21a.hn html.
Nikolay Karpov, Alexander Porshnev, and Kirill Rudakov. NRU-HSE at SemFval-2016 task 4: Comparative analysis of two iterative methods using quantification library. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 171-177, San Diego, California, June 2016. As- 000533/11S16610000 11/16-10225 URL https: aclweb. org/
sociation for Computational Linguistics. doi: 10. 10.18653/v1
anthology/S161100
Achim Klenke. Probability Theory: A Comprehensive Course. Springer, 2014.
John K. Kruschke. Bayesian analysis reporting guidelines. Nature Human Behaviour, 5(10):1282 1291, Oct 2021. ISSN 2397-3374. doi: 10.108//$11502-02--017771 URL https://doi. http:///dooorrg//0//// ort./0. 841562-021-011777--
John R. Lewis, Steven N. MacEachorn, and Yoonkyung Lec. Bayesian Restricted Likelihood Mothods: Conditioning on Insufficient Statistics in Bayesian Regression (with Discussion). Bayesion Analysis, 16 (4):1393 2854, 2021. doi: 10.1214/21-BAIT
URL https://dol. http://doo..or/.///// 1214/21-BA257.
Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. In Jennifer G. Dy and Andreas Krause (cds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmásson, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3128 3136. PMIR, 2018.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Simon Lyddon, Stephen Walker, and Chris C Holmes. Nonparametric learning from Bayesian models with randomized objective functions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Alejandro Moreo, Andrea Esuli, and Fabrizio Sebastiani. QuaPy: Python-based based framework for quantifica- tion. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4534-4544 4444,,453,.2 2021.
F Mölder, KP Jablonski, B Letcher, MB Hall, CH Tomkins- Tinch, V Sochat, J Forster, S Lee, SO Twardziok, A Kanitz, A Wilm, M Holtgrewe, S Rahmann, S Nahnsen, and J Köster. Sustainable data analysis with Snakemake. P1000Research, 10(33), 2021. doi:
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825- 2188255-800,, 2011.
C. Peters and W.A Coberly. The numerical evaluation of the maxinun-likelihood estimate of mixture proportions. Communications in Statistics Theory and Methods, 5:1 1-17271133, 1976.
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated proba- bilistic programming numpyro. Xrwi preprint 1912 11554, 2019.
Drago Plecko and Elias Bareinboim. Causal fairness analysis, 2022.
Jonathan K Pritchard, Matthew Stephens, and Peter Donnelly. Inference of Population Structure Using Multilocus Genotype Data. Genelics, 155(2):945-995) 959, 06 2000. ISSN 1943-2631. doi: 00.1093/genete 155.2.945. URL
1093/ genetics/155.2.900 2.945.
Marco Sacrens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to priori probabilities: A simple procedure. Neural Computation, 14:14 21, 2001.
Bernhard Schölkopf, Dominík Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12 12, pp. 459-466 Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
Shai Shalev-Shwartz and Yoram Singer. Efficient learning of label ranking by soft projections onto polyhedra.
J. Mach. Learn. Res., 7:1567-1599 1599, dec 2006. ISSN 1532-4435.
Amos Storkey When training and test sets are different: Characterizing learning transfer. In Joaquin Quidner--Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence (eds.), Dataset Shift in Machine Learning, chapter 1, pp. 3 28. The MIT Press, 2009. ISBN 0262170051.
Dirk Tasche. Fisher consistency for prior probability shift. Journal of Machine Learning Research, 18(95): 1 32, 2017. URL
Dirk Tasche. Confidence intervals for class prevalences under prior probability shift. Machine Learning and Knowledge Extraction, 1(3):805-83)) 2019. ISSN 2504-4990. doi: 10.3390/ 10.3390/make1000 makc1030047. URL https:
// httpppp.oo///4449//1/11111
Afonso Fernandes Vaz, Rafael Izbicki, and Rafael Bassi Stern. Quantification under prior probability shift: the ratio estimator and its extensions. Journal of Machine Learning Research, 20(79):1 1199):113.. 1-33, 2019. URL
http:///mmr argg/papers/v2011-45555 html.
James Watson and Chris Holmes. Approximate Models and Robust Decisions. Statistical Science, 31(4):465 489, 2016. doi: 10.1214/16-ST5502. URL https://dol: http:::///oo.or/11..../
Jack Xue and Gary Weiss. Quantification and semi-supervised classification methods for handling changes class distribution. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 897-906, 01 2009. doi: (0.1145/1557019.1555
16

---

Published in Transactions on Machine Learning Research (06/2024)
Andy B. Yoo, Morris A. Jette, and Mark Grondona. Shurm: Simple limux utility for resource management. In Dror Feitelson, Larry Rudolph, and Uwe Schwiegelshohn (eds.), Job Scheduling Stralegies for Parallel Processing, pp. 44 60, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. ISBN 978-3-540-33977--4.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference International Conference Machine Learning- Volume 28, ICML'13 13, pp. III 819 III 827. JMLR.org, 2013.
Yingdong Zhao, Ming-Chung Li, Mariam M. Konaté, Li Chen, Biswajit Das, Chris Karlovich, P. Mickey Williams, Yvonne A. Evrard, James H. Doroshow, and Lisa M. McShane. TPM, FPKM, or normalized counts? A comparative study of quantification measures for the analysis of RNA-seq data from the NCI patient-derived models repository. Journal of Translational Medicine, 19(1):269, Jun 2021. ISSN 1479- 5876. doi: 111186/112967-02111
URL https://doi.org/ otg/ 1101111111/1112700000-099------1
Albert Ziegler and Pawel Czyž. Unsupervised recalibration, 2020. URL https: httppp///aaxvvor/aab/////// 09157.
17

---

Published in Transactions on Machine Learning Research (06/2024)
A Strict linear independence of measures
We use the following definition:
Definition A.1. Let KI..... K2 be probability measures on measurable space X. We say that they are strictly linearly independent if for every non-zero X IRL, there erists measurable set Ax such that.
+ +.... (Ax)(Ax) #0.
This definition is especially useful for proving identifiability of a well-specified mixture model: Theorem A.2. Let K,,..., Kr. be striclly linearly independent probabilily measures space X. Them, for every two vectors of misture weights n. n E AL-1 such that
KiK1 + TL K1 .... T KL
it follows that n².
Proof. Consider the difference A = 77 n- E RR, If it not the zero vector, then for some set AĄ would have
KG(A)) +... kL
+ KL((((A))
Note also that A is not of positive measure with respect to both of the mixture measures, meaning that for well-specified model the discrepancy will eventually be visible according to the law of large numbers. Z
Although the definition above looks different from the original one proposed by Garg et al. (2020), they are essentially equivalent:
Lemma A.3. Lel be a o-finile mensure on X such that the Radon Nikudym derivalives ky exist. Then, the probability measures KI...., .....,,, Kr, are strictly linearly independent if and only if
/ x
Σ kx(x) du((z) + vEY
for every non-zero R4_
Proof. Consider any À + O and write A₁ + Kg for the signed measure. Using the standard rules of Radon Nikoddy calculus, the condition on the integral is equivalent to [p((2) # 0. If there exists AΛ such that v(Ax) # 0, then jx[(x') 2 [p((Ax) 2 [((A)] 0.
Conversely, assume that (x((x)) * and take the Hahn decomposition of X, with PnN Ø, v(P) 2 O, and v(N)= < We define now Hahn-Jordan decomposition of v into two positive measures, at (A) v(AnP) and x (A) -v(An (AANN), N), with the properties -t - and |p| = + - We conclude that (x((t) v(P) x(N) # O, that at least one of the sets P and N can be taken A2-
N such that
The above characterization of strict linear independence gives the following result:
Lemma A.4. Assume that X is standard Borel space and μ is strictly positive measure. Further, assume that the Madon-Nikodym derivatives ki...., all continuous functions C(X,TR) Then, if ki....,kr k, are linearly independent, then Ki,...,Ke K2 are strictly linearly independent.
a....,ka, KL are continuous functions, treated as vectors in the space of
Proof. Take any * and write |d +.... +...+.............. A C(X,R) From the linear independence follows that there exists 2o X such that u(za)= 0.
18

---

Published in Transactions on Machine Learning Research (06/2024)
We can use the continuity of to find open neighborhood A of such that for all A have u(x:) > 22 As u is non-negative and is strictly positive, we have pa( >> 0 and
J d(x)d 2 S ((z))d(( 2 u(zo)
m( 0.
2
B
Derivation of the sufficient statistic
Starting from the joint probability
N P(T,T',)) II (XX T) x II
N
(Y, (Y,
P (7),
==1
we need to derive
a' (V,CC), C.). (CJ)) P(YY,C) (C)) |,, (,),,",))((,,'''..)
The observed likelihood is given by marginalization of Y variables:
N ΣI
N II
(T,
Σ
P(C, (.,))))) T)
PIC - = = (T)
tyE
hey=
N
N
II
P(C, v, (T)
E NYYY
E 12. = 1T))
4Eyj=1
Each of these terms will be calculated separately.
We want to calculate
N
II P(C: = a =y) = yı ()
II
P(C. = a Y Yi = m1
P(Y = yb x).
A
A2
The term Ag is simple to calculate: as P(Y, = Vi π)
Ty. we have
N
A x..
II 1=1
(T))
where ny is the number of i E (1...., N}, such that y; = In particular, up to a factor N!/nz!.. nz!, this is the PMF of the multinomial distribution parametrised by æ evaluated at (TI....,
To calculate A we need observe that P(C, k p.
Pik- Hence,
A
II E=1
P(C: a: p, Y, vi) 1=1 k=1
where fik is the number of i E (1...., N), such that = 1 and e k. Observe that = Sa +... + fix. In particular, up to the factor
II
ny!
fu!... fix!
19

---

Published in Transactions on Machine Learning Research (06/2024)
this corresponds to the product of PMFs of L multinomial distributions parametrised by probabilities pt: evaluated at Jt:- Recall that
B: B= Σ
EY
h
We can use the sum-product identity
N N E (0.) II 40)
Σ EyY
hEy;=1
to reduce:
N B
II cl - =1
Because both Cj and y3 are parametrised with categorical distributions, we have
P(C = k 8. Y = = PP
and
P(Y = x²) ক/!
=
= n²) 7')
Hence,
N' B = II J=1
K (0 Dee
(10 (')
where n/ is the number of j E (I...., this is the PMF of the multinomial distribution parametrized by probabilities o x' evaluated at (nl,...,''')
N') such that cj = In particular, up factor of N'I/n/!... n'x.,
C Proof of asymptotic identifiability
In this section prove Theorem 3.1. We first need to establish two simple lemias regarding approximate left inverses:
Lemma C.1. Choose any norms the space of linear maps R + RK and RK - R.. Suppose K 2 L and that Ao: RR_ + RR is of full rank L. Then, for every > 0 there erists ò > such that if A: R- - RK is any matrix such that
|| Aoll < 6,
then the lest inverse A-1 (AT caisls and
| Ao <
Proof. First note that indeod the choice of norms does not matter, as all norms on finite-dimensional vector spaces are equivalent.
Then, observe that rank is a lower semi-contimuous function, so that for sufficiently small 8 the map A will be of rank as well.
Finally, is clear that the chosen formula for the left inverse is continuous function of A.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Lemma C.2. If 2 L and matric Ag: R. + RR is of full rank L, then for every 0 there eaist numbers 8 >> O and >> 0 such that for every linear mapping A: R + BBK and veclor TR if
||A Aoll
and
||A Agvoll 1
then
||u - voll E.
Proof. Again, the norm on either space can be chosen arbitrarily without any loss of generality. We will choose the p-norm for vectors and the induced matrix norms.
From the previous lemma we know that for any chosen B 0 we can take 8 > 0 such that A is left-invertible and
BB - Boll < 3,
where B A-1 and Bu Ap'
the left inverses in the form defined before.
Write w = Av and wi = Ao We have
|/0 voll |BB Bowel
|l(Bw Bow) + (Bow Bown)ll
II(B Bo)w + Bo(w wo)ll
S II(B Bo)w|l + || (w - wo)ll
S | Boll ||l||| + BBll || woll
S B||l|| + ||
We can bound each of these two terms by E/3 choosing appropriate / and v. Then, we can find à yielding appropriate B.
Now the proof of Theorem 3.1 will proceed in two steps:
1. We show than for any prescribed probability can find N and N' large enough that the maximum likelihood solution will be close to the true parameter values.
2. Then, we show that for reasonable priors the maximum posteriori solution will almost surely assymptoiccally converge to the maximum likelihood solution.
Let's assume that the data was sampled from the model with true parameters x², x"', xp" and take ó > and 0.
For any
the fact that likelikdl is given by
(T.,'.9)
Σ Ny log ग + ley
ΣΣ FiR log Pik +
No. log(e" T')k.
REC
REC
and by the strong law of large numbers we can find N and N' large enough that with probability at least 1- 8 we will have ||- -- and ||@ - s"1l v, and || x x'1l < v, where â, s, and #' is the
maximum likelihood estimate.
Basing on the previously established lemmas conclude that we can pick small enough that || -| x=1l < || - | < s, and || -1-1= <
Now note that we assume the PDF of the prior P(T, .p) to be continuous, we can take a compact neighborhood of (n*,, n't x*) inside L- x AL-1 x AK- x ... x K- with probability mass arbitrarily close to 1. Then, the log-prior defined on this set will be bounded and the marimum a posterioni estimate can be made arbitrarily close to the maximum likelihood estimate with any desired probability.
21

---

Published in Transactions on Machine Learning Research (06/2024)
C.1 Should one rely on the maximum posteriori estimate?
Although in the above section we discuss why the mazimum posteriori (MAP) estimate is consistent in the large-sample limit, we advise against using it: as the data is finite, the fully Bayesian approach is to use the full posterior (approximated by the Markov chain Monte Carlo samples) to understand the associated uncertainty on the provided estimates.
In settings in which the posterior distribution has to be summarized in terms of a single point estimate (e.g.: for the comparison with estimators providing point estimates in Section 4.2), we advise for using the posterior mean, which is available directly from the MCMC samples. Conversely; the MAP estimate requires optimization, rather than sampling, and may not be well-defined (e.g., in non-identifiable problems with non-invertible P(C Y) matrix or when the sample size is not large enough).
Moreover, posterior mean may be preferable over MAP on the basis of Bayesian decision theory. We present a short version of the standard decision-theorcte argument and refer to Bernardo & Smith (1994, Sec. 5.1.5) for a more detailed mreatment.
An approach employed in Bayesian decision theory is to define a loss function ((H'x"') quantifying risk associated with choosing the estimate #' when in reality the estimand attains x' value. Various applications involve different. loss functions, to be specified by the decision maker: for example, if the component ri describes the prevalence of a viral disease, one may prefer to choose loss function assigning higher loss to the predictions underestimating the disease prevalence (i.e., #1 < क|" should result in higher loss than #1 > )) The Bayesian decision theory argues then to provide an estimate minimizing the expected loss, I T(t', x d''''(x' data).
The MAP estimate (provided that it exists) corresponds to the limit of the losses 10 x²) 1[1
e) for e- 0+. The resulting discontinuous 0 --1 1 loss may not be of direct interest in the applied problems: for finite data the estimates are expected to be different from the ground-truth value and one instead may try to quantify by how much amount the estimate differs. For an ( = || - >²| loss, the minimum of the expected loss is attained at the posterior mean.
At the same time, we stress that preferable to analyze the whole posterior, rather than to rely on a single point estimate, especially that in reality the decision maker may not have an access to an oracle loss function and consider different candidate losses. In particular, in Section 4.2 and Appendix F.2 we quantify the algorithm performance using two losses for which the minimum may not be attained at the posterior mean. Nevertheless, the experimental results presented there confirm that the (arguably suboptimal) choice of the posterior mean as the point estimate is on par with the point estimates provided by other methods.
D Quantification algorithms
In this section we provide additional details on existing quantification methods, expanding on the description in Sec. 2.
D.1 Expectation-maxiiizatioo and the Gibbs sampler
In this section we analyse the expectation max.mmiaaiiiiiiiiiiini algorithm introduced by Saerens et al. (2001), which assumes access to a well-calibrated probabilistic classifier providing the probabilities r(z) P(Y X z, r). We assume a Dirichlet prior for P(='), so that the expectalion-maxxiizaatiee of the distribution P(T' (X)).r). The original algorithm Saerens et al. (2001) corresponds then to the uniform prior, P(r')= = Dirichlet(x'' 1, 1). Finally, we will show how adjust the expectation- maximization algorithm to obtain a Gibbs sampler, sampling from the posterior P(r' [X()r)) r). To improve readability we will generally drop conditioning on r, leaving it implicit.
largets maxinum a posteriori
22

---

Published in Transactions on Machine Learning Research (06/2024)
D.1.1 Expectation maximization Expectaioo-maxiimiatonn
Saerens et al. (2001) notice that if one has a well-calibrated classifier P(Y) x,7), then they also have an access to a distribution P(Y x')::
P(Y y X x= x P(Y y. x²) P((X r Y y, ,,a))((==yl))
P(X x Y = y) m.y
where the proportionality constant does not depend on y. Analogously,
P(Y = y X = x. T)) & P(X = x Y = y) ny.
As P(X = Y= =) is the same, we can take the ratio of both expressions and obtain
P(Y = y x= n, n.) & P(Y = y X = x, π
where the proportionality constant does not depend on y. This yields unnormalized probabilities, which can be easily rescaled so that they sum up to 1.
Expectation-maxxiization is an itcrative algorithm finding stationary point of the log- log-posterior
log P(x H[xj = x3})) = log P(T') + log P((x) = = x²)
N'
log P(T)) +
Σ log P(X; = dty (T))
In particular, by rumning the optimization procedure several times, we can aim at finding the maximum a posteriori estimate. Assume that at the current iteration the proportion vector is x1(), Then,
log P(X; = z'j xx) log y=1
Σ P(X;= d' YJ = T')
P(X; t/ = = =
log
P(Y | m(() ,(,,,,/'---- x zzj) = PYY) y ~(0 X₁ xf)
P(X) d.y = P(Y) = n10 x1
P(Y = y N11 x; = 22)) log
where the bound follows from Jensen's inequality. Hence,
N' log P(X = n'y x²)
log P((X) = 2//} |x)
Ø=1
N'
P(X; = = = = =
P(Y = y 110 X = 2/)) log
P(Y y ((0), X xf)
1=1 y=1
Now let
N'
L
= = PYY) = (+))) X = x3))
Q(T,= = log P(T)
+
Σ 1=
p(Y = (00 xx, = xzy) log
be a lower bound on the log- posterior We will define the value (+++) by optimizing this lower bound, i.c.,
((+1) argmax, Q(=², x0)).
23

---

Published in Transactions on Machine Learning Research (06/2024)
Define auxiliary variables Eyv PlY = 1110 X3 2z),, which can be calculated using the probabilistic classifier as above. Hence,
N'
Q(a', = log P(T')
+
Eig log P(X; a'y, = y) Eiv log Eiv
Note that the term Siy log Siy does not depend on n', so it does not have to be included in the optimization. Similarly, we can write log P(X; = Zj T²) = log P(X;- = zz YY y) + log T. and notice that log P(X) = 2' | = y) also does not depend on n². Hence, we have to optimize the expression
log P(
Es log Tgy
j=1y=1
where P(x') is modeled as the Dirichlet distribution, Dirichlet(a (i,...,aL). Hence, the optimization objective becomes
N (ay 1) +
Ety
log Tyy
==1
with the constraint #ं + TL 1. Saerens et al. (2001) the technique of Lagrange multipliers.
However, can optimise the first L coordinates and write J 1 - (5f + t.... + xL In this case, if
differentiate with respect to Ty. we obtain Ag/T(/ AL/TI, 0, where Ag ay +1 Σ==1
Hence, N'y = kAy for some constant k > 0. As
1. Ay
I Epr
Σ
0g
+
Σ y=1
By
L + N
9=1
y=1
we obtain
N
1
T (a1 + ... + a1) + N' T.
0gy
which is taken as the next iteration value, a((+1), The procedure is repeated until the sequence m((1) (approximately) converges to point.
D.1.2 Gibbs sampler
As typical for expectation-maxiiizaaiie algorithms, it possible implement Gibbs sampler targeting the sample from the posterior P(x' | {X})), rather than the mode (maximum a posteriori).
The Gibbs sampler will iteratively sample from the high-dimensional P(r', (Y/) (X})) distribution. Note that for Dirichlet prior we have
(rr
N 18 11 al + Σ 1[ 4
P(r [Y リリ XJ)) Dirichlet
Σ
The assignments of individual points are then sequentially sampled as
Y P(Y (i,..., Yk-1, (2),
which is possible due to the equality
P(Y Xt, T) = Categorical((a:
where Eky P(Y = X F4, T', n'), which is obtained using the probabilistic classifier similarly above.
24

---

The true and false positive rates are Published in Transactions on Machine Learning Research (06/2024)
D.2 Estimators employing auxiliary black-box classifiers
In this section we briefly review the main algorithms employing black-bos classifier.
D.2.1 Classify and count
When C = y and f: X + y' is classifier trained for given problem with good accuracy, the simplest approach is to count its predictions and normalize by the total mumber of examples in the unlabeled data set. However, as Tasche (2017) shows, this approach does not need to correctly estimate P(Y) even in the limit of infinite data.
D.2.2 Adjusted classify and count
Consider a case of imperfect binary classifier, with y -C -}. defined by
TPR P(C = + +) FPR P(C= + -)
and can be estimated using the labeled data set.
If 0 Prest (V +), we have
Ptest +)= TPR FPR. (1-0)
which can be estimated by applying the classifier to the unlabeled data set and counting positive outputs.
If we assume that TPR + FPR, i.e, the classifier has any predictive power, we obtain
0 Prest (C = +) FPR
TPR FPR
Then, Ptest(C = +) is estimated by counting the predictions of the classifier the unlabeled data set. As Tasche (2017) showed, is consistent in the limit of infinite data.
Two generalizations, extending it to the problems with more classes, are known the invariant ratio esti- mator and box shift estimator.
D.2.3 Invariant ratio estimator
Vaz et al. (2019) introduce the invariant ratio estimator, generalizing the Adjusted Classify and Count approach as well the "soft" version of proposed by Bella al. (2010).
Consider any function f: X 1 For example, if u: X - y is classifier predicting outputs in the set {1,..., LJ, we may define as the one-hot encoding of L - labels and assign the zero vector to the last label:
(1,0, .........)) 0) if ((x)) = 1,
(0,1,...., 0)
if (((x)) 2,
f(x)
(0,0, ..... 1) if ((x))) I- 0) if u(x) = L.
Analogously, for soft classifier 1).
A-1 C R f may be defined as fa( ux(x) for f1....,L
25

---

Published in Transactions on Machine Learning Research (06/2024)
Then the unrestricted estimator #' R is given by solving the linear system
Fiit +... FIrTL
fl
F1-1,17 +... FL-1LTT
= =fh-1
=1
where
1 N'
fk. N gx(r))
and
FRi 1
Σ (x(2),
| zESt
where St is the subset of the labeled data set with y; = I.
Note that adjusted classify and count is a special case of the invariant ratio estimator, for a hard classifier. Similarly, the algorithm proposed by Bella et al. (2010) a special case of invariant ratio estimator for a soft classifier.
The generalization for K o I is immediate, with & becoming (K- -1) matrix and 9 becoming vector of dimension K 1. Finally, Vaz et al. (2019) introduce a restricted estimator #R E A-11, which is given by a projection of ftJ onto the probability simplex. In our implementation we use the projection via sorting algorithm (Shalev-Shwarte & Singer, 2006; Blondel et al., 2014).
D.2.4 Black-box shift estimator
Black-Box shift estimators are also based on the observation that
Piet (C) P(C Peea
where P(C Y) matrix can be estimated using cither labeled or the unlabeled data set. Instead of solving this matrix equation directly by finding the (left) inverse, Lipton et al. (2018) estimate the pointwise ratio R(Y) = Ptest (Y) (Prrii(() by rewriting this equation as
Perst (C)) P Ptrain (C,Y)R(Y).
and estimate the joint probability matrix Perain (C,Y) using the labeled data set. Then, the equation be solved for R(Y). By pointwise multiplication by Perain (Y) (estimated using the labeled data set) the prevalence vector Prest (Y) is found.
Note that this approach naturally generalizes to the K + L case. Lipton et al. (2018) study the case K L and derive asymptotic error bounds. More rencently, Azizzadenesheli et al. (2019) introduced a regularized variant of this approach.
D.2.5 Unsupervised recalibration
Ziegler & Czyž (2020) study the quantification problem from the perspective of recalibration of given probabilistic classifier. Their method can be interpreted as partly a black-box shift estimator and partly as a likelihood- based estimator. Namely, they propose to use a black-bor classifier to predict the labels C. = f(X,) and C = f(X)) and estimate the probability table P(C Y) by using the plug-in estimator. However, they note that solving explicitly Eq. may suffer from numerical issues when condition number is high and instead they optimize the multinomial likelihood the observed counts Cz
26

---

Published in Transactions on Machine Learning Research (06/2024)
D.3 Other algorithms
Other quantification methods include the CDE-Iterate algorithm of Xue & Weiss (2009), which obtain good empirical performance selected problems (Karpov et al., 2016). However, Tasche (2017, Sec. 3.4) showed, it is not asymptotically consistent. Zhang et al. (2013) describes a kernel mean matching approach, with a provable theoretical guarantee. However, as Lipton et al. (2018, Sec. 6) observed, kernel methods may be challenging to scale to large data sets. Finally, Moreo et al. (2021) present. Python package for quantification problems.
E Experimental details and additional experiments
In this section we provide additional details on experimental protocols used in Sec. 4 logether with additional experimental results. Experiments described in Appendices E.1, E.4, and E.5 were run on a laptop with 32 GiB RAM and CPU cores clocked at 4680 MHz and finished under six hours. Experiments described in Appendices E.2 and E.3 require runs over many random seeds and require larger computing power (unless the number of random seeds is reduced). We them sequentially on cluster equipped with 384 GiB RAM and 128 CPU cores clocked at 2.25 3.7 GHz. As the cluster is shared between different researchers and uses the Slurm workload manager (Yoo et al., 2003) to distribute runs among different projects, this provides an upper bound on the computational resources used. These experiments have finished five hours.
E.1 Nearly non-identifiable model
We reproduced the experiment described in Sec. 4.1 S times varying the random seed to obtain different data samples (and, subsequently, different posterior and bootstrap samples) well the data set size under N = N' constraint. We noted that methods based on matrix inversion raised error whenever malrix estimated from the bootstrap sample was singular. We dropped such bootstrap samples.
In Fig. 6 we use N = = = 100, in Fig. 7 we use = N' = 10", and in Fig. 8 we use N = N = 10a.
We generally see that bootstrap for N N' {10?, 103} can result in negative prevalence estimates. On the other hand, restricted invariant ratio estimator (RIR) often does appropriately estimate the first component. Hence, consider the Bayesian posterior preferable in low-data settings. For N N' = 10* we see that the performance of all methods is comparable.
For each simulated data set, we ran four Markov chains with 500 warm-up steps and 1000 samples each using the NUTS algorithm of Hoffman & Gelman (2014). To flag potential convergence issues, we calculated the potential scale reduction factor ît (Gelman al., 2013, Sec. 11.4). For each variable it did not exceed 1.01. The effective sample size was over 3, 3,000 for all parameters and across all experimental settings.
E.2 Discrete categorical model benchmark
The default parameters introduced in Sec. 4.2 have been gathered in Table 1. For each data set we used four Markov chains with 500 warm-up steps and 1000 samples collected per chain. The maximal ît did not exceed 1.01 and the minimal effective sample size was 772.
Fig. represents the outcomes of the experiment where mean absolute error metric has been replaced with the root mean squared error:
RMSE(# (ISSE(",'''+>
Σ
T - 1/ y
V
Qualitatively, the conclusions do not change.
E.3
Misspecified model
We repeated the experiment described in Sec. 4.3 for N N' {10², 10 10*) samples with the results presented in Fig. 10. For each generated data set and each method we four Markov chains with 1,500
27

---

Published in Transactions on Machine Learning Research (06/2024)
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2 Class
2
3
1
2 Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
1
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2 Class
2
3
1
2 Class
Class
Class
Figure 6: For - 100 samples the posterior is not very precise around the first component. We note that for unrestricted estimators the bootstrap samples result in negalive probability estimates.
28

---

Published in Transactions on Machine Learning Research (06/2024)
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2 Class
2
3
1
2 Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
1
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
3
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2 Class
2
3
1
2 Class
Class
Class
Figure 7:
For = N' 103 Bayesian posterior concentrates around the ground-truth value of the first component. However, bootstrap samples often yield negalive probability estimates.
29

---

Published in Transactions on Machine Learning Research (06/2024)
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2 Class
2
3
1
2 Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
3
1
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2
3
1
2
3
2
3
1
2 Class
Class
Class
Class
Posterior samples
RIR (bootstrapped)
UIR (bootstrapped)
BBS (bootstrapped)
1.0
Prevalence
0.5
0.0
2 Class
3
1
2
2
3
1
2 Class
Class
Class
Figure 8: For N = No 104 samples the first component is perfectly determined. Bootstrap samples capture the uncertainty well and are qualitatively similar to the samples from the Bayesian posterior.
30

---

Published in Transactions on Machine Learning Research (06/2024)
b.
BAY
RMSE
0.1
BBS
CC
0.0
RIR
0.6
0.8
102
104
0.6
0.8 Classifier quality q
Prevalence rT'1
Unlabeled sample size N'
e.
RMSE
0.1
0.0
3
5
0.8 Misspecified quality q'
0.6
Classifier outputs
Number of labels L =
Figure 9: Quantification using simulated categorical black-box classifiers under different scenarios.
P[X | Y)
Ptrin( (X) and (Xes(())
Posteriar samples
Coverage
Y=
Train Test
Observed
0.75
Gaussian
Y=1
Student
0.50
Discrete bins)
0.25
Discrete (1D bins)
2
2
2
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
(Y
Exported
PIX | Y)
Perann (X) and (X)
Posterior samples
Coverage
1.00
Y-0
Train
Observed
Gaussian
Y-1
Student
0.50
Discrete bins)
0.25
Discrete (10 bins)
0.00
2
2
0.00
0.25
0.50
1.00
PaelY
Expected
PIX Y)
Perin (X) and Prese (X)
Posterior samples
Coverage
1.00
Y-0 0
Observed
0.75
Y=1
Test
Student Discrete bins)
0.25
Discrete (10 bins)
0.00
1.00
PuelY = 1)
Expected
Figure 10: Experiments with mixture of Student distributions. First column: conditional Student distribu-
tions P(X Y). Second column: train (labeled) and test (unlabeled) distributions. Third column: posterior according to differend models. Fourth column: coverage of high-density credible intervals measured over
5 = 200 simulations. Top row: = N' 100 samples. Middle row: N N = 103 samples. Bottom row:
= N' = 10a samples.
31

---

Published in Transactions on Machine Learning Research (06/2024)
warm-up steps and 2,000 samples collected per chain. We noticed occasional convergence issues: runs with at least one parameter such that R 2 1.02 were excluded from the coverage calculation (see Table 2). The convergence for all models is satisfactory (with the minimal number of simulations retained being 157 out of conducted 200), although we noticed that more runs were excluded for larger sample sizes and that misspecification (i.e., the Gaussian mixture model) resulted in more runs to be excluded.
We see that the coverage of the high-density credible intervals of the discrete model generally agrees with the nominal value. However, the posterior in the discrete model can be wider than in the properly specified model using the generative mechanisms P(X Y). Moreover, method may be preferable over using misspocified model when the large sample size is used: although for N = N" = 100 the (misspecified) Gaussian mixture model has coverage close to the nominal value, for N' E {102, } the coverage deteriorates quickly.
that using discretized quantification
( Parameters of the ground-truth mixture model We used (X = T(O, 0.5",, .3) and (X |Y 2) ~ [11, 0.52, 4), where o(,, is the location-scale t-distribution, i.e., the pushforward distribution of the standard Student t-distribution T(0,1, with v degroes of freedom by the affine mapping + μ + ax.
We sampled the number of labeled samples with label Y =1 using N1 Binomial 0.5) and then defined the number of samples with label Y = 2 by N N1. Similarly, generated an unlabeled data set, but with the class prevalence 0.2, rather than 0.5.
Priors the Gaussian and Student mixture models In both cases we used the uniform prior, a' ~ Dirichlet(1,1), 1), on the prevalence vector. We modeled the scale parameters o; [C((1) via the half- Cauchy prior and the location parameters via μ4 N (0,33) Additionally, the Student mixture had a positive prior on the degrees of freedom, V; ~ 1)., The Gaussian mixture model can be treated as a special case of this model with the constraint i4 = do for both components.
E.4 Single-cell data analysis
We downloaded the TPM-normalized (Zhao et al., 2021) data sequenced by Darmanis et al. (2017) from the Curated Cancer Cell Atlas. We applied the x +> log(1 + x) transform to all entries.
In Fig. 11 we visualize P(X YY) by projecting the gene expression X on the first four principal components (calculated using all samples pooled together). We see that the distribution P(X Y) differs between the samples, although the cell types seem roughly cluster together and the random forest classifier may distinguish well between different subtypes.
As random forest we used the SciKit-Learn implementation (Pedregosa et al., 2011, 1441) with default hyperparameters and 20 trees. Before training the random forest we reduced the dimensionality by projecting the training data onto the first 50 principal components. This projection (onto the components defined by the training data) is used for making the predictions on other samples, before a random forest is applied.
Similarly as in App. E.I, in this experiment we ran four chains with 500 warm-up steps and 1000 samples each. To flag potential convergence issues, we calculated the potential scale reduction factors Å to two decimal places, which did not exceed 1.02. The minimal corresponding effective sample size 298.
E.5 Sensitivity to the prior choice
In Sec. 4 we used uniform priors over all simplices, which is supposed to act as a reference prior (Gelman et al., 2013, Sec. 2.8) for problems without precise domain-specific information. In practice, the inference
Table 1: Default parameters used in the experiments.
N
N
r
9
L
K
1000
500
0.7
0.85
5
32

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Number of simulations (out of 200) with < 1.02 for cach sample size N N and model.
Sample Size
Discrete (10 bins)
Discrete (5 bins)
Gaussian
Student
100
200
200
192
200
1000
199
199
174
200
10000
199
196
157
197
BT_S1 [_
BT_S2
BT BT_S4
BT BT_S6
PC 2
PC
PC
PC
PC 1
PC 1
PC 1
PC 1
PC 4
PC 4
PC 4
PC 4
PC 3
PC 3
PC 3
PC 3
Figure 11: Projections onto first four principal components of the whole data set. Each column describes the projections of a specified sample on the 1st and 2nd (first row) or the 3rd and 4th (sccond row) component, coloured by the cell lype. We distributional differences and conelude that P(X | Y) is not invariant between samples.
may depend on the choice of the prior and a sensitivity analysis should be performed (see Kruschke (2021) Gelman et al. (2020, Sec. 6.3)).
We illustrate the prior choice problem in an example with L - K = 2 classes with imperfect classifier P 1) P(C = Y = 2) = 0.7 and prevalence vectors π (0.5, 0.5) and n' = (0.7,0.3). We sampled three data sets, differing by the number of points, N = N' E {50, 500, 5000} and we fitted three models, differing by the choice of the prior.
We used the symmetric Dirichlet priors Dirichlet a. over all simplices. While simultaneously controls the parameters of the &p and n' matrix, choice which may not be often desired in practice, we hope that it provides meaningful information
the sensitivity of the inference to the choice of the prior. encourages more sparsity:
For = 1 this prior is uniform and has been studied in Sec. 4. Choosing the prior on the x' vector concentrates near to the boundary of the simplex (i.e., (1, and (0,1) vectors). Simultaneously, the model assumes that & matrix corresponds to sharper predictions. For example, this means that one of the entries =P(C==11 = 1), P(C = 2 Y = =1)} should larger than the other, i.e., the true positive rate and false negative rate should be very different. For a the prior distributes the mass around the center, preferring balanced prevalence vectors n'. Under this prior, the false negative rate and the true positive rate are similar.
We changed E 1, and performed Bayesian inference. Similarly as in Appendices E. and E.4 we ran four Markov chains with 500 warm-up steps and 1,000 samples each. To flag potential convergence issues, we calculated the potential scale reduction factors A, which did not exceed 1.02. The minimal effective sample size was 362.
The posterior of different models visualised in Fig. 12. We see that for small sample size (N N' 50). the sparse prior (a 0.1) puts mass at the boundary. At the same time, the predictions under a
33

---

Published in Transactions on Machine Learning Research (06/2024)
N N' = 50
N = N' 500
N = N' 5000
111
a= 0.1
1.0
a= 10.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
n2
Sensitivity of the Bayesian inference with respect to the Dirichlet prior with concentration pa- rameter a under different sample sizes N'. We plot the posterior the form of histogram, together with a vertical line representing posterior mean. The dashed line represents the ground-truth value.
Figure 12:
and 10 priors are nearly identical. For larger N N' the effect of the prior vanishes, with the posterior concentrating around the true value, which is in line with Theorem 3.1.
34


Reviews:
[{"review_id": "pKXcT7hOd8", "reviewer": "Reviewer_6Rrn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a result of taking a Bayesian approach to the problem at hand, this\npaper has a pleasing simplicity. The assumed model is conveniently\nrepresented as a DAG, and \"learning\" reduces to finding (or\napproximating) a posterior distribution over the unknown(s) of\ninterest. The advance over the Bayesian approach (to the same problem)\nof Storkey is that it does not rely on a generative model\nP(X|Y). Whether a generative mechanism for (PX|Y) might be\nmisspecified is highly contingent: presumably one could have real\nsituations where it is possible to specify it reasonably correctly. As\nthe authors correctly state: \"M_{approx} loses information from the\nproblem, so we do not expect it to be as appropriate as a properly\nspecified generative model for P(X|Y).\"\n\nThe authors \"work with a given dimension reduction mapping\", and, as\nthey explain, much depends on the appropriateness of the choice of\nmapping and whether it leads to a sufficient statistic (as the authors\ndo explain).\n\nA motivation for dimensionality reduction (and thus losing\ninformation) is that: \"However, for high-dimensional $\\theta$, or when\nN and N' are large, MCMC will generally not be tractable\". This is a\nvague statement about using MCMC for this problem; it would be good to\nsee evidence of problems with MCMC (by eg citing some paper showing\nit, or doing the experiments yourselves). The statement: \"We did not\ncompare obtained posteriors in high-dimensional problems due to high\ncomputational costs associated with running MCMC on high-dimensional\ngenerative models.\" is not evidence.\n\nIt is true that $\\tilde{K}_{y}(.;\\phi)$ (and thus M_approx) is\nparticularly simple which is indeed an advantage of the presented\nmethod. In Section 3.2 the categorical distributions are effectively\nreplaced with multinomial ones, which makes sense.\n\nThe experiments are fairly small. It would be good to see what happens\nwith large problems - and to see where, if anywhere, we hit the limits\nof scalability for the presented method. Also, there is no discussion\nof convergence (or lack thereof) of the MCMC samplers. I see that the\nauthors use numpyro to do the MCMC; does this package not report, eg\nrhat values?\n\nMINOR POINTS\n\nThe \"panels\" in Fig 3 are individual sub-figures. I recommend just\nlabelling them as 3(a) - 3(f) and refer to them thus (rather than \"the\nfirst panel of the second row\", etc). The sixth \"panel\" (concerning\nmodel misspecification) is never explicitly referenced in the text.\n\n\"To formalize the problem, consider a probabilistic graphical model in\nFig. 1\" State that the model referred to is M_true, rather than\nrequiring the reader to deduce this from the text that follows this\nstatement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a paper well suited to TMLR since the focus is on \"technical\ncorrectness over subjective significance\" [jmlr.org/tmlr]. A problem\nwe might care about (\"quantification\") is addressed competently, and\nthe method is described reasonably clearly. Related work and competing\nmethods are described properly. I am not sufficiently well versed in\nthis problem to be sure that the competing approaches are the SOTA, so\nwill have to trust the authors on this. The authors are careful with\ntheir claims, including those concerning their proposed method. The\nword \"may\" appears very frequently.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Report on the convergence status of MCMC runs\n2. Do at least one experiment on a sufficiently large instance to cause computational problems for the presented method (or prove that no such instance exists).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bkc320EZKQ", "reviewer": "Reviewer_qVtw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nClear problem formulation and motivation. \n\nThe Bayesian approach is natural and provides a principled way for uncertainty quantification of the estimated P_test(Y)\n\nThe proposed method is assessed and demonstrated from various aspects with both simulated data and scRNA data.\n\nWeakness:\nFor the proposed method to be practically useful, there seems to be a lack of discussion on the choice of prior and feature extractor.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper formulated the quantification problem as a Bayesian inference problem, and proposed a Bayesian approach that utilizes blackbox feature extractor and allows exact uncertainty quantification, which is particularly favorable in small sample scenario.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical questions to clarify in the paper:\n\n- How robust is the inference to the prior specification? When there is limited prior knowledge, what are some recommendations on the prior? \n- How to choose the blackbox feature extractor f in practice?\n\n\nMinor changes / questions:\n- Clarify the notation P_lab and P_unl \n- Theorem 3.1 proved some desirable properties of MAP. Then why is the posterior mean instead of MAP used as the point estimate in Section 4.2?", "broader_impact_concerns": "The main contributions of the paper is on the methodology so to the best of my understanding, there is no major potential broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "js9LmCsLC4", "reviewer": "Reviewer_StyW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*:\n- This paper addresses an interesting problem - the quantification problem which tries to estimate the class prevalence under the prior shift.\n- This paper carefully analyzes the connections and pros and cons among different approaches to the quantification problem, and extends the black-box shift estimator to the Bayesian setting which mitigates several existing issues.\n- The authors have demonstrated the claimed improvement including the robustness and superior performance in low-data scenarios through empirical evaluations.\n \n*Weaknesses*:\n- Overall, the paper is not easy to follow, and the presentation could be improved.\n- The experiments could be more thorough and include datasets that are used in the literature. Please see Requested Changes for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the quantification problem and generalizes the point estimate methods like the black-box shift estimators to the Bayesian setting. The authors theoretically analyze the asymptotical consistency of the proposed maximum a posterior inference under weak assumptions. In particular, the proposed approach explicitly measures the uncertainty and does not suffer from the negative value problem in the low-data regime. Empirical evaluation on synthetic and real-world datasets shows that the proposal is competitive with existing non-Bayesian quantification methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- On the presentation side, the authors might want to provide a more coherent description of the proposed method, highlighting the connections and the differences from existing method and reasons why it has the ability to resolve issues of existing methods. It also would be nice to have some comparisons between the \"weak assumptions\" used in Theorem 3.1 and the literature. Moreover, more detailed description of the full efficient Markov chain Monte Carlo sampling would be much appreciated to be included in Section 3.2.\n- On the experimental side, the authors might want to 1) group the experiments according to the improvements of the proposed method and provide some discussion about the performance comparisons; 2) give clear definition of \"weak uniform priors\" used in the experiment and potentially some additional experiments to demonstrate the sensitivity to the different prior specifications; 3) include experiments used in the existing literature so that the audience could have a direct comparison; 4) present some summary quantitative metrics since the lines corresponds to different methods shown in Figure 3 and 5 are difficult to differentiate.\n- In Section 5, the authors \"stress the importance of a principal shift in perspective\" about treating f as an auxiliary feature extraction method. But since this has been introduced in the literature especially the black-box shift estimator method, the authors might want to give a clearer argument regarding additional insight/perspective advocated in this work.\n- Some notations: 1) \"lab\"/\"unl\" looks ambiguous to me, might consider \"source\"/\"target\" or just \"train\"/\"test\"; 2) it should be $C_{j}^{'}$ instead of $X_{j}^{'}$ in equation (3).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a sound and interesting approach to the quantification problem. The methodology is reasonable, and the authors provide sufficient support for the claims made in the paper. The topic is relevant to the field of distribution shifts, as well as statistics. Thus, the paper meets both the evidence and the audience criteria of TMLR. 

All reviewers are unanimously leaning towards acceptance. The consensus is that the paper is sound and provides a meaningful contribution to TMLR. During the discussion period, the authors addressed almost all of the concerns raised by the reviewers. 

**Requested revision**: Please address the remaining questions of reviewer 6Rrn ([link](https://openreview.net/forum?id=Ft4kHrOawZ&noteId=ZNteGLwnus)). In particular, please describe the compute necessary for running the methods. Ideally, please run the requested missing experiments with multiple MCMC chains and $\hat R$ estimates, or provide a discussion of why it is impractical to run these experiments.
Decision: Accept (poster)