Binary Losses for Density Ratio Estimation

## Abstract

Abstract Estimating the ratio of two probability densities from finitely many observations of the densities, is a central problem in machine learning and statistics.
A large class of methods constructs estimators from binary classifiers which distinguish observations from the two densities.
However, the error of these constructions depends on the choice of the binary loss function, raising the question of which loss function to choose based on desired error properties. In this work, we start from prescribed error measures in a class of Bregman divergences and characterize all loss functions that lead to density ratio estimators with a small error.
Our characterization provides a simple recipe for constructing loss functions with certain properties, such as loss functions that prioritize an accurate estimation of large values.
This contrasts with classical loss functions, such as the logistic loss or boosting loss, which prioritize accurate estimation of small values.
We provide numerical illustrations with kernel methods and test their performance in applications of parameter selection for deep domain adaptation.

## 

## Binary Losses for Density Ratio Estimation

## Binary Losses for Density Ratio Estimation

### 1 Introduction

Estimating the Radon-Nikodým derivative $\beta:=\frac{\mathop{}\!\mathrm{d}P}{\mathop{}\!\mathrm{d}Q}:\mathcal{X}\to
\mathbb{R}$ of some probability measures $P$ with respect to another probability measure $Q$, from iid observations $(x_{i})_{i=1}^{n}$ and $(x_{i}^{\prime})_{i=1}^{m}$ of the two measures, respectively, is a common machine learning task; with applications in two-sample testing  (Neyman & Pearson, ; Kanamori et al., ; Hagrass et al., ), anomaly detection Smola et al. (); Hido et al. (), divergence estimation Nguyen et al. (); Rhodes et al. (), covariate shift adaptation Shimodaira (); Sugiyama et al. (); Dinu et al. (), energy-based modeling Ceylan & Gutmann (); Gutmann & Hyvärinen (); Tu (); Grover & Ermon (), generative modeling Mohamed & Lakshminarayanan (); Grover et al. (); Kim et al. (), conditional density estimation Schuster et al. (), and classification from positive and unlabeled data Kato et al. ().
When both measures $P$ and $Q$ have densities w.r.t. the Lebesgue reference measure, then $\beta$ equals the ratio of their densities.

A large class of methods for density ratio estimation follows Algorithm 1 to construct an estimator $\widehat{\beta}$ of $\beta$ from binary classifiers
which separate the observations $(x_{i})_{i=1}^{n}$ of $P$ from the observations $(x_{i}^{\prime})_{i=1}^{m}$ of $Q$ (Qin, ; Bickel et al., ; Cheng & Chu, ; Sugiyama et al., ; Menon & Ong, ; Kato & Teshima, ; Zellinger et al., ).
The estimator $\widehat{\beta}$ constructed by Algorithm 1 converges (for $m,n\to\infty$) to the minimizer of a Bregman divergence(^1^11Assuming $\beta\in L^{1}(Q)$ and defining $F_{\phi}:L^{1}(Q)\to\mathbb{R}$ by $F_{\phi}(h):=\int_{\mathcal{X}}\phi(h(x))\mathop{}\!\mathrm{d}Q(x)$ identifies Eq. 1 as a Bregman divergence (Bregman, ) of the form $B_{\phi}(\beta,\widehat{\beta})=F_{\phi}(\beta)-F_{\phi}(\widehat{\beta})- \nabla F_{\phi}(\widehat{\beta})[\beta-\widehat{\beta}]$.)

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta}):=\mathbb{E}_{x\sim Q}\!\left[ \phi\!\left(\beta\left(x\right)\right)-\phi(\widehat{\beta}(x))-\phi^{\prime}( \widehat{\beta}(x))(\beta(x)-\widehat{\beta}(x))\right]$ (1) $$

with generator $\phi:\mathbb{R}\to\mathbb{R}$ (Sugiyama et al., ; Menon & Ong, ; Zellinger et al., ).

However, the error measure $B_{\phi}$ in Eq. 1 is only implicitly defined by the choice of the binary loss function $\ell$ used to instantiate Algorithm 1.
That is, choosing the wrong loss function might lead to unexpected behavior of the density ratio estimator.
For example, typical loss functions for Algorithm 1 are the logistic loss used in (Bickel et al., , Section 7), the boosting loss applied in (Menon & Ong, , Section 7), the square loss (Menon & Ong, , Table 1) and logarithm-based loss functions (Nguyen et al., , ; Sugiyama et al., ).
All of the loss functions above lead to error measures in Eq. 1 that put higher weight on smaller values of the density ratio (cf. (Menon & Ong, )), despite the fact that large values are crucial in many applications.

Figure: Algorithm 1 Density ratio estimation by binary classification

In this work, instead of choosing classical loss function for Algorithm 1, we propose to start from an error measure $B_{\phi}$ in Eq. 1.
Our first contribution is, given such an error measure, to characterize all loss functions that lead to a minimization of this measure $B_{\phi}$ by Algorithm 1.
More precisely, given some strictly convex $\phi:[0,\infty)\to\mathbb{R}$ and some $g:\mathbb{R}\to\mathbb{R}$, we provide the unique form of a (convex and strictly proper composite) loss function $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ and invertible link $\Psi:[0,1]\to\mathbb{R}$ such that Algorithm 1 minimizes $B_{\phi}(\beta,\widehat{\beta})$ for $\widehat{\beta}:=g\circ\widehat{f}$.
We also prove that the canonical choice $g_{\mathrm{can}}(\widehat{y}):=(\phi^{\prime})^{-1}(\widehat{y})$ leads to convex loss functions.

Our characterizations provide us with a simple recipe for constructing loss functions $\ell$ that allow Algorithm 1 to find estimators $\widehat{\beta}$ with certain properties.
Our second contribution is, using our characterization, to provide novel loss functions that (in contrast to classical losses) prioritize an accurate estimation of large density ratio values over smaller ones, see Figure 1.

One important application of density ratio estimation is importance weighting.
Our third contribution is to show that our novel losses can lead to improvements for the parameter selection procedures of Sugiyama et al. (); Dinu et al. () when their importance weights are computed with our novel loss function instead of the ones of Example 1.
More precisely, importance weighted validation (Sugiyama et al., ) (respectively importance weighted aggregation (Dinu et al., )) selects better parameters on two (respectively three) out of three datasets when it is based on our loss function instead of the ones in Example 1; where ”better” refers to a higher accuracy on average over 11 deep learning algorithms and all tasks of three datasets for text data (Blitzer et al., ), human body sensor signals (Stisen et al., ; Ragab et al., ) and images (Peng et al., ; Zellinger et al., ).

Figure: Figure 1: Left: Two piecewise constant probability measures $P$ and $Q$ on $[-1,1]$. Right: Estimators $\widehat{\beta}(x):=kx^{2}+d$ of density ratio $\beta=\frac{\mathop{}\!\mathrm{d}P}{\mathop{}\!\mathrm{d}Q}$ with $k,d>0$ computed by Algorithm 1 (for $m+n\to\infty$). Compared to LR (Bickel et al., ) and KuLSIF (Kanamori et al., ), our estimators originate from error measures $B_{\phi}$ in Eq. 1 with increasing weight functions $\phi^{\prime\prime}(c)=c,c^{6},e^{2c}$ and consequently obtain better estimates for large values $\beta(x),x\in[0.9,1]$, see Section 5.
Refer to caption: extracted/5703022/illustration.jpg

### 2 Related Work

It has been first observed by Sugiyama et al. () that many density ratio estimation methods minimize a Bregman divergence of the form of Eq. 1, see Example 1.

###### Example 1 .

- •
Following (Sugiyama et al., , Section 3.2.1), the kernel unconstrained least-squares importance fitting procedure (KuLSIF) of Kanamori et al. () uses $\ell(1,y):=-y$, $\ell(-1,y):=\frac{1}{2}y^{2}$ and the link function $\Psi^{-1}(y):=\frac{y}{1+y}$ in Algorithm 1. Eq. 1 is realized with $\phi(x)=(x-1)^{2}/2$ and $\widehat{\beta}(x)=\widehat{f}(x)$ (cf. Kanamori et al. (, Eq. (4))) such that
$B_{\phi}(\beta,\widehat{\beta})=\frac{1}{2}\big{\lVert}\beta-\widehat{\beta}
\big{\rVert}_{L^{2}(Q)}^{2}$.
- •
The logistic regression (Nelder & Wedderburn, ) approach (LR) applied in (Bickel et al., , Section 7) realizes Algorithm 1 with $\ell(1,y):=\log(1+e^{-y})$, $\ell(-1,y):=\log(1+e^{y})$ and link $\Psi^{-1}(y):=(1+e^{-y})^{-1}$.
The implicit error measure is Eq. 1 with $\phi(x)=x\log(x)-(1+x)\log(1+x)$ and $\widehat{\beta}(x)=e^{\widehat{f}(x)}$; see (Sugiyama et al., , Section 3.2.3).
- •
The Kullback-Leibler estimation procedure (KLest) of Nguyen et al. ()
uses $\ell(1,y):=-\log(x)$, $\ell(-1,y):=y$ and $\Psi^{-1}(y):=\frac{y}{1+y}$, leading to Eq. 1 with $\phi=x\log(x)-x$ and $\widehat{\beta}=\widehat{f}$; see (Sugiyama et al., , Section 3.2.4) and cf. (Sugiyama et al., ).
- •
The exponential loss (Boost) defined by $\ell(1,y):=e^{-y}$ and $\ell(-1,y):=e^{y}$, as applied in AdaBoost (Freund & Schapire, ), is used in (Menon & Ong, , Section 7) with link $\Psi^{-1}(y):=(1+e^{-2y})^{-1}$. Eq. 1 is used with $\phi(x)=x^{-3/2}$ and $\widehat{\beta}(x)=e^{2\widehat{f}(x)}$.

A general formula of the Bregman divergence for given (strictly proper composite) loss functions has been derived by  Menon & Ong ().
We extend their result by showing that the constructions in (Menon & Ong, , Appendix B) are in fact necessary if their loss representation (Menon & Ong, , Proposition 3) holds.
Our proofs are based on the theoretical works of Shuford Jr et al. (); Savage (); Reid & Williamson (, ); Bao () on proper loss functions.

There is a line of research concerned with the sampling behavior of algorithms following Eq. 1.
Specifically, for the special case of KuLSIF, error rates (for $m+n\to\infty$) have been discovered by Kanamori et al. (); Gizewski et al. (); Nguyen et al. ().
These results have been extended by Zellinger et al. (); Gruber et al. () to the general case of Eq. 1 for loss functions that satisfy a self-concordance property.
Although, our (continuous) analysis relies on similar theoretical arguments, except for the polynomial case of $k=0$, our novel loss functions are not self-concordant. That is, these results do not apply without modification.

Situations of quite dis-similar distributions have been recently discussed by Rhodes et al. () (density-chasm problem), Kato & Teshima (); Kiryo et al. () (PU-learning) and Srivastava et al. ().
Our work can be seen as analysis of a sub-problem of these works, where $P$ is large and $Q$ is small.
Choi et al. () propose an approach by training normalizing flows to obtain closer and simpler densities.

Another recent line of research applies density ratio estimation for correcting deep generative diffusion models (Kim et al., ).
In (Kim et al., ), the Bregman divergence approach in Eq. 1 is extended and further interesting Bregman divergences are identified.

###### Example 1 .

### 3 Notation and Problem

Let $P,Q\in\mathcal{M}_{+}^{1}(\mathcal{X})$ be two probability measures from the set $\mathcal{M}_{+}^{1}(\mathcal{X})$ of probability measures on a compact space $\mathcal{X}\subset\mathbb{R}^{d}$, $d\in\mathbb{N}$ such that $P$ is absolutely continuous with respect to $Q$, $P\ll Q$.
Sugiyama et al. () observed that many methods for estimating the density ratio $\beta:=\frac{\mathop{}\!\mathrm{d}P}{\mathop{}\!\mathrm{d}Q}$ are minimizations of estimated Bregman divergences Eq. 1.
Menon & Ong () further proved that there is in fact a general mathematical structure underlying this observation, which we review in the following.

We start with the data generation model of Reid & Williamson (); Menon & Ong ():
a probability measure $\rho$ on $\mathcal{X}\times\{-1,1\}$ with conditional probability measures(^3^33Existence follows from $\mathcal{X}\times\mathcal{Y}$ being Polish (cf. (Dudley, , Theorem 10.2.1), the product case).) defined by $\rho(x|y=1):=P(x)$ and $\rho(x|y=-1):=Q(x)$ and a marginal (w.r.t. $y$) probability measure $\rho_{\mathcal{Y}}$ defined as Bernoulli measure with probability(^4^44Our analysis can be generalized to probabilities $\pi\neq\frac{1}{2}$ using (Menon & Ong, , Appendix A).) $\frac{1}{2}$ for both events $y=1$ (”$x$ belongs to $P$”) and $y=-1$ (”$x$ belongs to $Q$”).
Our analysis is based on the following underlying assumption.

###### Assumption 1 ( Zellinger et al. ( 2023 ) ) .

The data

$$ $\displaystyle{\bf z}:=(x_{i},1)_{i=1}^{m}\cup(x_{i}^{\prime},-1)_{i=1}^{n}\in \mathcal{X}\times\mathcal{Y}$ (3) $$

is an i.i.d. sample of $\rho$.

Assumption 1 allows us to introduce a binary classification problem for the accessible data ${\bf z}$ as follows.
For a fixed loss function $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$, the goal of binary classification is to find, based on the given the data ${\bf z}$, a classifier $f:\mathcal{X}\to\mathbb{R}$ with a small expected risk

$$ $\displaystyle\mathcal{R}(f):=\mathbb{E}_{(x,y)\sim\rho}[\ell(y,f(x))]=\int_{ \mathcal{X}\times\mathcal{Y}}\ell(y,f(x))\mathop{}\!\mathrm{d}\rho(x,y).$ (4) $$

The loss function $\ell$ is a central object in our study.
In particular, we consider composite loss functions $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ such that $\ell(y,\widehat{y}):=\lambda(y,\Psi^{-1}(\widehat{y}))$ is composed of a loss function $\lambda:\{-1,1\}\times[0,1]\to\mathbb{R}$ and an invertible link function $\Psi:[0,1]\to\mathbb{R}$.
Composite loss functions allow to estimate $\rho(y=1|x)$ by functions $f:\mathcal{X}\to\mathbb{R}$ mapping outside of $[0,1]$ (Reid & Williamson, ).
A central object in the study of composite loss functions is the conditional risk

$$ $\operatorname{L}(\eta,\widehat{y}):=\eta\ell_{1}(\widehat{y})+(1-\eta)\ell_{-1 }(\widehat{y}),$ $$

where $\ell_{1}(\widehat{y}):=\ell(1,\widehat{y})$ and $\ell_{-1}(\widehat{y}):=\ell(-1,\widehat{y})$ denote the partial losses of $\ell$.
A composite loss function with invertible link $\Psi:[0,1]\to\mathbb{R}$ is called proper, if the conditional risk is minimized at $\eta=\Psi(\eta)$:

$$ $\displaystyle\operatorname{L}(\eta,\Psi(\eta))=\operatorname{\underline{L}}( \eta):=\inf_{\widehat{y}\in\mathbb{R}}\operatorname{L}(\eta,\widehat{y})$ $$

and it is strictly proper, if the Bayes risk $\operatorname{\underline{L}}(\eta)$ is achieved at a unique $\eta$.
The theoretical starting point of our work is the following key lemma for density ratio estimation.

###### Lemma 1 ( Menon & Ong ( 2016 , Proposition 3) ) .

Let $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ be a strictly proper composite loss function with invertible link function $\Psi:[0,1]\to\mathbb{R}$ and twice differentiable negative Bayes risk $-\operatorname{\underline{L}}:[0,1]\to\mathbb{R}$.
Then

$$ $\displaystyle\mathcal{R}(f)-\mathcal{R}(f^{\ast})=\frac{1}{2}B_{-\operatorname {\underline{L}}^{\diamond}}(\beta,g\circ f)$ (5) $$

with

$$ $\displaystyle g(y):=\frac{\Psi^{-1}(y)}{1-\Psi^{-1}(y)}\quad\text{and}\quad \phi^{\diamond}(z):=(1+z)\phi\!\left(\frac{z}{1+z}\right).$ (6) $$

For convenience of the reader, a proof of Lemma 3 is provide in Subsection A.3.
Lemma 3 specifies, for a given loss function, the asymptotic error $B_{-\operatorname{\underline{L}}^{\diamond}}(\beta,g\circ f)$ of the model $\widehat{\beta}:=g\circ\widehat{f}$ computed by Algorithm 1.
However, the error $B_{-\operatorname{\underline{L}}^{\diamond}}$ is only implicitly defined by the loss function $\ell$ and one might be interested in minimizing a prescribed error measure $B_{\phi}(\beta,g\circ f)$ for fixed $\phi$ and $g$.
The problem of this work is to answer the following question:

Which convex loss functions satisfy $\frac{1}{2}B_{\phi}(\beta,g\circ f)=\mathcal{R}(f)-\mathcal{R}(f^{\ast})$ for given $\phi$ and $g$?

Using the characterization of such loss functions, we are interested in designing loss functions that lead to error measures $B_{\phi}$ which put higher weight to large density ratio values than to small values, and, to test their effect in importance weighted parameter selection of deep neural networks.

###### Assumption 1 ( Zellinger et al. ( 2023 ) ) .

###### Lemma 1 ( Menon & Ong ( 2016 , Proposition 3) ) .

### 4 Characterization of Losses for Density Ratio Estimation

The following Theorem 1, see Subsection B.2 for a proof, answers our research question in Section 3 above for general (not necessarily convex) strictly proper composite loss functions.

###### Theorem 1 .

Let $\phi:[0,\infty)\to\mathbb{R}$ be strictly convex and twice differentiable and let $g:\mathbb{R}\to\mathbb{R}$ be strictly monotonically increasing.
Then, $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ is strictly proper composite and satisfies

$$ $\displaystyle\mathcal{R}(f)-\mathcal{R}(f^{\ast})=\frac{1}{2}B_{\phi}(\beta,g \circ f),\quad\forall P,Q\in\mathcal{M}_{1}^{+}(\mathcal{X}):P\ll Q,\forall f \in L^{1}(Q),$ (7) $$

if and only if

$$ $\displaystyle\ell(y,\widehat{y})=\begin{cases}\gamma\!\left(\Psi^{-1}(\widehat {y})\right)+\left(1-\Psi^{-1}(\widehat{y})\right)\gamma^{\prime}\!\left(\Psi^{ -1}(\widehat{y})\right)&\text{if}\ y=1\\ \gamma\!\left(\Psi^{-1}(\widehat{y})\right)-\Psi^{-1}(\widehat{y})\gamma^{ \prime}\!\left(\Psi^{-1}(\widehat{y})\right)&\text{if}\ y=-1\end{cases}$ (8) $$

with $\Psi^{-1}(\widehat{y}):=\frac{g(\widehat{y})}{1+g(\widehat{y})}$ and $\gamma(\widehat{\eta}):=-\phi\!\left(\frac{\widehat{\eta}}{1-\widehat{\eta}}
\right)(1-\widehat{\eta})+\widehat{\eta}c_{2}+c_{1}$ for some $c_{1},c_{2}\in\mathbb{R}$.

###### Remark 1 .

Theorem 1 extends (Menon & Ong, , Proposition 1), (Reid & Williamson, , Theorem 7) and Bao (, Corollary 3) in the following way.
We know from Menon & Ong (, Proposition 1)
that any strictly proper composite loss satisfies Eq. 7.
We also know from Reid & Williamson (, Theorem 7) (cf. (Reid & Williamson, , Theorem 17) and Bao (, Corollary 3)) that the loss in Eq. 8 constructed by any strictly concave $\gamma$ is strictly proper composite.
That is, the novelty of Theorem 1essentially consists of the fact that a strictly proper composite loss function satisfying Eq. 7 is necessarily of the form of Eq. 8.
We prove this necessity by our novel Lemma 4 (see Subsection B.1) combined with Theorem 3 originally proven by Savage ().

Theorem 1 expresses the expected Bregman divergence as an excess risk.
This allows us to estimate the Bregman divergence in a natural way by empirical risk minimization.
However, the minimization of non-convex functions is hard.
We therefore characterize convex loss functions by a corollary of Theorem 1 combined with Theorem 4 originally provided by Reid & Williamson ().

###### Corollary 1 .

Let $\phi:[0,\infty)\to\mathbb{R}$ be three times differentiable such that $\phi^{\prime\prime}(x)>0$ for all $x\in[0,\infty)$ and let $g:\mathbb{R}\to\mathbb{R}$ be strictly monotonically increasing.
Then, the loss $\ell$ in Eq. 8 is convex in its second argument for all $y\in\{-1,1\}$ if and only if

$$ $\displaystyle-\frac{1}{x}\leq\frac{\phi^{\prime\prime\prime}\!\left(x\right)}{ \phi^{\prime\prime}\!\left(x\right)}-\frac{\left(g^{-1}\right)^{\prime\prime} \!\left(x\right)}{\left(g^{-1}\right)^{\prime}\!\left(x\right)}\leq 0,\quad \forall x\in[0,\infty).$ (9) $$

Corollary 1 is proven in Subsection B.3.
Corollary 1 suggests a canonical

$$ $\displaystyle g_{\mathrm{can}}^{-1}(\widehat{y}):=\phi^{\prime}(\widehat{y})$ (10) $$

for constructing strictly convex loss functions by Eq. 8.

###### Remark 2 .

The canonical link $\Psi_{\mathrm{can}}^{\prime}(\operatorname{\widehat{\eta}}):=w(\operatorname{
\widehat{\eta}})=\operatorname{\widehat{\eta}}\ell^{\prime}_{1}(\Psi(
\operatorname{\widehat{\eta}}))$ of Buja et al. () (i.e., the matching loss of Kivinen & Warmuth (); Helmbold et al. ()) and the canonical $g_{\mathrm{can}}$ do not lead to the same loss function.
To see this, note that the equation $\Psi_{\mathrm{can}}^{-1}(\widehat{y})=\frac{g(\widehat{y})}{1+g(\widehat{y})}$ in Theorem 1 leads to

$$ $\left(g^{-1}\right)^{\prime}\!(z)=\Psi_{\mathrm{can}}^{\prime}\!\left(\frac{z} {1+z}\right)=w\!\left(\frac{z}{1+z}\right).$ $$

Buja et al. () show that differentiating a second time Eq. 19 gives $w(\operatorname{\widehat{\eta}})=-\operatorname{\underline{L}}^{\prime\prime}(
\operatorname{\widehat{\eta}})$ and therefore

$$ $\left(g^{-1}\right)^{\prime}\!(z)=-\operatorname{\underline{L}}^{\prime\prime} \!\left(\frac{z}{1+z}\right)=-\gamma^{\prime\prime}\!\left(\frac{z}{1+z}\right )=\phi^{\prime\prime}(z)(1+z)^{3}$ $$

which satisfies Eq. 9 but differs from $\left(g^{-1}_{\mathrm{can}}\right)^{\prime}=\phi^{\prime\prime}(z)$ by a factor $(1+z)^{3}$.

Thanks to Theorem 1 and Corollary 1, given a strictly convex Bregman generator $\phi$, we are now able to express the Bregman divergence $B_{\phi}(\beta,g_{\mathrm{can}}\circ f)$ as excess risk of a canonical convex loss function $\ell_{\mathrm{can}}$ defined by Eq. 8 and $g_{\mathrm{can}}$.
Using $\ell_{\mathrm{can}}$ and $\Psi^{-1}(\widehat{y})=\frac{g_{\mathrm{can}}(\widehat{y})}{1+g(wy)}$ to instantiate Algorithm 1, we arrive at minimizing $B_{\phi}(\beta,\widehat{\beta})$ for $\widehat{\beta}=g_{\mathrm{can}}\circ f$.

###### Theorem 1 .

###### Remark 1 .

###### Corollary 1 .

###### Remark 2 .

### 5 Loss Functions for Estimating Large Density Ratios

It has been observed by Menon & Ong () that no loss function in Example 1 prioritizes an accurate estimation of large density ratio values over smaller values.
To see this, note that the second derivative $\phi^{\prime\prime}$ of the Bregman generator $\phi$ satisfies the relation (see Subsection A.1 for a proof)

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta})=\mathbb{E}_{x\sim Q}\left[\int_{ 0}^{\infty}\phi^{\prime\prime}(c)\phi_{c}(\beta(x),\widehat{\beta}(x))\mathop{ }\!\mathrm{d}c\right]$ (11) $$

for the piecewise constant function $\phi_{c}$ defined in Eq. 16 that does not depend on $\phi$.
That is, the non-increasing weights $\phi^{\prime\prime}(c)=\frac{1}{c(1+c)}$ (LR), $\phi^{\prime\prime}(c)=c^{-1}$ (KLest), $\phi^{\prime\prime}(c)=c^{-3/2}$ (Exp) and $\phi^{\prime\prime}(c)=1$ (KuLSIF) of Example 1 do not realize a prioritization of estimating large values.
In contrast, we propose the following novel Bregman divergences as error measures for Algorithm 1, which prioritize an accurate estimation of large values, see Figure 1 for an illustration.

The polynomial weight functions $\phi^{\prime\prime}(c):=c^{2k}$ for $k\in\{0,1,2,3,\ldots\}$ and $g_{\mathrm{can}}^{-1}:=\phi^{\prime}$ in Eq. 10 lead to the novel Bregman divergences

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta})=\frac{1}{(1+k)(2+k)}\mathbb{E}_{ x\sim Q}\!\left[\beta(x)^{2+k}-(2+k)\beta(x)\widehat{\beta}(x)^{1+k}+(1+k) \widehat{\beta}(x)^{2+k}\right]$ (12) $$

which satisfy Eq. 7 with convex $\ell(1,y)=-y$ and ${\ell(-1,y)=\frac{((1+k)y)^{\frac{2+k}{1+k}}}{2+k}}$ and estimators ${\widehat{\beta}(x)=((1+k)\widehat{f}(x))^{\frac{1}{1+k}}}$.
Note that the class of divergences in Eq. 12 can be interpreted to extend KuLSIF which realizes Eq. 12 with $k=0$.

The exponential weight (EW) $\phi^{\prime\prime}(c):=e^{2c}$ and $g_{\mathrm{can}}^{-1}:=\phi^{\prime}$ lead to the novel Bregman divergence

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta})=\frac{1}{4}\mathbb{E}_{x\sim Q} \!\left[e^{2\beta(x)}+e^{2\widehat{\beta}(x)}(-1-2\beta(x)+2\widehat{\beta}(x) )\right]$ (13) $$

which satisfies Eq. 7 with $\ell(1,y)=-y$, $\ell(-1,y)=\frac{1}{2}y(\log(2y)-1)$ and $\widehat{\beta}(x)=\frac{1}{2}\log(2\widehat{f}(x))$.

### 6 Empirical Evaluations

#### 6.1 Numerical Illustrations

Before testing our novel loss functions for regularized kernel regression on real world datasets in Subsection 6.2 we illustrate the behavior of our methods on two experiments: Experiment (a) on the form of kernel estimators for a one-dimensional Gaussian density ratio and Experiment (b) on the behavior of the (analytically computed) density ratio estimators used as importance weights in kernel regression.

Figure: Figure 2: Estimation of density ratio (left, black) in Gaussian RKHS with Tikhonov penalty weighted by $\alpha$ for sample sizes $m+n=10$ (top right) and $m+n=100$ (lower right). Our loss function (blue) prioritizes accurate estimation of larger values and consequently produces flatter curves then KuLSIF (green).
Refer to caption: extracted/5703022/gauss.jpg

Experiment (a): Figure 2 shows the behavior of Algorithm 1 instantiated with the loss functions for KuLSIF (see example 1) and our novel loss function corresponding to Eq. 13, for the numerical illustration of Zellinger et al. ().
The function class $\mathcal{F}$ was fixed to a Gaussian RKHS with bandwidth parameter computed by the median heuristic.
We also added a Tikhonov penalty $+\alpha\lVert f\rVert_{\mathcal{H}}^{2}$ to Eq. 2 and computed the solution for different values of $\alpha\in\{10^{-6},10^{-4},10^{-2},1\}$ by the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, see, e.g., (Nocedal & Wright, ).

The equal weighting of all density ratio values in KuLSIF leads to exploding estimates for small sample size $m+n=10$ with small regularization parameter $\alpha\in\{10^{-6},10^{-4}\}$.
This problem does not appear for our loss function, which comes, however, at the cost of ”flatter” estimates due to a higher penalization of errors in larger density ratio values.

Experiment (b): Figure 3 extends the setting of Figure 1 to unsupervised domain adaptation under covariate shift (Shimodaira, ):
given iid source observations $(x_{i}^{\prime})_{i=1}^{m}$ from $Q$ together with labels $y_{i}^{\prime}=f_{P}(x_{i}^{\prime})+\epsilon_{i}$ corrupted by iid Gaussian noise observations $(\epsilon_{i})_{i=1}^{n}$ and unlabeled iid target observations $(x_{i})_{i=1}^{n}$ from $P$, the goal is to estimate the unknown Bayes predictor $f_{P}(x)=\sin(3x^{4})$.
This is done by importance weighted kernel least squares regression

$$ $\displaystyle\widehat{f}:=\operatorname*{arg\,min}_{f\in\mathcal{F}}\frac{1}{m +n}\sum_{i=1}^{m}\widehat{\beta}(x_{i}^{\prime})\left\lVert y_{i}^{\prime}-f(x _{i}^{\prime})\right\rVert_{2}^{2}+\alpha\lVert f\rVert_{\mathcal{F}}^{2}$ (14) $$

with $\widehat{\beta}$ of Figure 1 for LR and our method with exponential weight $\phi^{\prime\prime}(c)=e^{2c}$.
We used a degree-five polynomial RKHS $\mathcal{F}$, a small $\alpha=10^{-32}$ and computed the solution analytically thanks to the representer theorem Kimeldorf & Wahba ().

Our weight $\phi^{\prime\prime}(c)=e^{2c}$ is increasing, which leads to an error smaller than $0.040$ in squared $L^{2}(P)$-norm.
This can be explained by the fact that the $L^{2}(P)$-norm assigns higher weight to regions where the density ratio is large and that these regions are exactly the ones where our weight leads to more accurate estimates.
In contrast, the weight function of LR is decreasing, which leads to an error larger than $0.067$ in squared $L^{2}(P)$-norm.
The higher performance of our method in squared $L^{2}(P)$-norm comes at the cost of a higher squared $L^{2}(Q)$-norm ($0.798$) than for LR ($0.778$).
However, we note that in unsupervised domain adaptation we are interested in the target excess risk on $P$ which is nothing but the squared $L^{2}(P)$-norm.
The corresponding pointwise errors are depicted in Figure 3.

Figure: Figure 3: Using density ratio estimators from Figure 1 as sample weights for polynomial kernel least squares regression. Left: Bayes predictor (black, dashed) and observations ($\times$) from $Q$. Middle: regressors (blue: uniform weighting $\widehat{\beta}\equiv 1$, green: exact $\beta$, red: our estimate with $\phi^{\prime\prime}(c)=e^{2c}$ from Figure 1, yellow: LR estimate from Figure 1). Right: Pointwise errors for target observations from $P$. Exponential weight (ours) is higher for larger density ratio values; consequently achieves smaller errors on $[0.9,1]$.
Refer to caption: extracted/5703022/toy_experiment.png

#### 6.2 Real-World Datasets

In the following, we test the performance of our exponential weighted Bregman divergence in Eq. 13 compared to the performace of LR, KuLSIF and Boost (see Example 1) for calculating importance weights for parameter selection methods in unsupervised domain adaptation.
In total, we test the performance on 484 parameter selection tasks; one for each combination of a parameter selection method, a domain adaptation method (from three datasets) and a domain adaptation tasks (originating from three datasets). We trained 9174 neural networks in total, see Section C for details.

The unsupervised domain adaptation setting: given iid source observations $(x_{i}^{\prime})_{i=1}^{m}$ from $Q$ together with labels $y_{i}^{\prime}=f_{P}(x_{i}^{\prime})+\epsilon_{i}$ corrupted by general iid noise observations $(\epsilon_{i})_{i=1}^{n}$ and unlabeled iid target observations $(x_{i})_{i=1}^{n}$ from $P$, the goal is to estimate the generally unknown Bayes predictor $f_{P}:\mathcal{X}\to\mathcal{R}^{d_{y}}$, $d_{y}\in\mathbb{N}$.

Domain adaptation methods:
The goal is to select parameters from the methods Adversarial Spectral Kernel Matching (AdvSKM) (Liu & Xue, ), Deep Domain Confusion (DDC) (Tzeng et al., ), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et al., ), Central Moment Discrepancy (CMD) (Zellinger et al., ), Higher-order Moment Matching (HoMM) (Chen et al., ), Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman et al., ), Deep Subdomain Adaptation (DSAN) (Zhu et al., ), Domain-Adversarial Neural Networks (DANN) (Ganin et al., ), Conditional Adversarial Domain Adaptation (CDAN) (Long et al., ), DIRT-T (DIRT) (Shu et al., ) and Convolutional Deep Domain Adaptation specifically suitable for time-series (CoDATS) (Wilson et al., ).

Parameter selection methods:
We select these parameters by
Importance Weighted (Cross-)Validation (IWV) (Sugiyama et al., ) and Importance Weighted Aggregation (IWA) (Dinu et al., ).
Both methods, IWV and IWA, first compute several models $f_{1},\ldots,f_{l}:\mathcal{X}\to\mathbb{R}^{d_{y}}$, one for each parameter setting to try, and second compute the final model $\widehat{f}:\mathcal{X}\to\mathbb{R}^{d_{y}}$ by Eq. 14 for $\mathcal{F}=\{f_{1},\ldots,f_{l}\}$ and $\mathcal{F}=\{\sum_{k=1}^{l}c_{k}\cdot f_{k}|c_{1},\ldots,c_{k}\in\mathbb{R}\}$ for IWV and IWA, respectively.
Since Eq. 14 relies on an accurate estimate of $\beta=\frac{\mathop{}\!\mathrm{d}P}{\mathop{}\!\mathrm{d}Q}$ also IWV and IWA crucially depend on such an estimate.

Datasets and tasks:
We rely on three domain adaptation datasets with several domains and tasks.
The first dataset is the AmazonReviews dataset from Blitzer et al. () which consists of text reviews from four domains: books (B), DVDs (D), electronics (E), and kitchen appliances (K).
The reviews are encoded in feature vectors of bag-of-words unigrams and bigrams with binary labels indicating the sentiment of the rankings.
From the four domains we
obtain twelve domain adaptation tasks (B$\rightarrow$D, D$\rightarrow$E,…) where each domain serves once as source domain and once
as target domain.
The second dataset is the Heterogeneity Human Activity Recognition dataset (HHAR) from Stisen et al. () which investigates
sensor-, device- and workload-specific heterogeneities from 36 smartphones and smartwatches,
consisting of 13 different device models from four manufacturers.
We follow Ragab et al. () and obtain five domain adaptation tasks.
Our third dataset is the MiniDomainNet dataset from Zellinger et al. () which is a reduced version of the large scale DomainNet-2019 from Peng et al. () consisting of six
different image domains (Quickdraw, Real, Clipart, Sketch, Infograph, and Painting).
MiniDomainNet reduces the number of classes of DomainNet-2019 to the
top-five largest representatives in the training set of each class across all six domains.
MiniDomainNet is a multi-source single target dataset for which we obtain five tasks, one task for each domain chosen as target domain (and all others used as source domains).

Density ratio estimation methods: We estimate the density ratio by Algorithm 1 initialized by our loss function corresponding to Eq. 13 and the other loss functions from Example 1 excluding SQ which did not perform well in our experiments and also not in the one of Gruber et al. ().
Eq. 2 is optimized with the BFGS algorithm (100 iterations) in a Gaussian RKHS with bandwidth chosen by the median heuristic and Tikhonov regularization with parameter $\alpha\in\{10,0.1,10^{-3}\}$ chosen by cross-validation following Gruber et al. ().

Framework and benchmark protocol: Our framework(^5^55https://github.com/wzell/LargeDRE) extends the Pytorch framework of Gruber et al. (), by our exponential weight method in Eq. 13. The Pytorch framework of Gruber et al. () itself extends the one of Dinu et al. () which relies on the AdaTime benchmark suite of Ragab et al. ().
In all our remaining parameters, we exactly follow the statistical setup and data splits of Dinu et al. () together with the cross-validation based parameter choice for density ratio of Gruber et al. (), see Section C for details.

**Table 1: Mean ($\pm$ standard deviation) for three random initializations of model weights averaged over 12 domain adaptation tasks for AmazonReviews and five tasks for HHAR and MiniDomainNet; full tables in Subsections C.3, C.2 and C.3.**
| AmazonReviews |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
|  | SO | Boost | KuLSIF | LR | EW (ours) | Boost | KuLSIF | LR | EW (ours) | TB |
| AdvSKM | 0.766($\pm$.009) | 0.766($\pm$.010) | 0.767($\pm$.010) | 0.767($\pm$.013) | 0.770($\pm$.010) | 0.766($\pm$.013) | 0.773($\pm$.009) | 0.769($\pm$.012) | 0.778($\pm$.010) | 0.770($\pm$.010) |
| CDAN | 0.767($\pm$.012) | 0.774($\pm$.012) | 0.773($\pm$.011) | 0.772($\pm$.011) | 0.773($\pm$.009) | 0.768($\pm$.025) | 0.782($\pm$.014) | 0.775($\pm$.019) | 0.789($\pm$.011) | 0.777($\pm$.012) |
| CMD | 0.767($\pm$.009) | 0.773($\pm$.017) | 0.774($\pm$.013) | 0.774($\pm$.019) | 0.779($\pm$.011) | 0.745($\pm$.046) | 0.782($\pm$.013) | 0.772($\pm$.021) | 0.791($\pm$.010) | 0.785($\pm$.009) |
| CoDATS | 0.766($\pm$.012) | 0.785($\pm$.018) | 0.783($\pm$.017) | 0.781($\pm$.020) | 0.784($\pm$.017) | 0.777($\pm$.020) | 0.790($\pm$.012) | 0.784($\pm$.018) | 0.797($\pm$.010) | 0.791($\pm$.013) |
| DANN | 0.767($\pm$.012) | 0.780($\pm$.015) | 0.781($\pm$.017) | 0.777($\pm$.013) | 0.785($\pm$.011) | 0.772($\pm$.024) | 0.789($\pm$.011) | 0.779($\pm$.020) | 0.799($\pm$.008) | 0.798($\pm$.009) |
| DDC | 0.766($\pm$.011) | 0.769($\pm$.012) | 0.770($\pm$.010) | 0.770($\pm$.011) | 0.769($\pm$.012) | 0.765($\pm$.022) | 0.772($\pm$.014) | 0.769($\pm$.015) | 0.779($\pm$.011) | 0.770($\pm$.011) |
| DIRT | 0.764($\pm$.013) | 0.775($\pm$.020) | 0.780($\pm$.016) | 0.777($\pm$.017) | 0.783($\pm$.017) | 0.767($\pm$.022) | 0.786($\pm$.010) | 0.774($\pm$.016) | 0.794($\pm$.009) | 0.786($\pm$.015) |
| DSAN | 0.769($\pm$.012) | 0.778($\pm$.014) | 0.778($\pm$.013) | 0.777($\pm$.016) | 0.783($\pm$.013) | 0.769($\pm$.037) | 0.788($\pm$.014) | 0.782($\pm$.017) | 0.798($\pm$.008) | 0.789($\pm$.013) |
| DCoral | 0.766($\pm$.012) | 0.769($\pm$.012) | 0.770($\pm$.012) | 0.769($\pm$.011) | 0.771($\pm$.012) | 0.768($\pm$.020) | 0.778($\pm$.012) | 0.773($\pm$.014) | 0.783($\pm$.010) | 0.776($\pm$.011) |
| HoMM | 0.769($\pm$.012) | 0.765($\pm$.010) | 0.766($\pm$.012) | 0.766($\pm$.010) | 0.767($\pm$.011) | 0.762($\pm$.021) | 0.771($\pm$.013) | 0.765($\pm$.015) | 0.776($\pm$.009) | 0.769($\pm$.012) |
| MMDA | 0.767($\pm$.009) | 0.768($\pm$.014) | 0.772($\pm$.014) | 0.769($\pm$.012) | 0.775($\pm$.012) | 0.769($\pm$.018) | 0.778($\pm$.012) | 0.772($\pm$.018) | 0.789($\pm$.008) | 0.782($\pm$.012) |
| mean | 0.767($\pm$.011) | 0.773($\pm$.014) | 0.774($\pm$.013) | 0.773($\pm$.014) | 0.776($\pm$.012) | 0.766($\pm$.024) | 0.781($\pm$.012) | 0.774($\pm$.017) | 0.789($\pm$.010) | 0.781($\pm$.012) |
| HHAR |  |  |  |  |  |  |  |  |  |  |
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
|  | SO | Boost | KuLSIF | LR | EW (ours) | Boost | KuLSIF | LR | EW (ours) | TB |
| AdvSKM | 0.718($\pm$.044) | 0.729($\pm$.028) | 0.726($\pm$.025) | 0.729($\pm$.028) | 0.729($\pm$.028) | 0.682($\pm$.042) | 0.683($\pm$.053) | 0.684($\pm$.034) | 0.679($\pm$.039) | 0.749($\pm$.023) |
| CDAN | 0.728($\pm$.042) | 0.789($\pm$.035) | 0.765($\pm$.082) | 0.785($\pm$.033) | 0.781($\pm$.031) | 0.730($\pm$.051) | 0.747($\pm$.051) | 0.739($\pm$.047) | 0.755($\pm$.053) | 0.790($\pm$.025) |
| CMD | 0.748($\pm$.026) | 0.760($\pm$.040) | 0.757($\pm$.043) | 0.757($\pm$.043) | 0.768($\pm$.027) | 0.714($\pm$.038) | 0.728($\pm$.041) | 0.722($\pm$.040) | 0.723($\pm$.033) | 0.794($\pm$.015) |
| CoDATS | 0.710($\pm$.063) | 0.741($\pm$.052) | 0.720($\pm$.076) | 0.741($\pm$.052) | 0.739($\pm$.048) | 0.724($\pm$.042) | 0.731($\pm$.039) | 0.721($\pm$.028) | 0.732($\pm$.038) | 0.785($\pm$.027) |
| DANN | 0.757($\pm$.027) | 0.796($\pm$.030) | 0.793($\pm$.081) | 0.798($\pm$.027) | 0.798($\pm$.027) | 0.752($\pm$.044) | 0.759($\pm$.057) | 0.765($\pm$.037) | 0.767($\pm$.040) | 0.807($\pm$.023) |
| DDC | 0.716($\pm$.036) | 0.713($\pm$.052) | 0.701($\pm$.046) | 0.705($\pm$.039) | 0.712($\pm$.036) | 0.713($\pm$.042) | 0.702($\pm$.035) | 0.703($\pm$.043) | 0.703($\pm$.040) | 0.729($\pm$.035) |
| DIRT | 0.728($\pm$.030) | 0.748($\pm$.035) | 0.814($\pm$.060) | 0.748($\pm$.035) | 0.749($\pm$.035) | 0.718($\pm$.082) | 0.730($\pm$.066) | 0.725($\pm$.073) | 0.747($\pm$.047) | 0.820($\pm$.039) |
| DSAN | 0.721($\pm$.046) | 0.727($\pm$.066) | 0.717($\pm$.068) | 0.725($\pm$.062) | 0.723($\pm$.062) | 0.731($\pm$.030) | 0.736($\pm$.037) | 0.728($\pm$.034) | 0.731($\pm$.033) | 0.826($\pm$.023) |
| DCoral | 0.745($\pm$.039) | 0.747($\pm$.040) | 0.744($\pm$.038) | 0.748($\pm$.038) | 0.748($\pm$.038) | 0.673($\pm$.054) | 0.687($\pm$.035) | 0.682($\pm$.032) | 0.687($\pm$.031) | 0.776($\pm$.038) |
| HoMM | 0.739($\pm$.057) | 0.720($\pm$.041) | 0.730($\pm$.061) | 0.728($\pm$.037) | 0.728($\pm$.037) | 0.671($\pm$.061) | 0.690($\pm$.048) | 0.681($\pm$.053) | 0.703($\pm$.037) | 0.764($\pm$.020) |
| MMDA | 0.738($\pm$.053) | 0.726($\pm$.040) | 0.723($\pm$.031) | 0.726($\pm$.044) | 0.722($\pm$.044) | 0.685($\pm$.060) | 0.697($\pm$.039) | 0.690($\pm$.048) | 0.710($\pm$.029) | 0.785($\pm$.046) |
| mean | 0.732($\pm$.042) | 0.745($\pm$.042) | 0.744($\pm$.056) | 0.745($\pm$.040) | 0.745($\pm$.038) | 0.709($\pm$.050) | 0.717($\pm$.045) | 0.713($\pm$.043) | 0.722($\pm$.038) | 0.784($\pm$.028) |
| MiniDomainNet |  |  |  |  |  |  |  |  |  |  |
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
|  | SO | Boost | KuLSIF | LR | EW (ours) | Boost | KuLSIF | LR | EW (ours) | TB |
| AdvSKM | 0.509($\pm$.018) | 0.515($\pm$.014) | 0.514($\pm$.014) | 0.515($\pm$.014) | 0.516($\pm$.015) | 0.512($\pm$.012) | 0.513($\pm$.013) | 0.511($\pm$.015) | 0.512($\pm$.012) | 0.522($\pm$.020) |
| CDAN | 0.514($\pm$.015) | 0.511($\pm$.013) | 0.513($\pm$.019) | 0.510($\pm$.021) | 0.514($\pm$.018) | 0.519($\pm$.031) | 0.530($\pm$.014) | 0.531($\pm$.019) | 0.530($\pm$.021) | 0.542($\pm$.017) |
| CMD | 0.509($\pm$.022) | 0.525($\pm$.022) | 0.522($\pm$.023) | 0.522($\pm$.023) | 0.520($\pm$.021) | 0.517($\pm$.025) | 0.516($\pm$.025) | 0.519($\pm$.029) | 0.518($\pm$.031) | 0.533($\pm$.020) |
| CoDATS | 0.502($\pm$.032) | 0.514($\pm$.023) | 0.515($\pm$.020) | 0.512($\pm$.023) | 0.515($\pm$.020) | 0.527($\pm$.023) | 0.526($\pm$.015) | 0.525($\pm$.017) | 0.527($\pm$.016) | 0.529($\pm$.019) |
| DANN | 0.496($\pm$.019) | 0.511($\pm$.020) | 0.514($\pm$.018) | 0.512($\pm$.021) | 0.515($\pm$.018) | 0.515($\pm$.013) | 0.516($\pm$.014) | 0.516($\pm$.010) | 0.513($\pm$.014) | 0.532($\pm$.024) |
| DDC | 0.510($\pm$.021) | 0.513($\pm$.010) | 0.514($\pm$.010) | 0.513($\pm$.010) | 0.512($\pm$.012) | 0.516($\pm$.015) | 0.515($\pm$.018) | 0.515($\pm$.016) | 0.518($\pm$.018) | 0.521($\pm$.029) |
| DIRT | 0.499($\pm$.026) | 0.491($\pm$.033) | 0.495($\pm$.032) | 0.500($\pm$.039) | 0.484($\pm$.024) | 0.519($\pm$.031) | 0.523($\pm$.025) | 0.519($\pm$.020) | 0.519($\pm$.021) | 0.525($\pm$.042) |
| DSAN | 0.509($\pm$.022) | 0.504($\pm$.020) | 0.510($\pm$.028) | 0.502($\pm$.020) | 0.516($\pm$.013) | 0.518($\pm$.018) | 0.520($\pm$.014) | 0.522($\pm$.015) | 0.524($\pm$.018) | 0.563($\pm$.024) |
| DCoral | 0.505($\pm$.028) | 0.522($\pm$.012) | 0.522($\pm$.014) | 0.521($\pm$.013) | 0.522($\pm$.015) | 0.520($\pm$.012) | 0.520($\pm$.013) | 0.521($\pm$.013) | 0.521($\pm$.013) | 0.535($\pm$.017) |
| HoMM | 0.509($\pm$.023) | 0.518($\pm$.021) | 0.521($\pm$.025) | 0.516($\pm$.020) | 0.514($\pm$.019) | 0.521($\pm$.019) | 0.526($\pm$.013) | 0.523($\pm$.016) | 0.527($\pm$.014) | 0.537($\pm$.014) |
| MMDA | 0.509($\pm$.022) | 0.522($\pm$.020) | 0.518($\pm$.018) | 0.518($\pm$.018) | 0.519($\pm$.020) | 0.530($\pm$.012) | 0.529($\pm$.013) | 0.530($\pm$.016) | 0.528($\pm$.013) | 0.531($\pm$.013) |
| mean | 0.507($\pm$.022) | 0.513($\pm$.019) | 0.514($\pm$.020) | 0.513($\pm$.020) | 0.513($\pm$.018) | 0.519($\pm$.019) | 0.521($\pm$.016) | 0.521($\pm$.017) | 0.522($\pm$.017) | 0.534($\pm$.022) |

#### 6.3 Results of Empirical Evaluations

Beyond the obvious advantages of our novel methods illustrated in Figure 1, Experiment (a) and Experiment (b), we obtain the following results (see Subsections C.3, C.2 and C.3 for the full results):

- •
EW outperforms on average LR, KuLSIF and Boost for IWV and IWA on all three datasets, excluding IWV on MiniDomainNet for which all methods (Boost, KuLSIF, LR and EW) perform similarly.
- •
EW always obtains the highest average accuracy; see the ”mean” rows, i.e., averaged over all 11 domain adaptation algorithms in Table 1.
- •
EW is among the best methods in 56% of parameter selection problems, i.e., among all cells in all tables in Subsections C.3, C.2 and C.3; clearly outperforming KuLSIF (23%), LR (13%) and Boost (17%).

### 7 Conclusion and Future Work

A large class of methods for density ratio estimation minimizes a Bregman divergence between the true density ratio and an estimator.
However, the Bregman divergence is only implicitly determined and depends on the choice of a binary loss function used to instantiate the algorithm that underlies this class of methods.
In this work, we derived the unique formula of a (strictly proper composite convex) loss function for an explicitly given Bregman divergence that we want to minimize.
Our formula allows us to construct novel loss functions with the property of focusing on estimating large density ratio values, which is in contrast to related approaches.
We provide numerical illustrations and applications in deep domain adaptation.
The sample complexity of our novel methods is an open problem.

##### Acknowledgments

The author thanks Lukas Gruber and Marius-Constantin Dinu for helpful discussions on their Pytorch frameworks.

## Appendix

### Appendix A Auxiliaries

#### A.1 Weight Representation of Bregman Divergences

The second derivative $\phi^{\prime\prime}$ of the Bregman generator $\phi$ of Eq. 1 can be seen as a weight of the Bregman divergence as follows.

###### Lemma 2 ( Reid & Williamson ( 2009 ) ) .

Let $\phi:\mathbb{R}\to\mathbb{R}$ be strictly convex and twice differentiable.
Then

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta})=\mathbb{E}_{x\sim Q}\left[\int_{ 0}^{\infty}\phi^{\prime\prime}(c)\phi_{c}(\beta(x),\widehat{\beta}(x))\mathop{ }\!\mathrm{d}c\right]$ (15) $$

with

$$ $\displaystyle\phi_{c}(r,\widehat{r}):=\begin{cases}(r-c)&\text{if}\ \widehat{r }<c\leq r\\ (c-r)&\text{if}\ r<c\leq\widehat{r}\\ 0&\text{otherwise}\end{cases}.$ (16) $$

For the convenience of the reader, we provide a proof of Lemma 2 based on Taylor’s theorem.

###### Proof of Lemma 2 .

For any $s_{0},s\in[a,b]\subset\mathbb{R}$ and $\phi:[a,b]\to\mathbb{R}$, Taylor’s theorem grants us the expansion (cf. (Reid & Williamson, , Corollary 7))

$$ $\displaystyle\phi(s)=\phi(s_{0})+\phi^{\prime}(s_{0})(s-s_{0})+\int_{a}^{b} \phi_{c}(s,s_{0})\phi^{\prime\prime}(c)\mathop{}\!\mathrm{d}c.$ (17) $$

It follows that

$$ $\displaystyle B_{\phi}(\beta,\widehat{\beta})$ $\displaystyle=\mathbb{E}_{x\sim Q}\left[\phi(\beta(x))-\phi(\widehat{\beta}(x) )-\phi^{\prime}(\widehat{\beta}(x))(\beta(x)-\widehat{\beta}(x))\right]$ $\displaystyle=\mathbb{E}_{x\sim Q}\left[\int_{\widehat{\beta}(x)\wedge\beta(x) }^{\widehat{\beta}(x)\vee\beta(x)}\phi_{c}(\beta(x),\widehat{\beta}(x))\phi^{ \prime\prime}(c)\mathop{}\!\mathrm{d}c\right]$ $\displaystyle=\mathbb{E}_{x\sim Q}\left[\int_{0}^{\infty}\phi_{c}(\beta(x), \widehat{\beta}(x))\phi^{\prime\prime}(c)\mathop{}\!\mathrm{d}c\right].$ $$

∎

###### Lemma 2 ( Reid & Williamson ( 2009 ) ) .

###### Proof of Lemma 2 .

#### A.2 Characterization of Proper Losses

For convenience of the reader, in the following, we provide all mathematical statements we applied together with self-inclusive proofs of related works.

Menon & Ong () discovered a connection(^6^66The relation $\phi^{\prime\prime}(c)=\frac{1}{(1+c)^{3}}w\!\left(\frac{c}{1+c}\right)$ is proven in (Menon & Ong, , Lemma 4) with $w$ as defined in Theorem 2.)
between $\phi^{\prime\prime}$ in Lemma 2 and the weight in a fundamental characterization of proper loss functions due to Shuford Jr et al. (, Theorem 1) (cf. (Aczél & Pfanzagl, ; Staël von Holstein, )), which we re-state here following Buja et al. (); Reid & Williamson ().

###### Theorem 2 ( Shuford Jr et al. ( 1966 , Theorem 1) ) .

A loss function $\ell:\{-1,1\}\times[0,1]\to\mathbb{R}$ with differentiable partial losses $\ell_{1},\ell_{-1}$ is strictly proper if and only if for all $\eta\in(0,1)$

$$ $\displaystyle\frac{\ell_{1}^{\prime}(\eta)}{\eta-1}=\frac{\ell^{\prime}_{-1}( \eta)}{\eta}=w(\eta)$ (18) $$

for some weight function $w:(0,1)\to[0,\infty)$ such that $\int_{\epsilon}^{1-\epsilon}w(c)\mathop{}\!\mathrm{d}c<\infty$ for all $\epsilon>0$.

###### Proof of Theorem 2 .

If $\ell$ is proper, then $\operatorname{L}(\eta,\eta)=\inf_{\widehat{\eta}\in(0,1)}\operatorname{L}(\eta
,\widehat{\eta})$, which implies that

$$ $0=\frac{\partial}{\partial\widehat{\eta}}\operatorname{L}(\eta,\widehat{\eta}) \Bigr{|}_{\widehat{\eta}=\eta}=(1-\eta)\ell_{-1}^{\prime}(\eta)+\eta\ell_{1}^{ \prime}(\eta)$ $$

and consequently also Eq. 18.

Eq. 18 gives

$$ $\displaystyle\frac{\partial}{\partial\widehat{\eta}}\operatorname{L}(\eta, \widehat{\eta})$ $\displaystyle=(1-\eta)\ell_{-1}^{\prime}(\widehat{\eta})+\eta\ell_{1}^{\prime} (\widehat{\eta})$ $\displaystyle=\eta(\widehat{\eta}-1)w(\widehat{\eta})+(1-\eta)\widehat{\eta}w( \widehat{\eta})$ $\displaystyle=(\widehat{\eta}-\eta)w(\widehat{\eta}),$ (19) $$

which provides us a unique minimum $\operatorname{L}(\eta,\eta)=\inf_{\widehat{\eta}\in(0,1)}\operatorname{L}(\eta
,\widehat{\eta})$ since $w(\eta)>0$.
∎

Another fundamental characterization of proper loss functions is due to Savage (), which we re-state here in the notation of (Reid & Williamson, , Theorem 4).

###### Theorem 3 ( Savage ( 1971 ) ) .

A loss function $\ell:\{-1,1\}\times[0,1]\to\mathbb{R}$ is proper if and only if its pointwise Bayes risk $\operatorname{\underline{L}}$ is concave and for each $\eta\in[0,1]$ and $\widehat{\eta}\in(0,1)$

$$ $\operatorname{L}(\eta,\widehat{\eta})=\operatorname{\underline{L}}(\widehat{ \eta})+(\eta-\widehat{\eta})\operatorname{\underline{L}}^{\prime}(\widehat{ \eta}).$ $$

The next proof of Theorem 3 is given by Reid & Williamson ().

###### Proof of Theorem 3 .

To prove the first direction, note that the properness of the loss $\ell$ means that

$$ $\operatorname{L}(\eta,\eta)=\operatorname{\underline{L}}(\eta):=\inf_{\widehat {\eta}\in[0,1]}\operatorname{L}(\eta,\widehat{\eta})$ $$

for all $\eta\in[0,1]$.
Expanding the derivative of the pointwise Bayes risk gives

$$ $\displaystyle\operatorname{\underline{L}}^{\prime}(\eta)$ $\displaystyle=\frac{\partial}{\partial\eta}\operatorname{L}(\eta,\eta)$ $\displaystyle=\frac{\partial}{\partial\eta}\left((1-\eta)\ell(-1,\eta)+\eta \ell(1,\eta)\right)$ $\displaystyle=-\ell(-1,\eta)+(1-\eta)\ell^{\prime}(-1,\eta)+\ell(1,\eta)+\eta \ell^{\prime}(1,\eta)$ $\displaystyle=-\ell(-1,\eta)+\ell(1,\eta),$ $$

where the last equality follows from the fact that

$$ $\displaystyle 0=\frac{\partial}{\partial\widehat{\eta}}\operatorname{L}(\eta, \widehat{\eta})\Bigr{|}_{\widehat{\eta}=\eta}=(1-\eta)\ell^{\prime}(-1,\eta)+ \eta\ell^{\prime}(1,\eta).$ $$

Eq. 22 can now be obtained as follows:

$$ $\displaystyle\operatorname{\underline{L}}(\widehat{\eta})+(\eta-\widehat{\eta} )\operatorname{\underline{L}}^{\prime}(\widehat{\eta})$ $\displaystyle=(1-\widehat{\eta})\ell(-1,\widehat{\eta})+\widehat{\eta}\ell(1, \widehat{\eta})+(\eta-\widehat{\eta})(-\ell(-1,\widehat{\eta})+\ell(1,\widehat {\eta}))$ $\displaystyle=(1-\eta)\ell(-1,\widehat{\eta})+\eta\ell(1,\widehat{\eta})$ $$

where the last line is just the definition of $\operatorname{L}(\eta,\widehat{\eta})$.

To prove the other direction, assume that $\phi:[0,1]\to\mathbb{R}$ is a concave function and

$$ $\displaystyle\operatorname{L}(\eta,\widehat{\eta})=\phi(\widehat{\eta})+(\eta- \widehat{\eta})\phi^{\prime}(\widehat{\eta}).$ $$

Taylor’s theorem grants us the expansion (cf. (Reid & Williamson, , Corollary 7))

$$ $\displaystyle\phi(\eta)=\phi(\widehat{\eta})+\phi^{\prime}(\widehat{\eta})( \eta-\widehat{\eta})+\int_{0}^{1}\phi_{c}(\eta,\widehat{\eta})\phi^{\prime \prime}(c)\mathop{}\!\mathrm{d}c.$ $$

for $\phi_{c}$ as defined in Lemma 2.
It follows that

$$ $\displaystyle\operatorname{L}(\eta,\widehat{\eta})=\phi(\eta)-\int_{0}^{1}\phi _{c}(\eta,\widehat{\eta})\phi^{\prime\prime}(c)\mathop{}\!\mathrm{d}c\geq\phi( \widehat{\eta})$ (20) $$

since $\phi$ is concave and $\phi^{\prime\prime}(\eta)\leq 0$. The integral in Eq. 20 is minimized to zero in the case of $\widehat{\eta}=\eta$.
∎

###### Theorem 2 ( Shuford Jr et al. ( 1966 , Theorem 1) ) .

###### Proof of Theorem 2 .

###### Theorem 3 ( Savage ( 1971 ) ) .

###### Proof of Theorem 3 .

#### A.3 Bregman Divergence Representation of Excess Risk

To prove Lemma 1, we will need the following Bregman identity.

###### Lemma 3 ( Menon & Ong ( 2016 , Lemma 2) ) .

Let $\phi:[0,1]\to\mathbb{R}$ be twice differentiable strictly convex and denote by $d_{\phi}(y,\widehat{y}):=\phi(y)-\phi(\widehat{y})-\phi^{\prime}(\widehat{y})(
y-\widehat{y})$.
Then, for all $x,y\in[0,\infty)$, it holds that

$$ $\displaystyle(1+x)\cdot d_{\phi}\!\left(\frac{x}{1+x},\frac{y}{1+y}\right)=d_{ \phi^{\diamond}}(x,y),$ (21) $$

where $\phi^{\diamond}(z):=(1+z)\cdot\phi\!\left(\frac{z}{1+z}\right)$.

###### Proof of Lemma 3 (Menon & Ong, 2016 , Section 5.1) .

Eq. 17 gives, for any $s_{0},s\in\mathbb{R}$:

$$ $\displaystyle d_{\phi}(s,s_{0})=\int_{0}^{\infty}\phi_{c}(s,s_{0})\phi^{\prime \prime}(c)\mathop{}\!\mathrm{d}c$ $$

with $\phi_{c}(s,s_{0})=|s-c|\cdot\llbracket(s-c)(s_{0}-c)<0\rrbracket$, where $\llbracket\circ\rrbracket$ is $1$ if the statement $\circ$ is true and it is 0 0 otherwise.
For $s_{0}:=\frac{y}{1+y}$ and $s:=\frac{x}{1+x}$, we obtain

$$ $\displaystyle d_{\phi}\!\left(\frac{x}{1+x},\frac{y}{1+y}\right)=\int^{\frac{x \vee y}{1+x\vee y}}_{\frac{y\wedge x}{1+y\wedge x}}\left|\frac{x}{1+x}-z\right |\cdot\phi^{\prime\prime}(z)\mathop{}\!\mathrm{d}z.$ $$

Applying the substitution $z=\frac{u}{1+u}$ with $\mathop{}\!\mathrm{d}z=\frac{\mathop{}\!\mathrm{d}u}{(1+u)^{2}}$ gives

$$ $\displaystyle d_{\phi}\!\left(\frac{x}{1+x},\frac{y}{1+y}\right)$ $\displaystyle=\int_{y\wedge x}^{x\vee y}\left|\frac{x}{1+x}-\frac{u}{1+u} \right|\cdot\phi^{\prime\prime}\!\left(\frac{u}{1+u}\right)\frac{1}{(1+u)^{2}} \mathop{}\!\mathrm{d}u$ $\displaystyle=\int_{y\wedge x}^{x\vee y}\frac{\left|x-u\right|}{(1+x)(1+u)} \cdot\phi^{\prime\prime}\!\left(\frac{u}{1+u}\right)\frac{1}{(1+u)^{2}}\mathop {}\!\mathrm{d}u$ $\displaystyle=\frac{1}{1+x}\int_{y\wedge x}^{x\vee y}\left|x-u\right|\cdot\phi ^{\prime\prime}\!\left(\frac{u}{1+u}\right)\frac{1}{(1+u)^{3}}\mathop{}\! \mathrm{d}u$ $\displaystyle=\frac{1}{1+x}d_{\phi^{\diamond}}(x,y),$ $$

where the last line uses

$$ $\displaystyle\left(\phi^{\diamond}\right)^{\prime}(z)=\frac{1}{1+x}\phi^{ \prime}\!\left(\frac{z}{1+z}\right)+\phi\!\left(\frac{z}{1+z}\right)~{}\text{ and}~{}\left(\phi^{\diamond}\right)^{\prime\prime}(z)=\phi^{\prime\prime}\! \left(\frac{z}{1+z}\right)\frac{1}{(1+z)^{3}}.$ $$

∎

We are now ready to provide a proof of Lemma 1.
For convenience of the reader, we don’t just repeat the arguments of Menon & Ong (, Proposition 3), but we also include the required basis arguments of Reid & Williamson () and Reid & Williamson ().

###### Proof of Lemma 1 .

The properness of $\ell(y,\Psi^{-1}(\widehat{y}))$ and Theorem 3 imply

$$ $\displaystyle\operatorname{L}(\eta,\widehat{y})=\operatorname{\underline{L}}( \Psi^{-1}(\widehat{y}))+(\eta-\Psi^{-1}(\widehat{y}))\operatorname{\underline{ L}}^{\prime}(\Psi^{-1}(\widehat{y})).$ (22) $$

Following (Reid & Williamson, , Corollary 13), the excess risk can be expressed as an expected Bregman divergence as follows:

$$ $\displaystyle\mathcal{R}(f)\,-$ $\displaystyle\,\mathcal{R}(f^{\ast})=$ $\displaystyle=\mathbb{E}_{(x,y)\sim\rho}\left[\ell\!\left(y,f(x)\right)\right] -\mathbb{E}_{(x,y)\sim\rho}\left[\ell(y,f^{\ast}(x))\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[\operatorname{L}\! \left(\eta(x),f(x)\right)\right]-\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[ \operatorname{\underline{L}}(\eta(x))\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[\operatorname{ \underline{L}}\!\left(\Psi^{-1}(f(x))\right)+(\eta(x)-\Psi^{-1}(f(x))) \operatorname{\underline{L}}^{\prime}\!\left(\Psi^{-1}(f(x))\right)- \operatorname{\underline{L}}(\eta(x))\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[d_{-\operatorname{ \underline{L}}}\!\left(\eta(x),\Psi^{-1}(f(x))\right)\right]$ (23) $$

with $d_{\phi}$ as defined in Lemma 3.
Note that the Bregman generator $-\operatorname{\underline{L}}$ is strictly convex, since $\operatorname{\underline{L}}$ is concave due to Theorem 3 and it is strictly concave due to the uniqueness of the minimum $\operatorname{\underline{L}}(\eta)=\min_{\widehat{y}\in\mathbb{R}}
\operatorname{L}(\eta,\widehat{y})$.

Following the proof of Menon & Ong (, Proposition 3), in the next step, we apply the Radon-Nikodým theorem to obtain:

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[d_{-\operatorname{ \underline{L}}}\!\left(\eta(x),\Psi^{-1}(f(x))\right)\right]$ $\displaystyle~{}~{}=\frac{1}{2}\mathbb{E}_{x\sim P}\left[d_{-\operatorname{ \underline{L}}}\!\left(\eta(x),\Psi^{-1}(f(x))\right)\right]+\frac{1}{2} \mathbb{E}_{x\sim Q}\left[d_{-\operatorname{\underline{L}}}\!\left(\eta(x), \Psi^{-1}(f(x))\right)\right]$ $\displaystyle~{}~{}=\frac{1}{2}\mathbb{E}_{x\sim Q}\left[\beta(x)d_{- \operatorname{\underline{L}}}\!\left(\eta(x),\Psi^{-1}(f(x))\right)\right]+ \frac{1}{2}\mathbb{E}_{x\sim Q}\left[d_{-\operatorname{\underline{L}}}\!\left( \eta(x),\Psi^{-1}(f(x))\right)\right]$ $\displaystyle~{}~{}=\frac{1}{2}\mathbb{E}_{x\sim Q}\left[(1+\beta(x))d_{- \operatorname{\underline{L}}}\!\left(\eta(x),\Psi^{-1}(f(x))\right)\right].$ $$

Bayes’ theorem gives $\beta(x)=\frac{\eta(x)}{1-\eta(x)}$ which implies $\eta(x)=\frac{\beta(x)}{1+\beta(x)}$ (Bickel et al., ). Applying Lemma 3 gives the desired result.
∎

###### Lemma 3 ( Menon & Ong ( 2016 , Lemma 2) ) .

###### Proof of Lemma 3 (Menon & Ong, 2016 , Section 5.1) .

###### Proof of Lemma 1 .

#### A.4 Characterization of Convex Losses

A characterization of convex proper composite loss functions can be stated as follows.

###### Theorem 4 ( Reid & Williamson ( 2010 , Theorem 24) ) .

Consider a strictly proper composite loss function $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ with invertible link $\Psi:[0,1]\to\mathbb{R}$ such that $\left(\Psi^{-1}\right)^{\prime}(z)>0$ for all $z\in\mathbb{R}$ and denote by $w(\operatorname{\widehat{\eta}}):=\operatorname{\widehat{\eta}}\ell^{\prime}_{
1}(\Psi(\operatorname{\widehat{\eta}}))$.
Then $\ell_{1}$ and $\ell_{-1}$ are convex if and only if

$$ $\displaystyle-\frac{1}{\operatorname{\widehat{\eta}}}\leq\frac{w^{\prime}( \operatorname{\widehat{\eta}})}{w(\operatorname{\widehat{\eta}})}-\frac{\Psi^{ \prime\prime}(\operatorname{\widehat{\eta}})}{\Psi^{\prime}(\operatorname{ \widehat{\eta}})}\leq\frac{1}{1-\operatorname{\widehat{\eta}}},\quad\forall \operatorname{\widehat{\eta}}\in(0,1).$ (24) $$

Theorem 4 suggests a canonical link (Buja et al., ) for convex losses is $\Psi_{\mathrm{can}}^{\prime}(\operatorname{\widehat{\eta}}):=w(\operatorname{
\widehat{\eta}})$, which corresponds to the notion of ”matching loss” studied by Kivinen & Warmuth (); Helmbold et al. ().

The following proof from Reid & Williamson () reads in our notation as follows.

###### Proof of Theorem 4 .

Let us denote by $q(\widehat{y}):=\Psi^{-1}(\widehat{y})$.
Then, the definition of $\operatorname{L}$ and Thm. 2, give

$$ $\displaystyle\frac{\partial}{\partial z}\operatorname{L}(\eta,z)\Bigr{|}_{z= \widehat{y}}$ $\displaystyle=(1-\eta)\ell_{-1}^{\prime}(q(z))q^{\prime}(z)+\eta\ell_{1}^{ \prime}(q(z))q^{\prime}(z)\Bigr{|}_{z=\widehat{y}}$ $\displaystyle=(1-\eta)w(q(\widehat{y}))q(\widehat{y})q^{\prime}(z)+\eta(q( \widehat{y})-1)w(q(\widehat{y}))q^{\prime}(z)$ $\displaystyle=(q(\widehat{y})-\eta)w(q(\widehat{y}))q^{\prime}(\widehat{y}).$ (25) $$

A necessary and sufficient condition for $\ell_{y}$ being convex is that

$$ $\displaystyle\frac{\partial^{2}}{\partial z^{2}}\operatorname{L}(\eta,z)\Bigr{ |}_{z=\widehat{y}}\geq 0,\quad\forall\widehat{y}\in\mathbb{R},\forall\eta\in\{ 0,1\}.$ $$

From Eq. 25, we obtain

$$ $\displaystyle\left[w(q(\widehat{y}))q^{\prime}(\widehat{y})\right]^{\prime} \left(q(\widehat{y})-\llbracket y=1\rrbracket\right)+w(q(\widehat{y}))q^{ \prime}(\widehat{y})q^{\prime}(\widehat{y})\geq 0,\quad\forall\widehat{y}\in \mathbb{R},$ (26) $$

where

$$ $\left[w(q(\widehat{y}))q^{\prime}(\widehat{y})\right]^{\prime}:=\frac{\partial }{\partial\widehat{y}}w(q(\widehat{y}))q^{\prime}(\widehat{y}).$ $$

As it is noted in (Reid & Williamson, ), Eq. 26 can be found in (Buja et al., , Eq. (39)).
To proceed further, note that Eq. 26 leads, for $y\in\{-1,1\}$ to the pair of inequalities

$$ $\displaystyle\begin{split}w(q(\widehat{y}))(q^{\prime}(\widehat{y}))^{2}&\geq- q(\widehat{y})\left[w(q(\widehat{y}))q^{\prime}(\widehat{y})\right]^{\prime}\\ w(q(\widehat{y}))(q^{\prime}(\widehat{y}))^{2}&\geq(1-q(\widehat{y}))\left[w(q (\widehat{y}))q^{\prime}(\widehat{y})\right]^{\prime}\end{split}$ (27) $$

which holds for all $\widehat{y}\in\mathbb{R}$.
Note that Eq. 27 holds for $q(\cdot)=0$ and $1-q(\cdot)=0$ because $q^{\prime}(\cdot)>0$ and $w(\cdot)>0$.
Without loss of generality, in the following, we restrict our attention to the set $\Psi((0,1))=\{x\in\mathbb{R}\,|\,q(x)\neq 0,1-q(x)\neq 0\}$.
From Eq. 27 we equivalently have

$$ $\displaystyle\frac{(q^{\prime}(\widehat{y}))^{2}}{1-q(\widehat{y})}\geq\frac{ \left[w(q(\widehat{y}))q^{\prime}(\widehat{y})\right]^{\prime}}{w(q(\widehat{y }))}\geq\frac{-(q^{\prime}(\widehat{y}))^{2}}{q(\widehat{y})},\quad\forall \widehat{y}\in\Psi((0,1)),$ (28) $$

where we have used the fact that $-q(\widehat{y})=-\Psi^{-1}(\widehat{y})$ is always negative, that $q(\widehat{y}),1-q(\widehat{y})$ are non-zero for $\widehat{y}\in\Psi((0,1))$ and $w(\cdot)>0$ for strictly proper loss function $\ell$.
Using that

$$ $\displaystyle\left[w(q(\cdot))q^{\prime}(\cdot)\right]^{\prime}=w^{\prime}(q( \cdot))q^{\prime}(\cdot)q^{\prime}(\cdot)+w(q(\cdot))q^{\prime\prime}(\cdot)$ $$

we obtain, equivalently to Eq. 28,

$$ $\displaystyle\frac{1}{1-q(\widehat{y})}\geq\frac{w^{\prime}(q(\widehat{y}))}{w (q(\widehat{y}))}+\frac{q^{\prime\prime}(\widehat{y})}{(q^{\prime}(\widehat{y} ))^{2}}\geq\frac{-1}{q(\widehat{y})},\quad\forall\widehat{y}\in\Psi((0,1)).$ (29) $$

By substituting $\operatorname{\widehat{\eta}}:=q(\widehat{y})$ so that $\widehat{y}=q^{-1}(\operatorname{\widehat{\eta}})=\Psi(\operatorname{\widehat{
\eta}})$, we get

$$ $\displaystyle\frac{1}{1-\operatorname{\widehat{\eta}}}\geq\frac{w^{\prime}( \operatorname{\widehat{\eta}})}{w(\operatorname{\widehat{\eta}})}+\frac{q^{ \prime\prime}(\Psi(\operatorname{\widehat{\eta}}))}{(q^{\prime}(\Psi( \operatorname{\widehat{\eta}})))^{2}}\geq\frac{-1}{\operatorname{\widehat{\eta }}},\quad\forall\operatorname{\widehat{\eta}}\in(0,1).$ (30) $$

Note that $\frac{1}{q^{\prime}(\Psi(\operatorname{\widehat{\eta}}))}=\frac{1}{q^{\prime}(
q^{-1}(\operatorname{\widehat{\eta}}))}=(q^{-1})^{\prime}(\operatorname{
\widehat{\eta}})=\Psi^{\prime}(\operatorname{\widehat{\eta}})$ and Eq. 30 is equivalent to

$$ $\displaystyle\frac{1}{1-\operatorname{\widehat{\eta}}}\geq\frac{w^{\prime}( \operatorname{\widehat{\eta}})}{w(\operatorname{\widehat{\eta}})}+q^{\prime \prime}(\Psi(\operatorname{\widehat{\eta}}))\left(\Psi^{\prime}(\operatorname{ \widehat{\eta}})\right)^{2}\geq\frac{-1}{\operatorname{\widehat{\eta}}},\quad \forall\operatorname{\widehat{\eta}}\in(0,1).$ (31) $$

Observe that $q^{\prime}(\cdot)=(\psi^{-1})^{\prime}(\cdot)=\frac{1}{\Psi^{\prime}(\Psi^{-1}
(\cdot))}$ and therefore

$$ $\displaystyle q^{\prime\prime}(\cdot)=(\Psi^{-1})^{\prime\prime}(\cdot)=\left( \frac{1}{\Psi^{\prime}(\Psi^{-1}(\cdot))}\right)^{\prime}=\frac{-1}{\left(\Psi ^{\prime}(\Psi^{-1}(\cdot))\right)^{3}}\Psi^{\prime\prime}(\Psi^{-1}(\cdot)),$ $$

which implies that

$$ $\displaystyle q^{\prime\prime}(\Psi(\operatorname{\widehat{\eta}}))\left(\Psi^ {\prime}(\operatorname{\widehat{\eta}})\right)^{2}=\frac{-1}{(\Psi^{\prime}( \operatorname{\widehat{\eta}}))^{3}}\Psi^{\prime\prime}(\operatorname{\widehat {\eta}})(\Psi^{\prime}(\operatorname{\widehat{\eta}}))^{2}=-\frac{\Psi^{\prime \prime}(\operatorname{\widehat{\eta}})}{\Psi^{\prime}(\operatorname{\widehat{ \eta}})}.$ $$

Plugging this into Eq. 31, which is equivalent to the convexity of $\ell$ in its second argument, completes the proof.
∎

###### Theorem 4 ( Reid & Williamson ( 2010 , Theorem 24) ) .

###### Proof of Theorem 4 .

### Appendix B Proofs of Novel Results

#### B.1 Key Lemma on Conditional Bayes Risk

Before proving our main theorem, we characterize the conditional Bayes risk $\operatorname{\underline{L}}$ of loss functions that satisfy the questioned Bregman representation $\mathcal{R}(f)-\mathcal{R}(f^{\ast})=\frac{1}{2}B_{\phi}(\beta,g\circ f)$.
The final form of the loss function will then follow from the fact that two loss functions with the same conditional Bayes risk have the same derivatives (Reid & Williamson, , Theorem 1).

###### Lemma 4 .

Let $\ell:\{-1,1\}\times\mathbb{R}\to\mathbb{R}$ be strictly proper composite with invertible link function $\Psi:[0,1]\to\mathbb{R}$ and denote by $g(\widehat{y}):=\frac{\Psi^{-1}(\widehat{y})}{1-\Psi^{-1}(\widehat{y})}$.
If there is a strictly convex twice differentiable $\phi:[0,\infty)\to\mathbb{R}$ such that

$$ $\displaystyle\mathcal{R}(f)-\mathcal{R}(f^{\ast})=\frac{1}{2}B_{\phi}(\beta,g \circ f),\quad\forall P,Q\in\mathcal{M}_{1}^{+}(\mathcal{X}):P\ll Q,\forall f \in L^{1}(Q),$ (32) $$

then

$$ $\displaystyle\operatorname{\underline{L}}\!\left(\widehat{y}\right)=-\phi\! \left(\frac{\Psi^{-1}(\widehat{y})}{1-\Psi^{-1}(\widehat{y})}\right)(1-\Psi^{- 1}(\widehat{y}))+c_{2}\Psi^{-1}(\widehat{y})+c_{1},\quad\forall\widehat{y}\in \mathbb{R},$ (33) $$

for some $c_{1},c_{2}\in\mathbb{R}$.

###### Proof.

From Eq. 32 and Bayes’ theorem, we know that

$$ $\displaystyle 2(\mathcal{R}(f)-\mathcal{R}(f^{\ast}))=B_{\phi}\!\left(\beta,g \circ f\right)=\mathbb{E}_{x\sim Q}\left[d_{\phi}\!\left(\frac{\Psi^{-1}(f^{ \ast}(x))}{1-\Psi^{-1}(f^{\ast}(x))},\frac{\Psi^{-1}(f(x))}{1-\Psi^{-1}(f(x))} \right)\right].$ $$

Applying Lemma 3 with $\widetilde{\phi}^{\diamond}:=\phi$, $\widetilde{x}:=\frac{\Psi^{-1}(f^{\ast}(x))}{1-\Psi^{-1}(f^{\ast}(x))}$ and $\widetilde{y}:=\frac{\Psi^{-1}(f(x))}{1-\Psi^{-1}(f(x))}$, such that
$\frac{\widetilde{x}}{1+\widetilde{x}}=\Psi^{-1}(f^{\ast}(x))$ and $\frac{\widetilde{y}}{1+\widetilde{y}}=\Psi^{-1}(f(x))$, gives

$$ $\displaystyle d_{\phi}\!\left(\frac{\Psi^{-1}(f^{\ast}(x))}{1-\Psi^{-1}(f^{ \ast}(x))},\frac{\Psi^{-1}(f(x))}{1-\Psi^{-1}(f(x))}\right)=(1+\beta(x))d_{- \zeta}\!\left(\Psi^{-1}(f^{\ast}(x)),\Psi^{-1}(f(x))\right)$ $$

for

$$ $\displaystyle\zeta(z):=-\phi\!\left(\frac{z}{1-z}\right)(1-z),$ $$

where we have substituted $z=\frac{u}{1-u}$ in the definition of $\widetilde{\phi}^{\diamond}(z)$ in Lemma 3 to obtain

$$ $\widetilde{\phi}(u)=(1-u)\widetilde{\phi}^{\diamond}\!\left(\frac{u}{1-u} \right)=(1-u)\phi\!\left(\frac{u}{1-u}\right)=-\zeta(u),$ $$

which is twice differentiable and strictly convex as $\phi$ is twice differentiable and strictly convex.
Denoting by $u:=\Psi^{-1}(f^{\ast}(x))$ and $\widehat{u}:=\Psi^{-1}(f(x))$ and
summarizing the steps above, we have

$$ $\displaystyle 2(\mathcal{R}(f)$ $\displaystyle-\mathcal{R}(f^{\ast}))=\mathbb{E}_{x\sim Q}\left[(1+\beta(x))d_{ -\zeta}\!\left(u,\widehat{u}\right)\right].$ $$

The transformation used in the proof of (Menon & Ong, , Proposition 3) is the following:

$$ $\displaystyle\mathbb{E}_{x\sim Q}\left[(1+\beta(x))d_{-\zeta}\!\left(u, \widehat{u}\right)\right]$ $\displaystyle=\frac{1}{2}\mathbb{E}_{x\sim Q}\left[\beta(x)d_{-\zeta}\!\left(u ,\widehat{u}\right)\right]+\frac{1}{2}\mathbb{E}_{x\sim Q}\left[d_{-\zeta}\! \left(u,\widehat{u}\right)\right]$ $\displaystyle=\frac{1}{2}\mathbb{E}_{x\sim P}\left[d_{-\zeta}\!\left(u, \widehat{u}\right)\right]+\frac{1}{2}\mathbb{E}_{x\sim Q}\left[d_{-\zeta}\! \left(u,\widehat{u}\right)\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[d_{-\zeta}\!\left(u, \widehat{u}\right)\right],$ $$

which gives

$$ $\displaystyle 2(\mathcal{R}(f)$ $\displaystyle-\mathcal{R}(f^{\ast}))=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left [d_{-\zeta}\!\left(u,\widehat{u}\right)\right].$ (34) $$

At the same time, we know from Reid & Williamson (, Corollary 13) that (see Eq. 23 for a proof)

$$ $\displaystyle 2(\mathcal{R}(f)$ $\displaystyle-\mathcal{R}(f^{\ast}))=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left [d_{-\operatorname{\underline{L}}}\!\left(u,\widehat{u}\right)\right].$ (35) $$

Combining Eq. 34 and Eq. 35 gives

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[d_{-\operatorname{ \underline{L}}}\!\left(u,\widehat{u}\right)\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[d_{-\zeta}\!\left(u, \widehat{u}\right)\right]$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[-\operatorname{ \underline{L}}(u)+\operatorname{\underline{L}}(\widehat{u})+\operatorname{ \underline{L}}^{\prime}(\widehat{u})(u-\widehat{u})\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[-\zeta(u)+\zeta( \widehat{u})+\zeta^{\prime}(\widehat{u})(u-\widehat{u})\right]$ $$

holding for any $P,Q\in\mathcal{M}_{1}^{+}(\mathcal{X})$ with $P\ll Q$ and for any $f\in L^{1}(Q)$.
In particular, for $f(x):=1$ we have

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[\zeta(u)-\operatorname{ \underline{L}}(u)\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[(\zeta^{\prime}( \widehat{u})-\operatorname{\underline{L}}^{\prime}(\widehat{u}))(u-\widehat{u} )\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[u(\zeta^{\prime}( \widehat{u})-\operatorname{\underline{L}}^{\prime}(\widehat{u}))\right]+ \mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[-\widehat{u}(\zeta^{\prime}(\widehat {u})-\operatorname{\underline{L}}^{\prime}(\widehat{u}))\right]$ $\displaystyle=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\left[u\right]\left(\zeta^{ \prime}(\widehat{u})-\operatorname{\underline{L}}^{\prime}(\widehat{u})\right) -\widehat{u}(\zeta^{\prime}(\widehat{u})-\operatorname{\underline{L}}^{\prime} (\widehat{u}))=c,$ $$

where

$$ $\mathbb{E}_{x\sim\rho_{\mathcal{X}}}[u]=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\! \left[\Psi^{-1}(f^{\ast}(x))\right]=\mathbb{E}_{x\sim\rho_{\mathcal{X}}}[\rho( y=1|x)]=\frac{1}{2}$ $$

follows from the properness of $\ell$ and

$$ $c:=\frac{1}{2}\left(\zeta^{\prime}(\widehat{u})-\operatorname{\underline{L}}^{ \prime}(\widehat{u})\right)-\widehat{u}(\zeta^{\prime}(\widehat{u})- \operatorname{\underline{L}}^{\prime}(\widehat{u}))$ $$

is constant since $\widehat{u}=\Psi^{-1}(f(x))=\Psi^{-1}(1)\in[0,1]$.
To summarize the steps above, we may state with $\varphi:=\zeta-\operatorname{\underline{L}}$ that

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\!\left[\varphi\!\left(\rho(y =1|x)\right)\right]=c$ $$

for all $P,Q\in\mathcal{M}_{1}^{+}(\mathcal{X})$ with $P\ll Q$. In the following, we will construct probability density functions on $[0,1]$ witnessing the affinity of $\varphi$.
The case $[0,1]\subset\mathcal{X}$ then follows directly and the case $\mathcal{X}\subset[0,1]$ follows by scaling.

Let us define the probability density functions

$$ $\displaystyle p_{r}(x):=\begin{cases}r&\text{if}\ x\in[0,\frac{1}{r}]\\ 0&\text{otherwise}\end{cases}\quad\quad\quad q(x):=\begin{cases}1&\text{if}\ x \in[0,1]\\ 0&\text{otherwise}\end{cases}$ $$

for $r\in[1,\infty)$.
Then, it holds that

$$ $\rho(y=1|x)=\frac{\rho(x|y=1)\rho_{\mathcal{Y}}(y=1)}{\rho_{\mathcal{X}}(x)}= \frac{p_{r}(x)}{p_{r}(x)+q(x)}=\begin{cases}\frac{r}{r+1}&\text{if}\ x\in[0, \frac{1}{r}]\\ 0&\text{otherwise}\end{cases}$ $$

which gives

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\!$ $\displaystyle\left[\varphi(\rho(y=1|x))\right]=$ $\displaystyle=\frac{1}{2}\int_{0}^{1}\varphi(\rho(y=1|x))p_{r}(x)\mathop{}\! \mathrm{d}x+\frac{1}{2}\int_{0}^{1}\varphi(\rho(y=1|x))q(x)\mathop{}\!\mathrm{ d}x$ $\displaystyle=\frac{1}{2}\int_{0}^{\frac{1}{r}}\varphi\!\left(\frac{r}{r+1} \right)r\mathop{}\!\mathrm{d}x+\frac{1}{2}\int_{0}^{\frac{1}{r}}\varphi\!\left (\frac{r}{r+1}\right)1\mathop{}\!\mathrm{d}x+\frac{1}{2}\int_{\frac{1}{r}}^{1} \varphi\!\left(0\right)r\mathop{}\!\mathrm{d}x$ $\displaystyle=\frac{1}{2}\varphi\!\left(\frac{r}{r+1}\right)+\frac{1}{2r} \varphi\!\left(\frac{r}{r+1}\right)+\frac{1}{2}\varphi(0)\left(1-\frac{1}{r}\right)$ $\displaystyle=\varphi\!\left(b\right)\frac{1}{2b}+\varphi(0)\frac{2b-1}{2b}$ $$

for $b:=\frac{r}{r+1}\in[\frac{1}{2},1]$ with $b=1$ in the limit. As a consequence, we get

$$ $\displaystyle f(b)=b(2c-2\varphi(0))+\varphi(0)$ (36) $$

for all $b\in[\frac{1}{2},1]$.
To prove the relationship also for $b\in[0,\frac{1}{2}]$, consider the densities

$$ $\displaystyle p_{r}(x):=\begin{cases}r&\text{if}\ x\in[0,\frac{1}{2}]\\ 2-r&\text{otherwise}\end{cases}\quad\quad\quad q_{r}(x):=\begin{cases}2-r& \text{if}\ x\in[0,\frac{1}{2}]\\ r&\text{otherwise}\end{cases}$ $$

for $r\in[0,2]$. It follows that

$$ $\rho(y=1|x)=\begin{cases}\frac{r}{2}&\text{if}\ x\in[0,\frac{1}{2}]\\ 1-\frac{r}{2}&\text{otherwise}\end{cases}$ $$

which gives

$$ $\displaystyle\mathbb{E}_{x\sim\rho_{\mathcal{X}}}\!$ $\displaystyle\left[\varphi(\rho(y=1|x))\right]=$ $\displaystyle=\frac{r}{4}\varphi\!\left(\frac{r}{2}\right)+\frac{2-r}{4} \varphi\!\left(1-\frac{r}{2}\right)+\frac{2-r}{4}\varphi\!\left(\frac{r}{2} \right)+\frac{r}{2}\varphi\!\left(1-\frac{r}{2}\right)$ $\displaystyle=\frac{1}{2}\varphi\!\left(\frac{r}{2}\right)+\frac{1}{2}\varphi \!\left(1-\frac{r}{2}\right).$ (37) $$

Let $r\in[0,1]$, then $1-\frac{r}{2}\in[\frac{1}{2},1]$ and we get from Eq. 36

$$ $\displaystyle\varphi\!\left(1-\frac{r}{2}\right)=\left(1-\frac{r}{2}\right)(2c -2\varphi(0))+\varphi(0).$ $$

The combination with Eq. 37 gives

$$ $\displaystyle\varphi\!\left(b\right)=b(2c-2\varphi(0))+\varphi(0)$ (38) $$

for $b:=\frac{r}{2}\in[0,\frac{1}{2}]$.
The statement follows with $c_{1}:=\varphi(0)$ and $c_{2}:=(2c-2\varphi(0))$.
∎

###### Lemma 4 .

###### Proof.

#### B.2 Main Characterization of Losses

Next, we prove our characterization of convex losses by properties of $\phi,g$.

###### Proof of Theorem 1 .

To prove sufficiency of Eq. 8, note that the conditional Bayes risk of the loss function in Eq. 8 satisfies

$$ $\displaystyle\operatorname{L}\!\left(\eta,\widehat{y}\right)$ $\displaystyle=\eta\ell(1,\widehat{y})+(1-\eta)\ell(-1,\widehat{y})$ $\displaystyle=\gamma(\Psi^{-1}(\widehat{y}))+\left(\eta-\Psi^{-1}(\widehat{y}) \right)\gamma^{\prime}(\Psi^{-1}(\widehat{y}))$ $$

with strictly concave $\gamma$ defined by the strictly convex function $\phi$.
It follows from Theorem 3 that $\ell$ with $\operatorname{\underline{L}}=\gamma\circ\Psi^{-1}$ is proper composite with link $\Psi:[0,1]\to\mathbb{R}$.
Moreover, $\ell$ is strictly proper composite since $\operatorname{\underline{L}}^{\prime\prime}(\widehat{y})=\gamma(\Psi^{-1}(
\widehat{y}))<0$ for all $\widehat{y}\in\mathbb{R}$.
Applying Lemma 1 gives

$$ $\displaystyle\mathcal{R}(f)-\mathcal{R}(f^{\ast})=\frac{1}{2}B_{-\operatorname {\underline{L}}^{\diamond}}(\beta,g\circ f)=\frac{1}{2}B_{-\gamma^{\diamond}}( \beta,g\circ f)=\frac{1}{2}B_{\phi}(\beta,g\circ f)$ $$

because $\gamma^{\diamond}(z)=\phi(z)-c_{2}z-c_{1}$, the fact that Bregman divergences are unique up to affine terms of the generator and the equality $g(y)=\frac{\Psi^{-1}(y)}{1-\Psi^{-1}(y)}$ which follows from the definition of $\Psi^{-1}$.

For proving necessity of Eq. 8 assume $\ell$ is a strictly proper composite loss function with invertible link function $\Psi:[0,1]\to\mathbb{R}$ satisfying Eq. 7.
From $\Psi^{-1}(\widehat{y})=\frac{g(\widehat{y})}{1+g(\widehat{y})}$ we obtain $g(\widehat{y})=\frac{\Psi^{-1}(\widehat{y})}{1-\Psi^{-1}(\widehat{y})}$.
That is, all requirements for Lemma 4 are satisfied and it’s application gives, for all $\widehat{\eta}\in[0,1]$,

$$ $\displaystyle\operatorname{\underline{L}}(\widehat{y})=\gamma\!\left(\Psi^{-1} (\widehat{y})\right).$ (39) $$

Next, we construct a strictly proper loss function in the same way as it is done in (Reid & Williamson, , Theorem 7) and (Bao, , Corollary 3).
We use Theorem 3 to obtain

$$ $\displaystyle\operatorname{L}\!\left(\eta,\widehat{y}\right)=\gamma(\Psi^{-1}( \widehat{y}))+\left(\eta-\Psi^{-1}(\widehat{y})\right)\gamma^{\prime}\!\left( \Psi^{-1}(\widehat{y})\right)$ $$

for all $\eta\in[0,1]$ and $\widehat{y}\in\mathbb{R}$, which is allowed since $\Psi^{-1}(\widehat{y})\in(0,1)$ as $g$ is strictly monotonically increasing.
The form of the loss function in Eq. 8 follows now from the definition

$$ $\displaystyle\operatorname{L}\!\left(\eta,\widehat{y}\right)$ $\displaystyle=\eta\ell(1,\widehat{y})+(1-\eta)\ell(-1,\widehat{y}),$ $$

for $\eta=0$ and $\eta=1$.
∎

###### Proof of Theorem 1 .

#### B.3 Characterization of Convex Losses

Next, we prove our characterization of convex losses by properties of $\phi,g$.

###### Proof of Corollary 1 .

Our aim is to apply Theorem 4.
In the following denote by $\operatorname{\widehat{\eta}}:=\Psi^{-1}(\widehat{y})$
Theorem 2 gives

$$ $\displaystyle\frac{\partial}{\partial\widehat{\widehat{y}}}\operatorname{L}( \eta,\widehat{y})$ $\displaystyle=(1-\eta)\ell_{-1}^{\prime}(\widehat{y})+\eta\ell_{1}^{\prime}( \widehat{y})$ $\displaystyle=\eta\left(\operatorname{\widehat{\eta}}-1\right)w\!\left( \operatorname{\widehat{\eta}}\right)+(1-\eta)\operatorname{\widehat{\eta}}w\! \left(\operatorname{\widehat{\eta}}\right)$ $\displaystyle=(\operatorname{\widehat{\eta}}-\eta)w\!\left(\operatorname{ \widehat{\eta}}\right)$ $$

and, together with Theorem 3, we have

$$ $\displaystyle w(\operatorname{\widehat{\eta}})$ $\displaystyle=\frac{1}{1-\operatorname{\widehat{\eta}}}\frac{\partial}{ \partial\widehat{y}}\operatorname{L}(\eta,\widehat{y})$ $\displaystyle=\frac{1}{1-\operatorname{\widehat{\eta}}}\frac{\partial}{ \partial\widehat{y}}\left(\operatorname{\underline{L}}(\widehat{y})+(\eta- \operatorname{\widehat{\eta}})\operatorname{\underline{L}}^{\prime}(\widehat{y })\right)$ $\displaystyle=\frac{1}{1-\operatorname{\widehat{\eta}}}\frac{\partial}{ \partial\widehat{y}}\left(\operatorname{\underline{L}}^{\prime}(\widehat{y})- \operatorname{\underline{L}}^{\prime}(\widehat{y})+(\eta-\operatorname{ \widehat{\eta}})\operatorname{\underline{L}}^{\prime\prime}(\widehat{y})\right)$ $\displaystyle=-\operatorname{\underline{L}}^{\prime\prime}(\widehat{y})$ $\displaystyle=-\gamma^{\prime\prime}(\operatorname{\widehat{\eta}}),$ $$

where the last equality follows from Lemma 4.
Note that

$$ $\displaystyle\gamma^{\prime}(\operatorname{\widehat{\eta}})$ $\displaystyle=\phi\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname {\widehat{\eta}}}\right)+\frac{1}{\operatorname{\widehat{\eta}}-1}\phi^{\prime }\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{\eta}}} \right)+c_{1}$ $\displaystyle\gamma^{\prime\prime}(\operatorname{\widehat{\eta}})$ $\displaystyle=\frac{1}{(\operatorname{\widehat{\eta}}-1)^{3}}\phi^{\prime \prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{ \eta}}}\right)$ $\displaystyle\gamma^{\prime\prime\prime}(\operatorname{\widehat{\eta}})$ $\displaystyle=\frac{1}{(\operatorname{\widehat{\eta}}-1)^{5}}\phi^{\prime \prime\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{ \widehat{\eta}}}\right)-\frac{3}{(\operatorname{\widehat{\eta}}-1)^{4}}\phi^{ \prime\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{ \widehat{\eta}}}\right)$ $$

and

$$ $\displaystyle\Psi(\operatorname{\widehat{\eta}})$ $\displaystyle=g^{-1}\!\left(\frac{\operatorname{\widehat{\eta}}}{1- \operatorname{\widehat{\eta}}}\right)$ $\displaystyle\Psi^{\prime}(\operatorname{\widehat{\eta}})$ $\displaystyle=\frac{1}{(1-\operatorname{\widehat{\eta}})^{2}}\left(g^{-1} \right)^{\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{ \widehat{\eta}}}\right)$ $\displaystyle\Psi^{\prime\prime}(\operatorname{\widehat{\eta}})$ $\displaystyle=\frac{1}{(1-\operatorname{\widehat{\eta}})^{4}}\left(g^{-1} \right)^{\prime\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1- \operatorname{\widehat{\eta}}}\right)-\frac{2}{(1-\operatorname{\widehat{\eta} })^{3}}\left(g^{-1}\right)^{\prime}\!\left(\frac{\operatorname{\widehat{\eta}} }{1-\operatorname{\widehat{\eta}}}\right),$ $$

which gives

$$ $\displaystyle\frac{w^{\prime}(\operatorname{\widehat{\eta}})}{w(\operatorname{ \widehat{\eta}})}$ $\displaystyle=\frac{\phi^{\prime\prime\prime}\!\left(\frac{\operatorname{ \widehat{\eta}}}{1-\operatorname{\widehat{\eta}}}\right)}{\phi^{\prime\prime} \!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{\eta}}} \right)(\operatorname{\widehat{\eta}}-1)^{2}}-\frac{3}{\operatorname{\widehat{ \eta}}-1}$ $\displaystyle\frac{\Psi^{\prime\prime}(\operatorname{\widehat{\eta}})}{\Psi^{ \prime}(\operatorname{\widehat{\eta}})}$ $\displaystyle=\frac{\left(g^{-1}\right)^{\prime\prime}\!\left(\frac{ \operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{\eta}}}\right)}{\left( g^{-1}\right)^{\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1- \operatorname{\widehat{\eta}}}\right)(\operatorname{\widehat{\eta}}-1)^{2}}- \frac{2}{\operatorname{\widehat{\eta}}-1}.$ $$

Theorem 4 grants convexity if and only if

$$ $\displaystyle-\frac{1}{\operatorname{\widehat{\eta}}}\leq\frac{\phi^{\prime \prime\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{ \widehat{\eta}}}\right)}{\phi^{\prime\prime}\!\left(\frac{\operatorname{ \widehat{\eta}}}{1-\operatorname{\widehat{\eta}}}\right)(\operatorname{ \widehat{\eta}}-1)^{2}}-\frac{\left(g^{-1}\right)^{\prime\prime}\!\left(\frac{ \operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{\eta}}}\right)}{\left( g^{-1}\right)^{\prime}\!\left(\frac{\operatorname{\widehat{\eta}}}{1- \operatorname{\widehat{\eta}}}\right)(\operatorname{\widehat{\eta}}-1)^{2}}- \frac{1}{\operatorname{\widehat{\eta}}-1}\leq\frac{1}{1-\operatorname{\widehat {\eta}}},\quad\forall\operatorname{\widehat{\eta}}\in(0,1).$ $$

Substituting $x:=\frac{\operatorname{\widehat{\eta}}}{1-\operatorname{\widehat{\eta}}}$ such that $\operatorname{\widehat{\eta}}=\frac{x}{1+x}$, $-\frac{1}{\operatorname{\widehat{\eta}}}=-\frac{1+x}{x}$ and $-\frac{1}{\operatorname{\widehat{\eta}}-1}=1+x$ gives

$$ $\displaystyle-\frac{1+x}{x}\leq\frac{\phi^{\prime\prime\prime}\!\left(x\right) }{\phi^{\prime\prime}\!\left(x\right)}(1+x)^{2}-\frac{\left(g^{-1}\right)^{ \prime\prime}\!\left(x\right)}{\left(g^{-1}\right)^{\prime}\!\left(x\right)}(1 +x)^{2}+(1+x)\leq 1+x,\quad\forall x\in[0,\infty)$ $$

and consequently

$$ $\displaystyle-\frac{(1+x)^{2}}{x}\leq\frac{\phi^{\prime\prime\prime}\!\left(x \right)}{\phi^{\prime\prime}\!\left(x\right)}(1+x)^{2}-\frac{\left(g^{-1} \right)^{\prime\prime}\!\left(x\right)}{\left(g^{-1}\right)^{\prime}\!\left(x \right)}(1+x)^{2}\leq 0,\quad\forall x\in[0,\infty).$ $$

∎

###### Proof of Corollary 1 .

### Appendix C Details on Empirical Evaluations

The computation of the results for the domain adaptation benchmark experiment is based on gradient-based training of overall 9174 models which we built following Gruber et al. () which uses the code-base of Dinu et al. ().
We obtain the number of trained models as follows.

AmazonReviews (text data):
$11$ methods $\times\ 14$ parameters $\times\ 12$ domain adaptation tasks $\times\ 3$ seeds $=5544$ trained models

MiniDomainNet (image data):
$11$ methods $\times\ 8$ parameters $\times\ 5$ domain adaptation tasks $\times\ 3$ seeds $=1320$ trained models

HHAR (sensory data):
$11$ methods $\times\ 14$ parameters $\times\ 5$ domain adaptation tasks $\times\ 3$ seeds $=2310$ trained models

Following Dinu et al. () we used $11$ domain adaptation methods from the AdaTime (Ragab et al., ) benchmark.
For each of these methods and domain adaptation modalities (text, image, sensory) we evaluate 4 density ratio estimators (KuLSIF, LR, Boost and EW).
For our experiments we follow the model implementations and experimental setup of Dinu et al. () which results in the usage of fully connected networks for Amazon Reviews and a pretrained ResNet-18 backbone for MiniDomainNet.

For training and selecting the density ratio estimation methods within this pipeline we follow Gruber et al. () perform an additional train/val split of $80/20$ on the datasets that are used for training the domain adaption methods.
The regularization parameter $\lambda$ is selected from $\{10,10^{-1},10^{-3}\}$ and the number of iterations of the BFGS algorithm is fixed with 100. We follow Kanamori et al. () in using the Gaussian kernel with kernel width set according to the median heuristic for all compared density ratio estimation methods.
Each experiment is run $3$ times.
To test the performance of the compared methods the classification accuracy on the respective test sets of the target distribution is evaluated.
The rcond parameter for inverting the Gram matrix in (Dinu et al., ), we used a value of $10^{-3}$.

#### C.1 Amazon Reviews

**Table 2: AdvSKM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.784($\pm$.004) | 0.773($\pm$.008) | 0.773($\pm$.008) | 0.772($\pm$.008) | 0.771($\pm$.002) | 0.772($\pm$.011) | 0.782($\pm$.003) | 0.779($\pm$.005) | 0.786($\pm$.007) | 0.790($\pm$.018) |
| B$\rightarrow$E | 0.754($\pm$.009) | 0.751($\pm$.008) | 0.752($\pm$.007) | 0.754($\pm$.011) | 0.752($\pm$.018) | 0.758($\pm$.012) | 0.755($\pm$.007) | 0.749($\pm$.012) | 0.762($\pm$.012) | 0.756($\pm$.023) |
| B$\rightarrow$K | 0.769($\pm$.021) | 0.768($\pm$.016) | 0.770($\pm$.011) | 0.770($\pm$.017) | 0.771($\pm$.012) | 0.766($\pm$.032) | 0.771($\pm$.016) | 0.774($\pm$.017) | 0.781($\pm$.014) | 0.774($\pm$.014) |
| D$\rightarrow$B | 0.779($\pm$.004) | 0.783($\pm$.004) | 0.779($\pm$.009) | 0.776($\pm$.004) | 0.782($\pm$.006) | 0.781($\pm$.010) | 0.789($\pm$.002) | 0.788($\pm$.007) | 0.792($\pm$.001) | 0.789($\pm$.007) |
| D$\rightarrow$E | 0.766($\pm$.013) | 0.772($\pm$.006) | 0.766($\pm$.011) | 0.766($\pm$.011) | 0.772($\pm$.004) | 0.773($\pm$.005) | 0.776($\pm$.008) | 0.772($\pm$.012) | 0.780($\pm$.006) | 0.776($\pm$.005) |
| D$\rightarrow$K | 0.777($\pm$.015) | 0.783($\pm$.009) | 0.780($\pm$.010) | 0.782($\pm$.008) | 0.789($\pm$.015) | 0.789($\pm$.007) | 0.788($\pm$.012) | 0.786($\pm$.009) | 0.792($\pm$.011) | 0.786($\pm$.009) |
| E$\rightarrow$B | 0.688($\pm$.015) | 0.694($\pm$.021) | 0.695($\pm$.019) | 0.694($\pm$.021) | 0.701($\pm$.009) | 0.696($\pm$.020) | 0.703($\pm$.020) | 0.691($\pm$.019) | 0.714($\pm$.020) | 0.707($\pm$.017) |
| E$\rightarrow$D | 0.714($\pm$.005) | 0.717($\pm$.004) | 0.717($\pm$.004) | 0.725($\pm$.017) | 0.735($\pm$.008) | 0.715($\pm$.021) | 0.726($\pm$.007) | 0.722($\pm$.013) | 0.732($\pm$.011) | 0.737($\pm$.011) |
| E$\rightarrow$K | 0.858($\pm$.004) | 0.859($\pm$.010) | 0.859($\pm$.010) | 0.863($\pm$.014) | 0.865($\pm$.007) | 0.861($\pm$.007) | 0.866($\pm$.013) | 0.869($\pm$.012) | 0.873($\pm$.012) | 0.865($\pm$.008) |
| K$\rightarrow$B | 0.713($\pm$.002) | 0.710($\pm$.009) | 0.710($\pm$.009) | 0.710($\pm$.009) | 0.708($\pm$.014) | 0.695($\pm$.009) | 0.722($\pm$.006) | 0.708($\pm$.014) | 0.722($\pm$.004) | 0.718($\pm$.007) |
| K$\rightarrow$D | 0.741($\pm$.010) | 0.728($\pm$.015) | 0.753($\pm$.020) | 0.742($\pm$.033) | 0.748($\pm$.021) | 0.732($\pm$.014) | 0.752($\pm$.006) | 0.741($\pm$.014) | 0.751($\pm$.016) | 0.756($\pm$.014) |
| K$\rightarrow$E | 0.848($\pm$.008) | 0.851($\pm$.005) | 0.848($\pm$.003) | 0.850($\pm$.007) | 0.846($\pm$.003) | 0.849($\pm$.012) | 0.849($\pm$.010) | 0.847($\pm$.011) | 0.856($\pm$.007) | 0.850($\pm$.002) |
| mean | 0.766($\pm$.009) | 0.766($\pm$.010) | 0.767($\pm$.010) | 0.767($\pm$.013) | 0.770($\pm$.010) | 0.766($\pm$.013) | 0.773($\pm$.009) | 0.769($\pm$.012) | 0.778($\pm$.010) | 0.775($\pm$.011) |

**Table 3: CDAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.784($\pm$.022) | 0.786($\pm$.007) | 0.793($\pm$.013) | 0.786($\pm$.007) | 0.782($\pm$.016) | 0.794($\pm$.015) | 0.793($\pm$.016) | 0.785($\pm$.007) | 0.797($\pm$.010) | 0.797($\pm$.001) |
| B$\rightarrow$E | 0.758($\pm$.013) | 0.770($\pm$.019) | 0.759($\pm$.009) | 0.758($\pm$.010) | 0.762($\pm$.007) | 0.772($\pm$.014) | 0.777($\pm$.008) | 0.766($\pm$.017) | 0.782($\pm$.009) | 0.775($\pm$.008) |
| B$\rightarrow$K | 0.769($\pm$.019) | 0.782($\pm$.007) | 0.782($\pm$.007) | 0.782($\pm$.007) | 0.783($\pm$.006) | 0.777($\pm$.042) | 0.787($\pm$.026) | 0.788($\pm$.027) | 0.792($\pm$.012) | 0.796($\pm$.011) |
| D$\rightarrow$B | 0.786($\pm$.010) | 0.785($\pm$.006) | 0.785($\pm$.006) | 0.785($\pm$.006) | 0.787($\pm$.007) | 0.780($\pm$.008) | 0.791($\pm$.010) | 0.787($\pm$.011) | 0.791($\pm$.005) | 0.794($\pm$.001) |
| D$\rightarrow$E | 0.761($\pm$.003) | 0.779($\pm$.015) | 0.775($\pm$.007) | 0.786($\pm$.023) | 0.773($\pm$.004) | 0.788($\pm$.012) | 0.797($\pm$.008) | 0.791($\pm$.010) | 0.803($\pm$.007) | 0.794($\pm$.008) |
| D$\rightarrow$K | 0.778($\pm$.018) | 0.785($\pm$.013) | 0.785($\pm$.013) | 0.789($\pm$.007) | 0.794($\pm$.015) | 0.805($\pm$.012) | 0.801($\pm$.016) | 0.799($\pm$.013) | 0.807($\pm$.014) | 0.801($\pm$.010) |
| E$\rightarrow$B | 0.692($\pm$.014) | 0.695($\pm$.029) | 0.702($\pm$.022) | 0.690($\pm$.024) | 0.693($\pm$.013) | 0.706($\pm$.020) | 0.705($\pm$.023) | 0.698($\pm$.022) | 0.719($\pm$.031) | 0.711($\pm$.029) |
| E$\rightarrow$D | 0.720($\pm$.018) | 0.723($\pm$.015) | 0.728($\pm$.012) | 0.728($\pm$.012) | 0.718($\pm$.006) | 0.683($\pm$.061) | 0.726($\pm$.019) | 0.705($\pm$.045) | 0.741($\pm$.003) | 0.735($\pm$.013) |
| E$\rightarrow$K | 0.860($\pm$.006) | 0.857($\pm$.013) | 0.857($\pm$.013) | 0.854($\pm$.010) | 0.860($\pm$.008) | 0.859($\pm$.025) | 0.868($\pm$.014) | 0.861($\pm$.020) | 0.875($\pm$.012) | 0.865($\pm$.011) |
| K$\rightarrow$B | 0.706($\pm$.004) | 0.721($\pm$.005) | 0.718($\pm$.009) | 0.721($\pm$.005) | 0.719($\pm$.013) | 0.697($\pm$.050) | 0.728($\pm$.009) | 0.719($\pm$.020) | 0.733($\pm$.013) | 0.724($\pm$.013) |
| K$\rightarrow$D | 0.748($\pm$.013) | 0.755($\pm$.018) | 0.755($\pm$.018) | 0.746($\pm$.013) | 0.752($\pm$.008) | 0.735($\pm$.022) | 0.757($\pm$.018) | 0.757($\pm$.016) | 0.772($\pm$.012) | 0.752($\pm$.018) |
| K$\rightarrow$E | 0.845($\pm$.007) | 0.848($\pm$.002) | 0.843($\pm$.006) | 0.843($\pm$.006) | 0.849($\pm$.007) | 0.825($\pm$.017) | 0.853($\pm$.006) | 0.842($\pm$.023) | 0.861($\pm$.005) | 0.850($\pm$.001) |
| mean | 0.767($\pm$.012) | 0.774($\pm$.012) | 0.773($\pm$.011) | 0.772($\pm$.011) | 0.773($\pm$.009) | 0.768($\pm$.025) | 0.782($\pm$.014) | 0.775($\pm$.019) | 0.789($\pm$.011) | 0.783($\pm$.010) |

**Table 4: CMD**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.772($\pm$.009) | 0.789($\pm$.025) | 0.789($\pm$.025) | 0.789($\pm$.025) | 0.772($\pm$.012) | 0.791($\pm$.017) | 0.797($\pm$.013) | 0.784($\pm$.035) | 0.794($\pm$.008) | 0.789($\pm$.018) |
| B$\rightarrow$E | 0.745($\pm$.012) | 0.761($\pm$.025) | 0.759($\pm$.021) | 0.761($\pm$.025) | 0.770($\pm$.005) | 0.739($\pm$.028) | 0.759($\pm$.020) | 0.756($\pm$.018) | 0.781($\pm$.011) | 0.780($\pm$.009) |
| B$\rightarrow$K | 0.763($\pm$.015) | 0.754($\pm$.048) | 0.778($\pm$.012) | 0.758($\pm$.052) | 0.787($\pm$.013) | 0.743($\pm$.085) | 0.792($\pm$.011) | 0.790($\pm$.012) | 0.796($\pm$.012) | 0.793($\pm$.009) |
| D$\rightarrow$B | 0.788($\pm$.010) | 0.784($\pm$.007) | 0.793($\pm$.007) | 0.783($\pm$.007) | 0.793($\pm$.007) | 0.715($\pm$.115) | 0.788($\pm$.002) | 0.786($\pm$.010) | 0.807($\pm$.007) | 0.794($\pm$.006) |
| D$\rightarrow$E | 0.768($\pm$.003) | 0.776($\pm$.016) | 0.774($\pm$.016) | 0.783($\pm$.008) | 0.784($\pm$.022) | 0.794($\pm$.016) | 0.792($\pm$.009) | 0.795($\pm$.013) | 0.806($\pm$.011) | 0.798($\pm$.016) |
| D$\rightarrow$K | 0.777($\pm$.012) | 0.787($\pm$.011) | 0.777($\pm$.012) | 0.797($\pm$.027) | 0.802($\pm$.019) | 0.757($\pm$.049) | 0.792($\pm$.014) | 0.799($\pm$.016) | 0.804($\pm$.008) | 0.811($\pm$.007) |
| E$\rightarrow$B | 0.699($\pm$.014) | 0.700($\pm$.010) | 0.705($\pm$.009) | 0.698($\pm$.013) | 0.713($\pm$.014) | 0.685($\pm$.014) | 0.727($\pm$.014) | 0.691($\pm$.013) | 0.726($\pm$.015) | 0.712($\pm$.011) |
| E$\rightarrow$D | 0.722($\pm$.008) | 0.742($\pm$.010) | 0.733($\pm$.020) | 0.742($\pm$.010) | 0.742($\pm$.010) | 0.713($\pm$.021) | 0.735($\pm$.009) | 0.724($\pm$.015) | 0.742($\pm$.004) | 0.738($\pm$.007) |
| E$\rightarrow$K | 0.860($\pm$.012) | 0.872($\pm$.013) | 0.859($\pm$.005) | 0.867($\pm$.012) | 0.860($\pm$.008) | 0.857($\pm$.012) | 0.867($\pm$.020) | 0.872($\pm$.016) | 0.875($\pm$.012) | 0.871($\pm$.014) |
| K$\rightarrow$B | 0.718($\pm$.005) | 0.725($\pm$.011) | 0.733($\pm$.004) | 0.734($\pm$.006) | 0.713($\pm$.005) | 0.646($\pm$.068) | 0.727($\pm$.018) | 0.698($\pm$.042) | 0.734($\pm$.011) | 0.740($\pm$.007) |
| K$\rightarrow$D | 0.748($\pm$.003) | 0.739($\pm$.010) | 0.739($\pm$.010) | 0.727($\pm$.026) | 0.755($\pm$.015) | 0.694($\pm$.081) | 0.753($\pm$.016) | 0.723($\pm$.061) | 0.762($\pm$.023) | 0.761($\pm$.026) |
| K$\rightarrow$E | 0.842($\pm$.005) | 0.847($\pm$.016) | 0.847($\pm$.016) | 0.844($\pm$.015) | 0.855($\pm$.007) | 0.812($\pm$.051) | 0.854($\pm$.004) | 0.851($\pm$.004) | 0.867($\pm$.004) | 0.858($\pm$.005) |
| mean | 0.767($\pm$.009) | 0.773($\pm$.017) | 0.774($\pm$.013) | 0.774($\pm$.019) | 0.779($\pm$.011) | 0.745($\pm$.046) | 0.782($\pm$.013) | 0.772($\pm$.021) | 0.791($\pm$.010) | 0.787($\pm$.011) |

**Table 5: CoDATS**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.783($\pm$.013) | 0.793($\pm$.014) | 0.793($\pm$.014) | 0.793($\pm$.014) | 0.788($\pm$.008) | 0.794($\pm$.013) | 0.799($\pm$.001) | 0.803($\pm$.006) | 0.806($\pm$.006) | 0.801($\pm$.014) |
| B$\rightarrow$E | 0.755($\pm$.005) | 0.766($\pm$.009) | 0.766($\pm$.009) | 0.780($\pm$.018) | 0.764($\pm$.019) | 0.763($\pm$.024) | 0.774($\pm$.020) | 0.773($\pm$.020) | 0.785($\pm$.007) | 0.802($\pm$.009) |
| B$\rightarrow$K | 0.771($\pm$.022) | 0.817($\pm$.008) | 0.794($\pm$.015) | 0.803($\pm$.008) | 0.798($\pm$.011) | 0.793($\pm$.021) | 0.811($\pm$.001) | 0.807($\pm$.008) | 0.807($\pm$.005) | 0.817($\pm$.007) |
| D$\rightarrow$B | 0.774($\pm$.001) | 0.777($\pm$.006) | 0.777($\pm$.006) | 0.781($\pm$.011) | 0.791($\pm$.011) | 0.785($\pm$.021) | 0.799($\pm$.008) | 0.795($\pm$.014) | 0.806($\pm$.009) | 0.800($\pm$.011) |
| D$\rightarrow$E | 0.769($\pm$.003) | 0.806($\pm$.022) | 0.788($\pm$.015) | 0.806($\pm$.022) | 0.811($\pm$.018) | 0.785($\pm$.019) | 0.788($\pm$.010) | 0.784($\pm$.009) | 0.807($\pm$.010) | 0.817($\pm$.011) |
| D$\rightarrow$K | 0.782($\pm$.021) | 0.806($\pm$.041) | 0.806($\pm$.041) | 0.806($\pm$.041) | 0.823($\pm$.019) | 0.795($\pm$.020) | 0.808($\pm$.018) | 0.811($\pm$.016) | 0.824($\pm$.012) | 0.828($\pm$.010) |
| E$\rightarrow$B | 0.687($\pm$.013) | 0.722($\pm$.045) | 0.732($\pm$.022) | 0.722($\pm$.045) | 0.718($\pm$.023) | 0.713($\pm$.049) | 0.725($\pm$.028) | 0.720($\pm$.045) | 0.721($\pm$.028) | 0.723($\pm$.033) |
| E$\rightarrow$D | 0.720($\pm$.013) | 0.721($\pm$.020) | 0.725($\pm$.026) | 0.725($\pm$.026) | 0.725($\pm$.026) | 0.732($\pm$.010) | 0.750($\pm$.027) | 0.741($\pm$.027) | 0.751($\pm$.020) | 0.736($\pm$.012) |
| E$\rightarrow$K | 0.859($\pm$.016) | 0.862($\pm$.014) | 0.867($\pm$.002) | 0.860($\pm$.011) | 0.865($\pm$.010) | 0.837($\pm$.035) | 0.868($\pm$.011) | 0.864($\pm$.022) | 0.879($\pm$.008) | 0.870($\pm$.012) |
| K$\rightarrow$B | 0.712($\pm$.012) | 0.731($\pm$.022) | 0.732($\pm$.042) | 0.703($\pm$.016) | 0.722($\pm$.043) | 0.735($\pm$.005) | 0.739($\pm$.005) | 0.721($\pm$.023) | 0.745($\pm$.005) | 0.751($\pm$.016) |
| K$\rightarrow$D | 0.730($\pm$.022) | 0.774($\pm$.005) | 0.774($\pm$.005) | 0.748($\pm$.024) | 0.749($\pm$.017) | 0.741($\pm$.015) | 0.754($\pm$.010) | 0.736($\pm$.021) | 0.762($\pm$.008) | 0.764($\pm$.010) |
| K$\rightarrow$E | 0.845($\pm$.000) | 0.849($\pm$.009) | 0.847($\pm$.009) | 0.846($\pm$.008) | 0.853($\pm$.002) | 0.855($\pm$.004) | 0.862($\pm$.004) | 0.858($\pm$.009) | 0.868($\pm$.008) | 0.856($\pm$.003) |
| mean | 0.766($\pm$.012) | 0.785($\pm$.018) | 0.783($\pm$.017) | 0.781($\pm$.020) | 0.784($\pm$.017) | 0.777($\pm$.020) | 0.790($\pm$.012) | 0.784($\pm$.018) | 0.797($\pm$.010) | 0.797($\pm$.012) |

**Table 6: DANN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.783($\pm$.004) | 0.789($\pm$.020) | 0.798($\pm$.009) | 0.789($\pm$.020) | 0.799($\pm$.008) | 0.787($\pm$.015) | 0.795($\pm$.004) | 0.801($\pm$.009) | 0.801($\pm$.007) | 0.813($\pm$.017) |
| B$\rightarrow$E | 0.752($\pm$.011) | 0.770($\pm$.018) | 0.770($\pm$.018) | 0.772($\pm$.016) | 0.800($\pm$.005) | 0.739($\pm$.044) | 0.769($\pm$.010) | 0.747($\pm$.022) | 0.790($\pm$.004) | 0.800($\pm$.005) |
| B$\rightarrow$K | 0.767($\pm$.022) | 0.791($\pm$.020) | 0.784($\pm$.013) | 0.795($\pm$.001) | 0.784($\pm$.029) | 0.779($\pm$.029) | 0.798($\pm$.013) | 0.785($\pm$.018) | 0.804($\pm$.007) | 0.820($\pm$.004) |
| D$\rightarrow$B | 0.779($\pm$.002) | 0.786($\pm$.012) | 0.784($\pm$.015) | 0.787($\pm$.007) | 0.798($\pm$.001) | 0.785($\pm$.010) | 0.787($\pm$.006) | 0.772($\pm$.028) | 0.806($\pm$.006) | 0.799($\pm$.001) |
| D$\rightarrow$E | 0.767($\pm$.016) | 0.802($\pm$.009) | 0.794($\pm$.009) | 0.794($\pm$.009) | 0.789($\pm$.006) | 0.800($\pm$.022) | 0.806($\pm$.007) | 0.795($\pm$.017) | 0.814($\pm$.006) | 0.815($\pm$.007) |
| D$\rightarrow$K | 0.784($\pm$.013) | 0.805($\pm$.007) | 0.800($\pm$.002) | 0.793($\pm$.024) | 0.807($\pm$.007) | 0.774($\pm$.011) | 0.806($\pm$.007) | 0.780($\pm$.028) | 0.823($\pm$.007) | 0.827($\pm$.005) |
| E$\rightarrow$B | 0.701($\pm$.019) | 0.715($\pm$.035) | 0.714($\pm$.035) | 0.714($\pm$.035) | 0.719($\pm$.030) | 0.719($\pm$.018) | 0.720($\pm$.025) | 0.713($\pm$.036) | 0.723($\pm$.014) | 0.721($\pm$.010) |
| E$\rightarrow$D | 0.736($\pm$.005) | 0.727($\pm$.003) | 0.723($\pm$.018) | 0.728($\pm$.002) | 0.736($\pm$.008) | 0.718($\pm$.025) | 0.747($\pm$.014) | 0.741($\pm$.012) | 0.749($\pm$.003) | 0.757($\pm$.032) |
| E$\rightarrow$K | 0.854($\pm$.016) | 0.855($\pm$.019) | 0.862($\pm$.009) | 0.853($\pm$.016) | 0.866($\pm$.013) | 0.856($\pm$.012) | 0.868($\pm$.012) | 0.867($\pm$.018) | 0.878($\pm$.014) | 0.875($\pm$.008) |
| K$\rightarrow$B | 0.711($\pm$.014) | 0.742($\pm$.017) | 0.742($\pm$.017) | 0.721($\pm$.004) | 0.729($\pm$.004) | 0.720($\pm$.030) | 0.751($\pm$.011) | 0.748($\pm$.005) | 0.752($\pm$.007) | 0.747($\pm$.021) |
| K$\rightarrow$D | 0.738($\pm$.006) | 0.729($\pm$.018) | 0.747($\pm$.049) | 0.735($\pm$.013) | 0.751($\pm$.018) | 0.741($\pm$.050) | 0.772($\pm$.021) | 0.754($\pm$.044) | 0.786($\pm$.013) | 0.762($\pm$.021) |
| K$\rightarrow$E | 0.837($\pm$.013) | 0.853($\pm$.004) | 0.853($\pm$.004) | 0.850($\pm$.007) | 0.849($\pm$.005) | 0.848($\pm$.016) | 0.854($\pm$.003) | 0.849($\pm$.005) | 0.866($\pm$.007) | 0.859($\pm$.013) |
| mean | 0.767($\pm$.012) | 0.780($\pm$.015) | 0.781($\pm$.017) | 0.777($\pm$.013) | 0.785($\pm$.011) | 0.772($\pm$.024) | 0.789($\pm$.011) | 0.779($\pm$.020) | 0.799($\pm$.008) | 0.800($\pm$.012) |

**Table 7: DDC**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.781($\pm$.016) | 0.776($\pm$.008) | 0.780($\pm$.009) | 0.779($\pm$.009) | 0.782($\pm$.005) | 0.775($\pm$.015) | 0.783($\pm$.004) | 0.785($\pm$.009) | 0.790($\pm$.005) | 0.786($\pm$.004) |
| B$\rightarrow$E | 0.752($\pm$.007) | 0.754($\pm$.008) | 0.754($\pm$.007) | 0.754($\pm$.008) | 0.753($\pm$.012) | 0.741($\pm$.013) | 0.753($\pm$.010) | 0.746($\pm$.015) | 0.766($\pm$.013) | 0.756($\pm$.012) |
| B$\rightarrow$K | 0.766($\pm$.015) | 0.773($\pm$.019) | 0.773($\pm$.019) | 0.773($\pm$.019) | 0.774($\pm$.018) | 0.767($\pm$.026) | 0.777($\pm$.026) | 0.772($\pm$.032) | 0.782($\pm$.017) | 0.780($\pm$.014) |
| D$\rightarrow$B | 0.781($\pm$.008) | 0.783($\pm$.007) | 0.786($\pm$.011) | 0.783($\pm$.007) | 0.779($\pm$.009) | 0.776($\pm$.023) | 0.782($\pm$.016) | 0.779($\pm$.016) | 0.790($\pm$.011) | 0.788($\pm$.009) |
| D$\rightarrow$E | 0.767($\pm$.015) | 0.772($\pm$.006) | 0.774($\pm$.011) | 0.772($\pm$.006) | 0.776($\pm$.004) | 0.771($\pm$.014) | 0.767($\pm$.009) | 0.770($\pm$.018) | 0.784($\pm$.002) | 0.778($\pm$.010) |
| D$\rightarrow$K | 0.782($\pm$.013) | 0.784($\pm$.010) | 0.782($\pm$.013) | 0.782($\pm$.013) | 0.783($\pm$.014) | 0.771($\pm$.031) | 0.781($\pm$.018) | 0.781($\pm$.013) | 0.793($\pm$.017) | 0.788($\pm$.008) |
| E$\rightarrow$B | 0.693($\pm$.019) | 0.685($\pm$.014) | 0.685($\pm$.014) | 0.685($\pm$.014) | 0.691($\pm$.019) | 0.708($\pm$.018) | 0.705($\pm$.027) | 0.702($\pm$.011) | 0.702($\pm$.023) | 0.704($\pm$.018) |
| E$\rightarrow$D | 0.725($\pm$.006) | 0.737($\pm$.028) | 0.727($\pm$.017) | 0.740($\pm$.024) | 0.736($\pm$.026) | 0.737($\pm$.024) | 0.742($\pm$.007) | 0.739($\pm$.009) | 0.737($\pm$.011) | 0.736($\pm$.028) |
| E$\rightarrow$K | 0.857($\pm$.011) | 0.860($\pm$.001) | 0.860($\pm$.002) | 0.860($\pm$.002) | 0.860($\pm$.008) | 0.841($\pm$.034) | 0.857($\pm$.016) | 0.854($\pm$.010) | 0.873($\pm$.009) | 0.864($\pm$.009) |
| K$\rightarrow$B | 0.713($\pm$.002) | 0.716($\pm$.004) | 0.716($\pm$.004) | 0.713($\pm$.005) | 0.707($\pm$.006) | 0.706($\pm$.023) | 0.719($\pm$.011) | 0.709($\pm$.019) | 0.722($\pm$.006) | 0.719($\pm$.005) |
| K$\rightarrow$D | 0.741($\pm$.018) | 0.745($\pm$.025) | 0.754($\pm$.016) | 0.751($\pm$.016) | 0.739($\pm$.016) | 0.737($\pm$.019) | 0.747($\pm$.018) | 0.744($\pm$.014) | 0.751($\pm$.011) | 0.748($\pm$.019) |
| K$\rightarrow$E | 0.841($\pm$.001) | 0.849($\pm$.010) | 0.844($\pm$.004) | 0.849($\pm$.010) | 0.846($\pm$.005) | 0.845($\pm$.018) | 0.854($\pm$.005) | 0.843($\pm$.014) | 0.857($\pm$.006) | 0.849($\pm$.011) |
| mean | 0.766($\pm$.011) | 0.769($\pm$.012) | 0.770($\pm$.010) | 0.770($\pm$.011) | 0.769($\pm$.012) | 0.765($\pm$.022) | 0.772($\pm$.014) | 0.769($\pm$.015) | 0.779($\pm$.011) | 0.775($\pm$.012) |

**Table 8: DCoral**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.778($\pm$.018) | 0.790($\pm$.011) | 0.790($\pm$.011) | 0.789($\pm$.014) | 0.781($\pm$.009) | 0.786($\pm$.008) | 0.796($\pm$.005) | 0.798($\pm$.005) | 0.799($\pm$.009) | 0.794($\pm$.020) |
| B$\rightarrow$E | 0.749($\pm$.013) | 0.755($\pm$.015) | 0.755($\pm$.015) | 0.755($\pm$.015) | 0.761($\pm$.011) | 0.745($\pm$.008) | 0.754($\pm$.013) | 0.750($\pm$.021) | 0.762($\pm$.014) | 0.767($\pm$.006) |
| B$\rightarrow$K | 0.769($\pm$.011) | 0.775($\pm$.015) | 0.776($\pm$.012) | 0.774($\pm$.012) | 0.773($\pm$.021) | 0.768($\pm$.028) | 0.783($\pm$.018) | 0.771($\pm$.030) | 0.782($\pm$.015) | 0.779($\pm$.016) |
| D$\rightarrow$B | 0.783($\pm$.009) | 0.780($\pm$.003) | 0.777($\pm$.002) | 0.780($\pm$.003) | 0.780($\pm$.012) | 0.775($\pm$.025) | 0.791($\pm$.002) | 0.782($\pm$.009) | 0.794($\pm$.003) | 0.786($\pm$.004) |
| D$\rightarrow$E | 0.760($\pm$.013) | 0.762($\pm$.014) | 0.771($\pm$.011) | 0.762($\pm$.014) | 0.775($\pm$.008) | 0.767($\pm$.021) | 0.771($\pm$.015) | 0.768($\pm$.015) | 0.784($\pm$.003) | 0.785($\pm$.005) |
| D$\rightarrow$K | 0.780($\pm$.016) | 0.780($\pm$.016) | 0.779($\pm$.015) | 0.780($\pm$.016) | 0.789($\pm$.017) | 0.761($\pm$.054) | 0.782($\pm$.027) | 0.774($\pm$.023) | 0.797($\pm$.012) | 0.789($\pm$.017) |
| E$\rightarrow$B | 0.696($\pm$.009) | 0.695($\pm$.008) | 0.702($\pm$.020) | 0.696($\pm$.009) | 0.702($\pm$.020) | 0.710($\pm$.017) | 0.711($\pm$.021) | 0.700($\pm$.013) | 0.709($\pm$.021) | 0.708($\pm$.009) |
| E$\rightarrow$D | 0.722($\pm$.009) | 0.731($\pm$.008) | 0.726($\pm$.015) | 0.734($\pm$.005) | 0.729($\pm$.006) | 0.735($\pm$.008) | 0.745($\pm$.004) | 0.744($\pm$.005) | 0.740($\pm$.003) | 0.739($\pm$.006) |
| E$\rightarrow$K | 0.859($\pm$.010) | 0.859($\pm$.010) | 0.859($\pm$.010) | 0.859($\pm$.010) | 0.861($\pm$.009) | 0.841($\pm$.044) | 0.870($\pm$.012) | 0.867($\pm$.015) | 0.874($\pm$.008) | 0.864($\pm$.005) |
| K$\rightarrow$B | 0.720($\pm$.010) | 0.721($\pm$.008) | 0.721($\pm$.008) | 0.721($\pm$.008) | 0.717($\pm$.001) | 0.729($\pm$.006) | 0.731($\pm$.011) | 0.725($\pm$.014) | 0.737($\pm$.006) | 0.740($\pm$.005) |
| K$\rightarrow$D | 0.733($\pm$.011) | 0.734($\pm$.032) | 0.743($\pm$.018) | 0.735($\pm$.020) | 0.738($\pm$.023) | 0.755($\pm$.005) | 0.750($\pm$.018) | 0.751($\pm$.012) | 0.760($\pm$.010) | 0.751($\pm$.008) |
| K$\rightarrow$E | 0.841($\pm$.010) | 0.844($\pm$.006) | 0.844($\pm$.006) | 0.844($\pm$.006) | 0.846($\pm$.006) | 0.841($\pm$.020) | 0.854($\pm$.002) | 0.848($\pm$.007) | 0.859($\pm$.008) | 0.849($\pm$.009) |
| mean | 0.766($\pm$.012) | 0.769($\pm$.012) | 0.770($\pm$.012) | 0.769($\pm$.011) | 0.771($\pm$.012) | 0.768($\pm$.020) | 0.778($\pm$.012) | 0.773($\pm$.014) | 0.783($\pm$.010) | 0.779($\pm$.009) |

**Table 9: DIRT**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.777($\pm$.010) | 0.796($\pm$.014) | 0.796($\pm$.014) | 0.796($\pm$.014) | 0.804($\pm$.009) | 0.816($\pm$.003) | 0.819($\pm$.001) | 0.819($\pm$.007) | 0.805($\pm$.008) | 0.809($\pm$.010) |
| B$\rightarrow$E | 0.756($\pm$.013) | 0.779($\pm$.022) | 0.765($\pm$.010) | 0.748($\pm$.014) | 0.753($\pm$.010) | 0.770($\pm$.007) | 0.777($\pm$.004) | 0.757($\pm$.018) | 0.777($\pm$.012) | 0.804($\pm$.005) |
| B$\rightarrow$K | 0.770($\pm$.013) | 0.772($\pm$.031) | 0.796($\pm$.020) | 0.789($\pm$.012) | 0.796($\pm$.020) | 0.745($\pm$.049) | 0.798($\pm$.005) | 0.784($\pm$.017) | 0.802($\pm$.009) | 0.814($\pm$.006) |
| D$\rightarrow$B | 0.773($\pm$.008) | 0.799($\pm$.013) | 0.810($\pm$.017) | 0.813($\pm$.015) | 0.811($\pm$.016) | 0.767($\pm$.042) | 0.796($\pm$.019) | 0.780($\pm$.020) | 0.808($\pm$.006) | 0.811($\pm$.017) |
| D$\rightarrow$E | 0.757($\pm$.020) | 0.768($\pm$.032) | 0.768($\pm$.032) | 0.768($\pm$.032) | 0.778($\pm$.024) | 0.776($\pm$.038) | 0.782($\pm$.030) | 0.778($\pm$.028) | 0.801($\pm$.008) | 0.812($\pm$.006) |
| D$\rightarrow$K | 0.775($\pm$.016) | 0.820($\pm$.015) | 0.820($\pm$.015) | 0.820($\pm$.015) | 0.814($\pm$.021) | 0.822($\pm$.008) | 0.826($\pm$.002) | 0.814($\pm$.007) | 0.828($\pm$.010) | 0.828($\pm$.010) |
| E$\rightarrow$B | 0.702($\pm$.018) | 0.689($\pm$.025) | 0.700($\pm$.011) | 0.689($\pm$.025) | 0.697($\pm$.011) | 0.691($\pm$.014) | 0.698($\pm$.017) | 0.687($\pm$.013) | 0.708($\pm$.021) | 0.708($\pm$.042) |
| E$\rightarrow$D | 0.716($\pm$.008) | 0.715($\pm$.014) | 0.715($\pm$.014) | 0.715($\pm$.014) | 0.718($\pm$.010) | 0.704($\pm$.010) | 0.721($\pm$.010) | 0.698($\pm$.010) | 0.722($\pm$.016) | 0.730($\pm$.001) |
| E$\rightarrow$K | 0.855($\pm$.013) | 0.871($\pm$.020) | 0.872($\pm$.020) | 0.871($\pm$.020) | 0.879($\pm$.019) | 0.848($\pm$.037) | 0.868($\pm$.018) | 0.859($\pm$.031) | 0.889($\pm$.005) | 0.889($\pm$.004) |
| K$\rightarrow$B | 0.709($\pm$.012) | 0.689($\pm$.037) | 0.719($\pm$.016) | 0.719($\pm$.016) | 0.735($\pm$.020) | 0.713($\pm$.018) | 0.735($\pm$.009) | 0.726($\pm$.027) | 0.745($\pm$.009) | 0.723($\pm$.007) |
| K$\rightarrow$D | 0.736($\pm$.012) | 0.750($\pm$.011) | 0.749($\pm$.009) | 0.741($\pm$.005) | 0.748($\pm$.022) | 0.704($\pm$.028) | 0.745($\pm$.005) | 0.728($\pm$.009) | 0.765($\pm$.000) | 0.756($\pm$.007) |
| K$\rightarrow$E | 0.842($\pm$.009) | 0.857($\pm$.012) | 0.855($\pm$.015) | 0.853($\pm$.019) | 0.861($\pm$.019) | 0.854($\pm$.008) | 0.865($\pm$.000) | 0.857($\pm$.003) | 0.873($\pm$.005) | 0.863($\pm$.013) |
| mean | 0.764($\pm$.013) | 0.775($\pm$.020) | 0.780($\pm$.016) | 0.777($\pm$.017) | 0.783($\pm$.017) | 0.767($\pm$.022) | 0.786($\pm$.010) | 0.774($\pm$.016) | 0.794($\pm$.009) | 0.795($\pm$.011) |

**Table 10: DSAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.779($\pm$.013) | 0.791($\pm$.007) | 0.794($\pm$.008) | 0.791($\pm$.008) | 0.798($\pm$.013) | 0.785($\pm$.017) | 0.795($\pm$.008) | 0.794($\pm$.008) | 0.805($\pm$.004) | 0.805($\pm$.006) |
| B$\rightarrow$E | 0.752($\pm$.005) | 0.766($\pm$.027) | 0.761($\pm$.019) | 0.766($\pm$.027) | 0.785($\pm$.002) | 0.780($\pm$.012) | 0.782($\pm$.023) | 0.780($\pm$.027) | 0.797($\pm$.005) | 0.796($\pm$.012) |
| B$\rightarrow$K | 0.768($\pm$.012) | 0.778($\pm$.033) | 0.794($\pm$.006) | 0.786($\pm$.031) | 0.783($\pm$.027) | 0.795($\pm$.028) | 0.806($\pm$.013) | 0.798($\pm$.036) | 0.811($\pm$.010) | 0.816($\pm$.005) |
| D$\rightarrow$B | 0.782($\pm$.008) | 0.784($\pm$.005) | 0.789($\pm$.009) | 0.784($\pm$.005) | 0.790($\pm$.006) | 0.782($\pm$.027) | 0.795($\pm$.007) | 0.793($\pm$.009) | 0.798($\pm$.008) | 0.802($\pm$.007) |
| D$\rightarrow$E | 0.771($\pm$.002) | 0.788($\pm$.018) | 0.798($\pm$.014) | 0.802($\pm$.009) | 0.804($\pm$.009) | 0.796($\pm$.011) | 0.795($\pm$.009) | 0.796($\pm$.005) | 0.808($\pm$.003) | 0.813($\pm$.002) |
| D$\rightarrow$K | 0.786($\pm$.013) | 0.800($\pm$.002) | 0.793($\pm$.012) | 0.783($\pm$.017) | 0.808($\pm$.015) | 0.657($\pm$.273) | 0.790($\pm$.038) | 0.788($\pm$.033) | 0.820($\pm$.014) | 0.830($\pm$.016) |
| E$\rightarrow$B | 0.702($\pm$.021) | 0.710($\pm$.010) | 0.704($\pm$.019) | 0.706($\pm$.007) | 0.706($\pm$.008) | 0.713($\pm$.022) | 0.724($\pm$.018) | 0.717($\pm$.024) | 0.719($\pm$.015) | 0.711($\pm$.010) |
| E$\rightarrow$D | 0.725($\pm$.007) | 0.736($\pm$.015) | 0.732($\pm$.019) | 0.725($\pm$.026) | 0.731($\pm$.019) | 0.743($\pm$.018) | 0.744($\pm$.014) | 0.744($\pm$.014) | 0.744($\pm$.011) | 0.732($\pm$.019) |
| E$\rightarrow$K | 0.865($\pm$.013) | 0.860($\pm$.005) | 0.857($\pm$.011) | 0.857($\pm$.011) | 0.876($\pm$.006) | 0.875($\pm$.006) | 0.874($\pm$.008) | 0.871($\pm$.007) | 0.885($\pm$.009) | 0.876($\pm$.006) |
| K$\rightarrow$B | 0.716($\pm$.016) | 0.729($\pm$.020) | 0.712($\pm$.014) | 0.730($\pm$.020) | 0.720($\pm$.021) | 0.710($\pm$.000) | 0.729($\pm$.012) | 0.706($\pm$.033) | 0.748($\pm$.004) | 0.739($\pm$.022) |
| K$\rightarrow$D | 0.739($\pm$.023) | 0.738($\pm$.016) | 0.744($\pm$.017) | 0.744($\pm$.017) | 0.743($\pm$.026) | 0.741($\pm$.027) | 0.765($\pm$.019) | 0.744($\pm$.005) | 0.775($\pm$.013) | 0.763($\pm$.026) |
| K$\rightarrow$E | 0.840($\pm$.010) | 0.853($\pm$.007) | 0.854($\pm$.007) | 0.848($\pm$.011) | 0.855($\pm$.009) | 0.852($\pm$.005) | 0.858($\pm$.004) | 0.848($\pm$.005) | 0.869($\pm$.005) | 0.857($\pm$.004) |
| mean | 0.769($\pm$.012) | 0.778($\pm$.014) | 0.778($\pm$.013) | 0.777($\pm$.016) | 0.783($\pm$.013) | 0.769($\pm$.037) | 0.788($\pm$.014) | 0.782($\pm$.017) | 0.798($\pm$.008) | 0.795($\pm$.011) |

**Table 11: HoMM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.778($\pm$.010) | 0.778($\pm$.011) | 0.781($\pm$.015) | 0.780($\pm$.013) | 0.781($\pm$.012) | 0.779($\pm$.012) | 0.793($\pm$.013) | 0.786($\pm$.013) | 0.792($\pm$.009) | 0.789($\pm$.009) |
| B$\rightarrow$E | 0.745($\pm$.014) | 0.753($\pm$.018) | 0.749($\pm$.017) | 0.749($\pm$.017) | 0.735($\pm$.014) | 0.747($\pm$.010) | 0.750($\pm$.024) | 0.750($\pm$.022) | 0.761($\pm$.012) | 0.758($\pm$.017) |
| B$\rightarrow$K | 0.770($\pm$.018) | 0.766($\pm$.017) | 0.760($\pm$.012) | 0.769($\pm$.022) | 0.774($\pm$.014) | 0.767($\pm$.040) | 0.770($\pm$.023) | 0.767($\pm$.037) | 0.779($\pm$.014) | 0.776($\pm$.016) |
| D$\rightarrow$B | 0.785($\pm$.006) | 0.769($\pm$.009) | 0.778($\pm$.016) | 0.781($\pm$.011) | 0.771($\pm$.013) | 0.765($\pm$.025) | 0.780($\pm$.006) | 0.776($\pm$.012) | 0.789($\pm$.007) | 0.787($\pm$.007) |
| D$\rightarrow$E | 0.772($\pm$.012) | 0.767($\pm$.004) | 0.759($\pm$.011) | 0.768($\pm$.006) | 0.768($\pm$.005) | 0.764($\pm$.012) | 0.768($\pm$.010) | 0.763($\pm$.008) | 0.777($\pm$.001) | 0.774($\pm$.004) |
| D$\rightarrow$K | 0.783($\pm$.015) | 0.781($\pm$.005) | 0.788($\pm$.012) | 0.783($\pm$.007) | 0.784($\pm$.012) | 0.764($\pm$.051) | 0.780($\pm$.021) | 0.779($\pm$.020) | 0.787($\pm$.018) | 0.786($\pm$.013) |
| E$\rightarrow$B | 0.702($\pm$.026) | 0.687($\pm$.017) | 0.686($\pm$.017) | 0.686($\pm$.016) | 0.696($\pm$.020) | 0.697($\pm$.010) | 0.694($\pm$.020) | 0.694($\pm$.024) | 0.699($\pm$.015) | 0.703($\pm$.030) |
| E$\rightarrow$D | 0.733($\pm$.003) | 0.723($\pm$.013) | 0.732($\pm$.009) | 0.725($\pm$.009) | 0.735($\pm$.013) | 0.730($\pm$.005) | 0.734($\pm$.016) | 0.728($\pm$.006) | 0.734($\pm$.009) | 0.735($\pm$.015) |
| E$\rightarrow$K | 0.862($\pm$.009) | 0.855($\pm$.008) | 0.862($\pm$.009) | 0.856($\pm$.005) | 0.859($\pm$.005) | 0.844($\pm$.024) | 0.866($\pm$.007) | 0.861($\pm$.011) | 0.873($\pm$.006) | 0.863($\pm$.004) |
| K$\rightarrow$B | 0.715($\pm$.008) | 0.711($\pm$.003) | 0.712($\pm$.010) | 0.710($\pm$.005) | 0.708($\pm$.003) | 0.707($\pm$.022) | 0.715($\pm$.006) | 0.700($\pm$.016) | 0.723($\pm$.006) | 0.717($\pm$.005) |
| K$\rightarrow$D | 0.745($\pm$.011) | 0.745($\pm$.003) | 0.741($\pm$.005) | 0.741($\pm$.005) | 0.746($\pm$.006) | 0.738($\pm$.022) | 0.748($\pm$.009) | 0.738($\pm$.005) | 0.748($\pm$.002) | 0.745($\pm$.011) |
| K$\rightarrow$E | 0.844($\pm$.009) | 0.848($\pm$.008) | 0.849($\pm$.008) | 0.847($\pm$.011) | 0.846($\pm$.010) | 0.839($\pm$.016) | 0.852($\pm$.006) | 0.843($\pm$.009) | 0.855($\pm$.007) | 0.849($\pm$.007) |
| mean | 0.769($\pm$.012) | 0.765($\pm$.010) | 0.766($\pm$.012) | 0.766($\pm$.010) | 0.767($\pm$.011) | 0.762($\pm$.021) | 0.771($\pm$.013) | 0.765($\pm$.015) | 0.776($\pm$.009) | 0.774($\pm$.011) |

**Table 12: MMDA**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| B$\rightarrow$D | 0.775($\pm$.005) | 0.783($\pm$.016) | 0.788($\pm$.009) | 0.788($\pm$.009) | 0.789($\pm$.009) | 0.776($\pm$.014) | 0.786($\pm$.002) | 0.783($\pm$.008) | 0.800($\pm$.001) | 0.792($\pm$.013) |
| B$\rightarrow$E | 0.752($\pm$.006) | 0.768($\pm$.013) | 0.758($\pm$.008) | 0.754($\pm$.014) | 0.758($\pm$.011) | 0.754($\pm$.032) | 0.766($\pm$.024) | 0.758($\pm$.037) | 0.771($\pm$.019) | 0.781($\pm$.006) |
| B$\rightarrow$K | 0.765($\pm$.009) | 0.787($\pm$.012) | 0.791($\pm$.017) | 0.783($\pm$.010) | 0.784($\pm$.021) | 0.796($\pm$.011) | 0.793($\pm$.014) | 0.794($\pm$.019) | 0.797($\pm$.013) | 0.802($\pm$.019) |
| D$\rightarrow$B | 0.783($\pm$.007) | 0.783($\pm$.006) | 0.786($\pm$.010) | 0.779($\pm$.014) | 0.777($\pm$.009) | 0.769($\pm$.034) | 0.787($\pm$.004) | 0.784($\pm$.008) | 0.795($\pm$.003) | 0.790($\pm$.001) |
| D$\rightarrow$E | 0.764($\pm$.003) | 0.767($\pm$.017) | 0.769($\pm$.020) | 0.774($\pm$.009) | 0.774($\pm$.009) | 0.771($\pm$.018) | 0.781($\pm$.016) | 0.788($\pm$.008) | 0.789($\pm$.004) | 0.799($\pm$.006) |
| D$\rightarrow$K | 0.775($\pm$.012) | 0.777($\pm$.005) | 0.777($\pm$.005) | 0.777($\pm$.005) | 0.811($\pm$.007) | 0.785($\pm$.023) | 0.789($\pm$.012) | 0.787($\pm$.015) | 0.810($\pm$.006) | 0.800($\pm$.013) |
| E$\rightarrow$B | 0.701($\pm$.015) | 0.703($\pm$.013) | 0.698($\pm$.005) | 0.698($\pm$.005) | 0.699($\pm$.009) | 0.707($\pm$.008) | 0.711($\pm$.008) | 0.692($\pm$.018) | 0.719($\pm$.013) | 0.707($\pm$.013) |
| E$\rightarrow$D | 0.738($\pm$.010) | 0.709($\pm$.022) | 0.733($\pm$.011) | 0.732($\pm$.014) | 0.729($\pm$.008) | 0.732($\pm$.017) | 0.741($\pm$.019) | 0.737($\pm$.022) | 0.739($\pm$.003) | 0.738($\pm$.020) |
| E$\rightarrow$K | 0.855($\pm$.015) | 0.857($\pm$.011) | 0.857($\pm$.011) | 0.857($\pm$.011) | 0.858($\pm$.010) | 0.855($\pm$.009) | 0.869($\pm$.003) | 0.864($\pm$.008) | 0.878($\pm$.005) | 0.867($\pm$.004) |
| K$\rightarrow$B | 0.715($\pm$.008) | 0.710($\pm$.017) | 0.729($\pm$.032) | 0.710($\pm$.017) | 0.735($\pm$.023) | 0.726($\pm$.009) | 0.726($\pm$.020) | 0.717($\pm$.005) | 0.743($\pm$.010) | 0.730($\pm$.007) |
| K$\rightarrow$D | 0.736($\pm$.014) | 0.740($\pm$.031) | 0.740($\pm$.031) | 0.740($\pm$.031) | 0.747($\pm$.009) | 0.737($\pm$.023) | 0.747($\pm$.016) | 0.735($\pm$.042) | 0.766($\pm$.007) | 0.750($\pm$.017) |
| K$\rightarrow$E | 0.842($\pm$.008) | 0.831($\pm$.009) | 0.834($\pm$.010) | 0.831($\pm$.008) | 0.843($\pm$.021) | 0.823($\pm$.024) | 0.839($\pm$.006) | 0.823($\pm$.021) | 0.858($\pm$.009) | 0.847($\pm$.010) |
| mean | 0.767($\pm$.009) | 0.768($\pm$.014) | 0.772($\pm$.014) | 0.769($\pm$.012) | 0.775($\pm$.012) | 0.769($\pm$.018) | 0.778($\pm$.012) | 0.772($\pm$.018) | 0.789($\pm$.008) | 0.784($\pm$.011) |

#### C.2 HHAR

**Table 13: AdvSKM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.661($\pm$.035) | 0.692($\pm$.019) | 0.696($\pm$.019) | 0.692($\pm$.019) | 0.692($\pm$.019) | 0.646($\pm$.033) | 0.642($\pm$.036) | 0.650($\pm$.041) | 0.604($\pm$.046) | 0.731($\pm$.013) |
| 1$\rightarrow$6 | 0.821($\pm$.046) | 0.806($\pm$.026) | 0.806($\pm$.026) | 0.806($\pm$.026) | 0.806($\pm$.026) | 0.737($\pm$.045) | 0.732($\pm$.046) | 0.740($\pm$.040) | 0.735($\pm$.049) | 0.838($\pm$.014) |
| 2$\rightarrow$7 | 0.455($\pm$.070) | 0.497($\pm$.067) | 0.482($\pm$.062) | 0.497($\pm$.067) | 0.497($\pm$.067) | 0.490($\pm$.053) | 0.484($\pm$.099) | 0.475($\pm$.022) | 0.496($\pm$.034) | 0.574($\pm$.112) |
| 3$\rightarrow$8 | 0.790($\pm$.011) | 0.799($\pm$.005) | 0.794($\pm$.008) | 0.799($\pm$.005) | 0.799($\pm$.005) | 0.768($\pm$.034) | 0.771($\pm$.049) | 0.768($\pm$.037) | 0.770($\pm$.026) | 0.818($\pm$.012) |
| 4$\rightarrow$5 | 0.862($\pm$.055) | 0.850($\pm$.025) | 0.852($\pm$.010) | 0.850($\pm$.025) | 0.850($\pm$.025) | 0.771($\pm$.043) | 0.785($\pm$.035) | 0.785($\pm$.028) | 0.790($\pm$.039) | 0.895($\pm$.014) |
| mean | 0.718($\pm$.044) | 0.729($\pm$.028) | 0.726($\pm$.025) | 0.729($\pm$.028) | 0.729($\pm$.028) | 0.682($\pm$.042) | 0.683($\pm$.053) | 0.684($\pm$.034) | 0.679($\pm$.039) | 0.771($\pm$.033) |

**Table 14: CDAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.717($\pm$.004) | 0.726($\pm$.032) | 0.643($\pm$.159) | 0.726($\pm$.032) | 0.726($\pm$.032) | 0.690($\pm$.096) | 0.692($\pm$.101) | 0.683($\pm$.082) | 0.690($\pm$.086) | 0.718($\pm$.012) |
| 1$\rightarrow$6 | 0.742($\pm$.090) | 0.942($\pm$.011) | 0.942($\pm$.011) | 0.942($\pm$.011) | 0.942($\pm$.011) | 0.789($\pm$.064) | 0.821($\pm$.034) | 0.807($\pm$.051) | 0.829($\pm$.036) | 0.946($\pm$.000) |
| 2$\rightarrow$7 | 0.554($\pm$.031) | 0.600($\pm$.041) | 0.564($\pm$.064) | 0.583($\pm$.029) | 0.562($\pm$.019) | 0.490($\pm$.014) | 0.542($\pm$.056) | 0.507($\pm$.044) | 0.567($\pm$.066) | 0.624($\pm$.014) |
| 3$\rightarrow$8 | 0.770($\pm$.069) | 0.801($\pm$.004) | 0.771($\pm$.069) | 0.801($\pm$.004) | 0.801($\pm$.004) | 0.803($\pm$.063) | 0.812($\pm$.058) | 0.815($\pm$.035) | 0.814($\pm$.049) | 0.987($\pm$.006) |
| 4$\rightarrow$5 | 0.859($\pm$.014) | 0.875($\pm$.087) | 0.904($\pm$.107) | 0.875($\pm$.087) | 0.875($\pm$.087) | 0.880($\pm$.016) | 0.870($\pm$.006) | 0.882($\pm$.022) | 0.876($\pm$.026) | 0.982($\pm$.002) |
| mean | 0.728($\pm$.042) | 0.789($\pm$.035) | 0.765($\pm$.082) | 0.785($\pm$.033) | 0.781($\pm$.031) | 0.730($\pm$.051) | 0.747($\pm$.051) | 0.739($\pm$.047) | 0.755($\pm$.053) | 0.851($\pm$.007) |

**Table 15: CMD**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.703($\pm$.040) | 0.700($\pm$.085) | 0.743($\pm$.021) | 0.743($\pm$.021) | 0.743($\pm$.021) | 0.537($\pm$.029) | 0.618($\pm$.031) | 0.550($\pm$.059) | 0.572($\pm$.046) | 0.724($\pm$.049) |
| 1$\rightarrow$6 | 0.861($\pm$.035) | 0.868($\pm$.023) | 0.868($\pm$.023) | 0.868($\pm$.023) | 0.868($\pm$.023) | 0.793($\pm$.080) | 0.844($\pm$.042) | 0.857($\pm$.013) | 0.861($\pm$.010) | 0.925($\pm$.004) |
| 2$\rightarrow$7 | 0.573($\pm$.018) | 0.577($\pm$.011) | 0.586($\pm$.009) | 0.577($\pm$.011) | 0.577($\pm$.011) | 0.632($\pm$.034) | 0.591($\pm$.058) | 0.609($\pm$.041) | 0.610($\pm$.017) | 0.603($\pm$.016) |
| 3$\rightarrow$8 | 0.799($\pm$.016) | 0.799($\pm$.016) | 0.790($\pm$.022) | 0.799($\pm$.016) | 0.799($\pm$.016) | 0.814($\pm$.005) | 0.794($\pm$.030) | 0.790($\pm$.036) | 0.753($\pm$.040) | 0.822($\pm$.010) |
| 4$\rightarrow$5 | 0.806($\pm$.024) | 0.854($\pm$.063) | 0.799($\pm$.143) | 0.799($\pm$.143) | 0.854($\pm$.063) | 0.796($\pm$.045) | 0.792($\pm$.043) | 0.803($\pm$.051) | 0.819($\pm$.053) | 0.961($\pm$.024) |
| mean | 0.748($\pm$.026) | 0.760($\pm$.040) | 0.757($\pm$.043) | 0.757($\pm$.043) | 0.768($\pm$.027) | 0.714($\pm$.038) | 0.728($\pm$.041) | 0.722($\pm$.040) | 0.723($\pm$.033) | 0.807($\pm$.021) |

**Table 16: CoDATS**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.640($\pm$.096) | 0.689($\pm$.069) | 0.615($\pm$.129) | 0.689($\pm$.069) | 0.689($\pm$.069) | 0.640($\pm$.092) | 0.638($\pm$.069) | 0.638($\pm$.078) | 0.642($\pm$.073) | 0.735($\pm$.010) |
| 1$\rightarrow$6 | 0.775($\pm$.091) | 0.896($\pm$.069) | 0.858($\pm$.119) | 0.896($\pm$.069) | 0.896($\pm$.069) | 0.839($\pm$.017) | 0.846($\pm$.022) | 0.835($\pm$.017) | 0.835($\pm$.025) | 0.947($\pm$.009) |
| 2$\rightarrow$7 | 0.527($\pm$.076) | 0.472($\pm$.019) | 0.472($\pm$.019) | 0.472($\pm$.019) | 0.472($\pm$.019) | 0.501($\pm$.016) | 0.515($\pm$.029) | 0.491($\pm$.009) | 0.501($\pm$.014) | 0.558($\pm$.041) |
| 3$\rightarrow$8 | 0.783($\pm$.015) | 0.789($\pm$.010) | 0.783($\pm$.015) | 0.789($\pm$.010) | 0.789($\pm$.010) | 0.794($\pm$.052) | 0.796($\pm$.043) | 0.806($\pm$.008) | 0.845($\pm$.051) | 0.987($\pm$.002) |
| 4$\rightarrow$5 | 0.827($\pm$.039) | 0.861($\pm$.094) | 0.871($\pm$.098) | 0.861($\pm$.094) | 0.849($\pm$.074) | 0.846($\pm$.033) | 0.859($\pm$.031) | 0.837($\pm$.027) | 0.837($\pm$.025) | 0.979($\pm$.006) |
| mean | 0.710($\pm$.063) | 0.741($\pm$.052) | 0.720($\pm$.076) | 0.741($\pm$.052) | 0.739($\pm$.048) | 0.724($\pm$.042) | 0.731($\pm$.039) | 0.721($\pm$.028) | 0.732($\pm$.038) | 0.841($\pm$.014) |

**Table 17: DANN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.704($\pm$.037) | 0.718($\pm$.006) | 0.719($\pm$.005) | 0.718($\pm$.006) | 0.718($\pm$.006) | 0.694($\pm$.069) | 0.737($\pm$.027) | 0.717($\pm$.037) | 0.732($\pm$.013) | 0.711($\pm$.006) |
| 1$\rightarrow$6 | 0.833($\pm$.036) | 0.929($\pm$.004) | 0.833($\pm$.159) | 0.929($\pm$.004) | 0.929($\pm$.004) | 0.839($\pm$.013) | 0.839($\pm$.038) | 0.849($\pm$.021) | 0.819($\pm$.049) | 0.939($\pm$.015) |
| 2$\rightarrow$7 | 0.591($\pm$.023) | 0.618($\pm$.029) | 0.586($\pm$.080) | 0.626($\pm$.018) | 0.626($\pm$.018) | 0.597($\pm$.030) | 0.609($\pm$.055) | 0.606($\pm$.046) | 0.615($\pm$.060) | 0.635($\pm$.025) |
| 3$\rightarrow$8 | 0.809($\pm$.020) | 0.796($\pm$.030) | 0.909($\pm$.088) | 0.796($\pm$.030) | 0.796($\pm$.030) | 0.772($\pm$.064) | 0.805($\pm$.031) | 0.796($\pm$.037) | 0.807($\pm$.028) | 0.983($\pm$.002) |
| 4$\rightarrow$5 | 0.846($\pm$.022) | 0.922($\pm$.078) | 0.919($\pm$.076) | 0.922($\pm$.078) | 0.922($\pm$.078) | 0.859($\pm$.045) | 0.805($\pm$.135) | 0.859($\pm$.045) | 0.859($\pm$.050) | 0.980($\pm$.000) |
| mean | 0.757($\pm$.027) | 0.796($\pm$.030) | 0.793($\pm$.081) | 0.798($\pm$.027) | 0.798($\pm$.027) | 0.752($\pm$.044) | 0.759($\pm$.057) | 0.765($\pm$.037) | 0.767($\pm$.040) | 0.850($\pm$.010) |

**Table 18: DDC**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.575($\pm$.083) | 0.649($\pm$.012) | 0.619($\pm$.038) | 0.649($\pm$.012) | 0.650($\pm$.015) | 0.615($\pm$.031) | 0.604($\pm$.014) | 0.617($\pm$.033) | 0.622($\pm$.045) | 0.651($\pm$.010) |
| 1$\rightarrow$6 | 0.875($\pm$.014) | 0.856($\pm$.067) | 0.844($\pm$.084) | 0.856($\pm$.067) | 0.856($\pm$.067) | 0.842($\pm$.040) | 0.826($\pm$.037) | 0.836($\pm$.042) | 0.825($\pm$.044) | 0.899($\pm$.016) |
| 2$\rightarrow$7 | 0.487($\pm$.016) | 0.442($\pm$.136) | 0.455($\pm$.054) | 0.405($\pm$.073) | 0.438($\pm$.054) | 0.501($\pm$.045) | 0.464($\pm$.039) | 0.443($\pm$.051) | 0.464($\pm$.041) | 0.533($\pm$.107) |
| 3$\rightarrow$8 | 0.815($\pm$.019) | 0.827($\pm$.025) | 0.806($\pm$.019) | 0.827($\pm$.025) | 0.827($\pm$.025) | 0.794($\pm$.023) | 0.803($\pm$.033) | 0.806($\pm$.025) | 0.807($\pm$.035) | 0.822($\pm$.029) |
| 4$\rightarrow$5 | 0.831($\pm$.048) | 0.792($\pm$.020) | 0.777($\pm$.033) | 0.792($\pm$.020) | 0.792($\pm$.020) | 0.810($\pm$.070) | 0.811($\pm$.049) | 0.814($\pm$.064) | 0.797($\pm$.037) | 0.888($\pm$.037) |
| mean | 0.716($\pm$.036) | 0.713($\pm$.052) | 0.701($\pm$.046) | 0.705($\pm$.039) | 0.712($\pm$.036) | 0.713($\pm$.042) | 0.702($\pm$.035) | 0.703($\pm$.043) | 0.703($\pm$.040) | 0.758($\pm$.040) |

**Table 19: DCoral**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.692($\pm$.026) | 0.703($\pm$.038) | 0.703($\pm$.038) | 0.708($\pm$.029) | 0.708($\pm$.029) | 0.618($\pm$.025) | 0.600($\pm$.018) | 0.574($\pm$.048) | 0.586($\pm$.049) | 0.703($\pm$.030) |
| 1$\rightarrow$6 | 0.862($\pm$.018) | 0.864($\pm$.017) | 0.849($\pm$.034) | 0.864($\pm$.017) | 0.864($\pm$.017) | 0.785($\pm$.032) | 0.806($\pm$.040) | 0.790($\pm$.016) | 0.789($\pm$.024) | 0.911($\pm$.017) |
| 2$\rightarrow$7 | 0.509($\pm$.097) | 0.499($\pm$.079) | 0.487($\pm$.019) | 0.499($\pm$.079) | 0.499($\pm$.079) | 0.491($\pm$.051) | 0.522($\pm$.028) | 0.518($\pm$.035) | 0.515($\pm$.035) | 0.565($\pm$.117) |
| 3$\rightarrow$8 | 0.798($\pm$.016) | 0.799($\pm$.013) | 0.796($\pm$.018) | 0.799($\pm$.013) | 0.799($\pm$.013) | 0.743($\pm$.011) | 0.737($\pm$.013) | 0.745($\pm$.006) | 0.758($\pm$.008) | 0.822($\pm$.010) |
| 4$\rightarrow$5 | 0.862($\pm$.036) | 0.868($\pm$.053) | 0.884($\pm$.083) | 0.868($\pm$.053) | 0.868($\pm$.053) | 0.728($\pm$.148) | 0.771($\pm$.074) | 0.784($\pm$.054) | 0.789($\pm$.041) | 0.960($\pm$.010) |
| mean | 0.745($\pm$.039) | 0.747($\pm$.040) | 0.744($\pm$.038) | 0.748($\pm$.038) | 0.748($\pm$.038) | 0.673($\pm$.054) | 0.687($\pm$.035) | 0.682($\pm$.032) | 0.687($\pm$.031) | 0.792($\pm$.037) |

**Table 20: DIRT**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.708($\pm$.011) | 0.708($\pm$.011) | 0.708($\pm$.011) | 0.708($\pm$.011) | 0.708($\pm$.011) | 0.608($\pm$.154) | 0.649($\pm$.114) | 0.636($\pm$.119) | 0.717($\pm$.055) | 0.739($\pm$.016) |
| 1$\rightarrow$6 | 0.756($\pm$.058) | 0.814($\pm$.056) | 0.878($\pm$.104) | 0.814($\pm$.056) | 0.814($\pm$.056) | 0.768($\pm$.073) | 0.786($\pm$.032) | 0.785($\pm$.049) | 0.824($\pm$.047) | 0.942($\pm$.004) |
| 2$\rightarrow$7 | 0.531($\pm$.057) | 0.573($\pm$.083) | 0.637($\pm$.072) | 0.573($\pm$.083) | 0.579($\pm$.083) | 0.519($\pm$.067) | 0.542($\pm$.086) | 0.504($\pm$.092) | 0.499($\pm$.048) | 0.688($\pm$.004) |
| 3$\rightarrow$8 | 0.807($\pm$.008) | 0.807($\pm$.008) | 0.866($\pm$.106) | 0.807($\pm$.008) | 0.807($\pm$.008) | 0.832($\pm$.081) | 0.836($\pm$.051) | 0.845($\pm$.075) | 0.849($\pm$.047) | 0.911($\pm$.070) |
| 4$\rightarrow$5 | 0.839($\pm$.016) | 0.839($\pm$.016) | 0.982($\pm$.008) | 0.839($\pm$.016) | 0.839($\pm$.016) | 0.861($\pm$.034) | 0.837($\pm$.049) | 0.854($\pm$.029) | 0.845($\pm$.039) | 0.984($\pm$.004) |
| mean | 0.728($\pm$.030) | 0.748($\pm$.035) | 0.814($\pm$.060) | 0.748($\pm$.035) | 0.749($\pm$.035) | 0.718($\pm$.082) | 0.730($\pm$.066) | 0.725($\pm$.073) | 0.747($\pm$.047) | 0.853($\pm$.020) |

**Table 21: DSAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.653($\pm$.033) | 0.572($\pm$.103) | 0.557($\pm$.078) | 0.572($\pm$.103) | 0.572($\pm$.103) | 0.683($\pm$.019) | 0.683($\pm$.023) | 0.675($\pm$.019) | 0.678($\pm$.024) | 0.664($\pm$.010) |
| 1$\rightarrow$6 | 0.806($\pm$.083) | 0.908($\pm$.065) | 0.894($\pm$.057) | 0.908($\pm$.065) | 0.908($\pm$.065) | 0.817($\pm$.072) | 0.814($\pm$.081) | 0.815($\pm$.075) | 0.818($\pm$.081) | 0.940($\pm$.005) |
| 2$\rightarrow$7 | 0.490($\pm$.053) | 0.494($\pm$.030) | 0.476($\pm$.056) | 0.482($\pm$.012) | 0.476($\pm$.009) | 0.497($\pm$.020) | 0.522($\pm$.054) | 0.493($\pm$.030) | 0.503($\pm$.025) | 0.586($\pm$.083) |
| 3$\rightarrow$8 | 0.797($\pm$.016) | 0.816($\pm$.010) | 0.857($\pm$.091) | 0.816($\pm$.010) | 0.816($\pm$.010) | 0.828($\pm$.014) | 0.829($\pm$.008) | 0.832($\pm$.010) | 0.836($\pm$.004) | 0.982($\pm$.010) |
| 4$\rightarrow$5 | 0.861($\pm$.043) | 0.844($\pm$.122) | 0.799($\pm$.060) | 0.844($\pm$.122) | 0.844($\pm$.122) | 0.829($\pm$.022) | 0.832($\pm$.018) | 0.826($\pm$.035) | 0.822($\pm$.032) | 0.982($\pm$.005) |
| mean | 0.721($\pm$.046) | 0.727($\pm$.066) | 0.717($\pm$.068) | 0.725($\pm$.062) | 0.723($\pm$.062) | 0.731($\pm$.030) | 0.736($\pm$.037) | 0.728($\pm$.034) | 0.731($\pm$.033) | 0.831($\pm$.023) |

**Table 22: HoMM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.719($\pm$.041) | 0.704($\pm$.011) | 0.701($\pm$.006) | 0.715($\pm$.019) | 0.715($\pm$.019) | 0.583($\pm$.120) | 0.618($\pm$.070) | 0.586($\pm$.089) | 0.622($\pm$.037) | 0.733($\pm$.007) |
| 1$\rightarrow$6 | 0.787($\pm$.110) | 0.749($\pm$.060) | 0.768($\pm$.082) | 0.749($\pm$.060) | 0.749($\pm$.060) | 0.771($\pm$.012) | 0.753($\pm$.035) | 0.774($\pm$.006) | 0.774($\pm$.005) | 0.914($\pm$.010) |
| 2$\rightarrow$7 | 0.528($\pm$.103) | 0.497($\pm$.105) | 0.516($\pm$.089) | 0.530($\pm$.079) | 0.530($\pm$.079) | 0.454($\pm$.058) | 0.493($\pm$.045) | 0.469($\pm$.069) | 0.497($\pm$.067) | 0.546($\pm$.071) |
| 3$\rightarrow$8 | 0.797($\pm$.007) | 0.805($\pm$.010) | 0.770($\pm$.086) | 0.805($\pm$.010) | 0.805($\pm$.010) | 0.729($\pm$.080) | 0.763($\pm$.049) | 0.755($\pm$.063) | 0.792($\pm$.043) | 0.831($\pm$.022) |
| 4$\rightarrow$5 | 0.861($\pm$.022) | 0.844($\pm$.016) | 0.895($\pm$.041) | 0.844($\pm$.016) | 0.844($\pm$.016) | 0.816($\pm$.037) | 0.822($\pm$.040) | 0.822($\pm$.036) | 0.831($\pm$.033) | 0.940($\pm$.040) |
| mean | 0.739($\pm$.057) | 0.720($\pm$.041) | 0.730($\pm$.061) | 0.728($\pm$.037) | 0.728($\pm$.037) | 0.671($\pm$.061) | 0.690($\pm$.048) | 0.681($\pm$.053) | 0.703($\pm$.037) | 0.793($\pm$.030) |

**Table 23: MMDA**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| 0$\rightarrow$6 | 0.706($\pm$.042) | 0.699($\pm$.055) | 0.685($\pm$.028) | 0.699($\pm$.055) | 0.699($\pm$.055) | 0.571($\pm$.076) | 0.597($\pm$.025) | 0.574($\pm$.040) | 0.636($\pm$.037) | 0.732($\pm$.024) |
| 1$\rightarrow$6 | 0.792($\pm$.098) | 0.732($\pm$.009) | 0.722($\pm$.009) | 0.732($\pm$.009) | 0.732($\pm$.009) | 0.757($\pm$.030) | 0.751($\pm$.010) | 0.758($\pm$.027) | 0.765($\pm$.016) | 0.914($\pm$.028) |
| 2$\rightarrow$7 | 0.552($\pm$.073) | 0.549($\pm$.051) | 0.530($\pm$.071) | 0.549($\pm$.072) | 0.530($\pm$.071) | 0.494($\pm$.090) | 0.516($\pm$.048) | 0.496($\pm$.083) | 0.531($\pm$.035) | 0.552($\pm$.073) |
| 3$\rightarrow$8 | 0.790($\pm$.016) | 0.802($\pm$.018) | 0.820($\pm$.018) | 0.802($\pm$.018) | 0.802($\pm$.018) | 0.779($\pm$.090) | 0.801($\pm$.061) | 0.801($\pm$.075) | 0.796($\pm$.040) | 0.932($\pm$.005) |
| 4$\rightarrow$5 | 0.852($\pm$.036) | 0.846($\pm$.066) | 0.859($\pm$.031) | 0.846($\pm$.066) | 0.846($\pm$.066) | 0.826($\pm$.016) | 0.820($\pm$.051) | 0.824($\pm$.014) | 0.824($\pm$.018) | 0.947($\pm$.022) |
| mean | 0.738($\pm$.053) | 0.726($\pm$.040) | 0.723($\pm$.031) | 0.726($\pm$.044) | 0.722($\pm$.044) | 0.685($\pm$.060) | 0.697($\pm$.039) | 0.690($\pm$.048) | 0.710($\pm$.029) | 0.815($\pm$.030) |

#### C.3 MiniDomainNet

**Table 24: AdvSKM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.544($\pm$.017) | 0.546($\pm$.031) | 0.546($\pm$.031) | 0.546($\pm$.031) | 0.546($\pm$.031) | 0.546($\pm$.017) | 0.546($\pm$.017) | 0.546($\pm$.017) | 0.555($\pm$.017) | 0.582($\pm$.046) |
| R$\rightarrow$I | 0.380($\pm$.018) | 0.394($\pm$.021) | 0.394($\pm$.021) | 0.394($\pm$.021) | 0.394($\pm$.021) | 0.400($\pm$.009) | 0.403($\pm$.018) | 0.400($\pm$.021) | 0.395($\pm$.003) | 0.395($\pm$.020) |
| R$\rightarrow$P | 0.723($\pm$.009) | 0.711($\pm$.015) | 0.711($\pm$.015) | 0.711($\pm$.015) | 0.712($\pm$.017) | 0.707($\pm$.018) | 0.705($\pm$.021) | 0.705($\pm$.019) | 0.705($\pm$.021) | 0.723($\pm$.009) |
| R$\rightarrow$Q | 0.322($\pm$.027) | 0.340($\pm$.004) | 0.333($\pm$.003) | 0.340($\pm$.004) | 0.340($\pm$.004) | 0.316($\pm$.015) | 0.317($\pm$.011) | 0.314($\pm$.015) | 0.315($\pm$.013) | 0.335($\pm$.004) |
| R$\rightarrow$S | 0.579($\pm$.017) | 0.586($\pm$.001) | 0.586($\pm$.001) | 0.586($\pm$.001) | 0.586($\pm$.001) | 0.592($\pm$.003) | 0.592($\pm$.000) | 0.591($\pm$.001) | 0.588($\pm$.004) | 0.592($\pm$.009) |
| mean | 0.509($\pm$.018) | 0.515($\pm$.014) | 0.514($\pm$.014) | 0.515($\pm$.014) | 0.516($\pm$.015) | 0.512($\pm$.012) | 0.513($\pm$.013) | 0.511($\pm$.015) | 0.512($\pm$.012) | 0.525($\pm$.018) |

**Table 25: CDAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.557($\pm$.008) | 0.574($\pm$.008) | 0.549($\pm$.036) | 0.549($\pm$.036) | 0.549($\pm$.036) | 0.593($\pm$.026) | 0.598($\pm$.028) | 0.598($\pm$.030) | 0.598($\pm$.038) | 0.615($\pm$.008) |
| R$\rightarrow$I | 0.386($\pm$.010) | 0.357($\pm$.027) | 0.370($\pm$.014) | 0.357($\pm$.027) | 0.374($\pm$.015) | 0.326($\pm$.079) | 0.383($\pm$.003) | 0.382($\pm$.011) | 0.388($\pm$.007) | 0.386($\pm$.010) |
| R$\rightarrow$P | 0.709($\pm$.022) | 0.688($\pm$.014) | 0.708($\pm$.025) | 0.708($\pm$.025) | 0.710($\pm$.023) | 0.717($\pm$.020) | 0.714($\pm$.012) | 0.719($\pm$.016) | 0.709($\pm$.025) | 0.716($\pm$.055) |
| R$\rightarrow$Q | 0.342($\pm$.006) | 0.338($\pm$.008) | 0.338($\pm$.008) | 0.338($\pm$.008) | 0.338($\pm$.008) | 0.351($\pm$.016) | 0.352($\pm$.013) | 0.351($\pm$.024) | 0.351($\pm$.024) | 0.442($\pm$.114) |
| R$\rightarrow$S | 0.576($\pm$.027) | 0.595($\pm$.008) | 0.599($\pm$.011) | 0.599($\pm$.011) | 0.599($\pm$.011) | 0.608($\pm$.012) | 0.605($\pm$.013) | 0.606($\pm$.015) | 0.605($\pm$.013) | 0.651($\pm$.056) |
| mean | 0.514($\pm$.015) | 0.511($\pm$.013) | 0.513($\pm$.019) | 0.510($\pm$.021) | 0.514($\pm$.018) | 0.519($\pm$.031) | 0.530($\pm$.014) | 0.531($\pm$.019) | 0.530($\pm$.021) | 0.562($\pm$.049) |

**Table 26: CMD**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.555($\pm$.042) | 0.557($\pm$.038) | 0.555($\pm$.042) | 0.555($\pm$.042) | 0.549($\pm$.036) | 0.541($\pm$.030) | 0.525($\pm$.028) | 0.541($\pm$.030) | 0.544($\pm$.037) | 0.617($\pm$.025) |
| R$\rightarrow$I | 0.377($\pm$.009) | 0.395($\pm$.010) | 0.383($\pm$.010) | 0.382($\pm$.009) | 0.379($\pm$.007) | 0.388($\pm$.021) | 0.403($\pm$.024) | 0.406($\pm$.030) | 0.397($\pm$.029) | 0.389($\pm$.032) |
| R$\rightarrow$P | 0.704($\pm$.020) | 0.727($\pm$.023) | 0.727($\pm$.023) | 0.727($\pm$.023) | 0.727($\pm$.023) | 0.743($\pm$.014) | 0.740($\pm$.018) | 0.735($\pm$.021) | 0.739($\pm$.023) | 0.734($\pm$.014) |
| R$\rightarrow$Q | 0.332($\pm$.015) | 0.355($\pm$.035) | 0.355($\pm$.035) | 0.355($\pm$.035) | 0.355($\pm$.035) | 0.316($\pm$.041) | 0.312($\pm$.040) | 0.315($\pm$.045) | 0.316($\pm$.046) | 0.378($\pm$.008) |
| R$\rightarrow$S | 0.578($\pm$.022) | 0.592($\pm$.005) | 0.592($\pm$.005) | 0.592($\pm$.005) | 0.592($\pm$.005) | 0.599($\pm$.019) | 0.599($\pm$.013) | 0.599($\pm$.017) | 0.594($\pm$.018) | 0.590($\pm$.026) |
| mean | 0.509($\pm$.022) | 0.525($\pm$.022) | 0.522($\pm$.023) | 0.522($\pm$.023) | 0.520($\pm$.021) | 0.517($\pm$.025) | 0.516($\pm$.025) | 0.519($\pm$.029) | 0.518($\pm$.031) | 0.542($\pm$.021) |

**Table 27: CoDATS**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.538($\pm$.072) | 0.566($\pm$.025) | 0.557($\pm$.028) | 0.557($\pm$.028) | 0.557($\pm$.028) | 0.579($\pm$.033) | 0.566($\pm$.014) | 0.557($\pm$.014) | 0.560($\pm$.005) | 0.617($\pm$.066) |
| R$\rightarrow$I | 0.365($\pm$.037) | 0.362($\pm$.034) | 0.377($\pm$.015) | 0.362($\pm$.034) | 0.377($\pm$.015) | 0.360($\pm$.041) | 0.368($\pm$.025) | 0.368($\pm$.027) | 0.371($\pm$.032) | 0.406($\pm$.021) |
| R$\rightarrow$P | 0.689($\pm$.024) | 0.723($\pm$.020) | 0.723($\pm$.020) | 0.723($\pm$.020) | 0.723($\pm$.020) | 0.726($\pm$.015) | 0.728($\pm$.016) | 0.727($\pm$.019) | 0.726($\pm$.020) | 0.732($\pm$.017) |
| R$\rightarrow$Q | 0.322($\pm$.022) | 0.329($\pm$.023) | 0.329($\pm$.023) | 0.329($\pm$.023) | 0.329($\pm$.023) | 0.359($\pm$.022) | 0.357($\pm$.015) | 0.364($\pm$.020) | 0.364($\pm$.018) | 0.417($\pm$.025) |
| R$\rightarrow$S | 0.597($\pm$.004) | 0.588($\pm$.012) | 0.588($\pm$.012) | 0.588($\pm$.012) | 0.588($\pm$.012) | 0.611($\pm$.004) | 0.611($\pm$.006) | 0.609($\pm$.006) | 0.614($\pm$.008) | 0.627($\pm$.008) |
| mean | 0.502($\pm$.032) | 0.514($\pm$.023) | 0.515($\pm$.020) | 0.512($\pm$.023) | 0.515($\pm$.020) | 0.527($\pm$.023) | 0.526($\pm$.015) | 0.525($\pm$.017) | 0.527($\pm$.016) | 0.560($\pm$.027) |

**Table 28: DANN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.519($\pm$.013) | 0.552($\pm$.017) | 0.552($\pm$.017) | 0.552($\pm$.017) | 0.552($\pm$.017) | 0.560($\pm$.005) | 0.563($\pm$.005) | 0.557($\pm$.000) | 0.560($\pm$.017) | 0.631($\pm$.038) |
| R$\rightarrow$I | 0.376($\pm$.016) | 0.379($\pm$.036) | 0.388($\pm$.025) | 0.379($\pm$.036) | 0.389($\pm$.024) | 0.370($\pm$.023) | 0.379($\pm$.026) | 0.380($\pm$.016) | 0.371($\pm$.012) | 0.400($\pm$.025) |
| R$\rightarrow$P | 0.705($\pm$.011) | 0.699($\pm$.006) | 0.703($\pm$.011) | 0.703($\pm$.011) | 0.710($\pm$.015) | 0.718($\pm$.004) | 0.717($\pm$.004) | 0.719($\pm$.000) | 0.712($\pm$.006) | 0.713($\pm$.013) |
| R$\rightarrow$Q | 0.300($\pm$.043) | 0.332($\pm$.015) | 0.332($\pm$.015) | 0.332($\pm$.015) | 0.329($\pm$.013) | 0.317($\pm$.023) | 0.315($\pm$.025) | 0.318($\pm$.022) | 0.322($\pm$.023) | 0.392($\pm$.005) |
| R$\rightarrow$S | 0.581($\pm$.014) | 0.596($\pm$.024) | 0.596($\pm$.024) | 0.596($\pm$.024) | 0.596($\pm$.024) | 0.608($\pm$.010) | 0.607($\pm$.011) | 0.605($\pm$.011) | 0.601($\pm$.013) | 0.657($\pm$.015) |
| mean | 0.496($\pm$.019) | 0.511($\pm$.020) | 0.514($\pm$.018) | 0.512($\pm$.021) | 0.515($\pm$.018) | 0.515($\pm$.013) | 0.516($\pm$.014) | 0.516($\pm$.010) | 0.513($\pm$.014) | 0.559($\pm$.019) |

**Table 29: DDC**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.555($\pm$.045) | 0.560($\pm$.017) | 0.563($\pm$.013) | 0.563($\pm$.013) | 0.546($\pm$.025) | 0.563($\pm$.025) | 0.563($\pm$.033) | 0.555($\pm$.029) | 0.571($\pm$.034) | 0.585($\pm$.076) |
| R$\rightarrow$I | 0.374($\pm$.014) | 0.379($\pm$.009) | 0.379($\pm$.009) | 0.379($\pm$.009) | 0.386($\pm$.009) | 0.397($\pm$.017) | 0.394($\pm$.023) | 0.398($\pm$.020) | 0.394($\pm$.023) | 0.416($\pm$.014) |
| R$\rightarrow$P | 0.709($\pm$.025) | 0.706($\pm$.012) | 0.707($\pm$.013) | 0.706($\pm$.012) | 0.706($\pm$.012) | 0.717($\pm$.002) | 0.717($\pm$.002) | 0.719($\pm$.003) | 0.724($\pm$.005) | 0.717($\pm$.023) |
| R$\rightarrow$Q | 0.334($\pm$.002) | 0.333($\pm$.005) | 0.333($\pm$.005) | 0.333($\pm$.005) | 0.333($\pm$.005) | 0.323($\pm$.023) | 0.324($\pm$.022) | 0.324($\pm$.022) | 0.324($\pm$.025) | 0.337($\pm$.004) |
| R$\rightarrow$S | 0.576($\pm$.022) | 0.587($\pm$.008) | 0.586($\pm$.009) | 0.586($\pm$.009) | 0.586($\pm$.009) | 0.580($\pm$.006) | 0.579($\pm$.009) | 0.577($\pm$.008) | 0.577($\pm$.005) | 0.585($\pm$.021) |
| mean | 0.510($\pm$.021) | 0.513($\pm$.010) | 0.514($\pm$.010) | 0.513($\pm$.010) | 0.512($\pm$.012) | 0.516($\pm$.015) | 0.515($\pm$.018) | 0.515($\pm$.016) | 0.518($\pm$.018) | 0.528($\pm$.027) |

**Table 30: DCoral**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.555($\pm$.045) | 0.566($\pm$.008) | 0.566($\pm$.008) | 0.566($\pm$.008) | 0.579($\pm$.021) | 0.546($\pm$.005) | 0.552($\pm$.009) | 0.546($\pm$.013) | 0.549($\pm$.008) | 0.601($\pm$.056) |
| R$\rightarrow$I | 0.376($\pm$.012) | 0.377($\pm$.005) | 0.377($\pm$.015) | 0.373($\pm$.007) | 0.373($\pm$.007) | 0.379($\pm$.014) | 0.379($\pm$.018) | 0.379($\pm$.011) | 0.376($\pm$.005) | 0.386($\pm$.007) |
| R$\rightarrow$P | 0.711($\pm$.021) | 0.719($\pm$.011) | 0.719($\pm$.011) | 0.719($\pm$.011) | 0.709($\pm$.009) | 0.732($\pm$.003) | 0.728($\pm$.006) | 0.734($\pm$.004) | 0.739($\pm$.011) | 0.734($\pm$.014) |
| R$\rightarrow$Q | 0.316($\pm$.041) | 0.363($\pm$.023) | 0.363($\pm$.023) | 0.363($\pm$.023) | 0.363($\pm$.023) | 0.334($\pm$.026) | 0.333($\pm$.021) | 0.337($\pm$.029) | 0.332($\pm$.032) | 0.388($\pm$.002) |
| R$\rightarrow$S | 0.569($\pm$.020) | 0.585($\pm$.014) | 0.585($\pm$.014) | 0.585($\pm$.014) | 0.585($\pm$.014) | 0.611($\pm$.014) | 0.608($\pm$.010) | 0.609($\pm$.009) | 0.611($\pm$.012) | 0.658($\pm$.035) |
| mean | 0.505($\pm$.028) | 0.522($\pm$.012) | 0.522($\pm$.014) | 0.521($\pm$.013) | 0.522($\pm$.015) | 0.520($\pm$.012) | 0.520($\pm$.013) | 0.521($\pm$.013) | 0.521($\pm$.013) | 0.554($\pm$.023) |

**Table 31: DIRT**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.530($\pm$.013) | 0.538($\pm$.005) | 0.555($\pm$.031) | 0.566($\pm$.050) | 0.541($\pm$.025) | 0.574($\pm$.022) | 0.568($\pm$.026) | 0.566($\pm$.008) | 0.563($\pm$.013) | 0.587($\pm$.037) |
| R$\rightarrow$I | 0.373($\pm$.038) | 0.363($\pm$.038) | 0.379($\pm$.021) | 0.379($\pm$.021) | 0.360($\pm$.011) | 0.397($\pm$.046) | 0.421($\pm$.012) | 0.410($\pm$.013) | 0.418($\pm$.020) | 0.431($\pm$.045) |
| R$\rightarrow$P | 0.704($\pm$.031) | 0.663($\pm$.030) | 0.663($\pm$.030) | 0.663($\pm$.030) | 0.663($\pm$.030) | 0.711($\pm$.027) | 0.712($\pm$.028) | 0.711($\pm$.026) | 0.708($\pm$.029) | 0.715($\pm$.065) |
| R$\rightarrow$Q | 0.340($\pm$.020) | 0.326($\pm$.086) | 0.315($\pm$.069) | 0.326($\pm$.086) | 0.286($\pm$.049) | 0.323($\pm$.028) | 0.323($\pm$.028) | 0.319($\pm$.022) | 0.316($\pm$.029) | 0.382($\pm$.027) |
| R$\rightarrow$S | 0.550($\pm$.028) | 0.566($\pm$.008) | 0.566($\pm$.008) | 0.566($\pm$.008) | 0.569($\pm$.004) | 0.589($\pm$.033) | 0.592($\pm$.033) | 0.591($\pm$.032) | 0.592($\pm$.016) | 0.588($\pm$.033) |
| mean | 0.499($\pm$.026) | 0.491($\pm$.033) | 0.495($\pm$.032) | 0.500($\pm$.039) | 0.484($\pm$.024) | 0.519($\pm$.031) | 0.523($\pm$.025) | 0.519($\pm$.020) | 0.519($\pm$.021) | 0.541($\pm$.041) |

**Table 32: DSAN**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.555($\pm$.042) | 0.538($\pm$.029) | 0.563($\pm$.053) | 0.538($\pm$.029) | 0.568($\pm$.021) | 0.574($\pm$.030) | 0.566($\pm$.022) | 0.571($\pm$.026) | 0.582($\pm$.036) | 0.648($\pm$.014) |
| R$\rightarrow$I | 0.377($\pm$.009) | 0.341($\pm$.047) | 0.360($\pm$.053) | 0.341($\pm$.047) | 0.383($\pm$.021) | 0.350($\pm$.015) | 0.368($\pm$.009) | 0.370($\pm$.014) | 0.365($\pm$.007) | 0.404($\pm$.016) |
| R$\rightarrow$P | 0.705($\pm$.021) | 0.711($\pm$.012) | 0.711($\pm$.012) | 0.711($\pm$.012) | 0.711($\pm$.012) | 0.723($\pm$.013) | 0.723($\pm$.011) | 0.729($\pm$.004) | 0.730($\pm$.007) | 0.716($\pm$.000) |
| R$\rightarrow$Q | 0.332($\pm$.012) | 0.343($\pm$.008) | 0.329($\pm$.017) | 0.332($\pm$.011) | 0.332($\pm$.011) | 0.339($\pm$.016) | 0.338($\pm$.014) | 0.338($\pm$.018) | 0.339($\pm$.019) | 0.478($\pm$.013) |
| R$\rightarrow$S | 0.577($\pm$.026) | 0.587($\pm$.003) | 0.587($\pm$.003) | 0.587($\pm$.003) | 0.587($\pm$.003) | 0.607($\pm$.015) | 0.605($\pm$.012) | 0.603($\pm$.012) | 0.603($\pm$.019) | 0.639($\pm$.052) |
| mean | 0.509($\pm$.022) | 0.504($\pm$.020) | 0.510($\pm$.028) | 0.502($\pm$.020) | 0.516($\pm$.013) | 0.518($\pm$.018) | 0.520($\pm$.014) | 0.522($\pm$.015) | 0.524($\pm$.018) | 0.577($\pm$.019) |

**Table 33: HoMM**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.552($\pm$.041) | 0.568($\pm$.062) | 0.568($\pm$.062) | 0.568($\pm$.062) | 0.546($\pm$.041) | 0.574($\pm$.014) | 0.579($\pm$.017) | 0.577($\pm$.019) | 0.582($\pm$.022) | 0.607($\pm$.022) |
| R$\rightarrow$I | 0.373($\pm$.014) | 0.376($\pm$.021) | 0.391($\pm$.036) | 0.370($\pm$.015) | 0.380($\pm$.027) | 0.389($\pm$.044) | 0.409($\pm$.009) | 0.398($\pm$.024) | 0.413($\pm$.009) | 0.413($\pm$.041) |
| R$\rightarrow$P | 0.709($\pm$.026) | 0.714($\pm$.005) | 0.714($\pm$.005) | 0.714($\pm$.005) | 0.714($\pm$.005) | 0.723($\pm$.018) | 0.724($\pm$.015) | 0.722($\pm$.014) | 0.722($\pm$.014) | 0.721($\pm$.003) |
| R$\rightarrow$Q | 0.332($\pm$.009) | 0.355($\pm$.014) | 0.355($\pm$.014) | 0.355($\pm$.014) | 0.355($\pm$.014) | 0.323($\pm$.010) | 0.321($\pm$.009) | 0.324($\pm$.011) | 0.324($\pm$.010) | 0.370($\pm$.013) |
| R$\rightarrow$S | 0.579($\pm$.022) | 0.576($\pm$.006) | 0.576($\pm$.006) | 0.576($\pm$.006) | 0.576($\pm$.006) | 0.599($\pm$.012) | 0.598($\pm$.013) | 0.596($\pm$.012) | 0.596($\pm$.012) | 0.609($\pm$.011) |
| mean | 0.509($\pm$.023) | 0.518($\pm$.021) | 0.521($\pm$.025) | 0.516($\pm$.020) | 0.514($\pm$.019) | 0.521($\pm$.019) | 0.526($\pm$.013) | 0.523($\pm$.016) | 0.527($\pm$.014) | 0.544($\pm$.018) |

**Table 34: MMDA**
|  |  | Importance Weighted Validation | Importance Weighted Aggregation |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SO | Boost (IWV) | KuLSIF (IWV) | LR (IWV) | EW (IWV) | Boost (Agg) | KuLSIF (Agg) | LR (Agg) | EW (Agg) | TB |
| R$\rightarrow$C | 0.555($\pm$.042) | 0.568($\pm$.021) | 0.568($\pm$.021) | 0.568($\pm$.021) | 0.566($\pm$.022) | 0.582($\pm$.025) | 0.590($\pm$.025) | 0.590($\pm$.025) | 0.593($\pm$.021) | 0.585($\pm$.034) |
| R$\rightarrow$I | 0.377($\pm$.009) | 0.398($\pm$.040) | 0.379($\pm$.030) | 0.379($\pm$.030) | 0.379($\pm$.030) | 0.389($\pm$.009) | 0.374($\pm$.013) | 0.380($\pm$.018) | 0.370($\pm$.009) | 0.415($\pm$.016) |
| R$\rightarrow$P | 0.704($\pm$.020) | 0.709($\pm$.010) | 0.709($\pm$.010) | 0.709($\pm$.010) | 0.709($\pm$.010) | 0.728($\pm$.004) | 0.732($\pm$.007) | 0.732($\pm$.008) | 0.733($\pm$.007) | 0.735($\pm$.007) |
| R$\rightarrow$Q | 0.332($\pm$.015) | 0.353($\pm$.025) | 0.353($\pm$.025) | 0.353($\pm$.025) | 0.353($\pm$.025) | 0.343($\pm$.013) | 0.341($\pm$.012) | 0.343($\pm$.019) | 0.342($\pm$.020) | 0.392($\pm$.018) |
| R$\rightarrow$S | 0.578($\pm$.022) | 0.583($\pm$.006) | 0.583($\pm$.006) | 0.583($\pm$.006) | 0.588($\pm$.012) | 0.606($\pm$.010) | 0.606($\pm$.010) | 0.605($\pm$.009) | 0.603($\pm$.009) | 0.605($\pm$.008) |
| mean | 0.509($\pm$.022) | 0.522($\pm$.020) | 0.518($\pm$.018) | 0.518($\pm$.018) | 0.519($\pm$.020) | 0.530($\pm$.012) | 0.529($\pm$.013) | 0.530($\pm$.016) | 0.528($\pm$.013) | 0.546($\pm$.017) |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Could you briefly outline the most significant theoretical contribution of your paper and the challenges involved in achieving it?

2. Could you explain the intuition behind the flatness of your method in Figure 2? If the method aims to estimate the ratio accurately for high values, shouldn’t it follow the top of the curve closely? Furthermore, in the lower row of the figure for $\alpha=0.01$, I am not sure I understand why one would prefer your estimate over KuLSIF.

3. Since you mention that standard methods prioritize estimating smaller values and you focus on higher values, what happens if one applies standard methods to the inverse ratio, i.e., estimating $dQ/dP$?

Typo: In footnote 2, the next-to-last equation contains probabilities that should be conditioned on $x$: it should be $\rho(y=1∣x)\rho(x)$ instead of $\rho(x,y=1)\rho(x)$.
soundness: 3
strengths: 1. The paper appears to be self-contained and introduces all the tools and notation it uses.

2. The experimental results are fairly extensive for a theoretical paper.

3. The theoretical results, especially Lemma 4 in the appendix, seem to rely on non-trivial applications of several previously established results.
summary: The authors characterize the set of loss functions that, when used in density ratio estimation for binary classification, lead to the minimization of a particular Bregman divergence. This approach is motivated by the observation that some commonly used losses (such as exponential or logistic) yield density ratio estimates that minimize a similar Bregman divergence expression. After identifying these losses, they design a new family of losses aimed at accurately estimating large values of the density ratio, in contrast to standard losses that focus on estimating small ratio values. They apply these designed losses to estimate density ratios in Gaussian RKHS and for unsupervised domain adaptation.
weaknesses: 1. The presentation of the paper could be significantly improved. The paper uses heavy notation — for example, understanding $B_{-\underline{L}^\circ}(\beta,g\circ f)$ requires substantial effort. Another clear example is Remark 1, which is completely incomprehensible unless the reader is already familiar with everything it covers.

2. The novelty of this work is unclear. I believe the authors would agree that the main contribution of this paper is theoretical and primarily represented by Theorem 1. However, given Remark 1, it is not evident how substantial this theorem’s contribution really is.

3. It is not clear why one should focus on minimizing equation (1). According to the beginning of Section 2, there are "many" density ratio estimation methods that lead to minimizing (1), with four examples provided. What does "many" mean in this context, and why should one limit themself to this specific type of minimizers?

4. Some parts of the experimental results, like Section 6.2, contain too much irrelevant information for readers, making it easy to miss the main points. I would consider moving some of this information to the appendices.

---

confidence: 2
contribution: 1
initial_rating: 3
presentation: 2
questions: - Considering major weaknesses 1 and 2 discussed above, could you provide more additional discussions to clarify the novel contributions of your study?
- Considering major weaknesses 3 and 4 discussed above, could you provide further detailed information to elucidate the effectiveness of your approach discussed in Section 5?
soundness: 2
strengths: References to existing research are sufficiently provided.
summary: The authors present theoretical results characterizing strictly proper binary loss functions that lead to minimizers of Bregman divergences in probability density estimation. According to these theoretical results, they propose a novel loss function that prioritizes accurate estimation of large density ratio values over smaller ones. They also empirically validate the effectiveness of their proposed loss function through numerical experiments, demonstrating that the novel loss function can lead to improvements in parameter selection for domain adaptation tasks.
weaknesses: #### Major Weaknesses:
1. There are concerns regarding the novelty of the theoretical results presented in this study. Specifically, results such as the necessity of Equation (8) in Theorem 1 appear to be easily derived from findings  in prior work referenced by this study ([1], [2], and [3]). A detailed examination of this issue is provided below.
2. Additionally, the canonical form of the density ratio link, given in Equation (10), does not constitute a new result, as it can be derived from results presented in prior studies ([1]). A detailed examination of this issue is provided below.
3. The proposed loss function, derived from Equation (11) in Section 5, lacks a clear connection to the theoretical results previously established in Section 4.
4. Moreover, it is unclear how the proposed loss function specifically addresses shortcomings associated with existing loss functions. The authors are encouraged to include mathematical analysis, such as theorems, to clarify the properties of the proposed loss function.

#### Minor Weakness:
5. Figures 2 and 3:
    - The axis titles are missing, making it difficult to interpret the graphs.
    - In particular, Figure 3 lacks explanatory labels for each axis, which are needed to understand these experimental results.


---
Hereafter, details of the major weaknesses, specifically Weaknesses 1 and 2, are discussed.

#### About major weakness 1:
Equation (8) can be derived as follows:
1. From Theorem 4 in [2], we know that $B_{\phi} = B_{\phi'}$ for any $\phi'$ with $\phi'(y) = \phi(y) + c_2 y + c_1$, where $c_2$ and $c_1$ are constants. This fact implies that terms such as $\hat{\eta} c_2 $ and $c_1$ in the definition of $\gamma(\cdot)$ (line 236) are redundant.
2. From Theorem 4 in [1], we know that $L(\eta, \mu) = \underline{L}(z) + (\eta - \mu) \underline{L}'(\mu)$.
3. Additionally, we have $\underline{L}(\eta) = - \phi(\eta)$ because $\underline{L} = L(\eta, \eta) = \eta l_1(\eta) + (1 - \eta) l_2(\eta) = \gamma(\eta) = - \phi(\eta)$.
4. Thus, $L(\eta, \Psi^{-1}(y)) = - \phi(\Psi^{-1}(y)) - (\eta - \Psi^{-1}(y)) \phi' (\Psi^{-1}(y))$, where Equation (8) represents the cases for $\eta = 0$ and $\eta = 1$ in this equation.

#### About major weakness 2:
From Corollary 3 in [1] and the discussion in Section 6.1 of [1], it follows that $(g^{-1}_{can})' (c) = w(c) = - \underline{L}''(c) = \phi''(c)$. There appears to be no significant difference between Equation (10) and this equation.

---

[1] Reid, M. D., & Williamson, R. C. (2010). Composite binary losses. The Journal of Machine Learning Research, 11, 2387-2422.

[2] Reid, M. D., & Williamson, R. C. (2011). Information, Divergence and Risk for Binary Experiments. Journal of Machine Learning Research, 12(3).

[3] Menon, A., & Ong, C. S. (2016, June). Linking losses for density ratio and class-probability estimation. In International Conference on Machine Learning (pp. 304-313). PMLR.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the existence of the density ratio always guaranteed? For example, if the two densities have no overlapping support, in which case, the definition seems to fail.  Can the proposed method perform a good estimation?
2. If one distribution has a light tail and the other a heavy tail, how would that impact the estimation?
3. It seems the author considered only a one-dimensional case in the experiment, how about a multi-dimensional case when $d>0$? This is more common in covariate shift problems.
4. What are the biggest difficulties and challenges for deriving the the sample complexity of the proposed methods?
soundness: 3
strengths: 1. Compared to the related literature, this paper introduces a new framework for constructing novel loss functions, prioritizing an accurate estimation of large density ratio values over smaller ones.
2. It provides a thorough mathematical foundation, characterizing the types of loss functions that align with specific error measures derived from Bregman divergences. The comparison with the related literature is good.
3.  The work shows large practical implementation through empirical data and real application in deep domain adaptation. The simulation work is extensive to demonstrate the effectiveness of the proposed method.
summary: The paper addresses the challenge of estimating the ratio of two probability densities from observations. Typically, this is done using binary classifiers, but the efficacy of the estimators is significantly affected by the choice of the binary loss function used. The authors characterize loss functions that result in statistically favorable density ratio estimations, particularly focusing on achieving low errors in large density ratio values—a departure from classical approaches that perform well on small values. They introduce novel loss functions and demonstrate their application in parameter selection for deep domain adaptation tasks. Numerical experiments and real-world applications illustrate the practical benefits of these loss functions.
weaknesses: 1. The paper does not delve into the sample complexity of the proposed methods, which could be critical for understanding their efficiency in various scenarios.
2. While it improves estimation for large density values, the impact on performance for smaller values isn't thoroughly explored.
3. A more detailed introduction of the experiments should be considered.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Theorem 1 extends previous results. What is the main challenge in extending these, and what is the key novelty in the proof?
2. Table 1 shows that EW consistently performs best for Amazon Reviews under Importance Weighted Aggregation. Is there any intuition behind this outcome?
soundness: 3
strengths: The theoretical analysis appears solid, and several real datasets are used to demonstrate the proposed loss function. The authors provided detailed comparison between new results and previous analysis.
summary: This work examines the estimation of the Radon-Nikodym derivative, which is the ratio of two probability densities. In classical algorithms, an incorrect choice of binary loss function can lead to biased estimates. The author first derived the necessary properties for an appropriate loss function. Based on this analysis, novel loss functions were proposed, demonstrating improved parameter selection in both simulated and real data examples.
weaknesses: Although the author discussed how their results improve upon previous work, they did not elaborate on how their proof differs from prior analyses. Consequently, the challenges of the proof, as well as the novelty and contributions of the theoretical analysis, remain unclear. Additionally, the writing could be improved; including more high-level explanations of the motivation and results would make it easier to follow.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: In the conclusion, you mention that the sample complexity of these tasks is not known. Do you mean to say that the empirical risk minimizer of the loss converges to the minimizer of the Bregman divergence only as the number of samples goes to infinity, but we do not know finite-sample error bounds for the empirical minimizer (similar to how in PAC learning theory, this would correspond to an additional "complexity of F"/#samples error term)?
soundness: 4
strengths: The characterization provided by the authors significantly completes the picture laid out in prior work by Menon and Ong 2016, and work by Reid and Williamson (see Remark 1 for specifics). The motivation for considering the inverted problem is convincing---one can imagine ascribing certain desired properties of an estimator to the minimizer of a Bregman divergence, and thereafter, using the characterization derived in the paper, obtain the correct loss function to minimize on the data that realizes the minimization of this Bregman divergence (and hence has the desried property). The authors specifically consider the property "small errors on large density ratios", and obtain strong empirical results for minimizing the loss functions through their characterization. In my view, this is a valuable contribution that enhances our toolbox for estimating density ratios in a principled manner. The writing of the paper is also geneally good, although at some places, it becomes dense with a lot of assumed context.
summary: A standard technique to estimate the ratio between densities of two probability distributions $P$ and $Q$ from samples from each of the distributions is to train a binary classifier that distinguishes the samples. Namely, one labels samples from $P$ with the label $1$, and samples from $Q$ with the label $-1$, and empirically minimizes a loss function on these samples. Depending on the loss function that is chosen, it is known that the as the number of samples tends to infinity, the empirical minimizer effectively minimizes a certain Bregman divergence between the true density ratio and the estimated density ratio.

However, the convex potential of this Bregman divergence depends on the chosen loss function. Namely, the Bregman divergence to the true density ratio that our estimator ends up minimizing depends heavily on the loss function that we chose. As it turns out, most commonly used loss functions (like the logistic loss), correspond to Bregman divergences that do not appropriately penalize discrepancies at large density ratio values, instead penalizing density ratio errors at smaller values---this might lead to suboptimal density ratio estimates in many applications.

This paper takes an inverted approach: given a convex potential $\phi$ (and a ``probability link function'' $g$), the paper characterizes a unique loss function $l_{\phi, g}$, such that upon empirically minimizing $l_{\phi, g}$, as the number of samples grows, the Bregman divergence with potential $\phi$ is minimized. The paper furthermore proposes a canonical link function $g$, which induces a convex loss function $l$, and is hence computationally amenable to empirical risk minimization.

One convenient application of the characterization in this work is that it gives a way for a practitioner to design a loss function that would have the properties they desire. For example, as elaborated in Section 5 by the authors, if one cares about penalizing errors in larger density ratios more than errors in smaller density ratios, one can specify a potential $\phi$ for a Bregman divergence $D_\phi$ that ensures this, and thereafter using the characterization in the paper, obtain the associated loss function $l_\phi$. If one now minimizes $l_\phi$ on the samples, then in the limit, one would be minimizing $D_\phi$ (which was chosen so as to prioritize accurate estimation of large density ratios). In particular, the authors specifically propose two convex potentials $\phi$ for this purpose--the Exponential Weight (EW) function and polynomial weight functions, and derive the associated loss functions from their characterization.

Finally, the authors empirically validate minimizing these loss functions as compared to the standard loss functions on a variety of synthetic as well as real-world datasets. They show that minimizing their loss functions leads to better performance on importance weighting tasks on a range of datasets. The experimental evaluation appears quite thorough and extensive.
weaknesses: Up until you mention your first contribution, the reader has only looked at the pseudocode of Algorithm 1, which uses a probability link function $\Psi$. There has been no mention of $g$ as yet. Correct me if I am wrong, but my understanding is that $g(x)$ in Algorithm 1 is simply $\Psi^{-1}(x)/1-\Psi^{-1}(x)$---it would be helpful to at least mention this before introducing "Density ratio link $g$" in line 78. Because otherwise, the reader, who has just seen $\Psi$ in Algorithm 1, is a little confused about where $g$ sprang out of nowhere, and how it is relevant.

---

Minor/typos: \
Line 78: I believe the loss function for an arbitrary $g$ is not convex, but only strictly roper composite (the loss function for the canonical $g$, as stated in the next sentence, is convex).

Line 273: I believe in the denominator, there is a typo (should be $g_{can}$ instead of $g$)

Meta-review: The paper addresses an important problem, definitely in a proper way (no pun intended). In my decision to accept the paper, I have taken into account the initial opinions and their revised version, the responses of the authors, and the paper (I read it).

It is important to acknowledge that the paper adopts a first principle approach to the DR problem. I would have loved the paper to go all the way forward, up to deriving in a more formal way the content of Section 5, but this part is definitely a tricky bit (this is not to say that there is no principled / general way to address it). At least, the problem of tackling large DRs is approached in a right way, from the standpoint of the design of the loss function and it is an interesting work that tries to alleviate the problem of estimating large DRs by pointing at specific losses using this theory. I commend the authors not just for their result, but also for the non-trivial path adopted. Reviewers (in particular 1wrz) had a legitimate question on the significance of the material brought to the paper, but I think the authors have made a decent explanation of their contribution. Personally, I do not just consider the result but also the *path* chosen by the authors to addressing their problem as a positive contribution which deserves publication. I can only encourage the authors to make this even more clear in the camera ready version.

The authors may remark that addressing the sample complexity of their method may take advantage of the property of Bregman divergences that authorizes to link DR and class probability estimation (a reduction may also be useful to tackle this additional problem). This property is in fact a specific case of a property on the perspective transform of Bregman divergences. For a general treatment and further use, see "A scaled Bregman Theorem with Applications" by Nock, Menon and Ong, NeurIPS'16.

Decision: Accept (Poster)