$\alpha$-Divergence Loss Function for Neural Density Ratio Estimation

## Abstract

Abstract Density ratio estimation (DRE) is a fundamental machine learning technique for capturing relationships between two probability distributions. State-of-the-art DRE methods estimate the density ratio using neural networks trained with loss functions derived from variational representations of f 𝑓 f italic_f -divergence.
However, existing methods face optimization challenges, such as overfitting due to lower-unbounded loss functions, biased mini-batch gradients, vanishing training loss gradients, and high sample requirements for Kullback-Leibler (KL) divergence loss functions.
To address these issues, we focus on α 𝛼 \alpha italic_α -divergence, which provides a suitable variational representation of f 𝑓 f italic_f -divergence.
Subsequently, a novel loss function for DRE, the α 𝛼 \alpha italic_α -divergence loss function ( α 𝛼 \alpha italic_α -Div), is derived. α 𝛼 \alpha italic_α -Div is concise but offers stable and effective optimization for DRE.
The boundedness of α 𝛼 \alpha italic_α -divergence provides the potential for successful DRE with data exhibiting high KL-divergence.
Our numerical experiments demonstrate the effectiveness in optimization using α 𝛼 \alpha italic_α -Div.
However, the experiments also show that the proposed loss function offers no significant advantage over the KL-divergence loss function in terms of RMSE for DRE. This indicates that the accuracy of DRE is
primarily determined by the amount of KL-divergence in the data and is less dependent on α 𝛼 \alpha italic_α -divergence.

## 1 Introduction

Density ratio estimation (DRE), a fundamental technique in various machine learning domains, estimates the density ratio $r^{*}(\mathbf{x})=q(\mathbf{x})/p(\mathbf{x})$ between two probability densities using two sample sets drawn separately from $p$ and $q$. Several machine learning methods, including generative modeling (Goodfellow et al., ; Nowozin et al., ; Uehara et al., ), mutual information estimation and representation learning (Belghazi et al., ; Hjelm et al., ), energy-based modeling (Gutmann & Hyvärinen, ), and covariate shift and domain adaptation (Shimodaira, ; Huang et al., ), involve problems where DRE is applicable. Given its potential to enhance a wide range of machine learning methods, the development of effective DRE techniques has garnered significant attention.

Recently, neural network-based methods for DRE have achieved state-of-the-art results. These methods train neural networks as density ratio functions using loss functions derived from variational representations of $f$-divergence (Nguyen et al., ), which are equivalent to density-ratio matching under Bregman divergence (Sugiyama et al., ). The optimal function for a variational representation of $f$-divergence, through the Legendre transform, corresponds to the density ratio.

However, existing neural network methods suffer from several issues. First, an overfitting phenomenon, termed train-loss hacking by Kato & Teshima (), occurs during optimization when lower-unbounded loss functions are used.
Second, the gradients of loss functions over mini-batch samples provide biased estimates of the full gradient when using naive loss functions derived merely from the variational representation of $f$-divergences (Belghazi et al., ). Third, loss function gradients can vanish when the estimated probability ratios approach zero or infinity (Arjovsky & Bottou, ). Finally, optimization with a Kullback–Leibler (KL)-divergence loss function often fails on high KL-divergence data because the sample requirement for optimization increases exponentially with the true amount of KL-divergence (Poole et al., ; Song & Ermon, ; McAllester & Stratos, ).

To address these problems, this study focuses on $\alpha$-divergence, a subgroup of $f$-divergence, which has a sample complexity independent of its ground truth value. We then present a Gibbs density representation for a variational form of the divergence to obtain unbiased mini-batch gradients, from which we derive a novel loss function for DRE, referred to as the $\alpha$-divergence loss function ($\alpha$-Div). Despite its conciseness, $\alpha$-Div offers stable and effective optimization for DRE.

Furthermore, this study provides technical justifications for the proposed loss function. $\alpha$-Div has a sample complexity that is independent of the ground truth value of $\alpha$-divergence and provides unbiased mini-batch gradients of training losses. Additionally, $\alpha$-Div is lower-bounded when $\alpha$ is within a specific interval, thereby preventing train-loss hacking during optimization. By selecting $\alpha$ from this interval, we also avoid vanishing gradients in neural networks when they reach extreme local minima. We empirically validate our approach through numerical experiments using toy datasets, which demonstrate the stability and efficiency of the proposed loss function during optimization.

However, we observe that the root mean squared error (RMSE) of the estimated density ratios increases significantly for data with higher KL-divergence when using the proposed loss function. The same phenomenon is observed with the KL-divergence loss function. These results suggest that the accuracy of DRE is primarily determined by the amount of KL-divergence in the data and is less influenced by the $\alpha$-divergence, providing insights into the accuracy of downstream tasks in DRE using $f$-divergence loss functions.

The key contributions of this study are as follows: (1) A novel loss function for DRE, $\alpha$-Div, is proposed, offering a concise solution to the instability and biased gradient issues in existing $f$-divergence loss functions. (2) Technical justifications for the proposed loss function are presented. (3) We empirically confirm the stability and efficiency of the proposed loss function in optimization. (4) We find that RMSE in DRE increases significantly as the KL-divergence in the data rises when using the $\alpha$-divergence loss function, indicating that DRE and KL-divergence estimation accuracy is primarily determined by the amount of KL-divergence in the data, rather than the $\alpha$-divergence.

## 2 Related Work

Nguyen et al. () and Sugiyama et al. () proposed DRE and divergence estimation using variational representations of $f$-divergences. Sugiyama et al. () also introduced density-ratio matching under Bregman divergence.
As noted by Sugiyama et al. (), DRE using variational representations of $f$-divergences and density-ratio matching under Bregman divergence are equivalent.
Recently, Kato & Teshima () proposed a correction method for Bregman divergence loss functions to prevent overfitting.
Birrell et al. () derived an $\alpha$-divergence loss function from R’enyi’s $\alpha$-divergence, while Cai et al. () used a naive variational representation of Amari’s $\alpha$-divergence with $\alpha<0$ or $\alpha>1$.
Kwon & Baek () presented the same $\alpha$-divergence loss function as proposed in this study, using the Gibbs density expression to measure entropy in thermodynamics. In contrast, we offer the proposed loss function to address the biased gradient problem.

## 3 Problem Setup

Problem definition. $P$ and $Q$ are probability distributions on $\Omega\subset\mathbb{R}^{d}$ with unknown probability densities $p$ and $q$, respectively.
We assume $p(\mathbf{x})>0\Leftrightarrow q(\mathbf{x})>0$ at almost everywhere $\mathbf{x}\in\Omega$.
The goal of DRE is to accurately estimate $r^{*}(\mathbf{x})=q(\mathbf{x})/p(\mathbf{x})$ from given i.i.d. samples $\hat{\mathbf{X}}_{P[R]}=\{\mathbf{x}_{i}^{p}\}_{i=1}^{R}\sim p$ and $\hat{\mathbf{X}}_{Q[S]}=\{\mathbf{x}_{i}^{q}\}_{i=1}^{S}\sim q$.

Additional notation.
$E_{P}[\cdot]$ denotes an expectation under the distribution $P$:
$E_{P}[\phi(\mathbf{x})]=\int_{\Omega_{p}}\phi(\mathbf{x})dP(\mathbf{x})$, where $\phi(\mathbf{x})$ is a measurable function over $\Omega$.
$\hat{E}_{P[R]}[\cdot]$ denotes the empirical expectation of $\hat{\mathbf{X}}_{P[R]}$:
$\hat{E}_{P[R]}[\phi(\mathbf{x})]=\sum_{i=1}^{R}\phi(\mathbf{x}_{i}^{p})/R$.
Variables of a function or the superscript variable “$[R]$” of $\hat{E}{P[R]}$ are omitted when unnecessary and represented as $E_{P}[\phi]$ or $\hat{E}{P}[\phi]$.
Similarly, notations $E_{Q}[\cdot]$ and $\hat{E}_{Q[S]}[\cdot]$ are defined.
$E[\cdot]$ is written for $E_{P}[E_{Q}[\cdot]]$.

## 4 DRE via f 𝑓 f italic_f -divergence variational representations and its major problems

In this section, we introduce DRE using $f$-divergence variational representations and $f$-divergence loss functions. First, we review the definition of $f$-divergences. Next, we identify four major issues with existing $f$-divergence loss functions: the overfitting problem with lower-unbounded loss functions, biased mini-batch gradients, vanishing training loss gradients, and high sample requirements for Kullback-Leibler (KL) divergence loss functions.

### 4.1 DRE with f 𝑓 f italic_f -divergence variational representation

First, we review the definition of $f$-divergences.

###### Definition 4.1 ( $f$ -divergence) .

The $f$-divergence $D_{f}$ between two probability measures $P$ and $Q$, which is induced by a convex function $f$ satisfying $f(1)=0$, is defined as
$D_{f}(Q||P)=E_{P}[f(q(\mathbf{x})/p(\mathbf{x}))]$.

Many divergences are specific cases obtained by selecting a suitable generator function $f$.
For example, $f(u)=u\cdot\log u$ corresponds to KL-divergence.

Then, we derive the variational representations of $f$-divergences
using the Legendre transform of the convex conjugate of a twice differentiable convex function $f$,
$f^{*}(\psi)=\sup_{u\in\mathbb{R}}\{\psi\cdot u-f(u)\}$ (Nguyen et al., ):

$$ $D_{f}(Q||P)=\sup_{\phi\geq 0}\Big{\{}E_{Q}\big{[}f^{\prime}(\phi)\big{]}-E_{P} \big{[}f^{*}(f^{\prime}(\phi))\big{]}\Big{\}},$ (1) $$

where the supremum is taken over all measurable functions $\phi:\Omega\rightarrow\mathbb{R}$
with $E_{Q}[\,|f^{\prime}(\phi)|\,]<\infty$ and $E_{P}[\,|f^{*}(f^{\prime}(\phi))|\,]<\infty$.
The maximum value is achieved at $\phi(\mathbf{x})=q(\mathbf{x})/p(\mathbf{x})$.

By replacing $\phi$ with a neural network model $\phi_{\theta}$,
the optimal function for Equation (1) is trained
through back-propagation using an $f$-divergence loss function, such that

$$ $\mathcal{L}_{f}^{(R,S)}(\phi_{\theta})=-\left\{\hat{E}_{Q[S]}\big{[}f^{\prime} (\phi_{\theta})\big{]}-\hat{E}_{P[R]}\big{[}f^{*}(f^{\prime}(\phi_{\theta})) \big{]}\right\}$ (2) $$

Where $\phi_{\theta}$ is a real-valued function, the superscript variable “$(R,S)$” is omitted when unnecessary and represented as $\mathcal{L}_{f}(\cdot)$.
As shown in Table 2, we list pairs of convex functions and the corresponding loss functions $\mathcal{L}_{f}(\phi_{\theta})$ in Equation (2) for several $f$-divergences.

###### Definition 4.1 ( f 𝑓 f italic_f -divergence) .

### 4.2 Train-loss hacking problem

When $f$-divergence loss functions $\mathcal{L}_{f}(\phi_{\theta})$, as defined in Equation (2), are not lower-bounded, overfitting can occur during optimization.
For example, in the case of the Pearson $\chi^{2}$ loss function, $\mathcal{L}_{\mathrm{chi}\text{-}\mathrm{sq}}=-2\cdot\hat{E}_{Q}\big{[}\phi_{
\theta}\big{]}+\hat{E}_{P}\big{[}\phi_{\theta}^{2}\big{]}$, overfitting occurs as follows.
Considering the term $-2\cdot\hat{E}_{Q}\big{[}\phi_{\theta}\big{]}$ is not lower-bounded, it can approach negative infinity, causing the entire loss function to diverge to negative infinity as $\phi_{\theta}(\mathbf{x}_{i}^{q})\rightarrow\infty$ for $\mathbf{x}_{i}^{q}\in\hat{\mathbf{X}}_{Q[S]}$.
Therefore, $\mathcal{L}_{\mathrm{chi}\text{-}\mathrm{sq}}\rightarrow-\infty$ when $\phi_{\theta}(\mathbf{x}_{i}^{q})\rightarrow\infty$ for some $\mathbf{x}_{i}^{q}\in\hat{\mathbf{X}}_{Q[S]}$.
As shown in Table 2, both the KL-divergence and Pearson $\chi^{2}$ loss functions are not lower-bounded, and hence, are prone to overfitting during optimization.
This phenomenon has been referred to as train-loss hacking by Kato & Teshima ().

### 4.3 Biased gradient problem

Neural network parameters are optimized by summarizing the gradients of the loss function for each mini-batch. It is desirable for these gradients to be unbiased, i.e., $E\big{[}\nabla_{\theta}\mathcal{L}_{f}(\theta)\big{]}=\nabla_{\theta}E\big{[}
\mathcal{L}_{f}(\theta)\big{]}$ holds. However, naive $f$-divergence loss functions derived solely from Equation (2) often result in biased gradients. To illustrate this, consider the case of a KL-divergence loss function, where $f(u)=u\cdot\log u$.
The loss function is given by $\mathcal{L}_{KL}(\phi_{\theta})=-\hat{E}_{Q}\big{[}\log\phi_{\theta}\big{]}+
\hat{E}_{P}\big{[}\phi_{\theta}\big{]}-1$, and the gradient is expressed as $\nabla_{\theta}\mathcal{L}_{KL}(\phi_{\theta})=-\hat{E}_{Q}\big{[}\nabla_{
\theta}(\log\phi_{\theta})\big{]}+\hat{E}_{P}\big{[}\nabla_{\theta}(\phi_{
\theta})\big{]}$.
Because $E_{Q}\big{[}\cdot\big{]}$ represents an integral over $\Omega$, it does not hold that $\nabla_{\theta}E_{Q}\big{[}\log\phi_{\theta}\big{]}=E_{Q}\big{[}\nabla_{\theta
}(\log\phi_{\theta})\big{]}$.
Conversely, this equality does hold for the sample mean $\hat{E}_{Q}$, as shown above. Therefore, we find that $\nabla_{\theta}E\big{[}\mathcal{L}_{KL}(\phi_{\theta})\big{]}\neq E\big{[}
\nabla_{\theta}\mathcal{L}_{KL}(\phi_{\theta})\big{]}$.
To mitigate this issue, Belghazi et al. () introduced a bias-reduction method for stochastic gradients in KL-divergence loss functions.

### 4.4 Vanishing gradients problem

The vanishing-gradient problem in optimizing divergences is well known in GANs (Arjovsky & Bottou, ). To address the vanishing gradients of training losses, loss functions with penalty terms have been proposed (Gulrajani et al., ; Roth et al., ).

We believe this problem occurs when the following two conditions are satisfied:
(i) The loss function causes slight updates to the model parameters, and
(ii) Updating the model parameters causes minimal changes in the model outputs.
Therefore, the problem can arise when the following relations hold:

$$ $\underbrace{E\big{[}\nabla_{\theta}\mathcal{L}_{f}(\phi_{\theta})\big{]}=\vec{ 0}}_{\text{(i)}}\ \ \text{\&}\ \ \underbrace{E_{Q}\big{[}\nabla_{\theta}\phi_{ \theta}\big{]}=\vec{0}\ \ \text{\&}\ \ E_{P}\big{[}\nabla_{\theta}\phi_{\theta }\big{]}=\vec{0}}_{\text{(ii)}},$ (3) $$

where $\vec{0}$ denotes a vector of the same length as the model gradient.

Consider a case where the estimated density ratio is either very small or very large, which leads to sufficient conditions for Equation (3) to hold.
Table 2 lists the gradient formulas for divergence loss functions from Table 2, along with the asymptotic behavior of the loss gradients as $\phi_{\theta}\rightarrow 0$ or $\phi_{\theta}\rightarrow\infty$.
These results demonstrate that major $f$-divergence loss functions satisfy the conditions for Equation (3), such that $\nabla_{\theta}\mathcal{L}_{f}(\phi_{\theta})\rightarrow c_{1}\cdot E_{Q}\big{
[}\nabla_{\theta}\phi_{\theta}\big{]}+c_{2}\cdot E_{P}\big{[}\nabla_{\theta}
\phi_{\theta}\big{]}$, with constants $c_{1}$ and $c_{2}$, as $\phi_{\theta}\rightarrow 0$ or $\phi_{\theta}\rightarrow\infty$.

In summary, all the divergence loss functions in Tables 2 and 2
can experience vanishing gradients when the estimated density ratio approaches extreme local minima.

### 4.5 Sample size requirement problem for KL-divergence

The sample complexity of KL-divergence is $O(e^{KL(Q||P)})$, which implies that

$$ $\lim_{N\rightarrow\infty}N\cdot\mathrm{Var}\Big{[}\widehat{KL^{N}}(Q||P)\Big{] }\geq e^{KL(Q||P)}-1,$ (4) $$

where $\widehat{KL^{N}}(Q||P)$ represents an arbitrary KL-divergence estimator for sample size $N$ using a variational representation of the divergence,
and $KL(Q||P)$ denotes the true value of KL-divergence (Poole et al., ; Song & Ermon, ; McAllester & Stratos, ).
That is, when using KL-divergence loss functions, the sample size of the training data must increase exponentially as the true amount of KL-divergence increases in order to sufficiently train a neural network.
To address this issue, existing methods divide the estimation of high divergence values into multiple smaller divergence estimations (Rhodes et al., ).

**Table 1: List of $f$-divergence loss functions $\mathcal{L}_{f}(\phi_{\theta})$ in Equation (2) together with convex functions and their status of the lower-boundedness. Part of the list of divergences and their convex functions is based on Nowozin et al. ().**
| Name | convex function $f$ | $\mathcal{L}_{f}(\phi_{\theta})$ | Is lower<br>bounded? |
| --- | --- | --- | --- |
| KL | $u\cdot\log u$ | $-\hat{E}_{Q}\big{[}\log(\phi_{\theta})\big{]}+\hat{E}_{P}\big{[}\phi_{\theta}<br>\big{]}-1$ | No |
| Peason $\chi^{2}$ | $(u-1)^{2}$ | $-2\cdot\hat{E}_{Q}\big{[}\phi_{\theta}\big{]}+\hat{E}_{P}\big{[}\phi_{\theta}^<br>{2}\big{]}$ | No |
| Squared Hellinger | $(\sqrt{u}-1)^{2}$ | $\hat{E}_{Q}\big{[}\phi_{\theta}^{-1/2}\big{]}+\hat{E}_{P}\big{[}\phi_{\theta}^<br>{1/2}\big{]}-2$ | Yes |
| GAN | $u\cdot\log u$<br>$\ -(u+1)\log(u+1)$ | $\hat{E}_{Q}\big{[}\log(1+\phi_{\theta}^{-1})\big{]}$<br>$+\ \hat{E}_{P}\big{[}\log(1+\phi_{\theta})\big{]}$ | Yes |

## 5 DRE using a neural network with an α 𝛼 \alpha italic_α -divergence loss

Figure: Algorithm 1 Training for DRE wtih $\alpha$-Div

In this section, we derive our loss function from a variational representation of $\alpha$-divergence and present the training and prediction methods using this loss function. The exact claims and proofs for all theorems are provided in B.2 in the Appendix.

### 5.1 Derivation of our loss function for DRE

Here, we define $\alpha$-divergence (Amari’s $\alpha$-divergence), which is a subgroup of $f$-divergence, as (Amari & Nagaoka, ):

$$ $D_{\alpha}(Q||P)=E_{P}\left[\frac{1}{\alpha\cdot(1-\alpha)}\cdot\left\{\left( \frac{q(\mathbf{x})}{p(\mathbf{x})}\right)^{1-\alpha}-1\right\}\right],$ (5) $$

where $\alpha\in\mathbb{R}\setminus\{0,1\}$.
From Equation (5), Hellinger divergence is obtained as $\alpha=1/2$, and $\chi^{2}$ divergence by $\alpha=-1$.

Then, we achieve the following variational representation of $\alpha$-divergence :

###### Theorem 5.1 .

A variational representation of $\alpha$-divergence is given as

$$ $D_{\alpha}(Q||P)=\sup_{\phi\geq 0}\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\frac {1}{\alpha}\cdot E_{Q}\Big{[}\phi^{\alpha}\Big{]}-\frac{1}{1-\alpha}\cdot E_{P }\Big{[}\phi^{\alpha-1}\Big{]}\right\},$ (6) $$

where supremum is considered as the overall measurable functions with $E_{P}[\phi^{1-\alpha}]<\infty$ and $E_{Q}[\phi^{-\alpha}]<\infty$.
The maximum value is achieved at $\phi(\mathbf{x})=q(\mathbf{x})/p(\mathbf{x})$.

From the right-hand side of Equation (6), we obtain a naive $\alpha$-divergence loss function as

$$ $\mathcal{L}_{\text{naive}}^{(R,S)}(\phi_{\theta}\,;\,\alpha)=\frac{1}{\alpha} \cdot\hat{E}_{Q}\Big{[}\phi_{\theta}^{\alpha}\Big{]}+\frac{1}{1-\alpha}\cdot \hat{E}_{P}\Big{[}\phi_{\theta}^{\alpha-1}\Big{]}.$ (7) $$

The above loss function has a biased gradient because
$\nabla_{\theta}E_{Q}\big{[}\phi_{\theta}^{\alpha}\big{]}\allowbreak\neq
\allowbreak E_{Q}\big{[}\nabla_{\theta}(\phi_{\theta}^{\alpha})\big{]}$
and $\nabla_{\theta}E_{P}\big{[}\phi_{\theta}^{\alpha-1}\big{]}\allowbreak\neq
\allowbreak E_{P}\big{[}\nabla_{\theta}(\phi_{\theta}^{\alpha-1})\big{]}$ are generally observed.
To obtain unbiased gradients for our function, we rewrite the terms $\phi_{\theta}^{\alpha}$ and $\phi_{\theta}^{\alpha-1}$ of the equation in Gibbs density form.
Then, we have another variational representation of $\alpha$-divergence as

###### Theorem 5.2 .

A variational representation of $\alpha$-divergence is given as

$$ $D_{\alpha}(Q||P)=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha \cdot(1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\Big{[}e^{\alpha\cdot T}\Big{]}- \frac{1}{1-\alpha}\cdot E_{P}\Big{[}e^{(\alpha-1)\cdot T}\Big{]}\right\},$ (8) $$

where supremum is considered as the overall measurable function $T:\Omega\rightarrow\mathbb{R}$ with
$E_{P}[e^{(\alpha-1)\cdot T}]<\infty$ and $E_{Q}[e^{\alpha\cdot T}]<\infty$.
The equality holds for $T^{*}$ satisfying $e^{-T^{*}(\mathbf{x})}=q(\mathbf{x})/p(\mathbf{x})$.

Subsequently, we obtain our loss function for DRE, called $\alpha$-Divergence loss function ($\alpha$-Div).

###### Definition 5.3 ( $\alpha$ -Div) .

$\alpha$-Divergence loss is defined as:

$$ $\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}(T_{\theta}\,;\,\alpha)=\frac{1}{\alpha }\cdot\hat{E}_{Q[S]}\Big{[}e^{\alpha\cdot T_{\theta}}\Big{]}+\frac{1}{1-\alpha }\cdot\hat{E}_{P[R]}\Big{[}e^{(\alpha-1)\cdot T_{\theta}}\Big{]}.$ (9) $$

The superscript “$(R,S)$” is dropped when unnecessary, and is given as $\mathcal{L}_{\alpha\text{-Div}}(T_{\theta}\,;\,\alpha)$.

###### Theorem 5.1 .

###### Theorem 5.2 .

###### Definition 5.3 ( α 𝛼 \alpha italic_α -Div) .

### 5.2 Training and predicting with α 𝛼 \alpha italic_α -Div

We train a neural network with $\alpha$-Div in Algorithm 1.
In practice, neural networks rarely achieve the maximum in Equation (8).
The following theorem suggests that normalization of the estimated values,
$q(\mathbf{x})/p(\mathbf{x})=e^{-T_{\theta}(\mathbf{x})}/\hat{E}_{P}\left[e^{-T
_{\theta}}\right]$, improves the optimization of the neural networks.

###### Theorem 5.4 .

For a fixed function $T:\Omega\rightarrow\mathbb{R}$,
let $c^{*}$ be the optimal scaler value for the following infimum:

$$ $c^{*}=\arg\inf_{c\in\mathbb{R}}E\Big{[}\mathcal{L}_{\alpha\text{-Div}}(T-c) \Big{]}=\arg\inf_{c\in\mathbb{R}}\left\{\frac{1}{\alpha}\cdot E_{Q}\Big{[}e^{ \alpha\cdot(T-c)}\Big{]}+\frac{1}{1-\alpha}\cdot E_{P}\Big{[}e^{(\alpha-1) \cdot(T-c)}\Big{]}\right\}.$ (10) $$

Then, $c^{*}$ satisfies $E_{P}\left[e^{-T-c^{*}}\right]=1$.
That is, $e^{-T-c^{*}}=e^{-T}/E_{P}\left[e^{-T}\right]$.

###### Theorem 5.4 .

## 6 Theoretical results for the proposed loss function

In this section, we provide theoretical results that justify our approach with $\alpha$-Div.
The exact claims and proofs for all the theorems are provided in Section B.3 in the Appendix.

### 6.1 Addressing train-loss hucking problem

**Table 3: Status of the lower-boundedness of $\alpha$-Div for each case of $\alpha<0$, $\alpha>1$ and $0<\alpha<1$.**
| Intervals of $\alpha$ | $\frac{1}{\alpha}\cdot\hat{E}_{Q[S]}$ | $+$ | $\frac{1}{1-\alpha}\cdot\hat{E}_{P}\left[e^{(\alpha-1)\cdot T}\right]$ | $=$ | $\mathcal{L}_{\alpha\text{-Div}}(T;\alpha)$ |
| --- | --- | --- | --- | --- | --- |
| $\alpha<0$ | $\downarrow-\infty$ (as $e^{T}\uparrow\infty$) |  | $\geq 0$ |  | lower-unbounded |
| $\alpha>1$ | $\geq 0$ |  | $\downarrow-\infty$ (as $e^{T}\uparrow\infty$) |  | lower-unbounded |
| $0<\alpha<1$ | $\geq 0$ |  | $\geq 0$ |  | lower-bounded |

$\alpha$-Div avoids the train-loss hucking problem with $\alpha$ within $(0,1)$.
Table 3 summarizes the status of the lower-boundedness of $\alpha$-Div for each case of $\alpha<0$, $0<\alpha<1$ or $\alpha>1$.
$\alpha$-Div is lower-bounded when $0<\alpha<1$ whereas it is not lower-bounded when $\alpha>1$ and $\alpha<0$.
Thus, the train-loss hucking problem is avoided when $\alpha$ is selected from interval $(0,1)$.

### 6.2 Unbiaseness of gradients

The following Theorem 6.1 guarantees unbiased gradients of $\alpha$-Div.

###### Theorem 6.1 (brief and informal) .

Let $T_{\theta}(\mathbf{x}):\Omega\rightarrow\mathbb{R}$ be a function such that
the map $\theta=(\theta_{1},\theta_{2},\ldots,\theta_{p})\in\Theta\mapsto T_{\theta}(
\mathbf{x})$ is differentiable for all $\theta$ and
$\mu$-almost every $\mathbf{x}\in\Omega$.
Then, under some regular conditions for $T_{\theta}$, the following holds

$$ $E\Big{[}\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta};\alpha)|_{ \theta=\bar{\theta}}\Big{]}=\nabla_{\theta}E\Big{[}\mathcal{L}_{\alpha\text{- Div}}(T_{\theta};\alpha)\Big{]}|_{\theta=\bar{\theta}}.$ (11) $$

###### Theorem 6.1 (brief and informal) .

### 6.3 Addressing gradient vanishing problem

**Table 4: List of the destinations of $E\big{[}\nabla_{\theta}\mathcal{L}_{\text{naive}}(\phi_{\theta})\big{]}$ and $E\big{[}\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta})\big{]}$ as estimated probability ratios go to zero or $\infty$ for each case of $\alpha<0$, $\alpha>1$ and $0<\alpha<1$.**
|  | $E\big{[}\nabla_{\theta}\mathcal{L}_{\text{naive}}(\phi_{\theta})\big{]}<br>\rightarrow\text{?}$ | $E\big{[}\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta})\big{]}<br>\rightarrow\text{?}$ |  |  |
| --- | --- | --- | --- | --- |
| Intervals of $\alpha$ | $E_{P}\big{[}\phi_{\theta}\big{]}\rightarrow 0$ | $E_{P}\big{[}\phi_{\theta}\big{]}\rightarrow\infty$ | $E_{P}\big{[}e^{T_{\theta}}\big{]}\rightarrow 0$ | $E_{P}\big{[}e^{T_{\theta}}\big{]}\rightarrow\infty$ |
| $\alpha<0$ | $\vec{\infty}$ | $\mathbf{\vec{0}}$ | $\vec{\infty}-\vec{\infty}$ | $\mathbf{\vec{0}}$ |
| $\alpha>1$ | $\mathbf{\vec{0}}$ | $-\vec{\infty}$ | $\mathbf{\vec{0}}$ | $\vec{\infty}-\vec{\infty}$ |
| $0<\alpha<1$ | $\vec{\infty}$ | $\mathbf{\vec{0}}$ | $-\vec{\infty}$ | $\vec{\infty}$ |

When $\alpha$ is within $(0,1)$, $\alpha$-Div avoids vanishing gradients of its training losses.
Below, we describe how gradient vanishing does not occur in this case.

First, we obtain the gradients of a naive alpha-divergence loss as Equation (7) and $\alpha$-Div.

$$ $\displaystyle\nabla_{\theta}\mathcal{L}_{\text{naive}}(\phi_{\theta})$ $\displaystyle=\hat{E}_{Q}\Big{[}\nabla_{\theta}\phi_{\theta}\cdot{\phi_{\theta }}^{\alpha-1}\Big{]}-\hat{E}_{P}\Big{[}\nabla_{\theta}\phi_{\theta}\cdot{\phi_ {\theta}}^{-\alpha}\Big{]}$ (12) $\displaystyle\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(e^{T_{\theta}})$ $\displaystyle=\hat{E}_{Q}\Big{[}\nabla_{\theta}T_{\theta}\cdot e^{\alpha\cdot T _{\theta}}\Big{]}-\hat{E}_{P}\Big{[}\nabla_{\theta}T_{\theta}\cdot e^{(\alpha- 1)\cdot T_{\theta}}\Big{]}.$ (13) $$

Next, we consider the case in which the estimated probability ratios, $\phi_{\theta}$ and $e^{T_{\theta}}$, are nearly zero or large for some point $\mathbf{x}$.
Notably, under the assumption that $p(\mathbf{x})>0\Leftrightarrow q(\mathbf{x})>0$ for all $\mathbf{x}\in\Omega$, we observe that $E_{Q}[e^{T_{\theta}}]\rightarrow 0\Leftrightarrow E_{P}[e^{T_{\theta}}]\rightarrow
0$ and $E_{Q}[e^{T_{\theta}}]\rightarrow\infty\Leftrightarrow E_{P}[e^{T_{\theta}}]\rightarrow\infty$.
Subsequently, the behavior of $\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta};\alpha)$ when $E_{P}[e^{T_{\theta}}]\rightarrow 0$ or $E_{P}[e^{T_{\theta}}]\allowbreak\rightarrow\allowbreak\infty$, under certain regular conditions for $T_{\theta}$, is summarized in Table 4.

In all cases except for $\alpha$-Div with $0<\alpha<1$, vanishing of the loss gradients
is observed such that $E\big{[}\nabla_{\theta}\mathcal{L}_{\text{naive}}(\phi_{\theta})\big{]}\rightarrow
0$
or $E\big{[}\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta})\big{]}\rightarrow
0$.
This implies that, during optimization, neural networks may remain stuck at extreme local minima when their estimations for density ratios are either 0 0 or $\infty$.
However, this issue is avoided when $\alpha$ is within the interval $(0,1)$.
Additionally, $\alpha$ within the interval $(0,1)$ does not cause numerical instability in cases where $E\big{[}\nabla_{\theta}\mathcal{L}_{\alpha\text{-Div}}(T_{\theta};\alpha)\big{
]}\rightarrow\vec{\infty}-\vec{\infty}$ for $\alpha>1$ and $\alpha<0$.

### 6.4 Sample size requirements for optimizing α 𝛼 \alpha italic_α -divergence

We present the exact upper bound of the sample requirement for minimizing $\alpha$-Div in Theorem 6.2, which corresponds to Equation (4) for KL-divergence loss functions.
The sample requirement for minimizing $\alpha$-Div is upper-bounded depending on the value of $\alpha$.
Intuitively, this property arises from the boundedness of Amari’s $\alpha$-divergence:
$0\leq D_{\alpha}\leq 1/(\alpha\cdot(1-\alpha))$.

###### Theorem 6.2 .

Let $T^{*}=-\log(q(\mathbf{x})/p(\mathbf{x}))$ and $N=\min\{R,S\}$. Subsequently, let

$$ $\hat{D}^{(N)}(Q||P\,;\,\alpha)=\frac{1}{\alpha\cdot(1-\alpha)}-\mathcal{L}^{(N ,N)}_{\alpha\text{-Div}}(T^{*}\,;\,\alpha).$ (14) $$

Then,

$$ $\sqrt{N}\cdot\left\{\hat{D}^{(N)}(Q||P\,;\,\alpha)-D(Q||P\,;\,\alpha)\right\} \xrightarrow{\ \ d\ \ }\mathcal{N}\big{(}0,\sigma_{\alpha}\big{)}$ (15) $$

holds, where

$$ $\displaystyle\sigma_{\alpha}^{2}$ $\displaystyle=C^{1}_{\alpha}\cdot D(Q||P\,;\,2\alpha)+C^{2}_{\alpha}\cdot D(Q| |P\,;\,2\alpha-1)$ $\displaystyle\ \,\quad+C^{3}_{\alpha}\cdot D(Q||P\,;\,\alpha)^{2}+C^{4}_{ \alpha}\cdot D(Q||P\,;\,\alpha)+C^{5}_{\alpha},$ $$

and $C^{1}_{\alpha}=2\alpha\cdot(1-2\alpha)/\alpha^{2}$, $C^{2}_{\alpha}=2\alpha\cdot(1-2\alpha)/(1-\alpha)^{2}$,
$C^{3}_{\alpha}=-1/\alpha^{2}-1/(1-\alpha)^{2}$, $C^{4}_{\alpha}=2/\alpha^{2}+2/(1-\alpha)^{2}$,
and $C^{5}_{\alpha}=(1/\alpha^{2}+1/(1-\alpha)^{2})\cdot(2-2\alpha\cdot(1-\alpha))$.

Unfortunately, despite the sample requirement stated in Equation (15), we empirically find that the estimation accuracy for $\alpha$-Div and KL-divergence loss functions is roughly the same in downstream tasks of DRE, including KL-divergence estimation, as discussed in Section 7.3.

###### Theorem 6.2 .

## 7 Experiments

We evaluated the performance of our approach using synthetic datasets. First, we assessed the stability of the proposed loss function due to its lower-boundedness for $\alpha$ within $(0,1)$. Second, we validated the effectiveness of our approach in addressing biased gradients in the training losses.
Finally, we examined the $\alpha$-divergence loss function for DRE using high KL-divergence data. Details on the experimental settings and neural network training are provided in Section C in the Appendix.

In addition to the results presented in this section, we conducted two additional experiments: a comparison of $\alpha$-Div with existing DRE methods and experiments using real-world data. These additional experiments are reported in Section D in the Appendix.

### 7.1 Experiments on the Stability of Optimization for Different Values of α 𝛼 \alpha italic_α

We empirically confirmed the stability of optimization using $\alpha$-Div, as discussed in Section 6.1:This includes addressing the potential divergence of training losses for $\alpha>1$ and $\alpha<0$, and observing the stability of optimization when $\alpha$ is within $(0,1)$.
Subsequently, we conducted experiments using synthetic datasets to examine the behavior of training losses during optimization across different values of $\alpha$ at each learning step.

Experimental Setup.
First, we generated 100 training datasets from two 5-dimensional normal distributions, $P=\mathcal{N}(\mu_{p},I_{5})$ and $Q=\mathcal{N}(\mu_{q},\Sigma_{q})$, where $\mu_{p}=\mu_{q}=(0,0,\ldots,0)$, and $I_{5}$ denotes the $5$-dimensional identity matrix. The covariance matrix $\Sigma_{q}=(\sigma_{ij})_{i=1}^{5}$ is defined as
$\sigma_{ii}=1$, and $\sigma_{ij}=0.8$ for $i\neq j$.
Subsequently, we trained neural networks using the synthetic datasets by optimizing $\alpha$-Div for $\alpha=-3.0,\allowbreak-2.0,\allowbreak-1.0,\allowbreak 0.2,\allowbreak 0.5,
\allowbreak 0.8,\allowbreak 2.0,\allowbreak 3.0$, and $4.0$, while measuring training losses at each learning step. For each value of $\alpha$, 100 trials were performed.
For each value of $\alpha$, 100 trials were performed.
Finally, we reported the median of the training losses at each learning step, along with the ranges between the 45th and 55th percentiles and between the 2.5th and 97.5th percentiles.

Results.
As shown in Figure 2, we present the training losses of $\alpha$-Div across the learning steps for $\alpha=-2.0,\allowbreak 3.0$, and $0.5$. Results for other values of $\alpha$ are provided in Section C.1 in the Appendix.
The figures on the left ($\alpha=-2.0$) and in the center ($\alpha=3.0$) show that the training losses diverged to negative infinity when $\alpha<0$ or $\alpha>1$. In contrast, the figure on the right ($\alpha=0.5$) demonstrates that the training losses successfully converged.
These results highlight the stability of $\alpha$-Div’s optimization when $\alpha$ is within the interval $(0,1)$, as discussed in Section 6.1.

### 7.2 Experiments on the Improvement of Optimization Efficiency by Removing Gradient Bias

Figure: Figure 1: Results from Section 7.1. The left ($\alpha=-2.0$), center ($\alpha=3.0$), and right ($\alpha=0.5$) graphs show the training losses (y-axis) over learning steps (x-axis) during optimization using $\alpha$-Div for different $\alpha$ values. The solid blue line represents the median training losses, the dark blue shaded area shows the 45th to 55th percentiles, and the light blue shaded area represents the 2.5th to 97.5th percentiles.
Refer to caption: x1.png

Unbiased gradients of loss functions are expected to enhance the optimization of neural network parameters by ensuring that updates are made in directions that are closer to the ideal ones at each iteration, compared to biased gradient loss functions. We empirically compared the efficiency of minimizing training losses between the proposed loss function and the naive $\alpha$-divergence loss function derived from Equation (12). Our observations indicated that the proposed loss function was more effective in minimizing training loss than the biased-gradient $\alpha$-divergence loss function. Additionally, we found that the estimated density ratios using the naive $\alpha$-divergence loss function remained close to zero, whereas those obtained using $\alpha$-Div did not. This finding is consistent with the discussion in Section 6.3.

Experimental Setup. We first generated 100 training datasets from two normal distributions, $P=\mathcal{N}(\mu_{p},I_{5})$ and $Q=\mathcal{N}(\mu_{q},I_{5})$, where $I_{5}$ denotes the 5-dimensional identity matrix. The means were set as $\mu_{p}=(-5/2,0,0,0,0)$ and $\mu_{q}=(5/2,0,0,0,0)$. We then trained neural networks using three different loss functions: the naive $\alpha$-divergence loss function defined in Equation (12), $\alpha$-Div, and deep direct DRE (D3RE) (Kato & Teshima, ). Training losses were measured at each learning step. D3RE is designed to prevent issues such as train-hacking associated with Bregman divergence loss functions, as described in Section 4.4, by addressing the lower-unboundedness of the loss functions. The hyperparameter for D3RE was set to $C=2$. We chose the Bregman divergence least-squares importance fitting (nnBD-LSIF) loss function for D3RE due to its unbiased gradient and expected stable optimization. For both the naive $\alpha$-divergence loss and $\alpha$-Div, we used $\alpha=0.1$. Finally, we reported the median training losses at each learning step, along with ranges between the 45th and 55th percentiles and between the 2.5th and 97.5th percentiles.

##### Results.

Figure 2 illustrates the training losses at each learning step for each loss function. The center and right panels show that $\alpha$-Div and nnBD-LSIF are more effective at minimizing training losses compared to the naive $\alpha$-divergence loss function.
These findings indicate that the unbiased gradient of $\alpha$-Div, like nnBD-LSIF, leads to more efficient neural network optimization than the biased gradient of the naive $\alpha$-divergence loss function. Additionally, while the training losses for the naive $\alpha$-divergence loss function diverged after 400 steps, those for $\alpha$-Div remained stable. According to Equation (12), the estimated density ratio $\phi_{\theta}$ in Equation (12) approaches 0 0 or $\infty$ as $\mathcal{L}_{\text{naive}}(\phi_{\theta})\rightarrow\infty$. However, $\phi_{\theta}\rightarrow\infty$ does not occur because there is neither a stable point where $\mathcal{L}_{\text{naive}}(\phi_{\theta})\rightarrow c$ for some constant $c$ nor does $\mathcal{L}_{\text{naive}}(\phi_{\theta})$ decrease monotonically to an extremely negative value. Therefore, it is considered that $\phi_{\theta}\rightarrow 0$, which causes the gradients of $\mathcal{L}_{\text{naive}}(\phi_{\theta})$ to vanish, i.e., $\nabla_{\theta}\mathcal{L}_{\text{naive}}(\phi_{\theta})\rightarrow 0$, as shown for $0<\alpha<1$ in Table 4. In contrast, this issue was not observed for $\alpha$-Div, consistent with the discussion in Section 6.3.

### 7.3 Experiments on the Estimation Accuracy Using High KL-Divergence Data

Figure: Figure 3: Results of Section 7.3. The $x$-axis represents the ground truth KL-divergence of the data. The $y$-axis of the left and right graphs represents the RMSE and estimated KL-divergence, respectively. The plot shows the median $y$-axis values for the ground truth KL-divergence. Vertical lines indicate the interquartile range (25th to 75th percentiles) of the $y$-axis values.
Refer to caption: x3.png

In Section 4.5, we examined the $\alpha$-divergence loss function, hypothesizing that its boundedness could address sample size issues in high KL-divergence data. Theorem 6.2, based on this boundedness, suggests that $\alpha$-Div can be minimized regardless of the true KL-divergence, indicating its potential for effective DRE with high KL-divergence data. To validate this hypothesis, we assessed DRE and KL-divergence estimation accuracy using both $\alpha$-Div and a KL-divergence loss function.

Unfortunately, the RMSE of DRE using $\alpha$-Div increased significantly with higher KL-divergence in the test datasets, similar to the results obtained with the KL-divergence loss function. Additionally, both methods produced nearly identical KL-divergence estimations. These findings suggest that the accuracy of DRE and KL-divergence estimation is primarily influenced by the true amount of KL-divergence in the data rather than the $\alpha$-divergence.

Experimental Setup. We generated 100 training and 100 test datasets, each containing 10,000 samples. The datasets were drawn from two normal distributions, $P=\mathcal{N}(\mu_{p},\sigma^{2}\cdot I_{3})$ and $Q=\mathcal{N}(\mu_{q},4^{2}\cdot I_{3})$, where $\mu_{p}=(-3/2,-3/2,-3/2)$ and $\mu_{q}=(3/2,3/2,3/2)$, with $I_{3}$ denoting the 3-dimensional identity matrix. The values of $\sigma$ were set to 1.0, 1.1, 1.2, 1.4, 1.6, 2.0, 2.5, and 3.0. Correspondingly, the ground truth KL-divergence values of the datasets were 31.8, 25.6, 21.0, 14.5, 10.4, 5.8, 3.1, and 1.8 nats(^1^11A ’nat’ is a unit of information measured using the natural logarithm (base $e$)), reflecting the increasing $\sigma^{2}$ values. The true density ratios of the test datasets are known for this experimental setup. We trained neural networks on the training datasets by optimizing both $\alpha$-Div with $\alpha=0.5$ and the KL-divergence loss function. After training, we measured the root mean squared error (RMSE) of the estimated density ratios using the test datasets. Additionally, we estimated the KL-divergence of the test datasets based on the estimated density ratios using a plug-in estimator. Finally, we reported the median RMSE of the DRE and the estimated KL-divergence, along with the interquartile range (25th to 75th percentiles), for both the KL-divergence loss function and $\alpha$-Div.

Results. Figure 3 shows the experimental results.
The x-axis represents the true KL-divergence values of the test datasets, while the y-axes of the graphs display the RMSE (left) and estimated KL-divergence (right) for the test datasets.
We empirically observed that the RMSE for DRE using $\alpha$-Div increased significantly as the KL-divergence of the datasets increased. A similar trend was observed for the KL-divergence loss function. Additionally, the KL-divergence estimation results were nearly identical for both methods. These findings indicate that the accuracy of DRE and KL-divergence estimation is primarily determined by the amount of KL-divergence in the data and is less influenced by $\alpha$-divergence.
Therefore, we conclude that the approach discussed in Section 6.4 offers no advantage over the KL-divergence loss function in terms of the RMSE for DRE with high KL-divergence data.
However, we believe that these empirical findings contribute to a deeper understanding of the accuracy of downstream tasks in DRE using $f$-divergence loss functions.

## 8 Conclusion

This study introduced a novel loss function for DRE, $\alpha$-Div, which is both concise and provides stable, efficient optimization. We offered technical justifications and demonstrated its effectiveness through numerical experiments. The empirical results affirmed the efficiency of the proposed loss function. However, experiments with high KL-divergence data revealed that the $\alpha$-divergence loss function does not offer a significant advantage over the KL-divergence loss function in terms of RMSE for DRE. These findings contribute to a deeper understanding of the accuracy of downstream tasks in DRE when using $f$-divergence loss functions.

## Appendix A Organization of the Supplementary Document

The organization of this supplementary document is as follows:
Section B presents the theorems and proofs referenced in this study.
Section C provides details of the experiments conducted. Section C presents additional experimental results.
Additionally, the code used in the numerical experiments is included as supplementary material.

## Appendix B Proofs

In this section, we present the theorems and proofs referenced in this study. First, we define $\alpha$-Div within a probabilistic theoretical framework. Following that, we provide the theorems and proofs cited throughout the study.

##### Capital, small and bold letters.

Random variables are denoted by capital letters. For example, $X$.
Small letters are used for values of the random variables of the corresponding capital letters;
$a$ denotes a value of the random variable $X$.
Bold letters $\mathbf{X}$ and $\mathbf{x}$ represent sets of random variables and their values.

### B.1 Definition of α 𝛼 \alpha italic_α -Div

###### Definition B.1 ( $\alpha$ -Divergence loss) .

Let $\mathbf{X}^{1}_{P},\mathbf{X}^{2}_{P},\ldots,\mathbf{X}^{R}_{P}$ denote $R$ i.i.d. random variables drawn from $P$, and let
$\mathbf{X}^{1}_{Q},\mathbf{X}^{2}_{Q},\ldots,\mathbf{X}^{S}_{Q}$ denote $S$ i.i.d. random variables drawn from $Q$.
Then, the $\alpha$-Divergence loss $\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}(\cdot\,;\,\alpha)$ is defined as follows:

$$ $\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}(T\,;\,\alpha)=\frac{1}{\alpha}\cdot \frac{1}{S}\cdot\sum_{i=1}^{S}e^{\alpha\cdot T(\mathbf{X}^{i}_{Q})}+\frac{1}{1 -\alpha}\cdot\frac{1}{R}\cdot\sum_{i=1}^{R}e^{(\alpha-1)\cdot T(\mathbf{X}^{i} _{P})},$ (17) $$

where $T$ is a measurable function over $\Omega$ such that $T:\Omega\rightarrow\mathbb{R}$.

###### Definition B.1 ( α 𝛼 \alpha italic_α -Divergence loss) .

### B.2 Proofs for Section 5

In this section, we provide the theorems and proofs referenced in Section 5.

###### Theorem B.2 .

A variational representation of $\alpha$-divergence is given as

$$ $\displaystyle D(Q||P\,;\,\alpha)$ $\displaystyle=\sup_{\phi\geq 0}\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1} {\alpha}\cdot E_{Q}\left[\phi^{-\alpha}\right]-\frac{1}{1-\alpha}\cdot E_{P} \left[\phi^{1-\alpha}\right]\right\},$ (18) $$

where the supremum is taken over all measurable functions with $E_{P}[\phi^{1-\alpha}]<\infty$ and $E_{Q}[\phi^{-\alpha}]<\infty$.
The maximum value is achieved at $\phi=dQ/dP$.

###### Proof of Theorem B.2 .

Let $f_{\alpha}(t)=\{t^{1-\alpha}-(1-\alpha)\cdot t-\alpha\}/\{\alpha\cdot(\alpha-1)\}$ for $\alpha\neq 0,1$, then

$$ $\displaystyle E_{P}\left[f_{\alpha}\left(\frac{dQ}{dP}\right)\right]$ $\displaystyle=E_{P}\left[\frac{1}{\alpha\cdot(1-\alpha)}\cdot\left(\frac{dQ}{ dP}\right)^{1-\alpha}+\frac{1}{\alpha}\cdot\left(\frac{dQ}{dP}\right)+\frac{1} {1-\alpha}\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}\cdot E_{P}\left[\left(\frac{dQ}{ dP}\right)^{1-\alpha}\right]+\frac{1}{\alpha}+\frac{1}{1-\alpha}$ $\displaystyle=D(Q||P\,;\,\alpha).$ (19) $$

Note that, the Legendre transform of $g_{\alpha}(x)=x^{1-\alpha}/(1-\alpha)$ is obtained as

$$ $g_{\alpha}^{*}(x)=\frac{\alpha}{\alpha-1}\cdot x^{1-\frac{1}{\alpha}},$ (20) $$

and for the Legendre transforms of functions, it holds that

$$ $\{C\cdot h(x)\}^{*}=C\cdot h^{*}\left(\frac{x}{C}\right)\quad\text{and}\quad\{ h(x)+C\cdot x+D\}^{*}=h^{*}(x-C)-D.$ (21) $$

Here, $A^{*}$ denotes the Legendre transform of $A$.

From Equations (20) and (21), we have

$$ $\displaystyle f_{\alpha}^{*}(t)$ $\displaystyle=\left\{\frac{1}{(-\alpha)}\cdot g_{\alpha}(t)+\frac{1}{\alpha} \cdot t+\frac{1}{1-\alpha}\right\}^{*}$ $\displaystyle=\frac{1}{(-\alpha)}\cdot g_{\alpha}^{*}\left(-\alpha\cdot\left\{ t-\frac{1}{\alpha}\right\}\right)-\frac{1}{1-\alpha}$ $\displaystyle=-\frac{1}{\alpha}\cdot g_{\alpha}^{*}\left(1-\alpha t\right)+ \frac{1}{\alpha-1}$ $\displaystyle=-\frac{1}{\alpha}\cdot\left\{\frac{\alpha}{\alpha-1}\cdot\left(1 -\alpha t\right)^{1-\frac{1}{\alpha}}\right\}+\frac{1}{\alpha-1}$ $\displaystyle=\frac{1}{1-\alpha}\cdot\left(1-\alpha t\right)^{1-\frac{1}{ \alpha}}+\frac{1}{\alpha-1}.$ (22) $$

By differentiating $f_{\alpha}(t)$, we obtain

$$ $f_{\alpha}^{\prime}(t)=-\frac{1}{\alpha}\cdot t^{-\alpha}+\frac{1}{\alpha}.$ (23) $$

Thus,

$$ $\displaystyle E_{Q}\big{[}f_{\alpha}^{\prime}(\phi)\big{]}=E_{Q}\left[-\frac{1 }{\alpha}\cdot\phi^{-\alpha}+\frac{1}{\alpha}\right].$ (24) $$

From (22) and (23), we have

$$ $\displaystyle E_{P}\big{[}f_{\alpha}^{*}(f_{\alpha}^{\prime}(\phi))\big{]}$ $\displaystyle=E_{P}\left[\frac{1}{1-\alpha}\cdot\left\{1-\alpha\cdot\left(- \frac{1}{\alpha}\cdot\phi^{-\alpha}+\frac{1}{\alpha}\right)\right\}^{1-\frac{1 }{\alpha}}+\frac{1}{\alpha-1}\right]$ $\displaystyle=E_{P}\left[\frac{1}{1-\alpha}\cdot\phi^{1-\alpha}+\frac{1}{ \alpha-1}\right].$ (25) $$

In addition, from Equations (24) and (25),
we observe that $E_{P}\big{[}\phi^{1-\alpha}\big{]}<\infty$ is equivalent to
$E_{P}\big{[}\,\big{|}f_{\alpha}^{*}(f_{\alpha}^{\prime}(\phi))\big{|}\,\big{]}<\infty$.
Similarly, $E_{Q}\big{[}\phi^{-\alpha}\big{]}<\infty$ is equivalent to $E_{Q}\big{[}\,\big{|}f_{\alpha}^{\prime}(\phi)\big{|}\,\big{]}<\infty$.

Finally, by substituting Equations (24) and (25) into Equation (1), we get

$$ $\displaystyle D(Q||P\,;\,\alpha)$ $\displaystyle=\sup_{\phi\geq 0}\Big{\{}E_{Q}\big{[}f_{\alpha}^{\prime}(\phi) \big{]}-E_{P}\big{[}f_{\alpha}^{*}(f_{\alpha}^{\prime}(\phi))\big{]}\Big{\}}$ $\displaystyle=\sup_{\phi\geq 0}\left\{E_{Q}\left[-\frac{1}{\alpha}\cdot\phi^{- \alpha}+\frac{1}{\alpha}\right]-E_{P}\left[\frac{1}{1-\alpha}\cdot\phi^{1- \alpha}+\frac{1}{\alpha-1}\right]\right\}$ $\displaystyle=\sup_{\phi\geq 0}\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1} {\alpha}\cdot E_{Q}\left[\phi^{-\alpha}\right]-\frac{1}{1-\alpha}\cdot E_{P} \left[\phi^{1-\alpha}\right]\right\}.$ $$

This completes the proof.
∎

###### Theorem B.3 (Theorem 5.2 in Section 5 restated) .

The $\alpha$-divergence is represented as

$$ $D(Q||P\,;\,\alpha)=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha \cdot(1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T}\right]- \frac{1}{1-\alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T}\right]\right\},$ (26) $$

where the supremum is taken over all measurable functions $T:\Omega\rightarrow\mathbb{R}$ with
$E_{P}[e^{(\alpha-1)\cdot T}]<\infty$ and $E_{Q}[e^{\alpha\cdot T}]<\infty$.
The equality holds for $T^{*}$ satisfying

$$ $\frac{dQ}{dP}=e^{-T^{*}}.$ (27) $$

###### proof of Theorem 5.2 .

Substituting $e^{-T}$ into $\phi$ in Equation (18), we have

$$ $\displaystyle D(Q||P\,;\,\alpha)$ $\displaystyle=\sup_{\phi\geq 0}\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1} {\alpha}\cdot E_{Q}\left[\phi^{-\alpha}\right]-\frac{1}{1-\alpha}\cdot E_{P} \left[\phi^{1-\alpha}\right]\right\}$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha\cdot( 1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left[\left\{e^{-T}\right\}^{-\alpha} \right]-\frac{1}{1-\alpha}\cdot E_{P}\left[\left\{e^{-T}\right\}^{1-\alpha} \right]\right\}$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha\cdot( 1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T}\right]-\frac{1}{ 1-\alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T}\right]\right\}.$ (28) $$

Finally, from Theorem B.2, the equality for Equation (28) holds if and only if

$$ $\frac{dQ}{dP}=e^{-T^{*}}.$ (29) $$

This completes the proof.
∎

###### Lemma B.4 .

For a measurable function $T:\Omega\rightarrow\mathbb{R}$ with $E_{P}[e^{(\alpha-1)\cdot T}]<\infty$ and $E_{Q}[e^{\alpha\cdot T}]<\infty$, let

$$ $\displaystyle\widetilde{l}_{\alpha\text{-Div}}\left(T(\mathbf{x})\,;\,\alpha\right)$ $\displaystyle=\frac{1}{\alpha}\cdot e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ }{d\mu}(\mathbf{x})+\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot T(\mathbf{x})} \cdot\frac{dP}{d\mu}(\mathbf{x}).$ (30) $$

Then the optimal function $T^{*}$ for $\inf_{T:\Omega\rightarrow\mathbb{R}}\widetilde{l}_{\alpha\text{-Div}}\left(T\,
;\,\alpha\right)$ is obtained as $T^{*}=-\log dQ/dP$, $\mu$-almost everywhere.

###### proof of Lemma B.4 .

First, note that it follows from Jensen’s inequality that

$$ $\log(p\cdot X+q\cdot Y)\geq p\cdot\log(X)+q\cdot\log(Y),$ (31) $$

for $X,Y>0$ and $p,q>0$ with $p+q=1$, and equality holds when $X=Y$.

Substitute $X=e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ}{d\mu}(\mathbf{x})$,
$Y=e^{(\alpha-1)\cdot T(\mathbf{x})}\cdot\frac{dP}{d\mu}(\mathbf{x})$,
$p=1-\alpha$, and $q=\alpha$ into Equation (31), we obtain

$$ $\log(p\cdot X+q\cdot Y)=\log\left(\frac{1}{\alpha\cdot(1-\alpha)}\cdot \widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right),$ $$

and $\log\left(\frac{1}{\alpha\cdot(1-\alpha)}\cdot\widetilde{l}_{\alpha\text{-Div}
}(T\,;\,\alpha)\right)$ is minimized when
$e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ}{d\mu}(\mathbf{x})=e^{(\alpha-1)
\cdot T(\mathbf{x})}\cdot\frac{dP}{d\mu}(\mathbf{x})$,
$\mu$-almost everywhere.
Therefore, $\inf_{T:\Omega\rightarrow\mathbb{R}}\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)$ is achieved at $e^{-T^{*}(\mathbf{x})}=\frac{dQ}{dP}(\mathbf{x})$, $\mu$-almost everywhere.
Thus, we obtain $T^{*}(\mathbf{x})=-\log\frac{dQ}{dP}(\mathbf{x})$, $\mu$-almost everywhere.

This completes the proof.
∎

###### Lemma B.5 .

For a measurable function $T:\Omega\rightarrow\mathbb{R}$ with $E_{P}[e^{(\alpha-1)\cdot T}]<\infty$ and $E_{Q}[e^{\alpha\cdot T}]<\infty$, let

$$ $\displaystyle\macc@depth\char 1\relax\frozen@everymath{\macc@group} \macc@set@skewchar\macc@nested@a 111{\mathcal{L}}_{\alpha\text{-Div}}(T\,;\,\alpha$ $\displaystyle=E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}\left(T(\mathbf{x} \,;\,\alpha)\right)\right]$ (32) $\displaystyle=\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T(\mathbf{x})} \right]+\frac{1}{1-\alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T(\mathbf{x})} \right],$ (33) $$

and let

$$ $\displaystyle\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha)$ $\displaystyle=\inf_{T:\Omega\rightarrow\mathbb{R}}\widetilde{l}_{\alpha\text{- Div}}(T\,;\,\alpha),\quad\text{and}$ (34) $\displaystyle\macc@depth\char 1\relax\frozen@everymath{\macc@group} \macc@set@skewchar\macc@nested@a 111{\mathcal{L}}_{\alpha\text{-Div}}^{*}(\alpha)$ $\displaystyle=\inf_{T:\Omega\rightarrow\mathbb{R}}\macc@depth\char 1\relax \frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{\mathcal{L} }_{\alpha\text{-Div}}(T\,;\,\alpha),$ (35) $$

where the infima of Equations (34) and (35) are considered over measurable functions $T:\Omega\rightarrow\mathbb{R}$ with $E_{P}[e^{(\alpha-1)\cdot T}]<\infty$ and $E_{Q}[e^{\alpha\cdot T}]<\infty$.

Then,

$$ $E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha)\right]=\macc@depth \char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 11 1{\mathcal{L}}_{\alpha\text{-Div}}^{*}(\alpha).$ (36) $$

Additionally, the equality in Equations (34) and (35) holds for $T^{*}(\mathbf{x})=-\log dQ/dP(\mathbf{x})$.

###### proof of Lemma B.5 .

Let $T^{*}(\mathbf{x})=-\log dQ/dP(\mathbf{x})$.

First, it follows from Lemma B.4 that

$$ $\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha)=\inf_{T:\Omega\rightarrow\mathbb {R}}\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)=\widetilde{l}_{\alpha\text {-Div}}(T^{*}\,;\,\alpha).$ (37) $$

Next, we obtain

$$ $\displaystyle E_{\mu}\left[\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{ \alpha\text{-Div}}^{*}(\alpha)\right]$ $\displaystyle=\int\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha \text{-Div}}^{*}(\alpha)\right\}d\mu$ $\displaystyle=\int\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega \rightarrow\mathbb{R}}\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right\}d\mu$ $\displaystyle=\int\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha \cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right\}d\mu.$ (38) $$

Let $T_{k}=T^{*}+1/k$.
Note that,

$$ $\lim_{k\rightarrow\infty}\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)= \inf_{T:\Omega\rightarrow\mathbb{R}}\widetilde{l}_{\alpha\text{-Div}}(T\,;\, \alpha)=\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha).$ (39) $$

Then, we have

$$ $\displaystyle\lim_{k\rightarrow\infty}\bigg{\{}\frac{1}{\alpha\cdot(1-\alpha)} -\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\bigg{\}}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\lim_{k\rightarrow\infty}\bigg{ \{}\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\bigg{\}}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega\rightarrow\mathbb {R}}\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\bigg{\{}\frac{1}{\alpha \cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\bigg{\}}.$ (40) $$

From Theorem B.3, we have

$$ $\displaystyle\lim_{k\rightarrow\infty}E_{\mu}\left[\frac{1}{\alpha\cdot(1- \alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\lim_{k\rightarrow\infty}E_{\mu} \left[\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\lim_{k\rightarrow\infty}\left\{ \frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T_{k}}\right]+\frac{1}{1- \alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T_{k}}\right]\right\}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega\rightarrow\mathbb {R}}\left\{\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T}\right]+\frac{1}{ 1-\alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T}\right]\right\}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega\rightarrow\mathbb {R}}E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right]$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha\cdot( 1-\alpha)}-E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right]\right\}$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}E_{\mu}\left[\frac{1}{\alpha \cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\right].$ (41) $$

Now, we have

$$ $\displaystyle\left|\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{ -Div}}(T_{k}\,;\,\alpha)\right|$ $\displaystyle=\ \left|\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot \left(\frac{dQ}{dP}\left(\mathbf{x}\right)\right)^{\alpha}\cdot\frac{dQ}{d\mu} \left(\mathbf{x}\right)\cdot e^{\frac{\alpha}{k}}\right.$ $\displaystyle\ \qquad\left.-\frac{1}{1-\alpha}\cdot\left(\frac{dQ}{dP}\left( \mathbf{x}\right)\right)^{\alpha-1}\cdot\frac{dP}{d\mu}\left(\mathbf{x}\right) \cdot e^{\frac{\alpha-1}{k}}\right|$ $\displaystyle\leq\ \frac{1}{\alpha\cdot(1-\alpha)}+\frac{1}{\alpha}\cdot e^{ \frac{\alpha}{k}}\cdot\left(\frac{dQ}{dP}\left(\mathbf{x}\right)\right)^{ \alpha}\cdot\frac{dQ}{d\mu}\left(\mathbf{x}\right)$ $\displaystyle\ \qquad+\frac{1}{1-\alpha}\cdot e^{\frac{\alpha-1}{k}}\cdot\left (\frac{dQ}{dP}\left(\mathbf{x}\right)\right)^{\alpha-1}\cdot\frac{dP}{d\mu} \left(\mathbf{x}\right).$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}+\left\{\frac{1}{\alpha}\cdot e^{ \frac{\alpha}{k}}+\frac{1}{1-\alpha}\cdot e^{\frac{\alpha-1}{k}}\right\}\cdot \left(\frac{dQ}{dP}\left(\mathbf{x}\right)\right)^{\alpha-1}\cdot\frac{dP}{d \mu}\left(\mathbf{x}\right),$ (42) $$

and let $\phi(\mathbf{x})$ denote the term on the right hand side of Equation (42).

Then, we observe that

$$ $\left|\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k}( \mathbf{x})\,;\,\alpha)\right|\leq\phi(\mathbf{x})\quad\text{and}\quad E_{\mu} \big{[}\phi(\mathbf{x})\big{]}<\infty.$ $$

That is, the following sequence is uniformly integrable for $\mu$:

$$ $\left\{\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k} \,;\,\alpha)\right\}_{k=1}^{\infty}.$ $$

Thus, from the property of the Lebesgue integral (, P188, Theorem 4), we have

$$ $E_{\mu}\left[\lim_{k\rightarrow\infty}\left\{\frac{1}{\alpha\cdot(1-\alpha)}- \widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right\}\right]=\lim_{k \rightarrow\infty}E_{\mu}\left[\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{ \alpha\text{-Div}}(T_{k}\,;\,\alpha)\right].$ (43) $$

From Equations (40), (41) and (43), we
obtain

$$ $\displaystyle\frac{1}{\alpha\cdot(1-\alpha)}-E_{\mu}\left[\widetilde{l}_{ \alpha\text{-Div}}^{*}(\alpha)\right]$ $\displaystyle=E_{\mu}\left[\frac{1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{ \alpha\text{-Div}}^{*}(\alpha)\right]$ $\displaystyle=E_{\mu}\left[\sup_{T:\Omega\rightarrow\mathbb{R}}\bigg{\{}\frac{ 1}{\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T\,;\,\alpha)\bigg {\}}\ \right]$ $\displaystyle=E_{\mu}\left[\lim_{k\rightarrow\infty}\left\{\frac{1}{\alpha \cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right\}\right]$ ( $\therefore$ Equation ( 40 )) $\displaystyle=\lim_{k\rightarrow\infty}E_{\mu}\left[\frac{1}{\alpha\cdot(1- \alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right]\qquad\text {($\therefore$ Equation (\ref{exchange_integral_sup}))}$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{E_{\mu}\left[\frac{1} {\alpha\cdot(1-\alpha)}-\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha) \right]\right\}$ ( $\therefore$ Equation ( 41 )) $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega\rightarrow\mathbb {R}}E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}(T_{k}\,;\,\alpha)\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\macc@depth\char 1\relax \frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{\mathcal{L} }_{\alpha\text{-Div}}^{*}(\alpha).$ $$

Here, we have

$$ $E_{\mu}\left[\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha)\right]=\macc@depth \char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 11 1{\mathcal{L}}_{\alpha\text{-Div}}^{*}(\alpha).$ (44) $$

From Equations
(32) and
(44, we have

$$ $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar \macc@nested@a 111{\mathcal{L}}_{\alpha\text{-Div}}(T^{*}\,;\,\alpha)=E_{\mu} \left[\widetilde{l}_{\alpha\text{-Div}}^{*}(\alpha)\right]=\macc@depth\char 1 \relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{ \mathcal{L}}_{\alpha\text{-Div}}^{*}(\alpha).$ (45) $$

This completes the proof.
∎

###### Theorem B.6 (Theorem 5.4 in Section 5 restated) .

For a fixed function $T:\Omega\rightarrow\mathbb{R}$,
let $c_{*}$ be the optimal scalar value for the following infimum:

$$ $\displaystyle c_{*}$ $\displaystyle=\arg\inf_{c\in\mathbb{R}}E[\mathcal{L}_{\alpha\text{-Div}}(T+c\, ;\,\alpha)]$ $\displaystyle=\arg\inf_{c\in\mathbb{R}}\left\{\frac{1}{\alpha}E_{Q}\left[e^{ \alpha\cdot(T+c)}\right]\right.$ $\displaystyle\left.\quad\qquad+\quad\frac{1}{1-\alpha}E_{P}\left[e^{(\alpha-1) \cdot(T+c)}\right]\right\},$ (46) $$

Then, $c_{*}$ satisfies $e^{c_{*}}=E_{P}\left[e^{-T}\right]$, or
equivalently, $e^{-(T+c_{*})}=e^{-T}/E_{P}\left[e^{-T}\right]$.

###### proof of Theorem B.6 .

Now, we have

$$ $\widetilde{l}_{\alpha\text{-Div}}(T+c\,;\,\alpha)=\frac{1}{\alpha}\cdot e^{ \alpha\cdot c}\cdot e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ}{d\mu}(\mathbf{ x})+\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot c}\cdot e^{(\alpha-1)\cdot T( \mathbf{x})}\cdot\frac{dP}{d\mu}(\mathbf{x}).$ $$

For Equation (31), let
$X=e^{\alpha\cdot c}\cdot e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ}{d\mu}(
\mathbf{x})$,
$Y=e^{(\alpha-1)\cdot c}\cdot e^{(\alpha-1)\cdot T(\mathbf{x})}\cdot\frac{dP}{d
\mu}(\mathbf{x})$,
$p=1-\alpha$ and $q=\alpha$.
Then, from Jensen’s inequality, $\widetilde{l}_{\alpha\text{-Div}}\left(T+c\,;\,\alpha\right)$ is minimized at $c_{*}$
such taht $e^{\alpha\cdot c_{*}}\cdot e^{\alpha\cdot T(\mathbf{x})}\cdot\frac{dQ}{d\mu}(
\mathbf{x})=e^{(\alpha-1)\cdot c_{*}}\cdot e^{(\alpha-1)\cdot T(\mathbf{x})}
\cdot\frac{dP}{d\mu}(\mathbf{x})$,
$\mu$-almost everywhere.

Hence,

$$ $e^{c_{*}}\cdot\frac{dQ}{d\mu}(\mathbf{x})=e^{-T(\mathbf{x})}\cdot\frac{dP}{d \mu}(\mathbf{x}).$ $$

By integrating both sides of the above equality over $\Omega$ with $\mu$, we obtain

$$ $e^{c_{*}}=E_{P}\left[e^{-T}\right].$ $$

This completes the proof.
∎

###### Theorem B.2 .

###### Proof of Theorem B.2 .

###### Theorem B.3 (Theorem 5.2 in Section 5 restated) .

###### proof of Theorem 5.2 .

###### Lemma B.4 .

###### proof of Lemma B.4 .

###### Lemma B.5 .

###### proof of Lemma B.5 .

###### Theorem B.6 (Theorem 5.4 in Section 5 restated) .

###### proof of Theorem B.6 .

### B.3 Proofs for Section 6

In this section, we provide the theorems and proofs referred to in Section 6.

###### Theorem B.7 (Theorem 6.1 in Section 6 restated) .

Let $T_{\theta}(\mathbf{x}):\Omega\rightarrow\mathbb{R}$ be a function such that
the map $\theta=(\theta_{1},\theta_{2},\ldots,\theta_{p})\in\Theta\mapsto T_{\theta}(
\mathbf{x})$ is differentiable for all $\theta$ and
$\mu$-almost every $\mathbf{x}\in\Omega$.
Assume for a point $\bar{\theta}\in\Theta$, it holds that
$E_{P}[e^{(\alpha-1)\cdot T_{\bar{\theta}}}]<\infty$ and $E_{Q}[e^{\alpha\cdot T_{\bar{\theta}}}]<\infty$,
and there exists a compact neighborhood of $\bar{\theta}$, denoted by $B_{\bar{\theta}}$,
and a constant value $L$, such that $|T_{\psi}(\mathbf{x})-T_{\bar{\theta}}(\mathbf{x})|<L\|\psi-\bar{\theta}\|$.

Then,

$$ $E\left[\nabla_{\theta}\,\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}(T\,;\,\alpha) \Big{|}_{\theta=\bar{\theta}}\right]=\nabla_{\theta}\,E\left[\mathcal{L}_{ \alpha\text{-Div}}^{(R,S)}(T\,;\,\alpha)\right]\Big{|}_{\theta=\bar{\theta}}.$ (47) $$

###### proof of Theorem B.7 .

We now consider the values, as $\psi\rightarrow\bar{\theta}$, of the following two integrals:

$$ $\int\frac{1}{\|\psi-\bar{\theta}\|}\left\{\frac{1}{\alpha}e^{\alpha\cdot T_{ \psi}}-\frac{1}{\alpha}e^{\alpha\cdot T_{\bar{\theta}}}\right\}dQ,$ (48) $$

and

$$ $\int\frac{1}{\|\psi-\bar{\theta}\|}\left\{\frac{1}{1-\alpha}e^{(\alpha-1)\cdot T _{\psi}}-\frac{1}{1-\alpha}e^{(\alpha-1)\cdot T_{\bar{\theta}}}\right\}dP.$ (49) $$

Note that it follows from the intermediate value theorem that

$$ $\left|\frac{1}{\alpha}e^{\alpha\cdot x}-\frac{1}{\alpha}e^{\alpha\cdot y} \right|=\big{|}x-y\big{|}\cdot e^{\alpha\cdot\{y+\tau\cdot(x-y)\}}\quad(\, \exists\tau\in[0,1]\,).$ (50) $$

By using the above equation with $x=T_{\psi}(\mathbf{x})$ and $y=T_{\bar{\theta}}(\mathbf{x})$ for the integrand of Equation (48), we have

$$ $\displaystyle\left|\frac{1}{\|\psi-\bar{\theta}\|}\cdot\left\{\frac{1}{\alpha} e^{\alpha\cdot T_{\psi}(\mathbf{x})}-\frac{1}{\alpha}e^{\alpha\cdot T_{\bar{ \theta}}(\mathbf{x})}\right\}\right|$ $\displaystyle=\frac{1}{\|\psi-\bar{\theta}\|}\cdot\big{|}T_{\psi}(\mathbf{x})- T_{\bar{\theta}}(\mathbf{x})\big{|}\cdot e^{\alpha\cdot\{T_{\bar{\theta}}( \mathbf{x})+\tau_{\mathbf{x}}\cdot(T_{\psi}(\mathbf{x})-T_{\bar{\theta}}( \mathbf{x}))\}}\quad\qquad(\,\tau_{\mathbf{x}}\in[0,1]\,)$ $\displaystyle=\frac{1}{\|\psi-\bar{\theta}\|}\cdot\big{|}T_{\psi}(\mathbf{x})- T_{\bar{\theta}}(\mathbf{x})\big{|}\cdot e^{\alpha\cdot\tau_{\mathbf{x}}\cdot( T_{\psi}(\mathbf{x})-T_{\bar{\theta}}(\mathbf{x}))}\cdot e^{\alpha\cdot T_{ \bar{\theta}}(\mathbf{x})}$ $\displaystyle\leq\frac{1}{\|\psi-\bar{\theta}\|}\cdot\big{|}T_{\psi}(\mathbf{x })-T_{\bar{\theta}}(\mathbf{x})\big{|}\cdot e^{\alpha\tau_{\mathbf{x}}|T_{\psi }(\mathbf{x})-T_{\bar{\theta}}(\mathbf{x})|}\cdot e^{\alpha\cdot T_{\bar{ \theta}}(\mathbf{x})}$ $\displaystyle\leq L\cdot e^{\alpha\cdot L\cdot\|\psi-\bar{\theta}\|}\cdot e^{ \alpha\cdot T_{\bar{\theta}}(\mathbf{x})},$ (51) $$

for all $\psi\in B_{\bar{\theta}}$.

Integrating the term on the left-hand side of Equation (51) with respect to $Q$, we have

$$ $\displaystyle\int\left|\frac{1}{\|\psi-\bar{\theta}\|}\cdot\left\{\frac{1}{ \alpha}\cdot e^{\alpha\cdot T_{\psi}(\mathbf{x}^{q})}-\frac{1}{\alpha}\cdot e^ {\alpha\cdot T_{\bar{\theta}}(\mathbf{x}^{q})}\right\}\right|dQ(\mathbf{x}^{q})$ $\displaystyle\leq\int L\cdot e^{\alpha\cdot L\cdot\|\psi-\bar{\theta}\|}\cdot e ^{\alpha\cdot T_{\bar{\theta}}(\mathbf{x}^{q})}dQ(\mathbf{x}^{q})$ $\displaystyle=L\cdot e^{\alpha\cdot L\cdot\|\psi-\bar{\theta}\|}\cdot E_{Q} \left[e^{\alpha\cdot T_{\bar{\theta}}}\right].$ (52) $$

Considering the supremum for $\psi\in B_{\bar{\theta}}$ in Equation (52), we obtain

$$ $\displaystyle\sup_{\psi\in B_{\bar{\theta}}}\Bigg{\{}\int\left|\frac{1}{\|\psi -\bar{\theta}\|}\cdot\left\{\frac{1}{\alpha}\cdot e^{\alpha\cdot T_{\psi}}- \frac{1}{\alpha}e^{\alpha\cdot T_{\bar{\theta}}}\right\}\right|dQ\Bigg{\}}$ $\displaystyle\leq\sup_{\psi\in B_{\bar{\theta}}}\bigg{\{}L\cdot e^{\alpha\cdot L \cdot\|\psi-\bar{\theta}\|}\,\cdot E_{Q}\left[e^{\alpha\cdot T_{\bar{\theta}}} \right]\bigg{\}}$ $\displaystyle=E_{Q}\left[e^{\alpha\cdot T_{\bar{\theta}}}\right]\cdot\sup_{ \psi\in B_{\bar{\theta}}}L\cdot e^{\alpha\cdot L\cdot\|\psi-\bar{\theta}\|}<\infty,$ (53) $$

since $B_{\bar{\theta}}$ is compact.

Therefore, the following set is uniformly integrable for $Q$:

$$ $\left\{\frac{1}{\|\psi-\bar{\theta}\|}\left\{\frac{1}{\alpha}\cdot e^{\alpha \cdot T_{\psi}(\mathbf{x}^{q})}-\frac{1}{\alpha}e^{\alpha\cdot T_{\bar{\theta} }(\mathbf{x}^{q})}\right\}\,:\,\psi\in B_{\bar{\theta}}\right\}.$ (54) $$

Similarly, for Equation (49), we have

$$ $\displaystyle\sup_{\psi\in B_{\bar{\theta}}}\int\left|\frac{1}{\|\psi-\bar{ \theta}\|}\cdot\left\{\frac{1}{1-\alpha}e^{(\alpha-1)\cdot T_{\psi}(\mathbf{x} ^{p})}-\frac{1}{1-\alpha}e^{(\alpha-1)\cdot T_{\bar{\theta}}(\mathbf{x}^{p})} \right\}\right|dP(\mathbf{x}^{p})$ $\displaystyle\leq\sup_{\psi\in B_{\bar{\theta}}}\Big{\{}L\cdot e^{(1-\alpha)L \cdot\|\psi-\bar{\theta}\|}\,\cdot E_{P}\left[e^{(1-\alpha)\cdot T_{\bar{ \theta}}}\right]\Big{\}}$ $\displaystyle=E_{P}\left[e^{(1-\alpha)\cdot T_{\bar{\theta}}}\right]\cdot\sup_ {\psi\in B_{\bar{\theta}}}L\cdot e^{(1-\alpha)L\cdot\|\psi-\bar{\theta}\|}<\infty.$ (55) $$

Therefore, the following set is uniformly integrable for $P$:

$$ $\left\{\frac{1}{\|\psi-\bar{\theta}\|}\cdot\left\{\frac{1}{1-\alpha}\cdot e^{( \alpha-1)\cdot T_{\psi}(\mathbf{x}^{p})}-\frac{1}{1-\alpha}\cdot e^{(\alpha-1) \cdot T_{\bar{\theta}}(\mathbf{x}^{p})}\right\}\,:\,\psi\in B_{\bar{\theta}} \right\}.$ (56) $$

Thus, the Lebesgue integral and $\lim_{\psi\rightarrow\bar{\theta}}$ are exchangeable for the set in Equation (56). Then, we have

$$ $\displaystyle\nabla_{\theta}E_{Q}\left[\frac{1}{\alpha}\cdot e^{\alpha\cdot T_ {\theta}(\mathbf{x}^{q})}\right]\Bigg{|}_{\theta=\bar{\theta}}$ $\displaystyle=\lim_{\psi\rightarrow\bar{\theta}}\int\frac{1}{\|\psi-\bar{ \theta}\|}\cdot\bigg{\{}\frac{1}{\alpha}\cdot e^{\alpha\cdot T_{\psi}(\mathbf{ x}^{q})}-\frac{1}{\alpha}\cdot e^{\alpha\cdot T_{\bar{\theta}}(\mathbf{x}^{q}) }\bigg{\}}dQ(\mathbf{x}^{q})$ $\displaystyle=\int\lim_{\psi\rightarrow\bar{\theta}}\left[\frac{1}{\|\psi-\bar {\theta}\|}\cdot\bigg{\{}\frac{1}{\alpha}\cdot e^{\alpha\cdot T_{\psi}(\mathbf {x}^{q})}-\frac{1}{\alpha}\cdot e^{\alpha\cdot T_{\bar{\theta}}(\mathbf{x}^{q} )}\bigg{\}}\right]dQ(\mathbf{x}^{q})$ $\displaystyle=E_{Q}\left[\nabla_{\theta}\left(\frac{1}{\alpha}\cdot e^{\alpha \cdot T_{\theta}(\mathbf{x}^{q})}\right)\Bigg{|}_{\theta=\bar{\theta}}\right].$ (57) $$

Similarly, we obtain

$$ $\displaystyle\nabla_{\theta}E_{P}\left[\frac{1}{1-\alpha}\cdot e^{(\alpha-1) \cdot T_{\theta}(\mathbf{x}^{p})}\right]\Bigg{|}_{\theta=\bar{\theta}}$ $\displaystyle=\lim_{\psi\rightarrow\bar{\theta}}\int\frac{1}{\|\psi-\bar{ \theta}\|}\cdot\bigg{\{}\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot T_{\psi}( \mathbf{x}^{p})}-\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot T_{\bar{\theta}}( \mathbf{x}^{p})}\bigg{\}}dP(\mathbf{x}^{p})$ $\displaystyle=\int\lim_{\psi\rightarrow\bar{\theta}}\left[\frac{1}{\|\psi-\bar {\theta}\|}\cdot\bigg{\{}\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot T_{\psi}( \mathbf{x}^{p})}-\frac{1}{1-\alpha}\cdot e^{(\alpha-1)\cdot T_{\bar{\theta}}( \mathbf{x}^{p})}\bigg{\}}\right]dP(\mathbf{x}^{p})$ $\displaystyle=E_{P}\left[\nabla_{\theta}\left(\frac{1}{1-\alpha}\cdot e^{( \alpha-1)\cdot T_{\theta}(\mathbf{x}^{p})}\right)\Bigg{|}_{\theta=\bar{\theta} }\right].$ (58) $$

From Equations (57) and (58), we have

$$ $\displaystyle E\left[\nabla_{\theta}\,\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}( T\,;\,\alpha)\Big{|}_{\theta=\bar{\theta}}\right]$ $\displaystyle=E_{P}\left[E_{Q}\left[\nabla_{\theta}\,\mathcal{L}_{\alpha\text{ -Div}}^{(R,S)}(T\,;\,\alpha)\Big{|}_{\theta=\bar{\theta}}\right]\right]$ $\displaystyle=E_{P}\left[E_{Q}\left[\nabla_{\theta}|_{\theta=\bar{\theta}} \left\{\frac{1}{\alpha}\cdot\frac{1}{S}\cdot\sum_{i=1}^{S}e^{\alpha\cdot T( \mathbf{x}_{i}^{q})}+\frac{1}{1-\alpha}\cdot\frac{1}{R}\cdot\sum_{i=1}^{R}e^{( \alpha-1)\cdot T(\mathbf{x}_{i}^{p})}\right\}\right]\right]$ $\displaystyle=E_{P}\left[E_{Q}\left[\frac{1}{\alpha}\cdot\frac{1}{S}\cdot\sum_ {i=1}^{S}\nabla_{\theta}\left(e^{\alpha\cdot T(\mathbf{x}_{i}^{p})}\right)\Big {|}_{\theta=\bar{\theta}}\right.\right.$ $\displaystyle\left.\left.\qquad\qquad\qquad+\ \frac{1}{1-\alpha}\cdot\frac{1}{ R}\cdot\sum_{i=1}^{R}\nabla_{\theta}\left(e^{(\alpha-1)\cdot T(\mathbf{x}_{i}^ {p})}\right)\Big{|}_{\theta=\bar{\theta}}\,\right]\right]$ $\displaystyle=\frac{1}{\alpha}\cdot\frac{1}{S}\cdot\sum_{i=1}^{S}E_{Q}\left[ \nabla_{\theta}\left(e^{\alpha\cdot T(\mathbf{x}_{i}^{q})}\right)\right]\Big{| }_{\theta=\bar{\theta}}$ $\displaystyle\qquad\qquad\qquad+\ \frac{1}{1-\alpha}\cdot\frac{1}{R}\cdot\sum_ {i=1}^{R}E_{P}\left[\nabla_{\theta}\left(e^{(\alpha-1)\cdot T(\mathbf{x}_{i}^{ p})}\right)\Big{|}_{\theta=\bar{\theta}}\right]$ $\displaystyle=\frac{1}{\alpha}\cdot\frac{1}{S}\cdot\sum_{i=1}^{S}\nabla_{ \theta}E_{Q}\left[e^{\alpha\cdot T(\mathbf{x}_{i}^{q})}\right]\Big{|}_{\theta= \bar{\theta}}$ $\displaystyle\qquad\qquad\qquad+\ \frac{1}{1-\alpha}\cdot\frac{1}{R}\cdot\sum_ {i=1}^{R}\nabla_{\theta}E_{P}\left[e^{(\alpha-1)\cdot T(\mathbf{x}_{i}^{p})} \right]\Big{|}_{\theta=\bar{\theta}}$ $\displaystyle=\nabla_{\theta}\,E_{Q}\left[\frac{1}{\alpha}\cdot\frac{1}{S} \cdot\sum_{i=1}^{S}e^{\alpha\cdot T(\mathbf{x}_{i}^{q})}\right]\Big{|}_{\theta =\bar{\theta}}$ $\displaystyle\qquad\qquad\qquad+\ \nabla_{\theta}\,E_{P}\left[\frac{1}{1- \alpha}\cdot\frac{1}{R}\cdot\sum_{i=1}^{R}e^{(\alpha-1)\cdot T(\mathbf{x}_{i}^ {p})}\right]\Big{|}_{\theta=\bar{\theta}}$ $\displaystyle=\nabla_{\theta}\,\left\{E_{P}\left[E_{Q}\left[\frac{1}{\alpha} \cdot\frac{1}{S}\cdot\sum_{i=1}^{S}e^{\alpha\cdot T(\mathbf{x}_{i}^{q})}\right .\right.\right.$ $\displaystyle\left.\left.\left.\qquad\qquad+\ \frac{1}{1-\alpha}\cdot\frac{1}{ R}\cdot\sum_{i=1}^{R}e^{(\alpha-1)\cdot T(\mathbf{x}_{i}^{p})}\right]\right] \right\}\Big{|}_{\theta=\bar{\theta}}$ $\displaystyle=\nabla_{\theta}\,E_{P}\left[E_{Q}\left[\mathcal{L}_{\alpha\text{ -Div}}^{(R,S)}(T\,;\,\alpha)\right]\right]\Big{|}_{\theta=\bar{\theta}}$ $\displaystyle=\nabla_{\theta}\,E\left[\mathcal{L}_{\alpha\text{-Div}}^{(R,S)}( T\,;\,\alpha)\right]\Big{|}_{\theta=\bar{\theta}}.$ (59) $$

This completes the proof.
∎

###### Theorem B.8 (Theorem 6.2 in Section 6 restated) .

Assume $E_{P}\big{[}(dQ/dP(\mathbf{X}))^{2\cdot\alpha}\big{]}<\infty$.
Let $T^{*}=-\log dQ/dP$. Subsequently, let

$$ $\hat{D}^{(N)}(Q||P\,;\,\alpha)=\frac{1}{\alpha\cdot(1-\alpha)}-\mathcal{L}_{ \alpha\text{-Div}}^{(N,N)}(T^{*}\,;\,\alpha).$ (60) $$

Then, it holds that as $N\rightarrow\infty$,

$$ $\sqrt{N}\left\{\hat{D}^{(N)}(Q||P\,;\,\alpha)-D(Q||P\,;\,\alpha)\right\}\quad \xrightarrow{\ \ d\ \ }\quad\mathcal{N}\big{(}0,\ \sigma_{\alpha}^{2}\big{)},$ (61) $$

where

$$ $\displaystyle\sigma_{\alpha}^{2}$ $\displaystyle=C^{1}_{\alpha}\cdot D(Q||P\,;\,2\alpha)+C^{2}_{\alpha}\cdot D(Q| |P\,;\,2\alpha-1)$ $\displaystyle\qquad\qquad+\ C^{3}_{\alpha}\cdot D(Q||P\,;\,\alpha)^{2}+C^{4}_{ \alpha}\cdot D(Q||P\,;\,\alpha)+C^{5}_{\alpha},$ (62) $$

and

$$ $\displaystyle C^{1}_{\alpha}$ $\displaystyle=\frac{2\alpha\cdot(1-2\alpha)}{\alpha^{2}},$ (63) $\displaystyle C^{2}_{\alpha}$ $\displaystyle=\frac{2\alpha\cdot(1-2\alpha)}{(1-\alpha)^{2}},$ (64) $\displaystyle C^{3}_{\alpha}$ $\displaystyle=-\frac{1}{\alpha^{2}}-\frac{1}{(1-\alpha)^{2}},$ (65) $\displaystyle C^{4}_{\alpha}$ $\displaystyle=\frac{2}{\alpha^{2}}+\frac{2}{(1-\alpha)^{2}},$ $\displaystyle C^{5}_{\alpha}$ $\displaystyle=\left(\frac{1}{\alpha^{2}}+\frac{1}{(1-\alpha)^{2}}\right)\cdot( 2-2\alpha\cdot(1-\alpha)).$ (66) $$

###### proof of Theorem B.8 .

First, note that

$$ $\displaystyle\hat{D}^{(N)}(Q||P\,;\,\alpha)$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot\left\{ \frac{1}{N}\cdot\sum_{i=1}^{N}e^{\alpha\cdot T^{*}(\mathbf{X}^{i}_{Q})}\right \}-\frac{1}{1-\alpha}\cdot\left\{\frac{1}{N}\cdot\sum_{i=1}^{N}e^{(\alpha-1) \cdot T^{*}(\mathbf{X}^{i}_{P})}\right\}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot\left\{ \frac{1}{N}\cdot\sum_{i=1}^{N}\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{Q})\right)^{ -\alpha}\right\}$ $\displaystyle\qquad\qquad\qquad\qquad\qquad-\ \frac{1}{1-\alpha}\cdot\left\{ \frac{1}{N}\cdot\sum_{i=1}^{N}\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{P})\right)^{ 1-\alpha}\right\}.$ (67) $$

On the other hand, from Lemma B.5, we obtain

$$ $\displaystyle D(Q||P\,;\,\alpha)$ $\displaystyle=\sup_{T:\Omega\rightarrow\mathbb{R}}\left\{\frac{1}{\alpha\cdot( 1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T}\right]-\frac{1}{ 1-\alpha}\cdot E_{P}\cdot\left[e^{(\alpha-1)\cdot T}\right]\right\}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\inf_{T:\Omega\rightarrow\mathbb {R}}\left\{\frac{1}{\alpha}\cdot E_{Q}\left[e^{\alpha\cdot T}\right]-\frac{1}{ 1-\alpha}\cdot E_{P}\left[e^{(\alpha-1)\cdot T}\right]\right\}$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\macc@depth\char 1\relax \frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{\mathcal{L} }_{\alpha\text{-Div}}^{*}(\alpha)$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left [e^{\alpha\cdot T^{*}}\right]-\frac{1}{1-\alpha}\cdot E_{P}\left[e^{(\alpha-1) \cdot T^{*}}\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot E_{Q}\left [\left(\frac{dQ}{dP}(\mathbf{x})\right)^{-\alpha}\right]-\frac{1}{1-\alpha} \cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\right]$ $\displaystyle=\frac{1}{\alpha\cdot(1-\alpha)}-\frac{1}{\alpha}\cdot\left\{ \frac{1}{N}\cdot\sum_{i=1}^{N}E_{Q}\left[\left(\frac{dQ}{dP}(\mathbf{x}_{i}) \right)^{-\alpha}\right]\right\}$ $\displaystyle\qquad\qquad\qquad\qquad\qquad-\ \frac{1}{1-\alpha}\cdot\left\{ \frac{1}{N}\cdot\sum_{i=1}^{N}E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x}_{i}) \right)^{1-\alpha}\right]\right\}.$ $$

Subtracting Equation (LABEL:D_alpha_represented_as_exp) from Equation (67), we have

$$ $\displaystyle\hat{D}^{(N)}(Q||P\,;\,\alpha)-D(Q||P\,;\,\alpha)$ $\displaystyle=\frac{1}{N}\cdot\sum_{i=1}^{N}\frac{1}{\alpha}\cdot\left\{\left( \frac{dQ}{dP}(\mathbf{X}^{i}_{Q})\right)^{-\alpha}-E_{Q}\left[\left(\frac{dQ}{ dP}(\mathbf{x}_{i})\right)^{-\alpha}\right]\right\}$ $\displaystyle\qquad+\ \frac{1}{N}\cdot\sum_{i=1}^{N}\frac{1}{1-\alpha}\cdot \left\{\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{P})\right)^{1-\alpha}-E_{P}\left[ \left(\frac{dQ}{dP}(\mathbf{x}_{i})\right)^{1-\alpha}\right]\right\}$ $\displaystyle=\frac{1}{N}\cdot\sum_{i=1}^{N}\frac{1}{\alpha}\cdot\left\{\left( \frac{dQ}{dP}(\mathbf{X}^{i}_{Q})\right)^{-\alpha}-E_{Q}\left[\left(\frac{dQ}{ dP}(\mathbf{x})\right)^{-\alpha}\right]\right\}$ $\displaystyle\qquad+\ \frac{1}{N}\cdot\sum_{i=1}^{N}\frac{1}{1-\alpha}\cdot \left\{\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{P})\right)^{1-\alpha}-E_{P}\left[ \left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\right]\right\}.$ (69) $$

Let $L_{Q}^{i}=\frac{1}{\alpha}\cdot\left\{\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{Q})
\right)^{-\alpha}-E_{Q}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{-\alpha}
\right]\right\}$.
Then $\{L_{Q}^{i}\}_{i=1}^{N}$ are independent and identically distributed variables whose means and variances are as follows:

$$ $\displaystyle E_{Q}\Big{[}L_{Q}^{i}\Big{]}$ $\displaystyle=0,$ (70) $$

and

$$ $\displaystyle\mathrm{Var}_{Q}\Big{[}L_{Q}^{i}\Big{]}$ $\displaystyle=E_{Q}\left[\frac{1}{\alpha^{2}}\cdot\left\{\ \left(\frac{dQ}{dP} (\mathbf{x})\right)^{-\alpha}-E_{Q}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right )^{-\alpha}\right]\right\}^{2}\right]$ $\displaystyle=\frac{1}{\alpha^{2}}\cdot E_{Q}\left[\left\{\left(\frac{dQ}{dP}( \mathbf{x})\right)^{-\alpha}\right\}^{2}\ \right]-\frac{1}{\alpha^{2}}\cdot \left\{E_{Q}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{-\alpha}\ \right] \right\}^{2}$ $\displaystyle=\frac{1}{\alpha^{2}}\cdot E_{P}\left[\frac{dQ}{dP}(\mathbf{x}) \cdot\left(\frac{dQ}{dP}(\mathbf{x})\right)^{-2\alpha}\ \right]-\frac{1}{ \alpha^{2}}\cdot\left\{E_{P}\left[\frac{dQ}{dP}(\mathbf{x})\cdot\left(\frac{dQ }{dP}(\mathbf{x})\right)^{-\alpha}\ \right]\right\}^{2}$ $\displaystyle=\frac{1}{\alpha^{2}}\cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf {x})\right)^{1-2\alpha}\right]-\frac{1}{\alpha^{2}}\cdot\left\{E_{P}\left[ \left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\ \right]\right\}^{2}$ $\displaystyle=\frac{1}{\alpha^{2}}\cdot\Bigg{\{}\ 2\alpha\cdot(2\alpha-1)\cdot \left(\frac{1}{2\alpha\cdot(2\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{dP}( \mathbf{x})\right)^{1-2\alpha}-1\right]\right)\ +1$ $\displaystyle\qquad\qquad-\ \alpha^{2}\cdot(1-\alpha)^{2}\cdot\left(\frac{1}{ \alpha\cdot(\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^ {1-\alpha}-1\ \right]\right)^{2}+1$ $\displaystyle\qquad\qquad+\ \alpha^{2}\cdot(1-\alpha)^{2}\cdot\left(\frac{2}{ \alpha\cdot(\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^ {1-\alpha}-1\ \right]\right)$ $\displaystyle\qquad\qquad\qquad-\ 2\alpha\cdot(1-\alpha)\ \Bigg{\}}.$ (71) $$

Similarly, let $L_{P}^{i}=\frac{1}{1-\alpha}\cdot\left\{\left(\frac{dQ}{dP}(\mathbf{X}^{i}_{P}
)\right)^{1-\alpha}-E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-
\alpha}\right]\right\}$.
Then $\{L_{P}^{i}\}_{i=1}^{N}$ are independent and identically distributed variables whose means and variances are as follows:

$$ $\displaystyle E_{P}\Big{[}L_{P}^{i}\Big{]}$ $\displaystyle=0,$ (72) $$

and

$$ $\displaystyle\mathrm{Var}_{P}\Big{[}L_{P}^{i}\Big{]}$ $\displaystyle=E_{P}\left[\frac{1}{(1-\alpha)^{2}}\cdot\left\{\ \left(\frac{dQ} {dP}(\mathbf{x})\right)^{1-\alpha}-E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x}) \right)^{1-\alpha}\right]\right\}^{2}\right]$ $\displaystyle=\frac{1}{(1-\alpha)^{2}}\cdot E_{P}\left[\left\{\left(\frac{dQ}{ dP}(\mathbf{x})\right)^{1-\alpha}\right\}^{2}\ \right]-\frac{1}{(1-\alpha)^{2} }\cdot\left\{E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\ \right]\right\}^{2}$ $\displaystyle=\frac{1}{(1-\alpha)^{2}}\cdot E_{P}\left[\left(\frac{dQ}{dP}( \mathbf{x})\right)^{2(1-\alpha)}\ \right]-\frac{1}{(1-\alpha)^{2}}\cdot\left\{ E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\ \right]\right\}^ {2}$ $\displaystyle=\frac{1}{(1-\alpha)^{2}}\cdot E_{P}\left[\left(\frac{dQ}{dP}( \mathbf{x})\right)^{1-(2\alpha-1)}\right]-\frac{1}{(1-\alpha)^{2}}\cdot\left\{ E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^{1-\alpha}\ \right]\right\}^ {2}$ $\displaystyle=\frac{1}{(1-\alpha)^{2}}\cdot\Bigg{\{}\ 2\alpha\cdot(2\alpha-1) \cdot\left(\frac{1}{2\alpha\cdot(2\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{ dP}(\mathbf{x})\right)^{1-(2\alpha-1)}-1\right]\right)\ +1$ $\displaystyle\qquad\qquad-\ \alpha^{2}\cdot(1-\alpha)^{2}\cdot\left(\frac{1}{ \alpha\cdot(\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^ {1-\alpha}-1\ \right]\right)^{2}+1$ $\displaystyle\qquad\qquad+\ \alpha^{2}\cdot(1-\alpha)^{2}\cdot\left(\frac{2}{ \alpha\cdot(\alpha-1)}\cdot E_{P}\left[\left(\frac{dQ}{dP}(\mathbf{x})\right)^ {1-\alpha}-1\ \right]\right)$ $\displaystyle\qquad\qquad\qquad-\ 2\alpha\cdot(1-\alpha)\ \Bigg{\}}.$ (73) $$

Now, we consider an asymptotical distribution of the following term:

$$ $\displaystyle\sqrt{N}\cdot\left\{\hat{D}^{(N)}(Q||P\,;\,\alpha)-D(Q||P\,;\, \alpha)\right\}$ $\displaystyle=\sqrt{N}\cdot\left(\frac{1}{N}\cdot\sum_{i=1}^{N}\Big{\{}L_{Q}^{ i}-E_{Q}\left[L_{Q}^{i}\right]\Big{\}}\right)+\sqrt{N}\cdot\left(\frac{1}{N} \cdot\sum_{i=1}^{N}\Big{\{}L_{P}^{i}-E_{P}\left[L_{P}^{i}\right]\Big{\}}\right).$ (74) $$

By the central limit theorem, we observe that as $N\rightarrow\infty$,

$$ $\sqrt{N}\cdot\left(\frac{1}{N}\cdot\sum_{i=1}^{N}\Big{\{}L_{Q}^{i}-E_{Q}\left[ L_{Q}^{i}\right]\Big{\}}\right)\quad\xrightarrow{\ \ d\ \ }\quad\mathcal{N} \big{(}0,\mathrm{Var}_{Q}\left[L_{Q}^{i}\right]\big{)},$ (75) $$

and

$$ $\sqrt{N}\cdot\left(\frac{1}{N}\cdot\sum_{i=1}^{N}\Big{\{}L_{P}^{i}-E_{P}\left[ L_{P}^{i}\right]\Big{\}}\right)\quad\xrightarrow{\ \ d\ \ }\quad\mathcal{N} \big{(}0,\mathrm{Var}_{P}\left[L_{P}^{i}\right]\big{)}.$ (76) $$

Therefore, from Equations (75) and (76), we obtain

$$ $\sqrt{N}\cdot\left\{\hat{D}^{(N)}(Q||P\,;\,\alpha)-D(Q||P\,;\,\alpha)\right\} \xrightarrow{\ \ d\ \ }\mathcal{N}\big{(}0,\sigma_{\alpha}^{2}\big{)},$ (77) $$

and

$$ $\displaystyle\sigma_{\alpha}^{2}$ $\displaystyle=\mathrm{Var}_{Q}\left[L_{Q}^{i}\right]+\mathrm{Var}_{P}\left[L_{ P}^{i}\right]$ $\displaystyle=C^{1}_{\alpha}\cdot D(Q||P\,;\,2\alpha)+C^{2}_{\alpha}\cdot D(Q| |P\,;\,2\alpha-1)$ $\displaystyle\qquad\quad+\ C^{3}_{\alpha}\cdot D(Q||P\,;\,\alpha)^{2}+C^{4}_{ \alpha}\cdot D(Q||P\,;\,\alpha)+C^{5}_{\alpha},$ (78) $$

where

$$ $\displaystyle C^{1}_{\alpha}$ $\displaystyle=\frac{2\alpha\cdot(1-2\alpha)}{\alpha^{2}},$ (79) $\displaystyle C^{2}_{\alpha}$ $\displaystyle=\frac{2\alpha\cdot(1-2\alpha)}{(1-\alpha)^{2}},$ (80) $\displaystyle C^{3}_{\alpha}$ $\displaystyle=-\frac{1}{\alpha^{2}}-\frac{1}{(1-\alpha)^{2}},$ (81) $\displaystyle C^{4}_{\alpha}$ $\displaystyle=\frac{2}{\alpha^{2}}+\frac{2}{(1-\alpha)^{2}},$ $\displaystyle C^{5}_{\alpha}$ $\displaystyle=\left(\frac{1}{\alpha^{2}}+\frac{1}{(1-\alpha)^{2}}\right)\cdot \big{(}2-2\alpha\cdot(1-\alpha)\big{)}.$ (82) $$

This completes the proof.
∎

###### Theorem B.7 (Theorem 6.1 in Section 6 restated) .

###### proof of Theorem B.7 .

###### Theorem B.8 (Theorem 6.2 in Section 6 restated) .

###### proof of Theorem B.8 .

## Appendix C Details of the experiments in Section 7

In this section, we provide details on the hyperparameter settings used in the experiments described in Section 7.

### C.1 Details of the experiments in Section 7.1

In this section, we detail the experiments reported in Section 7.1.

#### C.1.1 DataSets.

We generated the following 100 train datasets.
$P=\mathcal{N}(\mu_{p},I_{5})$ and $Q=\mathcal{N}(\mu_{q},\Sigma_{q})$ where
$\mu_{p}=\mu_{q}=(0,0,\ldots,0)$, and $I_{5}$ denotes the $5$-dimensional identity matrix, and
$\Sigma_{q}=(\sigma_{ij})_{i=1}^{5}$ with $\sigma_{ii}=1$, and $\sigma_{ij}=0.8$ for $i\neq j$.
The size of each dataset was 5000.

#### C.1.2 Experimental Procedure.

Neural networks were traind using the synthetic datasets by optimizing $\alpha$-Div for $\alpha=-3.0,\allowbreak-2.0,\allowbreak-1.0,\allowbreak 0.2,\allowbreak 0.5,
\allowbreak 0.8,\allowbreak 2.0,\allowbreak 3.0$, and $4.0$ while measuring the training losses for each learning step.
For each value of $\alpha$, 100 trials were conducted.
Finally, we reported the median, ranging between the 45th and 55th quartiles, and between the 2.5th and 97.5th quartiles of the training losses at each learning step.

#### C.1.3 Neural Network Architecture, Optimization Algorithm, and Hyperparameters.

A 5-layer perceptron with ReLU activation was used, with each hidden layer comprising 100 nodes. For optimization, the learning rate was set to 0.001, the batch size was 2500, and the number of epochs was 250.
The models for DRE were implemented using the PyTorch library (Paszke et al., ) in Python. Training was conducted with the Adam optimizer (Kingma, ) in PyTorch and an NVIDIA T4 GPU.

#### C.1.4 Results.

As shown in Figure 4, the training losses of $\alpha$-Div across learning steps are presented for $\alpha=-3,-2,-1,0.2,0.5,0.8,2.0,3.0$, and $4.0$.
The upper ($\alpha=-3.0,-2.0$, and $-1.0$) and middle ($\alpha=2.0,3.0$ and $4.0$) figures in Figure 4show that the training losses diverged to large negative values when $\alpha<0$ or $\alpha>1$.
In contrast, the bottom figure ($\alpha=0.2,0.5$, and $0.8$) Figure 2, the training losses of $\alpha$-Div converged,
illustrating the stability of optimization with $\alpha$-Div when $0<\alpha<1$.

Figure: Figure 4: All results of Section 7.1. for $\alpha=-3,-2,-1,0.2,0.5,0.8,2.0,3.0$, and $4.0$. Each graph displays the training losses (y-axis) against the learning steps (x-axis) during optimization using $\alpha$-Div for different values of $\alpha$ values. The solid blue line represents the median training losses. The dark blue area indicates the range between the 45th and 55th percentiles, while the light blue area shows the range between the 2.5th and 97.5th percentiles of the training losses.
Refer to caption: x4.png

### C.2 Details of the experiments in Section 2

In this section, we provide details about the experiments reported in Section 2.

#### C.2.1 DataSets.

We first generated 100 training datasets, each with a total size of $10000$ samples.
Each dataset was drawn from two normal distributions: $P=\mathcal{N}(\mu_{p},\cdot I_{5})$ and $Q=\mathcal{N}(\mu_{q},\cdot I_{5})$ where $I_{5}$ denotes the $5$-dimensional identity matrix, and $\mu_{p}=(-5/2,0,0,0,0,0)$ and $\mu_{q}=(5/2,0,0,0,0,0)$.

#### C.2.2 Experimental Procedure.

We trained neural networks using the training datasets, optimizing both $\alpha$-Div and the naive $\alpha$-divergence loss function defined in Equation (7) with $\alpha=0.5$, as well as nnBD-LSIF, while measuring the training losses at each learning step.
We conducted 100 trials and reported the median training losses, along with the ranges between the 45th and 55th percentiles, and between the 2.5th and 97.5th percentiles, at each learning step.

##### Loss functions used in the experiments.

We used $\alpha$-Div, the naive $\alpha$-divergence loss function, and the non-negative Bregman divergence least-squares importance fitting (nnBD-LSIF) loss function (Kato & Teshima, ) to train neural networks.
The naive $\alpha$-divergence loss function, presented in Equation (7), exhibits a biased gradient when $\alpha<1$.

nnBD-LSIF is an unbounded Bregman divergence loss function obtained from the deep direct DRE (D3RE) method proposed by Kato & Teshima (), which is defined as

$$ $\mathcal{L}_{\text{$\mathrm{nnBD}$-$\mathrm{LSIF}$}}(\phi)=-\hat{E}_{Q}\left[ \phi(\mathbf{x})-\frac{C}{2}\phi^{2}(\mathbf{x})\right]+\left(\frac{1}{2}\cdot \hat{E}_{P}\left[\phi^{2}(\mathbf{x})\right]-\frac{C}{2}\cdot\hat{E}_{Q}\left[ \phi^{2}(\mathbf{x})\right]\right)_{+},$ (83) $$

where $(a)_{+}=a$ if $a>0$ otherwise $(a)_{+}=0$ and $C$ is positive constant.
Note that, nnBD-LSIF has a unbiased gradient.
The optimization efficiency of nnBD-LSIF was observed to confirm the effectiveness of an unbiased gradient of an $f$-divergence loss function as well as $\alpha$-Div.

#### C.2.3 Neural Network Architecture, Optimization Algorithm, and Hyperparameters.

We used a 4-layer perceptron with ReLU activation, where each hidden layer contained 100 nodes. For optimization, the learning rate was set to 0.00005, the batch size was 2500, and the number of epochs was 1000.
We implemented all models for DRE using the PyTorch library (Paszke et al., ) in Python. Training was performed with the Adam optimizer (Kingma, ) in PyTorch, utilizing an NVIDIA T4 GPU.

### C.3 Details of the experiments in Section 7.3

In this section, we provide details on the experiments reported in Section 7.3.

#### C.3.1 DataSets.

Initially, we created 100 train and test datasets, each with a size of 10,000.
Each dataset is generated from two normal distributions $P=\mathcal{N}(\mu_{p},\sigma^{2}\cdot I_{3})$ and $Q=\mathcal{N}(\mu_{q},4^{2}\cdot I_{3})$ where $I_{3}$ denotes the $3$-dimensional identity matrix and $\sigma$ values were $1.0$, 1.1, 1.2, 1.4, 1.6, 2.0, 2.5, or 3.0, and $\mu_{p}=(-3/2,-3/2,-3/2)$ and $\mu_{q}=(3/2,3/2,3/2)$.
In the aforementioned setting, the ground truth KL-divergence amounts of the datasets is obtained as

$$ $\displaystyle KL(P||Q)$ $\displaystyle=E_{P}\left[\log\left(\frac{dP}{dQ}\right)\right]$ $\displaystyle=\frac{1}{2}\cdot\left[\log\frac{|\Sigma_{p}|}{|\Sigma_{q}|}-d+ \mathrm{Tr}(\Sigma_{p}^{-1}\cdot\Sigma_{q})+(\mu_{p}-\mu_{q})^{T}\cdot\Sigma_{ p}^{-1}\cdot(\mu_{p}-\mu_{q})\right]$ $\displaystyle=\frac{1}{2}\cdot\left[\log\frac{\sigma^{2}\cdot|I_{3}|}{4^{2} \cdot|I_{3}|}-3+\mathrm{Tr}(\sigma^{-2}\cdot I_{3}\cdot 4^{2}\cdot I_{3})+3 \cdot\mathbf{1}^{T}\cdot\sigma^{-2}\cdot I_{3}\cdot 3\cdot\mathbf{1}\right]$ $\displaystyle=\frac{1}{2}\cdot\Big{(}6\log\sigma-12\log 2-3+3\cdot\sigma^{-2} \cdot 16+27\cdot\sigma^{-2}\Big{)}$ $\displaystyle=3\log\sigma-6\log 2-\frac{3}{2}+\sigma^{-2}\cdot\frac{75}{2}.$ (84) $$

From Equation (84), we see that the ground truth KL-divergence amounts of the datasets were 31.8, 25.6, 21.0, 14.5, 10.4, 10.4, 5.8, 3.1, and 1.8, which correspond to the ascending $\sigma$ values, such that $\sigma=1.0,1.1,...3.0$.

#### C.3.2 Experimental Procedure.

We trained neural networks using the training datasets by optimizing both $\alpha$-Div with $\alpha=0.5$ and a KL-divergence loss function.
Details of the KL-divergence loss function used in the experiments are provided in the following paragraph.
Training was halted if the validation losses, measured using the validation datasets, did not improve during an entire epoch.
After training the neural networks, we measured the root mean squared error (RMSE) of the estimated density ratios using the test datasets.
We estimated the KL-divergence of the test datasets for each trial using the estimated density ratios and the plug-in estimation method, which is detailed below.
A total of 100 trials were conducted. Finally, we reported the median RMSE of the DRE and the estimated KL-divergence, along with the interquartile range (25th to 75th percentiles), for each KL-divergence loss function and $\alpha$-Div.

##### KL-divergence loss function.

A naive KL-divergence loss function is obtained as

$$ $\mathcal{L}_{\mathrm{naive}\text{-}\mathrm{KL}}(\phi)=\hat{E}_{P}\left[\phi \right]-\hat{E}_{Q}\left[\log\phi\right].$ (85) $$

In our pre-experiment, the naive KL-divergence loss function exhibited poor optimization performance, which we attribute to its biased gradients.
However, we found that the Gibbs density transformation, as described in Section 5.1, improved optimization performance for the KL-divergence loss function. Therefore, we used the following KL-divergence loss function, $\mathcal{L}_{\mathrm{KL}}(\cdot)$ in our experiments:

$$ $\mathcal{L}_{\mathrm{KL}}(T)=\hat{E}_{P}\left[e^{T}\right]-\hat{E}_{Q}\left[T \right].$ (86) $$

##### Plug-in KL-divergence estimation method using the estimated density ratios.

The KL-divergence of the test datasets was estimated by estimated predicted density ratios for the test datasets using a plug-in estimation, such that

$$ $\widehat{KL}(P||Q)=\hat{E}_{Q}\left[\log\hat{r}_{q}(\mathbf{x})\right],$ (87) $$

where $\hat{r}_{q}(\mathbf{x})=e^{T(\mathbf{x})}/\hat{E}_{Q}[e^{T(\mathbf{x})}]$.

#### C.3.3 Neural Network Architecture, Optimization Algorithm, and Hyperparameters.

The same neural network architecture, optimization algorithm, and hyperparameters were used for both $\alpha$-Div and the KL-divergence loss function.
A 4-layer perceptron with ReLU activation was employed, with each hidden layer consisting of 256 nodes. For optimization with the $\alpha$-Div loss function, the value of $\alpha$ was set to 0.5, the learning rate was 0.00005, and the batch size was 256. Early stopping was applied with a patience of 32 epochs, and the maximum number of epochs was set to 5000. For optimization using the KL-divergence loss function, the learning rate was 0.00001, with a batch size of 256. Early stopping was applied with a patience of 2 epochs, and the maximum number of epochs was 5000.
Pytorch (Paszke et al., ) library in Python was used to implement all models for DRE, with the Adam optimizer (Kingma, ) in PyTorch and an NVIDIA T4 GPU used for training the neural networks.

## Appendix D Additional Experiments

### D.1 Comparison with Existing DRE Methods

We empirically compare the proposed DRE method with existing DRE methods in terms of accuracy in DRE tasks.
This experiment followed the setup described in Kato & Teshima ().

#### D.1.1 Existing f 𝑓 f italic_f -Divergence Loss Functions for Comparison.

The proposed method was compared with the Kullback-Leibler importance estimation procedure (KLIEP) (Sugiyama et al., ), unconstrained least-squares importance fitting (uLSIF) (Kanamori et al., ), and deep direct DRE (D3RE) (Kato & Teshima, ).
The densratio library in R was used for KLIEP and uLSIF.(^2^22https://cran.r-project.org/web/packages/densratio/index.html)
For D3RE, the non-negative Bregman divergence least-squares importance fitting (nnBD-LSIF) loss function was employed.

#### D.1.2 Datasets.

For each $d=10,20,30,50$, and $100$, 100 datasets were generated, comprising training and test sets drawn from two $d$-dimensional normal distributions $P=\mathcal{N}(\mu_{p},I_{d})$ and $Q=\mathcal{N}(\mu_{q},I_{d})$, where $I_{d}$ denotes the $d$-dimensional identity matrix, $\mu_{p}=(0,0,\dots,0)$, and $\mu_{q}=(1,0,\dots,0)$.

#### D.1.3 Experimental Procedure.

Model parameters were trained using the training datasets, and density ratios for the test datasets were estimated.
The mean squared error (MSE) of the estimated density ratios for the test datasets was calculated based on the true density ratios.
Finally, the average mean and standard deviation of the MSE for each method were reported.

#### D.1.4 Neural Network Architecture, Optimization Algorithm, and Hyperparameters.

For both D3RE and $\alpha$-Div, a 3-layer perceptron with 100 hidden units per layer was used, consistent with the neural network structure employed in Kato & Teshima ().
For D3RE, the learning rate was set to 0.00005, the batch size was 128, and the number of epochs was 250 for each data dimension.
The hyperparameter $C$ was set to 2.0. For $\alpha$-Div, the learning rate was set to 0.0001, the batch size was 128, and the value of $\alpha$ was set to 0.5 for each data dimension.
The number of epochs was set to 40 for data dimensions of 10, 50 for dimensions of 20, 30, and 50, and 60 for a dimension of 100.
The PyTorch library (Paszke et al., ) in Python was used to implement all models for both D3RE and $\alpha$-Div.
The Adam optimizer (Kingma, ) in PyTorch, along with an NVIDIA T4 GPU, was used for training the neural networks.

##### Results.

**Table 5: Results of additional experiments described in Section D.1. The table reports the average mean and standard deviation of the MSE for DRE with each method. Results are presented in the format "mean (standard deviation)." The lowest MSE values are highlighted in bold.**
|  | Data dimentions ($d$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | $d=10$ | $d=20$ | $d=30$ | $d=50$ | $d=100$ |
| KLIEP | 2.141(0.392) | 2.072(0.660) | 2.005(0.569) | 1.887(0.450) | 1.797(0.419) |
| uLSIF | 1.482(0.381) | 1.590(0.562) | 1.655(0.578) | 1.715(0.446) | 1.668(0.420) |
| D3RE | 1.111(0.314) | 1.127(0.413) | 1.219(0.458) | 1.222(0.305) | 1.369(0.355) |
| $\alpha$-Div | 0.173(0.072) | 0.278(0.113) | 0.479(0.259) | 0.665(0.194) | 1.118(0.314) |

Table 5 summarizes the results for each method across different data dimensions.
Six cases where the MSE for KLIEP exceeded 1000 were excluded.
For all data dimensions, $\alpha$-Div consistently demonstrated superior accuracy compared to the other methods, achieving the lowest MSE values. However, it is important to note that the prediction accuracy of $\alpha$-Div significantly decreased as the data dimensions increased.
The curse of dimensionality in DRE was also observed in experiments with real-world data, which will be reported in the next section.

### D.2 Experiments Using Real-World Data

We present numerical experiments using real-world data to highlight important considerations for applying the proposed method. Specifically, we conducted experiments on Domain Adaptation (DA) for classification models using the Importance Weighting (IW) method (Shimodaira, ).
The IW method builds a prediction model for a target domain using data from a source domain, while adjusting the distribution of source domain features to match that of the target domain features by employing the density ratio between the source and target domains as sample weights.

In these experiments, we used Amazon review data (Blitzer et al., ) and employed two prediction algorithms: linear regression and gradient boosting. The hyperparameters for each algorithm were selected from a predefined set based on validation accuracy, using the Importance Weighted Cross Validation (IWCV) method (Sugiyama et al., ) on the source domain data.

Through these experiments, we observed a decline in prediction accuracy on test data from the target domain as the data dimensionality increased.
Specifically, there were instances where the accuracy worsened compared to models that did not use importance weighting—i.e., models trained solely on the source data.
These phenomena are likely due to two issues in DRE: the degradation in density ratio estimation accuracy as dimensionality increases, as noted in Section D.1, and the negative impact of high KL-divergence on density ratio estimation, as observed in Section 7.3.
It is important to note that KL-divergence increases with the number of features (i.e., data dimensions), unless all features are fully independent.

#### D.2.1 DataSets.

The Amazon review dataset (Blitzer et al., ) includes text reviews and rating scores from four domains: books, DVDs, electronics, and kitchen appliances. The text reviews are one-hot encoded, and the rating scores are converted into binary labels. Twelve domain adaptation classification tasks were conducted, with each domain serving once as the source domain and once as the target domain.

##### Notation.

We provide detailed explanations of the notations in the next section. $\mathbf{X}_{\text{S}}^{d}$ and $\mathbf{X}_{\text{T}}^{d}$ denote subsets of the original data for the source and target domains, respectively, for each feature dimension $d$, where the columns of $\mathbf{X}_{\text{S}}^{d}$ and $\mathbf{X}_{\text{T}}^{d}$ are identical. $y_{\text{S}}$ and $y_{\text{T}}$ represent the objective variables in the source and target domains, respectively—binary labels for each sample in the source and target domain data. $\mathbf{Z}_{\text{S}}^{d}$ and $\mathbf{Z}_{\text{T}}^{d}$ denote $d$-dimensional feature tables used to estimate the density ratio $\hat{r}(\mathbf{Z}_{\text{S}}^{d})$, which is the ratio of the target domain density to the source domain density. $\text{dim}(X)$ indicates the number of columns (features) in the data $X$.

#### D.2.2 Experimental Procedure.

##### Step 1. Creation of feature tables.

Many DA methods utilize feature embedding techniques to project high-dimensional data into a lower-dimensional feature space, facilitating the handling of distribution shifts between source and target domains (Ragab et al., ). However, our preliminary experiments revealed that model prediction accuracies were significantly influenced by the embedding procedures. To address these effects on DA task accuracies, we explored an embedding method with theoretical considerations detailed in the next section.

Specifically, we selected an identical set of columns from the original data of both the source and target domains for each feature dimension, $d=8,16,32,64$, and $128$, arranging the columns in ascending order of $d$. Let $\mathbf{X}_{\text{S}}^{d}$ and $\mathbf{X}_{\text{T}}^{d}$ denote the subsets of the original data for the source and target domains, respectively, determined by these selected columns for each $d$. We then generated a $d\times\text{dim}(\mathbf{X}_{\text{S}}^{d})$ matrix $A_{d}$ from a normal distribution. Finally, by multiplying $\mathbf{X}_{\text{S}}^{d}$ and $A_{d}$, and $\mathbf{X}_{\text{T}}^{d}$ and $A_{d}$, we obtained the feature tables $\mathbf{Z}_{\text{S}}^{d}$ and $\mathbf{Z}_{\text{T}}^{d}$, embedding the original source and target domain data into a $d$-dimensional feature space.
(^3^33In our experiments, we utilized matrices generated from the normal distribution as embedding maps, which is equivalent to random projection (Bingham & Mannila, ). However, the linearity of the map is not necessary for preserving the density ratios, as discussed in Section D.2.3. In contrast, linearity is a key requirement for the distance-preserving property of random projection.)

##### Step 2. Estimation of importance weights.

Using the proposed loss function with $\mathbf{Z}_{\text{S}}^{d}$ and $\mathbf{Z}_{\text{T}}^{d}$ obtained from the previous step, we estimated the probability density ratio $\hat{r}(\mathbf{Z}_{\text{S}}^{d})$ for each feature dimension $d$, where $r(\mathbf{Z}_{\text{S}}^{d})={q(\mathbf{Z}_{\text{T}}^{d})}/{p(\mathbf{Z}_{
\text{S}}^{d})}$. This ratio represents the density of the target domain relative to the source domain.

##### Step 3. Model construction.

We constructed the target model using the IW method. Specifically, we built a classification model using the training dataset $(\mathbf{X}_{\text{S}}^{d},y_{\text{S}})$, where the estimated density ratio $\hat{r}(\mathbf{Z}_{\text{S}}^{d})$ served as the sample weights for the IW method. Additionally, we constructed a prediction model using only the source data, i.e., a model built without importance weighting.

##### Step 4. Verification of prediction accuracy

To evaluate the prediction accuracy of the models, we selected the ROC AUC score, as it measures accuracy independently of the thresholds used for label determination.
For the classification tasks in domain adaptation, we employed two classification methods, each representing a different algorithmic approach: LogisticRegression from the scikit-learn library (Pedregosa et al., ) for linear classification, and LightGBM (Ke et al., ) for nonlinear classification.

The hyperparameter sets for both methods used to evaluate prediction accuracy on target domains were selected using the IWCV method (Sugiyama et al., ). These hyperparameter sets were defined as all combinations of the values listed in Table 6 for LogisticRegression and Table 7 for LightGBM, respectively.
Finally, the prediction accuracies on the target domain were assessed using the best model selected through IWCV, with the target domain data $(\mathbf{X}_{\text{T}}^{d},y_{\text{T}})$ used for reporting.

**Table 6: Hyperparameter values for LogisticRegression. "Hyperparameters" shows the hyperparameter names used in the library. Texts inside parentheses provide explanations of the parameters.**
| Hyperparameters | Values |
| --- | --- |
| l1_ratio (Elastic-Net mixing parameter) | 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0 |
| lambda (Inverse of regularization strength) | 0.0001, 0.001, 0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, and 5 |

**Table 7: Hyperparameter values for LightGBM. "Hyperparameters" shows the hyperparameter names used in the library. Texts inside parentheses provide explanations of the parameters.**
| Hyperparameters | Values |
| --- | --- |
| lambda_l1 ($L_{1}$ regularization) | 0.0, 0.25, 0.5, 0.75, and 1.0 |
| lambda_l2 ($L_{2}$ regularization) | 0.0, 0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, and 4.0 |
| num_leaves (Number of leaves in trees) | 64, 248, 1024, 2048, and 4096 |
| learning_rate (Learning rate) | 0.01, and 0.001 |
| feature_fraction (Ratio of featuresr used for modeling) | 0.4, 0.8, and 1.0 |

**Table 8: Original data dimensions ($\text{dim}(\mathbf{X})$) used to obtain feature dimensions ($\text{dim}(\mathbf{Z})$) by embedding.**
|  | Feature dimensions ($d=\text{dim}(\mathbf{Z})$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | $d=8$ | $d=16$ | $d=32$ | $d=64$ | $d=128$ |
| Original data dimensions ($\text{dim}(\mathbf{X})$) | 500 | 700 | 900 | 1700 | 4600 |

#### D.2.3 Consideration of the Feature Embedding Method

Let $f:\mathbf{X}\longmapsto\mathbf{Z}$ denote a $C^{1}$-class embedding map which transforms the original data $\mathbf{X}\subseteq\mathbb{R}^{N\times D}$ to the feature table $\mathbf{Z}\subseteq\mathbb{R}^{N\times d}$ with $d<D$.

We now demonstrate that if $f$ is injective for both the source and target domain data, it preserves the density ratio between the target and source domain densities when mapping from the original data to the embedded data.

To see this, we use the singular value decomposition (SVD) of the Jacobian matrix $J_{f}(\mathbf{x})$ of $f$, which gives

$$ $J_{f}(\mathbf{x})=U(\mathbf{x})\cdot\Sigma(\mathbf{x})\cdot V^{T}(\mathbf{x})$ $$

with

$$ $\Sigma(\mathbf{x})=\begin{pmatrix}\sigma_{1}(\mathbf{x})&0&\dots&0\\ 0&\sigma_{2}(\mathbf{x})&\dots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\dots&\sigma_{\text{dim}(\mathbf{z})}(\mathbf{x})\\ 0&0&\dots&0\\ \vdots&\vdots&\dots&\vdots\\ 0&0&\dots&0,\\ \end{pmatrix}$ $$

where $U(\mathbf{x})$ and $V^{T}(\mathbf{x})$ are orthogonal matrices in $\mathbb{R}^{\text{dim}(\mathbf{x})\times\text{dim}(\mathbf{x})}$ and $\mathbb{R}^{\text{dim}(\mathbf{z})\times\text{dim}(\mathbf{z})}$, respectively, and $\sigma_{i}(\mathbf{x})\neq 0$ for all $i$.
This leads to the following relationship for the probability densities between the original and embedded data:

$$ $p_{\mathbf{X}}(\mathbf{x})=\left(\prod_{i=1}^{\text{dim}(\mathbf{z})}\sigma_{i }(\mathbf{x})\right)p_{\mathbf{Z}}(f(\mathbf{x})).$ (88) $$

From Equation (88), the probability density ratio between the source and target domains of data embedded by $f$ is obtained as

$$ $\frac{q_{\mathbf{X}}(\mathbf{x})}{p_{\mathbf{X}}(\mathbf{x})}=\frac{\left( \prod_{i=1}^{\text{dim}(\mathbf{z})}\sigma_{i}(\mathbf{x})\right)\cdot q_{ \mathbf{Z}}(f(\mathbf{x}))}{\left(\prod_{i=1}^{\text{dim}(\mathbf{z})}\sigma_{ i}(\mathbf{x})\right)\cdot p_{\mathbf{Z}}(f(\mathbf{x}))}=\frac{q_{\mathbf{Z}} (\mathbf{z})}{p_{\mathbf{Z}}(\mathbf{z})}.$ $$

Therefore, $f$ preserves the density ratio from the original data to the embedded data.
Additionally, if $f$ is a matrix multiplication, its injectivity can be achieved for $\mathbf{X}$ by reducing its dimensionality sufficiently.
Reducing the dimensionality of $\mathbf{X}$ can induce the injectivity of $f$.

We heuristically detected the injectivity of our embedding by observing the following:
We identified the largest subset of columns in
$\mathbf{Z}_{\text{S}}^{d}$ such that a sufficient increase in the KL divergence between $P(\mathbf{Z}_{\text{S}}^{d})$ and $P(\mathbf{Z}_{\text{T}}^{d})$ increased significantly as the number of columns increased. Injectivity was assumed for columns within this subset.

Although our feature embedding procedure is based on heuristic observations and lacks thorough theoretical analysis, we find it adequate for evaluating the performance of the proposed method in DRE downstream tasks with real-world data as the number of features increases.

The number of columns in the original data used in the experiments is listed in Table 8.

##### Neural Network Architecture, Optimization Algorithm, and Hyperparameters.

A 5-layer perceptron with ReLU activation was used, with each hidden layer consisting of 256 nodes. For optimization, the value of $\alpha$ was set to 0.5, the learning rate was 0.0001, and the batch size was 128. Early stopping was applied with a patience of 1 epoch, and the maximum number of epochs was set to 5000.
The PyTorch library (Paszke et al., ) in Python was used to implement all models for DRE. The Adam optimizer (Kingma, ) in PyTorch, along with an NVIDIA T4 GPU, was used to train the neural networks.

##### Results.

The results are shown in Figure 5 (LogisticRegression) and Figure 6 (LightGBM). The domain names at the origin of the arrows in the figure titles represent the source domains, while those at the tip represent the target domains. The x-axis of each figure shows the number of features, and the y-axis represents the ROC AUC for the domain adaptation tasks.
The orange line (SO) represents models trained using source-only data, i.e., models trained using source data without importance weighting, while the blue line (IW) represents models trained using source data with importance weighting.

Prediction accuracy for the models trained solely on the source data improved as the number of features increased, which is expected since more features typically lead to better accuracy. However, for both Logistic Regression and LightGBM, the performance of the IW method deteriorated as the number of features increased. A more significant decline in performance with increasing features was observed for most domain adaptation (DA) tasks, except for "books $\rightarrow$ DVDs" and "kitchen $\rightarrow$ DVDs".
These results suggest that the distributions shifted by the estimated density ratios using the proposed method diverged further from the target domain as the number of features increased. Consequently, the accuracy of the density ratio estimation (DRE) likely worsened with more features.

Figure: Figure 5: Results of Section D.2 for LogisticRegression. In the figure titles, domain names at the origin of the arrows indicate the source domains, while those at the tip represent the target domains. The x-axis shows the number of features, and the y-axis represents the ROC AUC for the domain adaptation tasks. The orange line (SO) denotes models trained using source-only data (i.e., models trained on source data without importance weighting), while the blue line (IW) represents models trained using source data with importance weighting.
Refer to caption: x5.png

Figure: Figure 6: Results of Section D.2. Results of Section D.2 for LightGBM. In the figure titles, domain names at the origin of the arrows represent the source domains, while those at the tip indicate the target domains. The x-axis shows the number of features, and the y-axis represents the ROC AUC for the domain adaptation tasks. The orange line (SO) denotes models trained using source-only data (i.e., models trained on source data without importance weighting), whereas the blue line (IW) represents models trained using source data with importance weighting.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 1
strengths: - **S1.** The problem addressed by the authors is of great interest to a part of the ML community.
- **S2.** The authors gather a good summary of the problems that existing approaches have.
- **S3.** The authors make a great effort into explaining the details of their proposed loss, which looks like a sensible approach.
- **S4.** While I have only skimmed through them, the results in the appendix on real-world data looks good, and I am not sure why it is not in the main manuscript.
summary: This work studies the problem of density ratio estimation (DRE), i.e., estimating the ratio $p(x)/q(x)$ using neural networks, where both $p$ and $q$ are probability densities. The authors argue that the common approach of optimizing a variational formulation of the KL-divergence (whose optimum is $p/q$) has a number of difficulties, namely: overfitting, biased and vanishing gradients, and high effective sample size. The authors then consider an alternative formulation based on Amari's $\alpha$-divergences instead, and develop a parametrization which does not suffer from the same problem as the original one. The author demonstrate these properties empirically, yet they also show that in terms of regression error, there is no significant advantage of using their method instead of the KL-divergence loss.
weaknesses: - **W1.** The presentation could be improved, e.g., there are some clear typos that should be corrected ("hucking" instead of "hacking" in section 6.1, "Peason" in Table 1, "simbole" in line 231).
- **W2.** Some of the "intuitive explanations" of the paper are wrong or are questionable at best. For example:
  1. The problem of train-loss hacking is attributed here to the loss not being lower-boundered, but I'd say that the problem is explained through the fact that there is a finite number of training samples. You could still lower-bound the loss, if the lower-bound is only tight at the limit, use the same argument as in section 4.2.
  2. Line 161 is wrong. Under mild conditions, the linearity of the integral allows you to swap the integral and gradient, as long as $Q$ does not depend on $\theta$, which I understand is the case. 
  3. The arguments in section 4.4 sound cyclic, i.e., arguments (i) and (ii) of lines 181-184 are equivalent to the gradients vanishing.
  4. (more nitpicky, but:) The assumption in the appendix for the theorem 6.1 is local L-continuity, which is worth mentioning in the main paper.
- **W3.** I have found several errors and questionable explanations in the manuscript, which make me worry about whether the results presented in the manuscript hold after fixing them. For example:
  1. Line 269 is false under fairly general assumptions. You can swap gradients and integral as long as Q does not depend on $\theta$ (or if Q is reparametrizable).
  2. The last two columns of Table 2 are all wrong as far as I can see ($E$ should be $\hat E$ and the $Q$ in the last column should be $P$.) 
  3. While not "wrong" per se, as it does not change the arguments of the paper, writing a difference between vector infinities in Table 4 is rather concerning.
  4. I quickly skimmed through the proofs, and I do not understand Eq. 57 at all. Are the authors writing a gradient as the limit of a function as $\psi$ approaches $\theta$? Where is this definition coming from?
- **W4.** The baseline methods to compare their approach with change across experiments, which I cannot fully undestand. Why not using nnBD-LSIF for the experimerints in section 7.3?
- **W5.** This is more of a personal taste, but I really dislike some of the wording chosen in this manuscript, e.g., using "naive" and "merely" as freely as in lines 48-49. It downplays the contribution of previous works in an unfair manner.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Experiments:
  + Why are there different datasets for each experiment, how were the parameters of the distributions chosen?
  + How was $\alpha$ chosen in your experiments and how is this hyperparameter selected in general?
  + Did KL-divergence face any of the claimed issues in the third experiment?
  + Did nnBD-LSIF face any of the claimed issues in the second experiment?

Theory:
+ Biased gradients
   * I did not immediately see why the gradient for KL is biased, can you show this formally? Biased gradients are only claimed for KL, what about other f-divergences?
   * Correct me if I'm wrong, but most kernel-based methods for DRE don't suffer from biased gradients.
+ Vanishing gradients
   * I do not really understand (3): Isn't (i) already implying vanishing gradients?
   * For me it is not clear how the limits in Table 2 are computed, a more formal derivation is needed.
   * Why should (3) happen during training? Why do very small/large density ratios imply (3), in particular how does this determine the gradient in parameter space?
   * Please explain the correctness of the two equivalences in line 347 in more detail.
   * Please show the computation of the limits in Table 4 in more detail.
+ Sample rate
   * A bad sample rate is only shown for KL-divergence, what about the other f-divergences? How does this relate to fast sample rates for DRE as discussed in [4]?
   * Describe the implications of Theorem 6.2 for the sample rate in more detail.
soundness: 2
strengths: - The authors address important issues in DRE.
- Improving DRE is an important problem that has significant impact on other ML disciplines, e.g., OOD detection and two-sample testing.
- Interesting ideas of how to investigate possible improvements of f-divergences for DRE.
summary: The authors identify four major issues with existing f-divergence loss functions for DRE. They discuss train-loss hacking, biased gradients, vanishing gradients and sample rates. All of these issues are theoretically analyzed. The authors introduce an $\alpha$-divergence loss function to tackle these issues and show theoretical results for it. Experiments were conducted to investigate the effectiveness of new loss function.
weaknesses: + Related work for DRE should in general be discussed more elaborate and in more detail. Novelty in comparison to these works is unclear.
+ Insufficient comparison to other neural net or kernel-based DRE methods such as [1,2,3], insufficient comparison with other f-divergences mentioned in the submission.
+ Claimed issues of f-divergences loss functions do not seem to be a problem; as performance of $\alpha$-Divergence is worse than KL-divergence for DRE in the third experiment.

[1] Kato, Masahiro, and Takeshi Teshima. "Non-negative bregman divergence minimization for deep direct density ratio estimation." International Conference on Machine Learning. PMLR, 2021.
[2] Rhodes, Benjamin, Kai Xu, and Michael U. Gutmann. "Telescoping density-ratio estimation." Advances in neural information processing systems 33 (2020): 4905-4916.
[3] Menon, Aditya, and Cheng Soon Ong. "Linking losses for density ratio and class-probability estimation." International Conference on Machine Learning. PMLR, 2016.
[4] Gruber, Lukas, Holzleitner, Markus, Lehner, Johannes, Hochreiter, Sepp, and Zellinger, Werner (2024). Overcoming Saturation in Density Ratio Estimation by Iterated Regularization. arXiv preprint arXiv:2402.13891.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: Line 161 says that $\nabla_{\theta} E_{Q}[log \phi_{\theta}] = E_{Q}[\nabla_{\theta} log \phi_{\theta}]$ does not hold because $E_{Q}$ is an integral. Why? Note that the distribution $Q$ is independent of $log \phi_{\theta}$. Similarly, other statements about the biased gradient problem are questionable.

Line 193 states that "These results demonstrate that major f-divergence loss functions satisfy the conditions for Equation (3)", which is clearly not true. See the KL and Peason $\chi^2$ cases.

Where is the proof of Eq. (4)?
soundness: 2
strengths: The paper is easy to follow.

Stable and accurate DRE is essential for machine learning, especially for GANs. The authors have provided some valuable and original research results.

Conventional DRE methods, as well as the presented $\alpha$-Div, are compared through the lens of lower boundedness and the vanishing gradient problem.
summary: The paper presents a new $\alpha$-divergence-based method for density ratio estimation (DRE). Compared to existing DRE methods, the presented method addresses optimization challenges such as overfitting due to lower-unbounded loss functions, vanishing training loss gradients, etc. Numerical simulations demonstrate the effectiveness of the presented method.
weaknesses: The paper should be revised carefully. There are many typing errors. For example, see $\hat E P[R]$ and $\hat E P[\phi]$ in Line 107 and the word "simbole" in the title of Table 2. Also, either $E_{Q}[\phi^{\alpha}]$ in Eq. (6) or $E_{Q}[e^{\alpha T}]$ in Eq. (8) is incorrect.  

Some important statements are questionable. See Questions below.

Experiments are only done on simulations. It would be great if the presented method could be varied in practical scenarios associated with GANs.

Meta-review: This paper addresses the problem of density ratio estimation (DRE). The authors propose to use \alpha-divergence loss so that several problems (i.e., bias, overfitting, vanishing gradient) in standard methods can be resolved.  

The problem itself is important. This paper addresses an interesting problem. However, the writing can be much improved. There are typos and unclear logics. Comparison with existing work is not thoroughly given. These issues could be addressed but a substantial revision will be required.  

For these reasons, this paper cannot be accepted.

Decision: Reject