Projection Pursuit Density Ratio Estimation
Authors: Meilin Wang
Wei Huang
Mingming Gong
Zheng Zhang
Venue: ICML 2025 poster (2025)
Keywords: density ratio estimation
projection pursuit
covariate shift adaptation
Primary area: general_machine_learning
TL;DR: This paper proposes a projection pursuit-based method for density ratio estimation to address high-dimensionality and model misspecification challenges, with theoretical guarantees and superior experimental performance.

Abstract: *Density ratio estimation* (DRE) is a paramount task in machine learning, for its broad applications across multiple domains, such as covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimating the density ratio possibly lead to biased results if models are misspecified, while conventional non-parametric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose a novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the impact of high dimensionality while retaining the model flexibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator.  Experimental results demonstrate that our proposed method outperforms existing alternatives in various applications.

Projection Pursuit Density Ratio Estimation
Meilin Wang'
Wei Huang?
Mingming Gong? Zheng Zhang
Abstract
DRE and its applications in machine learning.
Density ratio estimation (DRE) is a paramount task in machine learning, for its broad applica- tions across multiple domains, such covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimat- ing the density ratio possibly lead to biased results if models are misspecified, while conventional non con-aaammctric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the im- pact of high dimensionality while retaining the model flcxibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator. Experimental results demonstrate that our proposed method out- performs existing alternatives in various applica- tions.
There are a variety of methods concerning the DRE prob- lem in the existing literature, which can be divided into three paradigms: the parametric methods (Liu et al., 2013; 2017; Nagumo & Fujisawa, 2024), the nonparametric lin- ear sieve methods (Kanamori et al., 2009; Sugiyama et al., 2008; 2010) and the nonparametric neural network methods (Nam & Sugiyama, 2015; Fang et al., 2020; Rhodes et al., 2020). The validity of the parametric modeling builds on the correct model specification for the density ratio. If the model is misspecified, the results may be severely biased. Nonparametric neural network methods (Nam & Sugiyama, 2015; Fang et al., 2020; Rhodes et al., 2020) can achieve superior performance without imposing parametric model assumptions; however, they require substantial number of training samples to learn the extensive parameters involved. Furthermore, theoretical guarantees for neural network esti- mations remain insufficient.
Nonparametric linear sieve estimation is a statistical tech- nique that combines the flexibility of nonparametric meth- ods with the simplicity of linear models (Chen, 2007b). The term "linear sieve" refers sequence of increasingly complex linear models that are used to approximate the un- known function of interest. With different choices of criteria, e.g. squared-loss (Kanamori et al., 2009; Sugiyama et al., 1; Yamada et al., 2013), Kullback-Leibh divergence (Sugiyama et al., 2008; Tsuboi et al., 2009), the method of sieves is highly flexible in estimating complicated models. Unlike nonparametric neural network methods that can be- come excessively complex and computationally intensive, the sieve approach seeks to balance model complexity with interpretability and computational feasibility.
1. Introduction
Density ratio estimation (DRE) is a fundamental concept in machine learning that focuses on directly estimating the ra- tio between two probability density functions (PDFs), avoid- ing the need estimate each density function individually. DRE has widespread applications, such as covariate shift adaptation (Shimodaira, 2000; Sugiyama et al., 2007), out- lier detection (Hido et al., 2011) 1), independence test (Ai et al., 2024), mutual information estimation (Suzuki et al., 2009; Ai et al., 2024), importance sampling (Meng & Wong, 1996; Sinha al., 2020), and treatment effect estimation in causal inference (Ai et al., 2021; Matsushita et al., 2023). See Sugiyama et al. (2012) for comprehensive review of
Although the linear sieve methods enjoy computational con- venience and theoretical support (Kanamori et al., 2010), they suffer from a significant challenge known as the curse of dimensionality. Specifically, as the dimension of the data increases, there is notable and substantial deterioration in the performance of these linear sieve methods. This phe- nomenon is clearly evidenced by both ratio pattern visualiza- tion and estimation error in our experimental results. Con- ventional linear sieve methods, such as KLIEP (Sugiyama et al., 2007) and ul SIF (Kanamori et al., 2009) can cor- rectly capture the pattern of density ratio functions when
Center for Applied Statisties, Institute of Statisties and Big Data, Renmin University of China, Beijing, China Schooo of Mathematies and Statistics, University of Mel- bourne, Melbourne, Australia. Correspondence to: Zheng Zhang <zhengzhang wuc..eeeeeeooeeeee edu.en>. @ruc.edu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Projection Pursuit Density Ratio Estimation
the dimension of variables is 2 (see in Figure 1). However, they experience significant deterioration in performance when the dimension of variables increases to 10 (see Figure 2). The root mean squared logarithmic error (RMSLE) for the DRE based on conventional linear sieve methods further quantifies this performance decline, exhibiting a dramatic escalation as the dimension of variables increases (see in Figure 3). This deterioration in performance is attributed to the sparsity of the data in high-dimensional spaces, which hinders the ability of the linear sieve methods to effectively learn from the data and generalize to new, unseen instances.
2. Related Work
While this work primarily addresses the curse of dimension- ality in DRE, the field has also made significant progress on two related challenges: stabilizing estimators through effective regularization strategies, and addressing geometric disparities between distributions known as density-chasm effects.
Regularized Kernel Learning Methods. The regular- ization scheme within reproducing Kernel Hilbert space (RKHS) has been developed for estimating the DRE prob- lem. Que & Belkin (2013) reformulated the DRE problem as an inverse problem in terms of integral operator corre- sponding to a kernel, then proposed a regularized estimation method with an RKHS norm penalty. Gizewski et al. (2022) applied the regularized kernel methods in the context of unsupervised domain adaptation under covariate shift and developed the convergence rates. Gruber et al. (2024) pro- posed iterated regularization and developed an improved error bounds faster than the non-iterated error bound under the Bregman distance and certain regular conditions (e.g. source condition and capacity condition). Nguyen et al. (2024) established the pointwise convergence rate of the reg- ularized estimator taking into account both the smoothness of the density ratio and the capacity of the space in which it is estimated.
To address the curse of dimensionality, a variety of D3 (Di- rect DRE with Dimension reduction) methods have been pro- posed (Sugiyama et al., 2010; 2011; Yamada & Sugiyama, 2011) by incorporating a dimension reduction step prior to applying a linear sieve estimation. These dimension duction techniques assume that the density ratio function is located in low-dimensional intrinsic space character- ized by a linear transform of the high-dimensional variables. However, this fundamental assumption can be restrictive in many practical scenarios.
To overcome these limitations, in this paper, we propose novel method for DRE based on the projection pursuit (PP) (Friedman & Tukey, 1974). The core idea is to approximate the density ratio function by product of PP functions and estimate them iteratively. Each of the PP functions is a pro- jection of the target function to a low-dimensional space. As a result, at each iteration, we only need to estimate semiparametric single-index function based on a univari- ate linear sieve basis. It is known that the approximation error vanishes as the number of iterations approaches infin- ity, mitigating the curse of dimensionality when the target function can be expressed as many (or infinitely many) low- dimensional projections along different directions (Diaconis & Shahshahani, 1984). We further provide a theoretical jus- tification for our method by establishing its consistency and convergence rate for the proposed estimator. In application, we apply the proposed method in causal inference, mutual information estimation and covariate shift adaptation, and find consistent improvements in performance.
Density-Chasm Problem Density ratio estimation faces additional challenge known as the density-chasm prob- lem, which occurs when the distributions differ substan- tially (Rhodes et al., 2020; Choi et al., 2021; 2022). This phenomenon arises because samples are less likely to be observed in the low-density regions between the two dis- tributions. To overcome this challenge, Choi et al. (2021) proposed invertible parametric transform mapping the data onto a shared feature space, thereby bringing the trans- formed densities become closer. Then they estimate the density ratio in the feature space based on the key property that, the ratio remains invariant under such invertible trans- formation. Notably, this invertible transformation preserves the original data dimensionality. Therefore, it is necessary to clarify that they neither map the data low-dimensional latent space nor address the curse of dimensionality as fo- cused by our paper.
This paper is organized as follows. Section 2 discusses recent advancements in the field through analysis of related work. Section 3 introduces the conventional methods for DRE from the perspective of model specification. In Section 4, we propose the projection pursuit DRE method that can achieve efficient estimation even in high-dimensional data settings, and we establish the theoretical properties. Section 5 demonstrates the superiority of our proposed method in various applications.
3. Density Ratio Estimation
Let p(x) and q(x) be two probability density functions of the target and reference datasets respectively, where e Rd is a d-dimensional variable. The density ratio estimation (DRE) problem is to estimate
r r(xx) p(x)
g(x)
2

---

Projection Pursuit Density Ratio Estimation
based on two independently and identically distributed (i.i..)) samples from the two referred distributions, i.e. linl
ratio function is of order p -- (Lorentz, 1986, Theorem 8), where is the Hölder-ssmoothnsss of r (x), thus it requires p larger than [1/e] to achieve é-еrrоr approximation accuracy. On the other hand, a large p will significantly en- large the variance of these estimators; indeed, the variance of the linear sieve estimator is of order vp/n (Li & Racine, 2023, Theorem 15.1).
(xf") fp iid. p(x) and (x
fy iid.
Fiml ~ (x). To make the den- sity ratio function r" r"(x) well-defined, we assume (xx dom- inates p(x), i.e. p(x)= > 0 implies that (xx) > 0, and Rd denotes the support of rxx).
Below, we briefly summarize the existing methods for the DRE problem and highlight their limitations.
Deep Neural Network Methods Several deep neural network-based methods have been proposed for the DRE problem (Nam & Sugiyama, 2015; Fang et al., 2020; Rhodes et al., 2020; Kato & Teshima, 2021; Choi et al., 2022), with the aim of improving performance in nigh-dimensione data. Despite their superior performance, these methods tend to lack good interpretability and theoretical guarantees. More- over, the effectiveness of deep neural network estimation is intrinsically dependent the availability of substantial training datasets, and its computational burden is much heavier than the parametric and linear sieve methods.
Parametric Methods The density ratio function is as- sumed to satisfy the following parametric model (Liu et al., 2013; 2017; Nagumo & Fujisawa, 2024):
(x) = (=xxp(0xx(0)x)) hxx)),
where C is a normalizing constant, hxxx) Rd R is the feature transformation function, p is known fixed positive integer. and 0 E R.' is the parameter of interest to be estimated. See Nagumo & Fujisawa (2024, Appendix A) for more detailed discussion on this parametric formulation. The parametric formulation is particularly useful for sparse estimation in high-dimensional problems. However, it relies heavily on the correct model specification and may fail to capture non-linear relationships or interactions within paramarretticc methods. This
4. Projection Pursuit DRE
Projection pursuit (PP) is a statistical technique used for mul- tidimensional data analysis. The key idea approximate high-dimensional function by progressively projecting it onto efficient lww-dimessional spaces that capture the most significant features of the data structure. It was first de- veloped by Friedman & Tukey (1974) for exploratory data analysis and has been applied to various problems such as classssiiiin (Lee et al., 2005; da Silva et al., 2021), PP- regression (Friedman & Stuetzle, 1981; Zhan et al., 2025), and PP-density estimation (Friedman et al., 1984; Aladjem, 2005). Being inspired by the strengths of PP in mitigating the curse of dimensionality (Huber, 1985), we propose a PP-based method for estimating the density ratio r'xx) and develop valid asymptotic theories
the data effectively limitation diminishes their adaptability and flexibility in handling complex or varying datasets.
Nonparametric Linear Sieve Methods The density ratio function is approximated by a sequence of increasingly complex linear models (Kanamori al., 2009; Sugiyama al., 2008; 2012):
xxxx = (x) == 0 (xx.
or log-linear models (Kanamori et al., 2010; Tsuboi et al., 2009):
r' (xx) = Hm (x) == xx (x)).
4.1. Projection Pursuit Approximation
where (x) RP is a user-specified basis function (also called "sieve" in statistical literature), E R is the p-dimensional coefficient parameter, and p E N is unknown and will go infinity at appropriate rate the sample size increases. The linear sieve methods adaptively learn the model parameters (0. p) from the data by minimizing the empirical discrepancy between the true density ratio function and its linear sieve estimators under some distance measure ge.g. the Kullback-Leeble divergence, the squared distance, and the general Bregman distance). The linear sieve methods seek to balance model complexity with inter- pretability and computational feasibility.
We propose to approximate the density ratio function r' r'(x) by the multiplicative projection pursuit:
K
r' r'(x) x Fk))) [
where K E EN denotes the number of projections, {ax) unit d-dimensional vectors indicating the projection direc- tions, and {fx | are unknown univariate pursuit functions. That is, the PP approximation (1) converts the estimation of r" (xx, whose direct estimation is challenging in the case of large-dimensional x, to simpler tasks of estimating projected univariate functions (ff x))
k=1 are
However, the performance of linear sieve methods will sig- nificantly deteriorate as the dimension of x becomes large.
Such a curse of dimensionality can be explained as fol- lows. The linear sieve approximation error for the density
Estimation of (fe(a
| ket can be carried out iteratively
based on the relation rx(x) fx) for
3

---

Projection Pursuit Density Ratio Estimation
(),..,K), where ro(x)= = 1. Specifically, let [ and EpI-] denote the expectation taken with respect to (w.r.t) the prob- ability densities g()) and p()), respectively, Suppose that the first k - terms are given, that is (x)= = -m fm x)) is given, the kth projection direction 4 and the kth pur- suit function fk can be determined by minimizing the 2
Proposition 4.1. For every fixed Rd, we have
B (a) := arg B:a).B:a)
B
(x(a) (a) + J1
Wx(a)
14
np
distance, E, ( ([r"(x) (x) - -x .))]a, We show Ap-
pendix E that minimizing the L'-distance w.r.t or a is
where
equivalent to minimizing
Zx(a): == (fx-1(x;) (xx11(x)00x(''xx xx2)x ERa
H(f.a)==BB 1 (x) x)]
(x(a) == (x-1(x))) (x) ER",
- 2=plrx-(x)f((a'r
(2)
E=1
For model identification, we assume a& E S in Wang & Yang (2009), where 8 is the d dimensional upper unit hemisphere, i.e. 2 (a (d1.-..,ad) Rd ||a|| = 1, a1 > 0), and ||||| denotes the Euclidean norm, i.e. for vector v, ||>|| == Vv
and 1 is an identity matrix of size Jg x
Proof. Using above notation, La (a. B:d) can be written as quadratic function of B:
1 Rg
Cc(a.B.))
B Px-1(x (x²))x >
4.2. Estimation Procedure
Bq
Since the function space for searching fx is infinitely dimen- sional, direct optimization based on the sample analogue of (2) is impossible. We consider seeking the estimator of the pursuit function fz),,, for Z {a x a E , EX), from the linear sieve class:
2 B np
Σ
Ap
PK- Fx-1
-Dx(a x²))
+ +1B B
=B' (a)
BB2B Wx(a).
+ 11 J2
B
ng
n
J ( (<) ( (,o) Fr. := B
Differentiating it with respect to B and setting the derivative to zero give the desired result.
J
F=1
where ( is sequence of univariate basis, (x(2)=
By Proposition 4.1 and (3), the estimators of [z) and a
can be represented respectively as
( (z). du T and B (BI....BB,,)) is the approx- imation coefficients. The rationale is that any continuous function be approximated arbitrarily well by a linear sieve in Fj, as Ju goes to infinity (see e.g., Chen, 2007b).
(z) := (a) (x(2),
(5)
and
For k= = l,...,k, K, based on (2) and the linear sieve class, we define the estimator in the kth iteration of (fx. by
â =argmin Lx(a,Br(a):A) acRd
fa( = B BT) (z),, (ax.Bx) == arg min (a. (3)
argmin acRd
| [Z)
a.B
2ng
where
B,(a).
np
Na
La(a,Bid)= =
R- (x IB (a (4)
Then, we have fa(z) = (2). the estimators of {re(x)) are defined by
n4
i=1
ap
Pk- (x (xx
fx(x) = - (x ke ().
Hp
is the regularized empirical loss, where (x)1(x) Ih (4,x)))) 2| == B is the f2-penalty on the model complexity, and A > 0 is a tuning parameter. Here, we use the fz-penalty to circumvent the over-fitting prob- lem, while maintaining a closed-form solution in B for the problem (4) when keeping a fixed.
and the estimator of the density ratio function r (xx) is given by fx (x). In practice, to ensure non-negativity of the esti- mated density ratio function, we truncate the negative values of fr to the minimum positive estimated values for every iteration. The complete process of our approach given tuned hyper-parameters is summarized in Algorithm

---

Projection Pursuit Density Ratio Estimation
In the following theorem, we focus on each iteration k and show that, given the estimate (x) of (x)) as the sample sizes Mp - co, fa, and åg converge respectively to
to "p and ng to guarantee the consistency of our estimators. Such conditions are standard in sieve regression. Assump- tion F.4 requires the Hessian matrix of the sieve approxima- tion of Hff a) to be positive definite at its minimum w.r.t. a, which is met when the minimum is in the interior of A.
la = argmin H(.a))
(6)
and
Algorithm 1 ppDRE
a: == argmin Ha.a,...)
(7)
1: Input: samples (x liz and (x,) number of basis functions Jg. learning rate 6, ridge penalty parameter A, maximum steps K.
Theorem 4.2. Suppose that Assumptions F.l to F.4 in Appendix F hold. Then, for each k = K, if
"p E (Fx-1(x;) - -xx- O (Em..-1). we have
2: Initialize: fo(x) =
for k 1 to do Initialize randomly a (0)),(0)) (0) and set 1.
sup sup -z) a
5: while not converge do
=0p (n(sa) (En,A-160(J)
6: Calculate
2 - (fk-1 (x D0 (t-1) x9:: (1-1))))
(JR5O(JW)
(8)
vgg Anp
"p w
pk- (x (a x ((-1))))))
(1-1)T
and
1=(00 11-1) 11 + VEnk-1
7:
Update
|ي_ -xxl= =0p
(9)
p - Z +2l
1 w
VJk
(1(2))
"p
yng A ARp!
Evaluate the loss function
where A c St is compact set containing ak. Z (axx E A and x X}, s denotes the number of continuous derivatives that fax(z)) possesses w.r.t. z I for any a E A. {o(kk) is a sequence of constants such that supzez [2)(z)] s Zn(Jx), (J))) is a se- quence of constants such that the maximum eigenvalue of k
L 1 (1) (0
n4 Bk 112
2 np
(1) pl w + a pbp.
Update (a) xx/ 2 with stochastic gradient de- scent algorithm, such Adam, with learning rate 6.
B[D k in a E A, and ng A np = mi))n...pp)
(1) (a ( (a T] is bounded by (12) uniformly
10: Update / 1+11
11: end while
12: Update fx(x) - fx-1 B (a ( (1) xxy(( xx)x())
(!)
Finally, we have
13: end for
Σ (s-1)
K
Je
14:
return fx(x).
sup xEX
ng Anp
x VIVV soouonGGooo)
Remark 4.3 (Selection of Tuning Parameters). In practice, suggest using cross-validation (CV) to determine the number of projections K as well as other tuning parameters, Specifically, in experiments and applications, we adopt the Gaussian basis (z(2) = = =exp(-(2-y)) where y is the location parameter of the Gaussian basis. For each k E (1....,K), K), we determine / together with à and Bg jointly by minimizing the loss function Lx. We monitor the minimal value of the loss function Lx in a validation set for a gradually growing K, and determine the optimal K until no further improvement is observed.
where )
The proof is available in Appendix F. Assumption F.1 im- poses compactness conditions on X and parameter spaces A. Assumption F.2 requires (2) to be bounded and bounded away from 0, s-times continuously differentiable w.r.t ZE Z and has bounded derivative w.r.t. a E A, which are some common regularity conditions in the literature. Assumption F.3 excludes near multicollinearity among the basis functions, dD (z), and regulates the rate of Jg relative
5

---

Projection Pursuit Density Ratio Estimation
Dere
"osdes"
x5r
oxtrs
ube
color
Cauriaaes
100%c
(a) Estimated f(x)
(b) Contour plot of F(x)
(a) c
(b) c 2e1
Figure 1. 2-D DRE Experiment. The warmer color represents higher estimated density ratio value. The top-left plot each panel shows the true density ratio, while the remaining plots illustrate the estimates using various methods.
Figure 2. Stabilized weights estimation (dx 10). The y-axis represents the treatment / and the x-axis represents c xx Each point represents sample data point, and the color indicates the magnitude of the estimated density ratio. The top-left plot in each panel shows the true density ratio, while other plots illustrate estimates using various methods.
5. Experiments and Applications
5.2. Application Causal Inference
We apply our proposed ppDRE method to estimate continu- ous treatment effects in the framework of Ai et al. (2021). Let T denote the observed continuous treatment status variable, with PDF fr (r). Let Y*()) denote the potential response when treatment is assigned, and let Y = Y denote the observed response. Let X Rdx denote vec- tor of observable covariates. To identify the causal effect, we assume the unconfoundedness condition that (1) and T are conditionally independent given X. We assume a parametric model g(:;0",, called the general dose-response function, for the potential outcome Y*):):
In this section, we compare our proposed projection pur- suit density ratio estimation (ppDRF) method with existing alternatives using experimental and real-world data. The compared methods include two classical linear sieve meth- ods, KLIEP (Sugiyama al., 2007) and uLSIF (Kanamori al., 2009), linear sieve method with dimension reduction, D'HLHSS (Sugiyama et al., 2011), probabilistic classifi- cation approach (Qin, 1998; Bickel et al., 2007) based on machine learning classifiers, two latest baselines, fDRE
(Choi et al., 2021) and RRND (Nguyen et al., 2024), and a neural network-based density ratio estimation (nnDRE) method, which is a variant of our framework in the sense that it replaces the projection pursuit with feedforward neural network to model the density ratio. Detailed descrip- tions of these baselines and implementation specifics are provided in Appendix A and Appendix B.
[ [[LLO (((00(()((()))))0)) =(::)))((((d)
0" aargmin BERP
aargmin B[ (YY-TXX)L(Y-((T0))) PERP
(10)
where L(-) is user-specified loss function, the second equality holds by the unconfoundedness condition, xo(1,x) is called the stabilized weights, defined by
5.1. 2-D DRE Experiment
We first consider toy example where p(x)= = N L.) and g(x) = (O,, 2))). where E) denotes a multivariate Gaussian probability density with mean and covariance matrix Σ, and I denotes an identity matrix of size d. We consider ciw-dimensional case with d 2. which facil- itates visualization of the estimated results. The sample sizes are Пp ny 5000. Estimates of this density ratio are shown in Figure la, and the corresponding contour plot is presented in Figure lb. These figures demonstrate that the proposed ppDRE method and the uLSIF method signifi- cantly outperform the other baseline approaches, yielding estimates that closely align with the true density ratio.
xn(1,x) = fx(1)x(x) fr(() fr,x(1,x) Frix (1(x))
where fr,x the joint PDF T and X, frix is the condi- tional PDF of given X, fr and fx are marginal PDFs of T and X, respectively.
The causal model (10) encompasses variety of contin- uous treatment effect parameters of interest. For exam- ple, with [(v) = 2 model (10) gives ((::0")=B((((())) the average dose-response function (ADRF). With L(v) = v(r [(v s 0)) for some r (0, 1), the model (10) gives
6

---

Projection Pursuit Density Ratio Estimation
3.0
ppDRE method consistently outperforms its competitors. Moreover, the advantage of ppDRE is more pronounced for larger dimension of covariates, Additional experimental results can be found in Appendix C.1.
ULSIF
KLIEP
Classificttion
2.5
D3-LHS5
AnDRE
2.0
5.2.2. DOSE RESPONSE FUNCTION ESTIMATION
RMSSE
This section investigates the performance of estimating both ADRF and QDRF using the semi-synthetic variant of the Infant Health and Development Program (IHDP) dataset. The original IHDP dataset (Hill, 2011) consists of 747 observations, each characterized by dx 25 covari- ates. Following Nie et al. (2020) and Gao et al. (2023), we generate the semi-synthetic IHDP-continuous dataset by leveraging the real-world covariates from the origi- nal IHDP dataset to simulate continuous dosages and re- sponses. A detailed description of the data generation pro- be found in Appendix D.1. Specifically, the domly assigned treatment T is generated from X by (12), and the potential outcome (r) htt, X) + 0.5c, where X) is defined in (13) and ~ NO0, 1). Then the true ADRF is g" g²(r) = Ex (h( and the true 7-th QDRF is g' (1) = inf[4 P (h(h,, X) + 0.55 2 q) s <].
2
10
30
50
100
Dimension
Figure 3. Average RMSLE over 50 replicates for stabilized weights estimation with varying dimensions of covariates.
the rth quantile dose-response function (QDRF) 8(1;0)== ((:+)) (y (T) iq P(Y (1) 2 q) r).
Note that, by definition, the stabilized weights xo((,x) can be viewed as a density ratio. We compare our method with different estimators of (t(xx)) based on simulated data sets in Section 5.2.1. In Section 5.2.2. we investigate the application of DRE in both ADRF and QDRF analysis based on semiisynthetic data set.
We estimate ADRF and QDRFs at various quantile levels (0.1, 0.225, 0.5, 0.75, 0.9) using parametric model g(:0) 00 +,/+ +0,22+000" +0,13 +0al,, , and obtain 0 by solving the empirical version of the optimization problem (10). To measure the accuracy of the estimated dose-response func- tions across the full dosage range, we compute the average squared error (ASE):
5.2.1. STABILIZED WEIGHTS ESTIMATION
In this simulation study, we design the treatment assignment model as T, = C X E,, where e E Rdx specified below, X, i.d. (0 lax) and E, iid. NO., I). In all simulation scenarios, the sample size is fixed to be 5000.
" ASE ΣΣ (s(T::)-
The mean and standard error of the ASE computed based on 100 replicates are reported in Table 1. The numerical results indicate that our ppDRE method attains the lowest mean ASE in most cases, demonstrating its superior performance compared to the other alternatives.
We first investigate the visual performance of the estimators for a fixed dimension of covariates, dx 10. To facilitate visualization, we consider two choices of the coefficient: c = 0.5e, and c = 2e,, respectively, where e is a vector with in the first component while others are zeros. In both scenarios, only the first component of x affects the treatment assignment. The estimated density ratios visualized in Figure 2, where the y-axis represents the treatment/ and the x-axis is e x. In both scenarios, the density ratio estimates produced by our PpDRE method show the closest alignment with the true density ratios.
5.3. Application in Mutual Information Estimation
The mutual information (MI) is measure of the mutual dependence between two continuous random vectors U and V, which is defined by
Ml I fu.v log
SU. (u,v) (u( dudr.
We further investigate the estimators' performances in terms of the root mean squared logarithmic error (RMSLE) de- fined in Appendix C. for varying dimensions of covariates dx E (2, 30, 50, 100). We set c 0.5 1dy to be a dx- dimensional vector whose components are all 0.5. Figure 3 presents the average RMSLE values of various estima- tors over 50 replications. The RMSLE of all estimators increases as the dimension dx grows, but our proposed
Given sample from (u.v can create another sample following the distribution fo (v) (v) by permuting the y vectors across the dataset. This enables the application of DRE methods to estimate the MI by
1 Σ np
Mlg.
log log
E=1

---

Projection Pursuit Density Ratio Estimation
Table 1. ASE in dose response function estimation with IHDP-contimuous dataset.
QDRF
Method
ARDF
T= 0.1
0.2.5
0.5
T= 0.75
T= 0.9
uL.IIF
0. 0.112(0.022)
0.338(0.058)
0.130(0.027)
0.045(0.016)
0.090(0.022)
0.278(0.064))
KLIEP
0.1 (2(0.22))
0.339(0.059)
0. 0.130(0.027)
0.045(0.016)
0.089(0.022)
0.278(0.064))
Classification
0.107)0.028)
0.341(0.065)
0.130(0.032)
0.045(0.018)
0.088(0.025)
0.276(0.066)
D'-LHSS
0.110(0.024)
0.336(0.058)
0. 0.128(0.030)
0.045(0.016)
0.088(0.022)
0.279(0.068)
fDRE
0.106(0.026)
0. 0.127(0.029) (0.36))
0.045(0.016)
0.088(0.021)
0.267(0.061)
RRND
0.117(0.022)
0.125(0.026) 0.307(0.053)
0.043(0.015)
0.090(0.021)
0.408(0.082)
anDRE
0.107(0.022)
0.333(0.059)
0.126(0.027)
0.043(0.015)
0.089(0.021) 0.297(0.066)
ppDRE
0.091(0.025)
0.338(0.066)
0.124(0.029)
0.041(0.016)
0.079(0.022)
0.266(0.060)
Table 2. MAE in mutual information estimation varying dimension p and correlation coefficient p.
2
p 10
20
Method
0.2
p 0.8
p=0.2
p p= 0.8
p= 0= 0.2
p= 0.8
uLSIF
0.044(0.001)
0.573(0.015)
0.200(0.008)
5.003)
0.415(0.023)
10.242(0.088)
KLIEP
0.044(0.000)
1.099(0.011)
0.204(0.007)
4.885(0.019)
0.516(0.032)
(0.91))
Classification
0.270(0.006)
0.19(0)))
0.381(0.015)
2.579(0.062)
0.272(0.021)
7.185(0.044)
D'HLHSS
0.339(0.077)
0.248(0.162)
0.282(0.084)
4.080(0.201)
0.164(0.132)
(.241(0.198)
fDRE
0.043(0.000)
0.199(0.007) 0.907(0.048)
4.283(0.014)
0.487(0.036)
8.710(0.059)
RRND
0.061(0.001)
0.926(0.006)
0.200(0.008)
5.092(0.031)
0.414(0.022)
10.241(0.089)
anDRE
0.038(0.037)
1.064(0.275)
1.183(1.725)
3.838(1.912)
1.671(2.202)
8.629(2.577)
PpDRE
0.051(0.044)
0.069(0.015)
0.164(0.038)
0.272(0.183)
0.074(0.098)
2.966(1.089)
with v),,v) = (u, and q(u.v) (u) (v) for r" ((,,v)= = (u/quu))
parameter, in model fx: 0) regarding the probability distribution of y given x, using standard learning techniques such empirical risk minimization (ERM) can become biased (Sugiyama et al., 2012). To mitigate this issue, the importance-weighte ERM is widely used. The core idea is to re- re-weight the training samples in order to learn a model that minimizes the loss on the test dataset:
We adopt the experimental setting in Belghazi et al. (2018) 8); Rhodes et al. (2020); Choi et al. (2022). Specifically, we con- sider two standard multivariate Gaussian random vectors, = Up))
R and = (V1..... Vp vp) RP, with component-wise correlation, corr(U;, Vj) = o;jP, where p e (- and 6 is Kronecker's 's delta. Performance is mea- sured by the mean absolute error, MAE := |M -Mly.v We consider three replicates with sample size n = 5000. The MAE averaged over three runs for various values of p and p reported in Table 2. Overall, our proposed ppDRE method demonstrates superior comparable performance in all circumstances, indicating its robustness and effective- ness in mutual information estimation.
B(x.)) (L)))-)) (x([(x;)),)]
(xc(x) L(F(x:0),))
= -B(x,y)-p((r.)))
(x)())
where L(.) is user-specified loss function, and the density ratio rxx) Pi: (x) (x)) referred as the importance, acting as adjusting weight during the training process.
5.4. Application in Covariate Shift Adaptation
To simulate the covariate shift setting, we adopt biased sampling scheme as described by Stojanov et al. (2019), on various classical benchmark regression datasets. In this setup, we introduce a sample selection variable s E (0, 1), and the probability sample being selected for the train-
Covariate shift refers to the change in the distribution of the input variables in the training and the test data sets, i.e. Pie (x) # pi (x). In the case of covariate shift, learning a
8

---

Projection Pursuit Density Ratio Estimation
Table 3. NMSE under covariate shift adaptation various henchmark datasets.
Method
Abalone
Billboard Spotify
Cancer Mortality
Computer Activity
Diamond Prices
Unweighted
0.544(0.057)
0.582(0.044)
0.704(0.069)
0.278(0.050)
0.231(0.082)
uLSSIF
0.525(0.057)
0.582(0.044)
0.704(0.069)
0.668(0.360)
0.239(0.068)
KLIEP
(0.05))
0.565(0.028)
0.696(0.058)
0.302(00.09)
0.228(0.080)
Classification
0.535(0.049)
0.579(0.042)
0.691(0.067)
0.273(0.051)
0.183(0.056)
D'-LHSS
0,520(0.054)
0.564(0.031)
0.672(0.066)
0.318
0.220(0.070)
fDRE
0.586(0. 136)
0.615(0.084)
0.664(0.064)
0.294(0.053)
0.233(0.078)
RRND
0.574(0.144)
0.616(0.087)
0.655(0.069)
0.279(0.051)
0.232(0.082)
unDRE
0.806(0.829)
0.822(0.861)
0.660(0.063)
0.301(0.033)
0.360(0.429)
ppDRE
0.514(0.049)
0.559(0.026)
0.661(0.048)
0.199(0.019)
0.133(0.024)
ing set is given by:
6. Conclusion
P(s= 1[xx (I+e")' with v 4 s (x- x) Tn T
We propose novel projection pursuit-based method for estimating the density ratio function, which does not require parametric assumptions, enjoys computational convenience, and can alleviate the curse of dimensionality. The asymp- totic consistency and the convergence rates are established to guarantee the validity of the proposed method. Numeri- cal experiments demonstrate that our method outperforms existing alternatives in variety of applications.
where F is the sample mean of the covariates, is ran- dom projection vector uniformly selected from the interval [-1, ljd and (x-x) is the standard deviation. Subse- quently, we learn kernel ridge regression model by mini- mizing the importance imprrtance-weighted empirical risk:
i = argmin Σ>>ne ((x))) [[(x}]0) - 2 + }
1 -
Density ratio estimation based on projection pursuit admits many exciting directions for future work. One particularly promising direction is the extension of ppDRE to indepen- dence testing. However, this extension requires rigorous theoretical development, particularly in establishing the es- timator's 's limiting distributions under the null hypothesis of independence. Furthermore, the method's iterative optimiza- tion process may still face challenges with computational complexity as the number of projections K increases. Devel- oping adaptive stopping criteria could alleviate the compu- tational burden by avoiding tuning K with cross validation.
0
F1 1=1
where f(x;0) En (xx,x, K(x,x') exp(-lix x'll//d the kernel basis, and P is an estimator of based on DRE methods.
To evaluate the performance, compute the normalized mean squared error (NMSE) the test dataset:
NMSE 1 the (y 5/22
- Ahe 1=1
where The is the number of test samples, y is the true value for the ith sample, ete is the predicted value, and of is the variance of the response variable the test dataset. Aver- age NMSE values across 15 random replicates are reported in Table 3. More details on the datasets can be found in Appendix D.2.
A cknooleegements
The research of Zheng Zhang is supported by the funds from the National Key R&D Program of China [grant num- ber 2022YFA 1008300), the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China [project number 23XNA025]. Mingming Gong is supported by ARC DE210101624, ARC DP240102088, and WIS-MBZUAI 142571.
As shown in Table 3, the unweighted regression model per- forms poorly across all datasets, underscoring the necessity for adjustments for the covariate shift problem. In contrast, proposed ppDRE method consistently outperforms matches the performance of the baseline methods across various datasets. This highlights the effectiveness of ppDRE in mitigating the impact covariate shift, positioning it as a promising approach for regression tasks affected by this issue.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning in high high-dimensional data. There are many potential societal consequences of our work, none

---

Projection Pursuit Density Ratio Estimation
which we feel must be specifically highlighted here.
shift. In Advances in Neural Information Processing Sys- tems, volume 33, pp. 11996-12007. Curran Associates, Inc., 2020.
References
Ai. C., Linton, O., Motegi, K., and Zhang, Z. unified framework for efficient estimation of general treatment models. Quantitative Economics, 12(3):77--816, 2021.
Friedman, J. and Tukey, J. A projection pursuit algorithm for exploratory data analysis. IEEE Transactions on Com- puters, C-23(9) 1974.
Ai, C., Sun, L.-H., Zhang, Z., and Zhu, L. Testing uncondi- tional and conditional independence via mutual informa- tion. Journal of Econometrics, 240(2):005335, 2024.
Friedman, J. H. and Stuetzle, W. Projection pursuit regres- sion. Journal of the American Statistical Association, 76 (376):817-823, 1981.
Aladjem, M. Projection pursuit mixture density estimation. IEEE Transactions on Signal Processing, 53(11):43766 4383, 2005.
Friedman, J. H., Stuetzle, W., and Schroeder, A. Projec- tion pursuit density estimation. Journal of the American Statistical Association, 79(387)5599-608 1984.
Gao, E., Bondell, H., Huang, W., and Gong, M. vari- ational framework for estimating continuous treatment effects with measurement error. In The Twelfth Interna- tional Conference Learning Representations, 2023.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In Proceedings of the 35th Interna- tional Conference on Machine Learning, pp. 531-540 PMLR, 2018.
Gizewski, E. R., Mayer, L., Moser, B. A., Nguyen, D. H., Pereverzyev, S., Pereverzyev, S. V., Shepeleva, N., and Zellinger, W. On regularization of unsupervised domain adaptation in rkhs. Applied and Computational Harmonic Analysis, 57:201-227, 2022. ISSN 1063-5203.
Bickel, S., Brückner, M., and Scheffer, T. Discriminative learning for differing training and test distributions. In Proceedings of the 24th International Conference on Ma- chine Learning, pp. 81-88, Corvalis Oregon USA, 2007. ACM.
Gruber, L., Holzleitner, M., Lehner, J., Hochreiter, S., and Zellinger, W. Overcoming saturation in density ratio estimation by iterated regularization. In Proceedings of the 41st International Conference Machine Learning. pp. 16502-16529, PMLR, 2024.
Chen, X. Large sample sieve estimation of semi- nonparametric models. Handbook of Econometrics, (B):5549-5632, 2007a.
Chen, X. Large sample sieve estimation of semi- nonparametric models. Handbook of Econometrics, Else- vier, 2007b.
Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., and Kanamori, Statistical outlier detection using direct density ratio estimation. Knowledge and Information Systems, 26(2):309-336. 2011.
Choi, K., Liao, M., and Ermon, S. Featurized density ratio estimation. In Proceedings of the Thirty-Seventh Confer- ence on Uncertainty in Artificial Intelligence, pp. 172- 182. PMLR, 2021.
Hill, J. I.. Bayesian nonparametric modeling for causal infer- ence. Journal of Computational and Graphical Statistics, 20(1):217-240. 2011.
Choi, K., Meng, C., Song. Y., and Ermon, S. Density ratio estimation via infinitesimal classification. In Proceed- ings of The 25th International Conference on Artificial Intelligence and Statistics, pp. 2552-2573. PMLR, 2022.
Huber, P. Projection pursuit. The Annals of Statistics, 13 (2):435-475 1985.
Kanamori, T., Hido, S., and Sugiyama, M. A least- least-squares approach direct importance estimation. Journal of Machine Learning Research, 10:1391-1444 2009.
da Silva, N., Cook, D., and Lec, E.-K. A projection pursuit forest algorithm for supervised classification. Journal of Computational and Graphical Statisties, 30(4): 168- 1180, 2021.
Kanamori, T., Suzuki, T., and Sugiyama, M. Theoretical analysis of density ratio estimation. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E93-A((4:787-798, 2010.
Diaconis, P. and Shahshahani, M. On nonlinear functions of linear combinations. SIAM Journal on Scientific and Statistical Computing, 511):175-191, 1984.
Kato, M. and Teshima, T. Non-negative bregman divergence minimization for deep direct density ratio estimation. In Proceedings of the 38th International Conference on Ma- chine Learning, pp.
Fang, T,, Lu, N., Niu, G., and Sugiyama, M. Rethinking im- portance weighting for deep learning under distribution
PMLR, 2021.
10

---

Projection Pursuit Density Ratio Estimation
Lee, E.-K., Cook, D., Klinke, S., and Lumley, T. Projection pursuit for exploratory supervised classification, Journal of Computational and Graphical Statistics, 14(4):831- 846, 2005.
Que, Q. and Belkin, M. Inverse density as inverse prob- lem: the fredholm equation approach. In Burges, C., Bottou, L.,, Welling, M., Chahramani, Z., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Li, Q. and Racine, J. S. Nonparametric econometrics: the- ory and practice. Princeton University Press, 2023.
Rhodes, B., Xu, K., and Gutmann, M. U. Telescoping density-ratio estimation. In Advances in Neural Infor- mation Processing Systems, volume 33, pp. 4905 4916. Curran Associates, Inc., 2020.
Liu, S., Yamada, M., Collier, N., and Sugiyama, M. Change- point detection in time-series data by relative density-ratio estimation. Neural Networks, 43:72-83 2013.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log-likelihood function. Jour- nal of Statistical Planning and Inference, 90(2):227-244, 2000.
Liu, S., Suzuki, T., Relator, R., Sese, J., Sugiyama, M., and Fukumizu, K. Support consistency of direct sparse- change learning in Markov networks. The Annals of Statistics, 45(3), 2017.
Sinha, A., O' Kelly, M., Tedrake, R., and Duchi, J. C. Neural bridge sampling for evaluating safety-critical autonomous systems. Advances in Neural Information Processing Systems, volume 33, pp. 6402-6416. Curran Associates, Inc., 2020.
Lorentz, G. Approximation functions. Chelsea Publishing Company, 1986.
Matsushita, Y., Otsu, T., and Takahata, K. Estimating den- sity ratio of marginals to joint: Applications to causal inference. Journal of Business Economic Statistics, 41 (2):467-481, 2023.
Stojanov, P., Gong, M., Carbonell, J., and Zhang, K. Low- dimensional density ratio estimation for covariate shift correction. In Proceedings of the Twenty-Second Interna- tional Conference on Artificial Intelligence and Statistics, pp. 3449-3458. PMLR, 2019.
Meng, X.-L. and Wong, W. H. Simulating ratios of nor- malizing constants via a simple identity: a theoretical exploration. Statistica Sinica, 1996.
Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P., and Kawanabe, M. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Nagumo, R. and Fujisawa, H. Density ratio estimation with doubly strong robustness. Proceedings of the 41st International Conference on Machine Learning. pp. 37260-37276. PMLR, 2024.
Sugiyama, M., Suzuki, T., Nakajima, S., Kashima, H., Von Bünau, P., and Kawanabe, M. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4))699-746, 2008.
Nam, H. and Sugiyama, M. Direct density ratio estimation with convolutional neural networks with application in outlier detection. IEICE Transactions Information and 2015.
Systems,
Sugiyama, M., Kawanabe, M., and Chui, P. L. Dimen- sionality reduction for density ratio estimation in high- dimensional spaces. Neural Networks, 23(1):44-59, 2010.
Newey, W. K. and McFadden, D. Large sample estimation and hypothesis testing. Handbook of Econometrics, 4: 2111 2111-2245, 1994.
Sugiyama, M., Yamada, M., Von Bünau, P., Suzuki, T. Kanamori, T., and Kawanabe, M. Direct density-ratio es- timation with dimensionality reduction via least-squares hetero-distributionall subspace search. Neural Networks, 24(2):183-198, 2011.
Nguyen, D. H., Zellinger, W., and Pereverzyev, S. On regularized radon-nikodym differentiation. Journal of Machine Learning Research, 25(1), January 2024. ISSN 1532-4435.
Sugiyama, M., Suzuki, T., and Kanamori, T. Density ratio
Nie, L., Ye, M., Liu, Q., and Nicolae, D. VCNet and func- tional targeted regularization for learning causal effects of continuous treatments. In International Conference on Learning Representations, 2020.
estimation in machine learning. Cambridge University Press, edition, 2012.
Suzuki, T., Sugiyama, M., and Tanaka, T. Mutual informa- tion approximation via maximum likelihood estimation of density ratio. In 2009 IEEE International Sympo- sium on Information Theory, pp. 463-467, 2009. doi: 0..1199/STT.2200955500
Qin, J. Inferences for case-control and semiparametric two- sample density ratio models. Biometrika, 95(3):619-630 1998.

---

Projection Pursuit Density Ratio Estimation
Tsuboi, Y., Kashima, H., Hido, S., Bickel, S., and Sugiyama, M. Direct density ratio estimation for large-scale covari- ate shift adaptation. Journal of Information Processing, 17:138- 155, 2009.
Vaart, A. W. and Wellner, J. A. Weak convergence and empirical processes. Springer, 1996.
Van der Vaart, A. W. Asymptotic statistics, volume 3. Cam- bridge University Press, 2000.
Wang, I.. and Yang, L. Spline estimation of single-index models. Statistica Sinica, 19(2)7655-783, 2009. ISSN 10170405, 19968507.
Yamada, M. and Sugiyama, M. Direct density-ratio estimation with dimensionality reduction via hetero- distributional subspace analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 25(1):549- 554, 2011.
Yamada, M., Suzuki, T., Kanamori, T., Hachiya, H., and Sugiyama, M. Relative density-ratio estimation for robust distribution comparison. Neural Computation, 25(5): 1324-1370, 2013.
Zhan, H., Zhang, M., and Xia, Y. Ensemble projection pursuit for general nonparametric regression. The Annals of Statistics, 2025.
12

---

Projection Pursuit Density Ratio Estimation
A. Baseline Methods
This section presents the baseline methods used in our numerical studies.
uLSIF The unconstrained Least- Squares Importance Fitting (uLSIF) method, proposed by Kanamori et al. (2009), models the density ratio r"(x) as follows:
Ap
r'(x) R 0' (x))= Σ (xK(x.xf))
(11)
where (x,xx)) is the Gaussian kernel basis, and the definitions of di (x) and are straightforward. The model parameter learned by minimizing the squared loss with ridge penalty. For implementation, employ the functions provided by the Python package densratio'..
The Kullback-Leible importance estimation procedure (KLIEP) method, proposed by Sugiyama al. (2008), uses the same Gaussian kernel model for approximating r' r'(x) as that in the uLSIF method (11), but the model parameter identified by minimizing the unnormalized Kullback-Leibler (UKL) divergence:
KLIEP
UKL' (r) [x[r(x]] - Bp [1ogr(x)]
With the fact that B3 [,[r(x]]= = 1, the estimation is implemented as follows:
"p Σ
1 ng
max 0
log
0 (xxx)
S.t.
Σ 0 (x)))x) and 0 0,
np
ap
where the inequality for vectors is applied in the dlement-wise manner. The matlab code of the KLIEP method is available online?.
Probabilistic Classification By assigning the label +1 to the sample from pxx) and y= to the sample from (xx) respectively (Qin, 1998; Bickel et al., 2007), the density ratio function r" r'(x) (xx)/4(xx can be expressed by
Fx = +1) - + +1(x)
r" (x) P(xly = P(y=+1)P(y= =
Given estimator of the posterior probability, p(y|x), the density ratio estimator f(x) can be constructed as
f(x) nq. p(y +(y=+1xx)
"p p(y==1|xx
In our experiments, use the LightGBM model the classifier, which is ensemble learning algorithm based on gradient boosting decision trees.
The Direct Density-ratio estimation with Dimensionality reduction via Least-squares Heterodistributional Subspace Search (D²-LHSS) method is proposed by Sugiyama al. (2011). This method assumes that the density ratio can be identified in low-dimensional space specified by projection matrix U e Rmxdd Given the projection matrix Ó obtained by the LHSS algorithm, the estimator of the density ratio is given by
D'HSSS
P(x) Σ (Ux).
where (0e)/ are the learned using the uLSIF method on the heterodistributional subspace corresponding Ú. For more details, we refer to Sugiyama et al. (2011), whose matlab implementation is available online?.
hanns:/ggiuna.ccmmooommmcoooaat py
13

---

Projection Pursuit Density Ratio Estimation
The featurized density ratio estimation (fDRE) framework, proposed by Choi al. (2021), addresses distributional discrepancy challenges through feature learning. By employing normalizing flow model fo : - Z with invertible transformation properties, this method embeds heterogeneous data distributions into unified feature space where density ratios remain preserved. The crucial invariance property is formally expressed as:
NRE
r' (x)) = p(x) p'(foxx) ((x) q'(fo(x))
where p' are the densities of (x and (xg) respectively, Xp pxx), and Xg qxx). This preservation enables direct application of classical density ratio estimators, such as KLIEP, in the transformed space Z, often achieving superior numerical stability compared to native space estimation.
The regularized Radon-Nikodym differentiation estimation (RRND) method, proposed by Nguyen et al. (2024), introduces a dernel-based regularization framework within reproducing kemee Hilbert spaces (RKHS) for accurate pointwise estimation of RadonnNikodym derivatives, which are equivalent to density ratio functions. This approach can extend the conventional kernel uLSIF through an iterative Lavrentiev regularization scheme, in which the core algorithm operates through successive approximations follows:
RRND
x =0,
x = SX (Sx, Sx, 1+1 1 I.N,
where / is the /-th iteration of the approximation of (x). The methodology leverages two sample operators:
Sx, (xp//=(/(E)).....fff
/=
and two adjoint operators defined through kernel embeddings:
1 Σ K( ((.xf))u; RE Hp
1 VER
Sx, (())
Sx v()) n4
For complete theoretical analysis and convergence properties, readers are directed the original work by Nguyen et al. (2024).
nnDRE We present neural network-based density ratio estimation (nnDRF) method. It differs from our proposed ppDRE method in that it utilizes a feedforward neural network to model the density ratio, i.c.
(x) ~ TT (x;0) =x)()F(x(())
where F(-10) represents the neural network with parameter 0. The estimation is proceeded by minimizing the squared loss based on the neural networks: SQ' (0) =x;((("<xx;)))) ( (x; (r;;))).
B. Implementation Details and Hyperparameter Selection Process
To ensure rigorous and fair comparisons, we implemented the following protocols.
For our proposed ppDRE method, as mentioned in Remark 4.3, the optimal hyperparameters are identified by the minimal validation loss CV. For clarity, we detail our approach as follows: across all experiments, we utilized 5-fold CV with random sampling. A grid search was conducted over a predefined set parameter ranges, which are outlined in Table
For open-source baseline methods (e.g., KLIEP, uLSIF, D -LHSS), we utilized their official implementations (c.g.. Python densratio package for uLSIF) and default hyperparameter search grids recommended in their original papers standard toolkits. For instance, uLSIF's hyperparameters (or, A) were selected from the grid 1e-3:10:1e: consistent with its standard implementation. These methods inherently incorporate CV-based hyperparameter selection in their standard workflows. We preserved these built-in CV mechanisms without modification.
For probabilistic classification approach and nnDRE, we implemented 5-fold CV (aligned with our ppDRE method) to identify the optimal hyperparameters within the search spaces outlined in Table
14

---

Projection Pursuit Density Ratio Estimation
Table Search Grid for ppDRE
Parameter
Description
Search Space
Number of PP iterations
(5, 10, 15} (20, 50, 70, 100, 150) (0.5, 1, 5, 10) {0.001, 0.01, 0.1)
Jk
Number of basis functions
(z-regularization strength
6
Gradient descent learning rate
Table 5. Search Grid for Probabilistic Classification approach and nnDRE
Method
Parameter
Search Space
n esstimmttoos
{100, 300)
learning_rate
(0.0001, 0.001, 0.01)
Classification
num num_leaves
(20, 30)
depth
(2, 3)
onDRE
width
{8, 32, 64)
learning rate
[0.0001, 0.001, 0.01)
For fDRE, we have implemented version that employs KLIEP as the second-stage DRE method. Adhering to the guidelines provided by the official open-source code and the original research paper (Choi et al., 2021), we trained the masked autoregressive flow (MAF) models with the configurations presented in Table 6. For the KLIEP method in the second stage, we have retained the original hyperparameter settings as per open-source implementation.
Table 6. Hyperparameter setting for the normalizing flow model in IDRE
Dataset
blocks
n_hidden
hidden_size
n_epochs
IHDP
5
100
100
Regression Benchmarks
5
100
100
MI Gaussians
5
100
200
For RRND, in the absence of open-source code, we implemented the algorithm by adhering to the implementation details outlined the Numerical Illustrations section in Nguyen al. (2024). The kernel function is assigned as K x(x,x') ++exp(((xxxx)) 22)2))). Utilizing 5-fold CV, the hyperparameter is chosen based on the quasi-optimality criterion, and the optimal iteration step & is determined by minimizing squared distance loss function in the validation set. Following the configurations in Nguyen al. (2024), the search grids are k E (1.2...., 10) and A (2e= Aop { 9...,9) with Ag 0.9. The decay factor (d.,, AA) 1// is derived from the lower bound A, (n -112 1/2 m 1/2), where are the sample sizes of
the numerator and denominator distributions respectively.
C. Addtionna Numerical Results
C.I. Stabilized Weights Estimation
Metrics We the mean squared error (RMSB) and the root mean squared logarithmic error (RMSLE) to evaluate the performance, which are defined follows:
" RMSE Σ ((xx)))
RMSLE
- (x;))
F
15

---

Projection Pursuit Density Ratio Estimation
Results We present additional numerical results, including the box plot of RMSLE in Figure and the RMSE values in Table 7. The box plot of RMSLE in Figure 4 shows the variation of all methods. In Table 7, due to the extremely large RMSE in some data replications, report the median of RMSE values in 50 replications, which offers evidence supporting the superior performance of the ppDRE method.
ULSIF
3.0
KLIEP
Classification
D3-LHSS
AnDRE
PpDRE
2.0
RMSLE
1.5
1.0
0.5
2
10
30
50
100
Dimension
Figure 4. Boxplot of RMSLE in stabilized weights estimation for different covariate dimension settings dx E 10,30,,, 100) estimation across 50 data replications
Table Median RMSE stabilized weights estimation for different covariate dimension settings dx e (2, 10,30,50, 50,
(Median) RMSE
Dimension of Covariates
Method
2
10
30
50
100
uLSIF
1.864
4.216
2,452
1.875
1.321
KLIEP
1.827
4.222
2,464
1.898
1.39
Classification
1.93
4.217
2.413
1.809
1.208
D'-LHSS
2.604
8.793
2.637
2,454
2.054
nnDRE
2.181
4.739
2.772
2.227
1.549
ppDRE
1.742
3.835
1.777
1.12
0.9
C.2. Computational Cost Evaluation
Through controlled numerical experimentation, we systematically evaluate the computational costs of all competing methodologies. Maintaining the experimental protocol established in Section 5.2.1, 1, we conduct dimensional scalability analyses over dx e (2, 10, 30, 50, 100) and measure absolute wall-clock execution times. The experiments were carried out on computing node utilizing dual AMD EPYC 7713 processors, providing total of 128 CPU cores. Table presents the mean computational duration across three statistically independent trials, with temporal measurements recorded in minutes.
We obtain the following key observations: (i) when the dimension of the data is low moderately large (dx 2, 10), the computation time of our PPDRE method is comparable to that of the uLSIF, KLIEP, and classification methods, which are suitable for low-dimensional data. Our computation time is much less than that of the fDRE, RRND, nDDRE and D'HLHSS methods, some of which are also designed for high-dimensional data. (ii) when the dimension of the data is large (dx = 30, 50, 100), the methods for oow-dimensional data fail work due the curse of dimensionality. Our PPDRE
16

---

Projection Pursuit Density Ratio Estimation
Table 8. Computation Time Comparison
Method
d = 2
d = 10
d
30
d 50
d = 100
uLSIF
0.38
0.35
0.38
0.43
0.46
KLIEP
0.36
0.39
0.49
0.57
0.72
Classification
0.21
0.16
0.19
0.23
0.40
IDRE
3.78
3.88
3.92
4.28
7.25
RRND
3.78
3.87
3.77
3.82
3,81
nnDRE
5.91
3.84
8.76
6.02
8.63
D
5.95
11.25
10.60
10.79
10.97
ppDRE
0.32
0.91
5.64
10.78
15.22
method consistently outperforms baseline methods in estimation accuracy, and the computation time is comparable that of existing methods for nigh-dimensional data. The observed increase in computation time expected, as higher dimensions naturally require more iterations reach convergence.
D. Datasets
D.1. Dataset in Dose Response Function Estimation
IHDP-contee ciDP-continuous Dataset The original IHDP dataset contains 25 covariates with binary treatments and continuous outcomes. We follow similar procedure as employed by Nie et al. (2020) and Gao et al. (2023), disregard the treatments and outcomes, and use the covariates to generate continuous dosages and treatments. Let .,..... Xdx denote first to the last element of where dx 25. The generating procedure for the semi-synthetic LHDP-continuous dataset is as follows:
Assigned treatment: 7= (1+ (1+xx(()))
(12)
where /= X1
max max X4.
+ +tanh Lier, X; |4l
- 0.50.
1+X2 0.2+ X3,X4. Xs)
Potential outcome: y" h(t, x) +0.5e,
where X),, -1.2- sinn--r))nnnnnn)))) sin 22) 0.5 0.5tanh tanh 5 Lier, Xi
1.2(X1-X) 0.1+min (2,, X3, Xa)
1-5
(13)
1/2
where E ~ N(O, (3,6, 10, 13, 13, and 12 (15, 16, 18, 19, 20, 21,,..2..2
D.2. Datasets Covariate Shift Adaptation
In this section, we introduce the basic information and the regression tasks the benchmark datasets that are used the covariate shift adaption experiments. These datasets are readily accessible via the Ready Tensor platform. The specific numbers of observations and features for each dataset are detailed in Table
Table 9, Observation and Feature Numbers of Regression Benchmark Datasets
Dataset
Abalone
Billboard Spotify
Cancer Mortality
Computer Activity
Diamond Prices
Observation
4177
8930
3047
8192
6000
Feature
18
31
21
7
Abalone The Abalone dataset, accessible onlinet, popular dataset used in machine learning and statistics predict the age of abalone from physical measurements. Abalone age is determined by cutting the shell, staining and counting the number of growth rings. However, this is destructive process, non-destructive method based on physical measurements is desirable.
17

---

Projection Pursuit Density Ratio Estimation
Billboard Spotify The Billboard Spotify dataset represents comprehensive collection of audio features for songs that made their mark various Billboard charts that span the years 1961 through 2022. For each song, the corresponding audio features were sourced from the Spotify API. The regression task for the Billboard Spotify dataset is to predict the danceability of song, which describes how suitable track is for dancing based on combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity.
Cancer Mortality The Cancer Mortality dataset, aggregated from authoritative sources including the American Com- munity Survey, clinicaltrials.ge and cancer.gov, is designed for regression tasks to predict cancer mortality rates in US counties from 2010 to 2016. It incorporates 2013 census data, including county-level features such as population, income, households, and other demographic attributes.
Computer Activity The Computer Activity databases consist of a collection of metrics related to the activity of computer systems. The data was collected from Sun Sparestation 20/712 with 128 Mbytes of memory running in multi-user university department. This dataset, available online", is utilized to estimate the proportion of time during which central processing units (CPUs) are engaged in user mode operations, based on the recorded system activity measures.
Diamond Prices
The Diamond Prices dataset is accessible via the PyCaret library within the Python programming environment. The objective of this dataset is to predict diamond prices based on set of attributes, including carat weight, cut, color, clarity, polish, symmetry, and the report issued by the grading agency that assessed the diamond's qualities.
E. L distance Minimization
In this section, we show that minimizing the distance minimizing H(f.a)= :=
xx)3² w.r.t or is equivalent
x)).
Note that, by definition, ((()=))))))())===0)))())))))) Then,
(4 2
dx
(1 (x-1(x)) p dx (1) xx)).
Since the first term [x(]r" is independent either f or a, we have that minimizing w.r.t. f or a is equivalent to minimizing H(f.a).
F. Proof of Theorem 4.2
F.1. Notations and Assumptions
To prove the theorem, we require the following notations and assumptions.
Notations of derivative. For any univariate function f. we let f() denote its jth derivative. For any multivariate function g(B. a), we denote 0ig( (B, a) to be its partial derivative with respect to w.r.t. the first argument B, and 0zg (B. a) to be its partial derivative (w.r.t)) the second argument a.
U (a'x) for 0, where Notations regarding matrices. Consider j 0, We let 2 (a) BBo S x)) the expectation can be taken w.r.t. the probability density p(-) or q()).
For the corresponding empirical version, let (II....,,,,) be i.i.d. from either the probability density p(-) (resp. n= np)) or
q()) (resp. n,g,, we define (()((a) = ( (a (aTx))..... ob2 (a xx)) e RnxJs and 22 (a) =n I (a) p()) (a).
(1)
Similarly, we define Σ,, (a) B, [r [ (a) xa)x() En (a) = n.a (a) Zx(a), Ln) = Σ,, (a) +N1 where (a) is defined Proposition 4.1.
18

---

Projection Pursuit Density Ratio Estimation
Moreover, let mmin and m/ (22) denote the minimum and maximum eigenvalues of any matrix Q2 respectively.
Notations regarding minimization criteria. Recall the definitions of H(f.a) and La(a.B:A) from (2) and (4), respectively. By definition, for any a E A,
1 Pp
mp Pk- (xxaaa))aa"')))))
x)) = (x(xxx), and Bx(a) = (E)).(aa))
(14)
We define
())aaxx))]---- -Bp (x) B px x)] to the approximation of H(f.a) and
the theoretical counterpart of Lx. We then define (a) := argmin H H (B,a) a) for any E A. a = argmin H (B) and
a&A
(a"Txx xx) for any x E E.
Assumption F.1. (a) The support X of x is a compact subset of Rd. (b) The parameters {ax) K k-
defined in (7) are in the
interior of a compact set A -
Assumption F.2. (a) For every a E A, there exists Bx. (a) & such that
a a 25 a.k (2) Bxx,
sup sup
(1) (2)
< x (5-1)
(8-1)
(15)
for some constants Co.j. > 0, and 0 j j < s, where Z == = {a x a E A and x E X); (b) For every 2 E Z, fa,x is continuously differentiable w.r.t. a, and sup2eZ 110a x (2) -= < <o; (c) fa (z) are uniformly bounded and bounded away from 0.
Assumption F.3. (a) The eigenvalues of 2 (a) are bounded and bounded away from zero uniformly in J2 and a e A.A. (b)
There exist sequences of constants (J)))), j = 1.2, such that the eigenvalues of 22// (a) are bounded by (JR) uniformly in Js and a E A (c) There exist sequences of constants {j(Jk), == 0,,,,2 such that supjsm supze Z 11 (1) (2))l S [m for m = 1,2 and all k. (d) As np - Jx - co, {o( (o(J)))NJx//(n (Jp/(pAnn)) - O. Jx (x-1) (Ex() (1/ Ang) 0 and d = 0( (JJ))/)))
O.
Assumption F.4 The minimum eigenvalue of 0211 II" ( (a).a) a=a; is bounded away from 0 uniformly in JR.
Assumption F.1 imposes compactness conditions on the densities and parameter spaces. This condition is convenient for deriving uniform convergence rates. Assumption F.2 includes regularity conditions. Assumption F:2 (a) can be satisfied if (2) is s-times continuously differentiable w.r.t. E Z for any E A (Lorentz, 1986). Assumption F.3 rules out near multicollinearity in the approximating basis functions. This condition is familiar in the sieve regression literature (Chen, 2007a). Assumption F.4 requires the Hessian matrix of the approximation of the theoretical minimization criteria to be positive definite at its minimum, dz. which is satisfied if a is in the interior of A.
E2. Outline of the proof
We prove the results in an inductive way. Note that the initial estimate fo(x;;p) = I, rx(x) = 1III -m fam. (a)x)) and Ax (x) fam (a]T)) and sa,e is estimated based on fx-1. Let rx( and fo(x) = fo(x: Bo) The following results hold for k k=
Σ
=0p(En,-1))
(16)
We shall show that, for any & E (1,2...., K), given (16) holds, we have
sup I (ā]xx fa. xEX
0p (J (1-((-1) VEnR-I+ (el v (17)
where (J²) is the rate of the maximum eigenvalue of 2 2 (a) and (Z)( - Then, using the fact that En,0 = O, we can inductively derive that
K Σ
K
sup (xx xEX
(x)
Jr
I i=t
ng App
[=)
19

---

Projection Pursuit Density Ratio Estimation
which establishes the last statement of Theorem 4.2.
Proof of (17). To prove (17), we will need to establish the results in (8) and (9), which are relegated to Sections F.3 and F.4,
respectively. Specifically, we decompose
(18)
Lap (a]x)= lar, (a (a fa f fac. xxxx)
(19)
(20)
For (18), we have supxex x (āīx) fa su supxex Sa x x) fa, x)).
For (19), under Assumption F.2 (b), we can use Taylor's expansion to expand fai.k around far.k and obtain suprex (aaxx)/a,.aaaa))) fa
0 -axl))
For (20), we can first decompose
(20) = [a,, (a]x) - fa (a. (a]x)) [La,, -a]x) [a aa [a.
The supremum norm of the first term can be bounded by Op (Ilâ: - axll), using Taylor's expansion expanding far.k (a, around far. xx. x). The second and the third term can also be bounded by (Ilâx - axll) using the same argument for (19). Thus, have
= fau, (xx) sup (aa
sup sup (ax Sac. ( (a] Rxx
axll)
x
Then, using (8) and (9), and noting that (o())) s Jn, we then have (17) holds. The outline of the proof is completed.
E3. Proof of (8)
Recalling the definitions of fa.k in (6) and fa, in Section F.I, decompose
sup sup 2E
fa. fa.x
s sup sup -() aeA
(21)
sup sup Sa, (z) z
(22)
We derive the convergence rates for (21) and (22).
Rate of (21).
We obtain the rate
(2)==0(73(6(33))
from Lemma F.9.
Rate for (22). By definition, we have
(2, (C),)
Recalling the representation of la, in (14), we decompose
sup sup La, fa, xx)]
xcAxee
Bp
@ x) (E))aaaa))))))
Pk- (x 1 (1al::/)((1aapo--////11000000000
= =sup sup xxX
"p
< sup sup xxxx
@a (a) (a))
(fx (2--
(23)
np
i=1
np
0 (TTx) (E),
rk- (x -x x) (xx (24)
+ sup sup as x
"p
1=1
+ sup sup xex
d (a xx) ((E) (()) (E,, (a)) (x) Tx)]]]]_
(25)
20

---

Projection Pursuit Density Ratio Estimation
We derive the rate for (23) (25) one by one.
Rate for (23). Note that the term in the absolute value sign in (23) is the Lz-projection of (Fx-1(x")-ra - to the linear space spanned by the sieve basis 4x (a)x). Given that A and X are compact sets and the sieve basis D2 is local basis, have (23) I supas o SuPxex
[ x)1l -
=0 = (5nx-1)
Rate for (24). Define (2) Dx(2)
and $ʃ₁ to be the Jx dimensional unit ball
3 == (b e R. ||b|| = 1}. Since, under Assumption F.3,
sup sup (1)x (2.. =Or((((())) a
the term (24) can be bounded follows:
(24) =0p ({o(JR)) sup sup
ba bela't))nn11(()000 (
xEA
Up i=
"p
s0p (50(J2))
sup (b.a)eS
nh-
(26)
XA
np
im]
Consider the measurable function class H4 (x + b (= (b. &3, XA) with its envelope denoted by Hg(x) sup(b.a) x |lb (a x)]l supas.a [| x)
xx))ll We denote ||H|| : (f H(x) fx)dxx) for any function H and
density f. Then || ())( Note that by the maximal inequality (e.g., Vaart & Wellner, 1996, Theorem 2.14.2 )
b bT Σ 1=1
B
sup (b.a) X
Up
C. C
Tmp L'
I+oogNnpe [x Hg. Hx-11- dl ||H
where (elle[Hellp He. II. (l is the minimum number of < -lIHeell-brackess brackets needed to cover Hy (Vaart Wellner, 1996, Section 2.1). Since |H S ||Hxlooe we have
Ne ll s Nn(e, Ilaa).
(27)
Then, by Lemma F.5 we deduce that
[ ++oogN1j(e ||H&llp. Hg. lI,)
sc S.' V 1-Js log
E /e-1Hx lIp
de
[o((2)
sCuJT (' V Jk - llog())) =
Hence,
1 up
Σ b r2 1x (x(") (x()x(())--))))))))))))))))))))) -Epl" (x)))))) =0p (VInso(se)
E
sup (b.a) x.
P1p
np
Consequently, by (26). we have
VJRCO(JR)
(24)=0p
(28)
Rate for (25).
21

---

Projection Pursuit Density Ratio Estimation
Note that, under Assumption F.3 and recalling the definition of 7/mx (+) and the results in Lemma F.6,
(25) s [o(Jk) sup Mmax acM
(r- x)]]
=O((o(JR)) sup
V Mmax (E,
E,, (a) + (2
aEA
sup Op((o(Jk))
{++ |ニ>((a) - Σn (a) (xx
aEA
=Op (V (JiSo(JR²
sup
|Bp (xx x|]
(29)
where the last equality follows from Lemma F.6 and the rate of A in Assumption F.3.
Let V(a)= Ep Bp f x) Px x)]. Under Assumption F33, we have the eigenvalues of Qr == Ep(0s x))aa(a r) are bounded and bounded away from 0 uniformly E A. Then
sup [|V(a))| = sup V(a) atA
acA =0(1)- sup V(a) (u, "u.
a
=0(1) sup Bp [VVa))
which is the L2-projection By (x)= x) the space linearly spanned by @x x). Therefore,
||( Ep s (x)
sup
a
Combining the results of (23) to (25), we have
(22) VEnk-15 2T2
Vpp Ang
Consequently, we have (8).
F.4. Proof of (9)
Consistency, We first show à P aæ as Rp + co. By definition, å (resp. a&) is the unique minimizer of (a (a(aaB(a)) (resp. a))., From the theory of Meestimation (Van der Vaart, 2000, Theorem 5.7), if the following condition holds:
())
-)) (a)t:a))H((Haaeeee a() a
sup
as Ap. co, then we have àg L ag. Since
Bg
2 mp
Ex(a,Br(a),d)-
Σ
(x
(xxx))
x-)- PR- (x.)))) Bx()).
ng
np i=1
22

---

Projection Pursuit Density Ratio Estimation
can decompose
sup aEл
1 ng
S sup atA
k-1 (x (x
(t)
4 i=1
+ 2 sup a π
x)
(30)
Pp i=1
sup ac R
My i=1
+ 2 sup
1
xx/1
(31)
acA
"p i=1
+1 sup (a)TBBx(0) at
(32)
Rate for (30). As imposed in Assumptions F.1 and F.2, A is compact and fa (a x) is continuous in a, applying the uniform law large numbers (Newey & McFadden, 1994, Lemma 2.4), we have
(30) = =00)=0((1// =/pp
Rate for (31). Note that
(a -a)
I (x))
(4,, (axxx))
1 Σ
11 k- (x
(2)) (axx))
ng
ny 1=1
-=
1 (a Pax (a (xTx)))1]]
ng
i=1
We apply the same decomposition to the second term of (31).
For any k E 21, K). given (16) that nty E"1 (x2-1 -x-1 - 0 (p(End-1)) using (8) and Cauchy- Schharr inequality, we can conclude that
(31) = ( + (4)
(1)
Rate for (32). Using the rate of 2 in Assumption F.3 and Lemma F.11, we have (32) = op)
Consequently, we have à as 0.
Convergence Rate. Let (a) La (a ()): Since à is the unique global minimizer the context function & (a), by the first order condition, we have 0g la (a)) = 0. By applying the mean value theorem, we obtain
= a & (ax) = Os (a.) (Q) -a;),
where à lies between à and ag. Hence,
à a = - -la
(33)
Equation (33) implies that the convergence rate of à is determined by the Hessian matrix 02642 (āx) and the score function 0₂ (a()) We derive the rate of these terms one by one. Recalling the definition of & () at the beginning of the subsection and applying the chain rule, we obtain
0 = La htw.aaaaaaaaoof..a.........aaaaa
=b
(34)
23

---

Projection Pursuit Density Ratio Estimation
where for any bivariate function g)x,y), (x, y) and d2g(x,y) denotes the partial derivative of g w.r.t. the first and the second arguments, and y, respectively. The second equality follows from the fact that Bx (a) is the unique global minimizer of the convex function (a, (a): for any A, so d (rL (a): for any A. Then, we have
- -
a, (a)
(35)
Asymptotics of the Hessian matrix 0g (ax). Taking derivative on both sides of 0g ()) = w.r.t yields
0
Since (a (a), substituting this expression into the above equation, obtain
(a) "B
(36)
Thus
Note that
ng
of (()))a
k- (x,) (xxxx)))xxx)))
fg i=l
2
k-1 (xa) ( (x)))
Σ
"p
(x (22) (xxx(xx))
+
ng
np E=1
F=1
and
2 ng
0.0 0,0,
(xx) (()) (TTx)) xx) (x)x))))))) a.kk
n4
ng
up
Σ (aa ( (x))
Σ
(x-1(x/) (a x(())))))
Note that 020 Le = (a.
(a):a).. Using calculations similar to those in the derivation for (31), the results of Lemmas F.6 and F.10, and the rate of Assumption F.3, we have
of (1)) 1r2 (x) (x)
(aT)
+ +2B, 1/2 (x) x) xxxxx)xx==
- 2Bp (x-1 x)x(x))
(1,
(1orri((aBB(()):))))) 2Bg r (x) ((l)) (a (aTx))(((a=
(37)
+ 2E (x), (a)))
() (x)) 2B, (a
(1)
and
( (a)ll o
24

---

Projection Pursuit Density Ratio Estimation
Consequently, have,
where the last equality follows from the fact that à lies between âg and ag, and âg ax - 0 and the continuous mapping theory. Using arguments similar to the proof of the consistency of à to as, we have a -a - 0. Thus,
Thus, under Assumption F.4, 0g (a) is asymptotically invertible and (1) Asymptoties of the score function dg (ax). Note that
(ax) =0, =1 (a (a..aiaa.B,(()):...
2 "g
Pr. (x)) (a (4)2)))))))))))))) Σ
Σ (x-
(()
P()
ng i=1
np im1
Using (16), Lemmas F.9 and F.10, and the arguments similar those in the proof of Lemma F.8, have
(ak)(ax) =2
[ (x) (a (a (x) -2Bp (r- (x-1 (x)
(38)
+ +0p ( + + (((J))-Je(mppA)))
By the definition in (6), we have, for any E A,
( (x) x))((a -2pp-2p x)] =0
(39)
for all integrable yp R R. Moreover, by the definition in (7) and chain rule, we have
0 H( az) fa + =0.
(40)
Note that, for any a A,
8 H.H((a. (1
Using (39), have
0 a()))aa))-0) - l=ak
=0.
Then (40) implies that (aa( a))
=2By
(x2 1
(. (x]x)) (4
(a]x)))))))))]] =0,
l-aa
which, combined with (38), implies that the score function
(alx(ax) =0p (1-(5-)+ +210))++ (h(J))-)/(mppna)
Then we have
| a1ll =0p (1-(0-1) + VEnk-i) (J() +((J))++((())))
25

---

Projection Pursuit Density Ratio Estimation
E.5. Technical Lemmas Lemma F.5. For any (1.2...., =
(l
Proof. For fixed E 0, we choose e/ e/ centering bI....,bM, bM, 3r, such that Sy can be covered, and e2[2sux (Jx)) centering ai..... M; such that A can be covered. Using the triangle inequality and the mean value theorem, we deduce that, for any (b, a) E S x A, there exist j E (l,-.,,Mi) and & E (1,".,,M, such that
b TT Dx (a -)
b [1 |||||| ||a aaaaalll arll
b byll [o(1+)+ + (1 || ||a -aa--a arll s.
Hence, {[b) (x( (a +e]) form set of e-brackets that cover the space Hy. Because j ranges from I to M1, and & raagess from to M2, we have total of M1 x M2 brackets. Therefore,
Ne. Ilo)) SM, XM2
N(ee Meee sup (1))
xeX
Since SJ₁ is a compact set in R and A is compact set in Rd, 11. -C- and N(e. II) s Ce Hence
He: :0)l>))0[[0ll oa
=0 (e/
Lemma F6.. For any k (1,2..... K}, suppose (16) holds. Under Assumption F.3. j =0 1,2, we have
sup 1122 1/4 (a) - - a
Similarly, we have
sup | (2) =0p (VEnA-150(JW)+ So(J2)NJ1 aЕЯ
1M Ang
Furthermore, we have
Mmnn (22(f) (a)) mm/ (22 (,
Mmin { (E,, (a) 21 E3 0a) VEn,E-15o(WW++ (o(Je)
nn Ang [T(Jk)
Mmin (ain(Essaa(a)))) mmin (E,, (a)}-O, VEnAAiZ0(11)+
np Ang
and
Mmax (52( Smaa (0)
Mmax (a) Smax (Σ,, +0 VEm.k-1Z((1)+ [o(Je)JJK
vmp Ang [o(Jk)JJk
Mmmx ( (a)} S1mmax ( (E, (a)) +0
(Enx-iZo(Ja)+
Vnp Ang
uniformly in a A.
26

---

Projection Pursuit Density Ratio Estimation
Proof. Under Assumption F.3, we have the eigenvalues of 2%/ are bounded for j = =0, 1.2 uniformly in a A. We can then derive that
Jr
2 x) (a x)1 (a (')= (a'))
sup E 1122 (a) - /2 (2)1
sup aEA
aE
j=1
J
Bl Bl(o) (a x)) (a x)}
- sup
#E.1
j=1
(a x)) Σ 10 (a
sup
a6 न
sup -B
(
x)
(a x) (a
{J()))
(5(Le)
sup tr
(7)
(a
x) U (a
(a'x)') (
k
a
Thus,
||
(2) -=)
sup a
then follows from the definition of the maximum and minimum eigenvalues that
M/min (a)
min (BT (a)(())++++ B (a) 22/ (a)]ß) 7/min (,)
118" ø1|
and
max (a)) S Mmax (1)
uniformly in E A. For Σx (a), we first define
Σ, (a) = 1 ng
Σ (a'x)))
n1q i=1
Then, we have
Σn (a) -0--2 (a)
-x
(fx- 1 - r4- (4
(41)
11
t=1
fg
h- (x (a Dz xx) (1
(42)
n4
For (41), using (16) and Cauchy- Sehch--Schhazz inequality, we can derive
1 -1x
sup a
||(41)]|
S
sup Mmax (2,, (a)) sup sup (a)xx)) aEA
11 i=1
a
=0p -Vm.A-i-Co(Jil
For (42), using the same arguments for 22,, (a), have
sup a
[[(2)] = (0p()))))))))))))
27

---

Projection Pursuit Density Ratio Estimation
The result then follows.
The results for (a) follows from the fact that (a) (a) and the rate of in Assumption F.3,
Lemma E.7. For any k (1,2..... K), suppose Assumptions F.2 - F.4 and (16) hold. Then,
sup (a) = acA
Proof. Note that (B.a) ()) is convex in B under Assumption F.3, and has unique global minimizer B (a). Given constant > C. J for some constant C > 0 (to be chosen later), for any B satisfying B3), = =d, we have
C (1--
C CJ H (B.a) If (L.o(a) d (Bx,ola) BB)a IT (B
Then, we have
CJ -I 21l (x.o(a) - CJ
- --(8x_0(a--P -B).a -Ix(Bxo(a)aa)
d
CJ (Bx,0la) -Bx) œx.a xx -H(aaxxaa)
= fa.k
CJ= C d (ee - +5(0).
where 5(0) := (Sak (T (B -, a) for any 0 RR Note that fa the minimizer of H(f a). Under Assumption F.3, Hafa HT ax. is globally convex 0 and attains the minimum O. Applying the Taylor's expansion of H(f,, f a) around 0, we have
CJ d -B) -H(fakra)
inf (B:18-Beoole Prola))l=d)
H fax [ffx-
C²J=2
inf (B:10- (BlIB-Prr((a)) -ro(a))==d) ||=d)
(aa
d²
2 171 2
where > is the minimum eigenvalue of ( under Assumptions F.2 and F.3.
By the definition of g(0), we can derive that, for | BB,((a))==
CJ (B,o(a) p1) -(0) d
-5(0) = 2C
d (BLo(a) B} "E,
-))Bxn(a))) Ox (2(aa))]]
22npCCoJ125.
uniformly in and X, where 1/2 is the maximum eigenvalue Bf (x)) xx) x)'] under Assump- tions F.2 and F.3. By choosing C² 20/2o//m /ns have H" (B.a)- (B 2 0 for all | BB,((a)) C- uniformly in A and x E X, which implies that H' (B. a) has a local minimum for (B-BBxx(a)) - C-J=s. Since (a) is the unique global minimizer of H" (B.a), we have
sup - - (x)l = CC. aEA
(43)
Lemma F88. For any k (1,2,.... .,,
K), suppose Assumptions F.2 F44 and (16) hold. For any a A, we have
VJk
(a) (a) 0p
1- 15- VEnA-I+
mg Alp
28

---

Projection Pursuit Density Ratio Estimation
Proof. Recall that
"
and Bx(a)= (E),((a)) np
Σ
h (x)1 xx)))
We decompose
(44)
(Ena(aa))
1
pR- (x=1ax))x(aaxx'))) -B((aa))
np
E=
Emna(a))
1
Mp
(x--
x9)
(45)
np
mg
E {a)a(a))
1P- (x - - x9) Sa)
(46)
ng
"p
1
+
(m)a(aa))
Tx-1(x²)) (xxax/"'''('')))) {E)
2
(47)
np
n4
(a)l
(48)
For (45). Defining i ( (f. R"r and r=
E ER"r by Lemma F.6, we have
7max (())aa(a)))) (1). Thus,
s0 10
(0000)))0000111---77000177710110 (0)(m |||- O 1 (--60-1) 1
where the last inequality follows from the fact that P(a))P(a)P(aaa)'(a(aa is projection matrix with maximum eigenvalue and last equality follows from (16). Using exactly the same arguments and the fact that supa. lfa (a x)] (1),, we have (46) O p ((45)).
For (47), we have
(a)
np
rk (x Dx
Σ ni 1=1
1
Bg
2 (x
np 1=l
=(E))((a)-1
Tk- (x 0x (1
(49)
np
1=1
+(8)ma(a)}]1
(x)) x)' (a) - Σ
kー (x (x (xx'')))aa x/)0x(a' x/) Bi(a)
(50)
ng im1
+
[ [E x)) - (x) (a')))(((a()(((((((()))))))
xx) x) B (a)]].
(51)
29

---

Projection Pursuit Density Ratio Estimation
Note that
Up
B
Rp
Bp
"p
1
$ Bp
|| (x)
"p
tr
[p [Bpl
(x) ) (a))
J/ np
Pp
where the last equality follows from Assumptions F.2 and F.3. Thus, by Chebyshev's inequality, we have
[I(4))] = Op(
7Tp
Similarly, we have
VJk
[|(50)]| = p
Vig
For (51), noting that f³ (a) is the globally unique minimizer of HI (a. B) for any E by the first order condition, we have
0= 0 ( (x) (1) x) B (a)) (x)- (a'x))
Thus, have (51)= = O. Consequently, we have
VJ&
Vnp A ni
Finally, under Assumption F.3, we have (48) OO((Jk/nn)) Thus, we obtain the result.
Lemma F9.. For any k (1,2..... K), suppose Assumptions F.1 F.4 and (16) hold. For any a A and = 0, 1,
((ا) (x)x)= -fakk (a x) = 0 (1 (5-1) ((()))
sup sup
a x
B Jakk (a xx)= fak (a (a x 0 (15
2(s 11--21--11,40 (os-niz(11 (J(()))
a.k (a x;) salh (a xx))2 0r (Ja" -2-20-1) (2((1)))
Proof. Note that by Lorentz (1986, Theorem 8) and Assumption F.2, for j < there exists a (a. E such that
sup sup (2 (B), (z) Co.15(1-11
(52)
a ze
for some constant Ca. > Then, we have
( (z) - (2)
(2)
sup sup
as.9 2
sup sup (a) (1) (z) (2)
as
S sup sup (B) Br, (a) (2)
as 2
sup sup Bk. aa) (z) (2)
ac. 7
sup [8 - Bx,, (a)l (7()2) -0 (1-1))
(53)
a
30

---

Projection Pursuit Density Ratio Estimation
For = 0, we have (53) x from Lemma F.7.
For = I, since Z is compact set, using the fundamental theorem of calculus, (52) implies
sup sup (x)x(=)- - (x., (1z acA
(54)
Then, using a similar argument Lemma F.7, we have the following result
sup = (1-11) a
(55)
Then have
053)= 2(- (J()))
We next prove Bl (a' (a'x) (a'xx p 2(06-11) (,00)) (J(1))
B B(( 1
(Be))())B((a)) " x ((aa'a)]]] (Bx, (a)
s (10 (1=26=1(2,((k))
= (1 (a) B (a) (a)) (1
(56)
Thus, using similar decomposition as (53), we obtain
Bllf2 (a (xxxx=))=--) - f (a x]
sal (a Ta x) Bx) of
2B
+B (BB,() -() "
(a)x))))
=0 2(0-111 (1(1z) +0 (1 21x-1) ( =0 (
2(()))
(57)
We finally prove n1 Ei 1/ (a x,) (xxx) - ffak (a 2xxx1) Op (J)))
Note that, for large enough Jz. e Bl15df (a sall ((axx))]]]] & C. (jkk,, for some constant C. Choosing C(e)= = 2C/e and using the Markov's s inequality, we have
-(6-1)
fak (a (xxxx))aa -Sakk Sall (axx)))=2 2-
n
B ( (a xx)xx)=> - - (a xx)
C(e) J 2-00--1 110--1)
2
which implies the third result.
Lemma F.10. For any k E (1,2..... K}. suppose Assumptions F.l F.4 and (16) hold. For any E A and j = 0, 1,2,
Jy(Jx)JJK
p()) (a x) |ج( (a x)) Op (1 (1-1) (
sup xeX
Vng Anp
+ 2₁(Ja)Je
B elf (a x) - - (a x2) =0p
( 22(--1)
ny np (1-66-1) ( +En.k-1 (J2) 2j(3K
1
Σ
Ain (a xx) -fall) )
(a xx) = Op
ng Anp
31

---

Projection Pursuit Density Ratio Estimation
Proof. Under Assumption F.3 and by Lemma F.8, have
((0 (a x) (a fak ( (axx)
sup x=X
sup (Bx (a) (aTx)
xEX
[ (a) - (a)ll
sup 1 n (t,z)l
z6Z
=0p (((W)) VEnx-11 (
JR nag A Anp
Similar to (56), by Lemma F.8, we deduce that
J lfat (a' x) (xxx)
Sal (a')
[0 - (.( dFx(x)
-Ba(a))BAb (((a)) B 00 _ (a (a'r) (a (a'x)" (B((a)-)11) (a))
= -r -i(0) (((0)) =0p (Ue (1-20-11 t (1 [J)++ - {j(JK) ny A Up
Similar to the end of the proof of Lemma F.9, using Markov's inequality, we also have
T - (xTaa)
E=1
Ba(a)' 0 1 (a x) (a
(a) 1 _ (x a) dFx(x))
=0p
2(x-1)1 (1-21--))+mma-11 +mm--1) [11) 21(32)J2
=0p
ng An
Lemma F.11. For any E (1,2...., ...,..,, K}, suppose Assumptions F. F.4 and (16) hold. We have sup (a)) and [10,B((a) = ( (1), for any a E A. aeM
Proof. First, from (8), we have
sup ( xax x() =0p (0p (J-(s-1) (0(1)) +201))3 /ng /mp (1).
ae ग.
Then, by Lemma F.6.
sup (a)l Ba(a)) Ba(a) s sup Mmin (22 10) (a)) sup Be (a) (a) '200 (a) a
a
acA
acA
=0(1) sup [ la x) (x) aE
I la (xx(=x=)dx(x)xxxxxx)))) x
s0(1) sup
=0, (1).
32

---

Projection Pursuit Density Ratio Estimation
From (36) and (37), using Lemma F.6, have
=1)(
where
M , (a))
2g (2_1 (x) (aa)
(a (a (x)) - 2Bp (a'))
=:M M2 M3.
where the definition of M, and M3 are clear.
By triangle inequality,
Er (a) MIl < (1) Mill+11 (M)l Mall-
Mill. Consider ||E, (E)) Define
V(a v(a"x)= =y ( Dx
(a) (x
which is the L2 projection of ( (x (x x)) T onto the space linearly spanned by Px(aa). Then we have
(a) M1 (M S Mmax (I,, (a) (0
00 (V(a x))((a'((a ( HI) s 0 1110 (4) ( (xla
=0(1).
Similarly, under Assumption F., we have ||E,, (a) 0(1) and || |ا (a) 0(1). The result follows.
33


Reviews:
[{"review_id": "qK7EJehhf5", "reviewer": "Reviewer_fbc2", "summary": "Parametric methods for DRE are susceptible to bias when model assumptions are misspecified, whereas traditional non-parametric approaches often struggle with the curse of dimensionality in high-dimensional settings. To overcome these limitations, the authors suggest using a projection pursuit (PP) based approach to estimate density ratios. They prove consistency and convergence rates for their suggested estimator. Their method is experimentally evaluated on diverse tasks and compared to several baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. Below, we address them point by point.\n\n*Q1: Comparison with projection pursuit density estimation*\n\n[1] and [2] focused on estimating a pdf $p(x)$. In contrast, we aim at estimating the ratio $r^*(x)=p(x)/q(x)$ between two probabilities densities, which is significantly different from [1] and [2] in problem formulation, estimation methods, and theoretical results. We will include these discussions in the revised manuscript.\n- Comparison with [1]:\n\n|   |  $\\qquad\\qquad\\qquad\\qquad$  [1]      |  $\\qquad\\qquad\\qquad\\qquad$  Ours       |\n| :-------: | :----------------------: |:----------------------: |\n|  Model  |  $p(x)\\approx p_K(x)=f_0(x)\\prod_{k=1}^Kf_k(\\theta_k^{\\top}x)$ |     $r^*(x)\\approx r_K(x)=\\prod_{k=1}^Kf_k(a_k^{\\top}x)$      |\n|   Identification   | They iteratively identify $f_k(\\theta_k^{\\top}x)$ by minimizing the KL distance between $p_K$ and $p$ | We iteratively identify $f_k(a_k^{\\top}x)$ by minimizing the $L^2$ distance between $r_K(x)$ and $r^*(x)$ |\n| Estimation | Monte Carlo Sampling |    For a fixed $a_k$, we derived a closed-form estimator of the pursuit function $f_k(·)$, which provides convenience compuation of the estimator.   |\n| Theory  | No any theoretical justification |   We developed convergence results for our proposed method  |\n\n- Comparison with [2]:\n\n[2] proposed a parametric projected probabilistic product model:\n$$p(x)=p(y,z)=\\prod_{i=1}^{d-J}\\mathcal{N}(y_i|0,1)\\prod_{j=1}^J\\mathcal{T}(z_j|\\alpha_j),$$\nwhere $y_i=v_i^{\\top}x$ and $z_j=w_j^{\\top}x$ are projections of $x$ onto $v_i$ and $w_j$ respectively, $\\mathcal{N}(y_i|0,1)$ is the standard Gaussian distribution, and $\\mathcal{T}(·|\\alpha_j)$ is a univariate parametric distribution indexed by $\\alpha_j$ such as the Student-t distribution. They further developed a sequential learning algorithm for this model. Note that, [2] employs a fully parametric approach, distinct from both [1] and our work which require estimation of unknown pursuit functions. Again, no theoretical results are provided in [2].\n\n*Q2: Hyperparameter selection process*\n\nWe will add a detailed description in the paper. Please kindly refer to our response to Q2 of reviewer WQAb. Due to space limitation, we addressed this point there.\n\n*Q3: Discussion of  featurized DRE*\n\n We will incorporate the fDRE method as a baseline (kindly refer to our response to reviewer qyuE for experimental results) and discuss it in the paper. We would like to clarify that [3] neither map the data to a low-dimensional latent space nor address the curse of dimensionality as focused by our paper. In fact, they tackled a different DRE challenge posed by significant distributional divergence between $p(x)$ and $q(x)$. To address this issue, they proposed an invertible parametric transform $f_{\\theta}:\\mathbf{R}^{d}\\to\\mathbf{R}^{d}$ such that the transformed densities become closer. Then $r^*(x)$ is estimated based on:\n$$r^*(x)= \\frac{p(x)}{q(x)}=\\frac{p'(f_{\\theta}(x))}{q'(f_{\\theta}(x))},$$ where $p'$ and $q'$ are densities of transformed data $f_{\\theta}(x_p)$ and $f_{\\theta}(x_q)$ respectively. Note that, the invertible transformation preserves the original data's dimensionality.\n\n*Q4: Theoretical results comparison with DRE with iterated regularization*\n\nWe will add a discussion on the theoretical results established in [4].  [4] proposed an estimation method for the density ratio $r^*(x)$ by minimizing the Bregman distance based on a RKHS $\\mathcal{H}$ with iterated regularization:\n$$r^{\\lambda,t+1}:=\\arg\\min_{r\\in\\mathcal{H}} B(r^*(x),r)+\\frac{\\lambda}{2}\\|r-r^{\\lambda,t}\\|^2.$$\nwith $r^{\\lambda,0}=0$. Their key theoretical result is an improved error bounds faster than the non-iterated error bound $(n_p+n_q)^{-1/3}$, under the Bregman distance and certain regular conditions (e.g. source condition and capacity condition). In contrast, we establish the convergence rates (after $k$th iteration) for our proposed estimator under sup-norm:\n$$\\sup_{x\\in\\mathcal{X}}|\\hat{r}\\_{k}(x)-r_{k}(x)|=O_{p}(\\sum_{\\ell=1}^k[\\\\{J_{\\ell}^{-(s-1)}+\\sqrt{\\frac{J_{\\ell}}{n_q\\wedge n_p}}\\\\}·\\Pi_{i=\\ell}^k\\\\{\\sqrt{\\tilde{\\zeta}\\_1(J_i)}\\vee\\zeta^2_0(J_i)\\\\}]),$$where $s$ is the smoothness of the pursuit functions $\\\\{f_j\\\\}\\_{j=1}^k$. Under sufficient conditions, our rate can also achieve $o_P(n_p+n_q)^{-1/3}$. However, as the function spaces, approximation space, and distance metrics considered in [4] and our paper are completely different, it is difficult/meaningless to explicitly compare the two theoretical results based on the same standard.", "claims_and_evidence": "Overall, the claims made in the paper are supported by theoretical and experimental evidence. Theorem 3.2 presents the theoretical findings, while Section 4 provides a detailed account of the experiments conducted on various datasets.", "methods_and_evaluation_criteria": "The proposed PP method is well-motivated for the DRE setting, with a clear intuitive rationale for its potential to enhance performance. The selection of benchmark datasets and evaluation metrics is appropriate and aligns with standard practices in assessing DRE methodologies.", "theoretical_claims": "The results in Proposition 3.1 seem to be valid and skimming over the proof of Theorem 3.2 I did not find any inconsistencies.", "experimental_designs_or_analyses": "The experimental setup appears to be well-structured for evaluating the stated claims. I could not find a detailed description of the hyperparameter selection process, including dataset splits, search methodology, and search grid specification, however.", "relation_to_broader_scientific_literature": "The primary contributions of this paper pertain to Projection Pursuit Density Estimation [1,2] and established DRE methods, which are utilized as baselines for comparison.\n\n[1] Friedman et al. A projection pursuit density estimation, 1984\n\n[2] Welling et al. Efficient Parametric Projection Pursuit Density Estimation, 2012", "essential_references_not_discussed": "To enhance the comprehensiveness of the study, it would be beneficial to discuss related approaches or incorporate them as baseline methods for comparison. For instance, the method proposed in [3] first maps the data to a low-dimensional latent space before performing DRE to mitigate the curse of dimensionality.\nAdditionally, it would be valuable to explore how the theoretical results presented in this manuscript compare to the convergence rates established in [4].\n\n[3] Choi et al. Featurized Density Ratio Estimation, 2021\n\n[4] Gruber et al. Overcoming Saturation in Density Ratio Estimation by Iterated Regularization, 2024", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "At first glance, PP for DRE seems to be an application of the PP algorithm conceptually very similar to how it is used in [1,2]. Could the authors emphasize the major differences/novelties compared to these approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YKgaB6eGox", "reviewer": "Reviewer_mSdz", "summary": "This paper introduces a novel projection pursuit (PP)-based method for density ratio estimation (DRE), a critical task in machine learning with applications in areas like causal inference and covariate shift adaptation. Addressing the limitations of parametric methods (potential bias) and non-parametric methods (curse of dimensionality), the proposed approach approximates the density ratio function as a product of functions, each representing a projection onto a low-dimensional space. These projections are iteratively estimated using semi-parametric single-index functions and a regularized empirical loss, leading to a consistent estimator with a demonstrable convergence rate. Experimental results showcase the method's superior performance compared to existing DRE techniques in various applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments.\n\n*Q1: Error bound for density ratio $\\hat{r}$*\n\nThe main result in the original manuscript gives the convergence rates of $\\hat{f}\\_{a,k}$ and $a_l$ given the estimate $\\hat{r}\\_{k-1}$. Since our estimator $\\hat{r}\\_k$ is computed iteratively, and $\\hat{r}\\_0=r_0=1$, we can establish the rate of $\\hat{r}\\_k$ in an inductive way. We first suppose for $k=1,\\ldots, K$, $n_p^{-1}\\sum^{n_p}\\_{i=1}\\\\{\\hat{r}\\_{k-1}(x_i^p)-r_{k-1}(x_i^p)\\\\}^2=O_p(\\xi_{n,k-1})$ for some $\\xi_{n,k-1}>0$. Using the arguments for establishing Theorem 3.2 in the original manuscript, we can establish the convergence rates:$$\\sup_{a\\in\\mathcal{A}}\\sup_{z\\in\\mathcal{Z}}|\\hat{f}\\_{a,k}(z)-f_{a,k}(z)|=O_p(J_k^{-s}\\zeta_0(J_k)+\\sqrt{\\xi_{n,k-1}}\\zeta_0(J_k)^2+\\frac{\\sqrt{J_k}\\zeta_0(J_k)^2}{\\sqrt{n_q\\wedge n_p}})\\tag{1}$$and $\\\\|\\hat{a}\\_k-a_k\\\\|=O_p(\\\\{J_k^{-(s-1)}+\\sqrt{\\xi_{n,k-1}}+\\frac{\\sqrt{J_k}}{\\sqrt{n_q\\wedge n_p}}\\\\}·\\sqrt{\\tilde{\\zeta}\\_1(J_k)})$\n\nwhere $\\tilde{\\zeta}_1(J)$ is a sequence of constants such that the maximum eigenvalue of $\\mathbb{E}[\\Phi_k^{(1)}(a^\\top x)\\Phi_k^{(1)}(a^\\top x)^\\top]$ is bounded by $\\tilde{\\zeta}_1(J)$ uniformly in $a\\in\\mathcal{A}$.\n\nWe can then decompose$$\\hat{f}\\_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{a_k,k}(a_k^\\top x)=\\hat{f}\\_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)\\tag{2}$$\n$$\\qquad+f_{\\hat{a}\\_k,k}(a_k^\\top x)-f_{a_k,k}(a_k^\\top x)\\tag{3}$$\n$$\\qquad+f_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{\\hat{a}\\_k,k}(a_k^\\top x)\\tag{4}$$\nFor (2),  we have $\\sup_{x\\in\\mathcal{X}}|\\hat{f}\\_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)|\\leq\\sup_{a\\in\\mathcal{A}}\\sup_{x\\in\\mathcal{X}}|\\hat{f}\\_{a,k}(a^\\top x)-f_{a,k}(a^\\top x)|$.\n\nFor (3), under Assumption D.2. (b), we can use Taylor's expansion to expand $f_{\\hat{a}\\_k,k}$ around $f_{a_k,k}$ and obtain $\\sup_{x\\in\\mathcal{X}}|f_{\\hat{a}\\_k,k}(a_k^\\top x)-f_{a_k,k}(a_k^\\top x)|=O_p(\\\\|\\hat{a}\\_k - a_k\\\\|)$.\n\nFor (4), we can first decompose it as$$(4) = [f_{a_k,k}(\\hat{a}\\_k^\\top x) - f_{a_k,k}(a_k^\\top x)] +[f_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x) - f_{a_k,k}(\\hat{a}\\_k^\\top x)] + [f_{a_k,k}(a_k^\\top x)-f_{\\hat{a}\\_k,k}(a_k^\\top x)].$$\nThe supremum norm of the first term can be bounded by $O_p(\\\\|\\hat{a}\\_k-a_k\\\\|)$, using Taylor's expansion expanding $f_{a_k,k}(\\hat{a}\\_k^\\top x)$ around $f_{a_k,k}({a}\\_k^\\top x)$. The second and the third term can also be bounded by $O_p(\\\\|\\hat{a}\\_k-a_k\\\\|)$ using the same argument for (3). Thus, we have$$\\sup_{\\in\\mathcal{X}}|\\hat{f}\\_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{a_k,k}(a_k^\\top x)|=O_p(\\sup_{a\\in\\mathcal{A}}\\sup_{x\\in\\mathcal{X}}|\\hat{f}\\_{a_k,k}(a_k^\\top x)-f_{a_k,k}(a_k^\\top x)|+\\\\|\\hat{a}\\_k-a_k\\\\|).$$\nNoting that $\\zeta_0(J_k)\\leq J_k$, we then have$$\\sup_{x\\in\\mathcal{X}}|\\hat{f}\\_{\\hat{a}\\_k,k}(\\hat{a}\\_k^\\top x)-f_{a_k,k}(a_k^\\top x)|=O_p([J_k^{-(s-1)}+\\sqrt{\\xi_{n,k-1}}+\\sqrt{\\frac{J_k}{n_q\\wedge n_p}}]·[\\sqrt{\\tilde{\\zeta}\\_1(J_k)}\\vee\\zeta^2_0(J_k)]),$$\nwhere $\\sqrt{\\tilde{\\zeta}_1(J_k)}\\vee \\zeta_0(J_k)=\\max[\\sqrt{\\tilde{\\zeta}_1(J_k)},\\zeta_0(J_k)]$.\n\nFinally, using the fact that $\\xi_{n,0}=0$, we can inductively derive that$$\\sup_{x\\in\\mathcal{X}}|\\hat{r}\\_{K}(x)-r_{K}(x)|=O_{p}(\\sum_{\\ell=1}^K[\\\\{J_{\\ell}^{-(s-1)}+\\sqrt{\\frac{J_{\\ell}}{n_q\\wedge n_p}}\\\\}·\\Pi_{i=\\ell}^K\\\\{\\sqrt{\\tilde{\\zeta}\\_1(J_i)}\\vee\\zeta^2_0(J_i)\\\\}]).\n$$We will add this result to the paper.\n\n*Q2\\&3: Clarification of norms*\n\nThank you for your careful comments. We use the Euclidean norm to measure the distance between $\\hat{a}\\_k$ and $a_k$. To make the notation consistent, we will clearly define the Euclidean norm $\\\\| v\\\\|:=\\sqrt{v^{\\top}v}$ (without the subscript \"2\" shown in $\\\\|·\\\\|_2$ in the previous manuscript) for a general vector $v$, and we will apply it to both $\\hat{a}\\_k-a_k$ and $a$, i.e. $\\\\|\\hat{a}\\_k-a_k\\\\|$ and  $\\\\| a\\\\|$.\n\n*Q4: Adding Baseline from [1]*\n\nWe will incorporate the regularized kernel method proposed in [1] into our experimental comparisons. Please kindly refer to our response to Reviewer qyuE. Due to space limitation, we addressed this point there.\n\n*Q5: Discussion of related work*\n\nThank you for your references. We will add the following discussion to the introduction:\n> **Regularized Kernel Learning Methods**. Recently, the regularization scheme within reproducing Kernel Hilbert space (RKHS) has been developed for estimating the DRE problem. [3] reformulated the DRE problem as an inverse problem in terms of an integral operator corresponding to a kernel, then proposed a regularized estimation method with an RKHS norm penalty [1] established the pointwise convergence rate of the regularized estimator taking into account both the smoothness of the density ratio and the capacity of the space in which it is estimated.  [2] applied the regularized kernel methods in the context of unsupervised domain adaptation under covariate shift and developed the convergence rates.", "claims_and_evidence": "Yes, the claims are made and supported by clear and convincing evidence", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense for the problem.", "theoretical_claims": "Yes, I have checked the proof of Proposition 3.1.", "experimental_designs_or_analyses": "Yes, I have checked. All experiments are designed to support the theory. However, it would be better if the up-to-date methods are added, such as in a recent paper [1].\n\n[1] Duc Hoan Nguyen, Werner Zellinger, and Sergei Pereverzyev. On regularized Radon-\nNikodym differentiation. Journal of Machine Learning Research, 25(266):1–24, 2024.", "relation_to_broader_scientific_literature": "The proposed approach approximates the density ratio function as a product of functions, each representing a projection onto a low-dimensional space, to address the limitations of parametric methods (potential bias) and non-parametric methods (curse of dimensionality).", "essential_references_not_discussed": "To put the paper in the proper place, it would be reasonable to include in the discussion the recent publications such as\n \n[1] Duc Hoan Nguyen, Werner Zellinger, and Sergei Pereverzyev. On regularized Radon-\nNikodym differentiation. Journal of Machine Learning Research, 25(266):1–24, 2024.\n\n[2] Elke R. Gizewski, Lukas Mayer, Bernhard A. Moser, Duc Hoan Nguyen, Sergiy\nPereverzyev, Sergei V. Pereverzyev, Natalia Shepeleva, and Werner Zellinger. On a\nregularization of unsupervised domain adaptation in RKHS. Applied and Computational\nHarmonic Analysis, 57:201–227, 2022.\n\n[3] Qichao Que, Mikhail Belkin. Inverse Density as an Inverse Problem: The Fredholm Equation Approach. Advances in Neural Information Processing System, 26 (2013).", "other_strengths_and_weaknesses": "Strengths: The authors provide the analysis of algorithms.", "other_comments_or_suggestions": "1. Line 163 in the right column page 3, $k$ in this fomula $\\hat{f}_k (\\hat{a}_k^Tx)$ should be $m$.\n2. In proposition 3.1, $I_{J_k}$ is not defined.", "questions_for_authors": "1. The main results are two estimations for $\\hat{f}_{a,k}$ and $\\hat{a}_k$, but there is no error bound for density ratio $\\hat{r}$. Can the authors provide a bound for $\\hat{r}$?\n2. Which norm do the authors measure the difference between $\\hat{a}_k - a_k$?\n3. Can the authors indicate the norms in which space? For example, in line 128 in the right column on page 3, the authors use “$||a||_2$”, but $||.||_2$ is not defined. In the proof of Theorem 3.2 in Appendix D, the authors use $||.||$, the readers cannot know what space the norm is considered.\n4. It will be beneficial if the authors add to the baseline in experiments the up-to-date method such as in [1]\n\n[1] Duc Hoan Nguyen, Werner Zellinger, and Sergei Pereverzyev. On regularized Radon-\nNikodym differentiation. Journal of Machine Learning Research, 25(266):1–24, 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhrcHGwJdX", "reviewer": "Reviewer_qyuE", "summary": "This paper proposes a non-parametric method for the density ratio estimation (DRE) task using a projection pursuit (PP) approximation. The method is computationally convenient and also helps alleviate the curse of dimensionality. The authors conduct extensive experiments demonstrating that the proposed method consistently outperforms other approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions.\n\n*Q1: Computational cost evaluation in high-dimensional cases*\n\nCompared to the conventional projection pursuit method (kindly refer to our response to Q1 of Reviewer fbc2) that requires additional Monte Carlo sampling to estimate the pursuit function, our estimators of $\\\\{f_{k}(·)\\\\}_{k=1}^K$ in equation (5) take closed-form expressions, enhancing computational efficiency. Moreover, we use Cholesky decomposition in computing the closed-form solution, improving numerical stability and reducing overall computation time.  \n\nWe evaluated computational costs via numerical experiments, which will be included in Appendix. Specifically, we will report the runtime of ppDRE method and all baseline methods (including newly added baselines, fDRE and RRND, suggested by Reviewers fbc2 and mSdz), in the same setup as Figure 3 in Section 4.2.1 with dimensions $d\\in\\\\{2,10,30,50,100\\\\}$. The experiments were conducted on a node with dual AMD EPYC 7713 CPUs, totaling 128 cores. The computation time (minutes) averaged over 3 runs is reported below:\n\nTable4. Computation Time Comparison\n\n|Method|$d=2$|$d=10$|$d=30$|$d=50$|$d=100$|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|uLSIF|0.38|0.35|0.38|0.43|0.46|\n|KLIEP|0.36|0.39|0.49|0.57|0.72|\n|Class|0.21|0.16|0.19|0.23|0.40|\n|fDRE|3.78|3.88|3.92|4.28|7.25|\n|RRND|3.78|3.87|3.77|3.82|3.81|\n|nnDRE|5.91|3.84|8.76|6.02|8.63|\n|$\\text{D}^3$-LHSS|5.95|11.25|10.60|10.79|10.97|\n|ppDRE|0.32|0.91|5.64|10.78|15.22|\n\nWe observe the following results: (i) when the dimension is low or moderately large ($d=2,10$),  the computation time of our ppDRE method is comparable to that of the uLSIF, KLIEP, and classification methods, which are suitable for low-dimensional data.  Our computation time is much less than that of the fDRE, RRND, nnDRE and $\\text{D}^3$-LHSS methods, some of which are also designed for high-dimensional data. (ii) When the dimension is large ($d=30,50,100$), the methods for low-dimensional data fail due to the curse of dimensionality. Our ppDRE method consistently outperforms all baseline methods in estimation accuracy, and the computation time is comparable to that of existing methods for high-dimensional data. The computation time increases significantly, which can be expected as we need more iterations to achieve convergence.\n\n*Q2: Adding fDRE and RRND as baselines*\n\nWe will make the following improvements to our experimental analysis in Section 4 of the paper.\n-  We will incorporate the two methods as baselines. \n1. We will add a discussion on the featurized DRE method (fDRE) in [1] (kindly refer to our response to Q3 of Reviewer fbc2). The fDRE method maps the data to a shared feature space via a flow model to mitigate inaccuracies caused by distribution discrepancy.\n2. We will add a discussion on the regularized Radon-Nikodym differentiation estimation method (RRND) in [2] (kindly refer to our response to Q5 of Reviewer mSdz). The RRND method employs a regularized kernel method in RKHS to estimate the density ratio.\n- We will add the following experimental results of both baselines across key applications to Tables 1, 2 and 3 in our paper. Implementation details are given in our response to Q2 of Reviewer WQAb.\n\nTable1. ASE in dose response function estimation\n\n|Method|ARDF|QRDF(0.1)|QRDF(0.25)|QRDF(0.5)|QRDF(0.75)|QRDF(0.9)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|fDRE|0.106(0.026)|0.331(0.056)|0.127(0.029)|0.045(0.016)|0.088(0.021)|0.267(0.061)|\n|RRND|0.117(0.022)|**0.307**(0.053)|0.125(0.026)|0.043(0.015)|0.090(0.021)|0.408(0.082)|\n\nTable2. MAE in mutual information estimation for varying dimension $p$ and correlation coefficient $ρ$\n\n|Method|$d=2,ρ=0.2$|$d=2,ρ=0.8$|$d=10,ρ=0.2$|$d=10,ρ=0.8$|$d=20,ρ=0.2$|$d=20,ρ=0.8$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|fDRE|0.043(0.000)|0.907(0.048)|0.199(0.007)|4.283(0.014)|0.487(0.036)|8.710(0.059)|\n|RRND|0.061(0.001)|0.926(0.006)|0.200(0.008)|5.092(0.031)|0.414(0.022)|10.241(0.089)|\n\nTable3. NMSE under covariate shift adaptation\n\n|Method|Abalone|Billboard Spotify|Cancer Mortality|Computer Activity|Diamond Prices|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|fDRE|0.586(0.136)|0.615(0.084)|0.664(0.064)|0.294(0.053)|0.233(0.078)|\n|RRND|0.574(0.144)|0.616(0.087)|**0.655**(0.069)|0.279(0.051)|0.232(0.082)|\n\nThe results indicate that our ppDRE maintains superior performance in most cases. The fDRE method (mapping data with a flow model prior to KLIEP) shows its effectiveness compared to the naive KLIEP method with enhanced performance in most cases, but remains inferior to ppDRE. The RRND method proves its merit by outperforming the non-regularized kernel methods (uLSIF, KLIEP) in several instances and even achieves top performance in some cases (bolded); however, its overall performance is less stable, ultimately underperforms ppDRE.\n\n[1] Choi et al. Featurized Density Ratio Estimation, 2021\n\n[2] Duc Hoan Nguyen, Werner Zellinger, and Sergei Pereverzyev. On regularized Radon- Nikodym differentiation. Journal of Machine Learning Research, 25(266):1–24, 2024.", "claims_and_evidence": "Yes, the claims are well supported by strong theoretical proofs and a large number of experiments.Yes, the claims are well supported by strong theoretical proofs and a large number of experiments.", "methods_and_evaluation_criteria": "Most of the evaluation metrics make sense, such as MSE or similar measures. However, there is no metric provided to demonstrate the computational cost of this method.", "theoretical_claims": "Yes. I checked the proofs for Proposition3.1 and AppendixC (L2-distance Minimization); they all appear to be correct.", "experimental_designs_or_analyses": "The datasets used in this paper are appropriate. The authors employ simulated data for a 2-D DRE experiment on stabilized weights \nestimation. They also use the real IHDP dataset for dose-response function estimation. In the section on covariate shift adaptation, they use 5 different benchmark datasets.", "relation_to_broader_scientific_literature": "This paper is a good application of the projection pursuit (PP) idea to the density ratio estimation problem. As stated in the paper, the approach is similar to nnDRE, except that the feedforward neural network (NN) is replaced by projection pursuit (PP).", "essential_references_not_discussed": "No, I believe all relevant references are adequately cited in this work.", "other_strengths_and_weaknesses": "Strengths: The paper is well written and presents the ideas clearly, with solid proofs and thorough experimental validation.\n\nWeaknesses: The main idea is not entirely new, but it is well integrated into this particular problem. There is no demonstration of computational cost in high-dimensional cases within the experiments", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGdF7wN9iu", "reviewer": "Reviewer_WQAb", "summary": "This paper considers the problem of density ratio estimation (DRE), where parametric methods may leads to bias, while non-parametric methods struggle with the curse of dimensionality in high-dimensional settings. The authors present an approach that leverages projection pursuit approximation to estimate the density ratio. Theoretical analysis establishes the estimator’s consistency and convergence rate. Additionally, experimental results demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments.\n\n*Q1: Theoretical support for the superiority of the proposed method*\n\nWe develop asymptotic convergence rates (Theorem 3.2) for our proposed projection pursuit density ratio estimator:\n$$ \\sup_{x\\in\\mathcal{X}}| \\hat{r}\\_{K}(x) - r_{K}(x)| =O_{p}(\\sum_{\\ell=1}^K[\\\\{J_{\\ell}^{-(s-1)}+ \\sqrt{\\frac{J_{\\ell}}{n_q \\wedge n_p}}\\\\}\\cdot \\Pi_{i=\\ell}^K\\\\{\\sqrt{\\tilde{\\zeta}\\_1(J_i)}\\vee \\zeta^2_0(J_i)\\\\}])$$\nPlease kindly refer to the details in our response to Q1 of Reviewer mSdz.  This rate does not depend on the dimension of the data, which highlights the strengths of projection pursuit modeling in mitigating the curse of dimensionality and may explain the superior performance compared to other methods based on direct nonparametric approximation. We agree with you that the non-asymptotic rates are more proper evidence for evaluating the performance of the estimator, but the proof presents significant challenges and is beyond the scope of this paper. We will pursue it in future work.\n\n*Q2: Hyperparameter selection process*\n\nTo ensure rigorous and fair comparisons, we implemented the following protocol.\n- For our proposed ppDRE method, as mentioned in Remark 3.3, the optimal hyperparameters are identified by the minimal validation loss in CV. For clarity, we detail our approach as follows: across all experiments, we utilized 5-fold CV with random sampling. A grid search was conducted over a predefined set of parameter ranges, which are outlined in the table below:\n\nTable1. Hyperparameters for ppDRE method\n\n|parameter|description|searchspace|\n|:-:|:-:|:-:|\n|$K$|Number of PP iterations|\\{5,10,15\\}|\n|$J_k$|Number of basis functions|\\{20,50,70,100,150\\}|\n|$\\lambda$|L2-penalty strength|\\{0.5,1,5,10\\}|\n|$\\delta$|Gradient descent learning rate|\\{0.001,0.01,0.1\\}|\n\n- For open-source baseline methods (e.g., KLIEP, uLSIF, $\\text{D}^3$-LHSS), we utilized their official implementations (e.g., Python densratio package for uLSIF) and default hyperparameter search grids as recommended in their original papers or standard toolkits. For instance, uLSIF's hyperparameters ($\\sigma$, $\\lambda$) were selected from the grid 1e-3:10:1e9, consistent with its standard implementation. These methods inherently incorporate CV-based hyperparameter selection in their standard workflows. We preserved these built-in CV mechanisms without modification.\n- For Classification and nnDRE, we implemented 5-fold CV (aligned with our ppDRE method) to identify the optimal hyperparameters within the search spaces outlined in the table below:\n\nTable2. Hyperparameters for Classification and nnDRE methods\n\n|     method     |   parameter   |      search space       |\n| :-: | :-: | :-: |\n| Classification | n_estimators  |       \\{100, 300}       |\n|                | learning_rate | \\{0.0001, 0.001, 0.01\\} |\n|                |  num_leaves   |        \\{20, 30}        |\n|     nnDRE      |     depth     |         \\{2, 3}         |\n|                |     width     |      \\{8, 32, 64\\}      |\n|                |   learning    | \\{0.0001, 0.001, 0.01\\} |\n\n- For fDRE, we have implemented a version that employs KLIEP as the second-stage DRE method. Adhering to the guidelines provided by the official open-source code and the original research paper [1], we trained the masked autoregressive flow (MAF) models with the configurations presented in the table below. For the KLIEP method in the second stage, we have retained the original hyperparameter settings as per the open-source implementation.\n\nTable3. Hyperparameters for the flow model in fDRE method\n\n|Dataset|n_blocks|n_hidden|hidden_size|n_epochs|\n|:-:|:-:|:-:|:-:|:-:|\n|IHDP-Continuous|5|1|100|100|\n|Regression Benchmarks|5|1|100|100|\n|MI Gaussians|5|1|100|200|\n\n- For RRND,  in the absence of open-source code, we implemented the algorithm by adhering to the implementation details outlined in the Numerical Illustrations section of [2]. The kernel function is assigned as $K(x,x^\\prime)=1+\\exp\\\\{-(x-x^\\prime)^2/2\\\\}$. Utilizing a 5-fold CV, the hyperparameter $\\lambda$ is chosen based on the quasi-optimality criterion, and the optimal iteration step $k$ is determined by minimizing squared distance loss function in the validation set. Following the configurations in [2], the search grids are $k\\in\\\\{1,2,\\ldots,10\\\\}$ and $\\lambda\\in\\\\{\\lambda_\\ell=\\lambda_0\\rho^\\ell,\\ell=1,\\ldots,w\\\\}$ with $\\lambda_0=0.9$ and $w=9$. The decay factor $\\rho=(\\lambda_w/\\lambda_0)^{1/w}$ is derived from the lower bound $\\lambda_w=(n^{-1/2}+m^{-1/2})$, where $n,m$ are the sample sizes of the numerator and denominator distributions respectively.\n\nWe will incorporate these implementation details into the Appendix of the revised manuscript to improve reproducibility.\n\n[1] Choi et al. Featurized Density Ratio Estimation, 2021\n\n[2] Duc Hoan Nguyen, Werner Zellinger, and Sergei Pereverzyev. On regularized Radon- Nikodym differentiation. Journal of Machine Learning Research, 25(266):1–24, 2024.", "claims_and_evidence": "The consistency and the convergence rate of the proposed algorithms is established in Theorem 3.2. \n\nThe effectiveness of the algorithm is presented in Table 1,2 and 3, where the proposed density ratio estimation method is applied in several different cases and achieves good permormance", "methods_and_evaluation_criteria": "The proposed method makes sense, since it has a clear motivation and mathematical derivation, and also has experimental support. For benchmark datasets, the authors use the dataset that has been adopted by previous works in the corresponding application, which is also reasonable.", "theoretical_claims": "I did not check every proof in detail.", "experimental_designs_or_analyses": "The experimental designs are in general sound. Only one question: how do you choose the hyper parameters of other algorithms? How can readers be convinced that the superiority performance of ppDRE is not an artifact of hyper parameter selection?", "relation_to_broader_scientific_literature": "The ppDRE method, as already shown in the paper, can be applied to dose response function estimation and other fields of science.", "essential_references_not_discussed": "Not finding any missing essential related works.", "other_strengths_and_weaknesses": "Strength: novel idea of leveraging projection pursuit in density ratio estimation. Clear motivation and explanation of the algorithm. Theoretical results are provided, justifying the consistency of the estimator. Experimental results are provided, showing the effectiveness of this algorithm, compared with prior works.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "I am wondering if you can see from your theory (Thm 3.2) of why your method achieves superior performance than other methods. Is your theory only about convergence rate and consistency, not any non-asymptotic result that can show your superiority?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the theoretical contributions made by the paper, as well as the potential practical relevance of the methodology for problems requiring density ratio estimation in higher dimensional applications where non-parametric techniques typically struggle.

The reviewers raised some concerns over the computational complexity/running time of the method, as well as querying the existence of finite sample bounds on accuracy. There were also multiple comments relating to the presentation of the paper contents, in that the contribution's position within the existing body of literature was not clearly and/or insufficiently justified.

I believe the authors have thoroughly addressed the vast majority of these matters in their rebuttal, and with the proposed modifications I believe the work may be a valuable contribution.
Decision: Accept (poster)