Fine-Grained Dynamic Framework for Bias-Variance Joint Optimization on Data Missing Not at Random
Authors: Mingming Ha
Taoxuewen
Wenfang Lin
QIONGXU MA
Wujiang Xu
Linxun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Advertising recommendation systems
Quantifying bias-variance joint optimization
Limitations of regularization
Primary area: causal_inference

Abstract: In most practical applications such as recommendation systems, display advertising, and so forth, the collected data often contains missing values and those missing values are generally missing-not-at-random, which deteriorates the prediction performance of models. Some existing estimators and regularizers attempt to achieve unbiased estimation to improve the predictive performance. However, variances and generalization bound of these methods are generally unbounded when the propensity scores tend to zero, compromising their stability and robustness. In this paper, we first theoretically reveal that limitations of regularization techniques. Besides, we further illustrate that, for more general estimators, unbiasedness will inevitably lead to unbounded variance. These general laws inspire us that the estimator designs is not merely about eliminating bias, reducing variance, or simply achieve a bias-variance trade-off. Instead, it involves a quantitative joint optimization of bias and variance. Then, we develop a systematic fine-grained dynamic learning framework to jointly optimize bias and variance, which adaptively selects an appropriate estimator for each user-item pair according to the predefined objective function. With this operation, the generalization bounds and variances of models are reduced and bounded with theoretical guarantees. Extensive experiments are conducted to verify the theoretical results and the effectiveness of the proposed dynamic learning framework.

Fine-Grained Dynamic Framework for Bias- Joint Optimization on Data Missing Not at Random
Mingming Ha MYbank, Ant Group Beijing, China hamingming. hwn@mybank.c.m cn
Xuewen Tao MYbank, Ant Group Shanghai, China
Wenfang Lin MYbank, Ant Group Hangzhou, China moxi 1wf @nybank cn
xuewen. txw@mybank.cmm
Qiongxu Ma MYbank, Ant Group Shanghai, China qiongxu www.:xxwwmmmmmbbbaa..o cn
Wujiang Xu
Linxun Chen
MYbank, Ant Group Shanghai, China
MYbank, Ant Group Beijing, China linxun. clx@mybank. cn
auwujiang xwj @eybank. cn
Abstract
In most practical applications such as recommendation systems, display advertising, and forth, the collected data often contains missing values and those missing values are generally performance of models. Some existing estimators and regularizers attempt achieve unbiased estimation to improve the predictive performance. However, variances and generalization bound of these methods are generally unbounded when the propensity scores tend to zero, compromising their stability and robustness. In this paper, we first theoretically reveal that limitations of regularization techniques. Besides, we further illustrate that, for more general estimators, unbiasedness will inevitably lead to unbounded variance. These general laws inspire us that the estimator designs is not merely about eliminating bias, reducing variance, or simply achieve varianee trade-off. Instead, it involves quantitative joint optimization of bias and variance. Then, develop systematic fine- grained dynamic learning framework to jointly optimize bias and variance, which adaptively selects appropriate estimator for each user-item pair according to the predefined objective function. With this operation, the generalization bounds and variances of models are reduced and bounded with theoretical guarantees. Extensive experiments are conducted to verify the thcoretical results and the effectiveness the proposed dynamic learning framework.
which deteriorates the prediction
Introduction
In virtually all real-world applications, the pieces of data we collected are partially missing with certain probabilities. A special case with the identical missing probability known as missing at random (MAR) [I] However, in online recommendation, search, and display advertising, there are lots of missing--ntt-t-randde missing with different probabilities, i.e., propensities. For example, in recommendation systems, a user usually clicks the items that she/he is likely to purchase and ignores other items with a low willingness to buy. Therefore, the observed click and conversion data is MNAR, which are not representative samples of all the events (5). When the MNAR data is used to train model, the prediction performance of this model on the MAR data is generally unacceptable. This is because MNAR data introduces sample selection bias 6 into the prediction model. To eliminate sample have been developed, e.g., Error- Eeror-Imputation-
(MNAR) click, conversion, and rating records 22
which are
selection bias, lots of debiasing estimators 3 6
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Based (EIB) approach [TO], Inverse Sospensity-Sooring (IPS) technique [6], Doubly Robust (DR) method and so forth.
However, in almost all debiased methods, the existence of propensities results in the high variance and
generalization bound. [1]11 Therefore, various methods [5 g 12 have been developed to reduce
estimation variances and improve the model stability. Even so, they still suffer from unbounded variances and generalization bounds when the propensity tends zero. For the high variance and generalization bound caused by small propensities, some approaches compromise to self-normalized addition, the overwhelming majority of previous works focus on the specific designs of the estimators or regularizers to reduce variance or eliminate bias while neglecting both the bias-variance relationship of estimators and the essence of the estimator
technique [22 [ at the expense of unbiasedness.
designs.
In this paper, we reveal limitations of general regularization techniques. We find that it is impossible to reduce variance without sacrificing unbiasedness by introducing regularizers, and that regularization cannot guarantee estimators to have bounded variance and generalization bound. Besides, for general estimators, unbiasedness will inevitably result in unbounded variance and generalization bound. To some extent, the generalization bound can reflect the predictive performance of estimator. Therefore, reducing and bounding the generalization bound can assist in improving the predictive performance of models. Since the generalization bounds of estimators contain the bias and variance terms, the essence of estimator design is not merely about eliminating bias, reducing variance, or simply achieving bias-variance trade-off but about the quantitative joint optimization of bias and variance. Then, we develop systematic dynamic learning framework to achieve this objective. To the best of our knowledge, this the first work to systematically reveal limitations of general regularizers and the design perspective of the quantitative bias-variance joint optimization. Our main contributions can be summarized as follows:
1) We theoretically elaborate limitations of regularization techniques, and the relationship of unbiasedness, variance and generalization bound of general estimators.
2) Based on the general laws, we elaborate a novel design perspective for the estimator, namely the quantitative bias- joint optimization;
3) We develop a comprehensive dynamic learning framework with the bounded variances and generalization error to optimize a weighted objective with respect to bias and variance for each user-item pair (u, i), which dynamically selects different estimators for different user-item pair from family of estimators according to the given objective function;
4) We conduct extensive experiments to verify the theoretical results and the performance of the dynamic regularizer and estimators.
2 Preliminaries
Data missing not at random. Denote the sets of users and items as {ug, u uM} } and I = {i1, i2. ... } respectively. The set of all user-item pairs is denoted as D = U x I. Define the true and prediction matrices Y E RMM and Y MMN where prediction tasks include rating, CTR and CVR predictions, and so forth. Each element Yo., in and each entry Du. in Y are the true label and predicted output of user u to an item i. general, it impossible to observe all entries in the matrix Y. The indicator entry of revealed elements is defined as ou,i E (0, 1}. If the true label yu. is revealed, the indicator entry of (u, i) satisfies Ou,i 1. If an entry Y is missing, then ou, = 0. The corresponding indicator set is denoted O {o.,, 1}. Considering the case that no entries are missing, the prediction inaccuracy
of Y is defined as
1 M
N
Lecal (Ý, Y)
Σ eu.i (a.))
Cu.
MN
where e is the prediction error. εu,i can be selected as mean absolute error (MAE), mean square error (MSE) or other measures. The objective of prediction problems is to minimize the prediction
inaccuracy L (Y)) 7 8 I 12 [4]. Actually, only the observed label set Y can be used to establish the prediction model. The naive approach uses y to minimize the following prediction inaccuracy:
Luaive (8, yy)
|0
Σ E
Cu,
O
Σ
Gu.
(U.))D

---

As mentioned in [II], if the probability of every entry Ya.i in Y being missing is identical, then the naive estimator is unbiased. that is Bo n = Lreal where O is taken to represent the random variable of observation. The unbiased estimation property of the naive approach is no longer valid when the data is MNAR, which even results in a large difference between Treal and 18g [
Quantitative Bias- Variance Joint Optimization. Considering the large difference between Lreal and Eo Liivell various unbiased estimation methods have been developed to overcome this problem. such as EIB [10], IPS estimator [6], DR method [Л], and various variations of them [5] 15.. The corresponding estimators are given as follows:
12
LEI((YY)) Y-)
Jou,seu + (1-
D
(u.I)ED
Lis((2,
Ous Puri
Cu,it
(u.))
LnR(F,Y'
Cuga + Ou.t (ea, - Emss)]
(en, Pu.i
(u.))ED
where ēu,i w/se.; - > for MAE or εu.; w(u,, - )-² for MSE of missing entries Va, is the imputed errors, and Pu.s E 10, the estimation of the observation propensity, i.e., Pa. Pr(ou.s = 1) E (0, 1). Note that w and are papeeeparameters [10]. For the naive, EIB. IPS, and DR estimators, their biases, variances and generalization bounds are summarized in Table (see Appendix for more details), where A,,i = 1 Pu. and o,,i Cu,i é., In general, the learning of the imputation model also involves the MNAR problem. Some joint learning algorithms [1] 12 employ the propensity model to overcome this problem. Therefore, propensity estimation has a crucial role in unbiasedness and robustness. Besides, is difficult accurately estimate imputed errors for all user-item pair (u, i) in the sense that it is difficult to achieve the unbiasedness of the EIB estimator. If the propensity estimation Pu.i is accurate, that is Pu,i = Pu. then IPS and DR estimators are unbiased. For new dataset, we cannot know in advance the range of the propensities in this dataset. Therefore, new dataset may introduce extremely small propensities lead to unbounded variances of IPS and DR, which will disrupt the stability of estimators, especially for larger datasets. is unacceptable for real industrial scenarios. Specifically, the smaller the propensity, the larger the variance. When the propensity tends to zero, the variance tends to infinity (see Appendix B for more details). Similarly, variances of other IPS bas-based and DR-based unbiased estimation methods [5 are also unbounded. On the other hand, although the variances of naive and EIB methods are bounded when the prediction error é,,; is bounded, it is difficult and even impossible to achieve an unbiased estimation. Other variance reduction estimation methods 5 7 to the expressions of estimators and Table 4 the bias and variance of an estimation are determined by the random variable O. We found that slightly relaxing the requirements for unbiasedness will lead to bounded variance for all propensities. Therefore, the core problem of estimation on MNAR data is the biri-variance joint optimization.
Pu.:
are generally biased. According
3 Fine-Grained Dynamic Framework for Quantitative Biri-nariance Joint Optimization
In this section, we first discuss limitations of regularization techniques and the relationship between unbiasedness of the generalized estimator and its generalization bound, which illustrate the core of the fine lineggrineed estimator design. Then, the dynamic estimation framework for quantitative bias striinariance optimization is present. Its generalization bounds and variances are reduced and bounded with theoretical guarantees.
3.1 Limitations of Regularization Techniques
Define the general form of the estimator with regularization as
LesteReg
[[(ouss [(oussPus)ee. Dus) + g(Ou.s Pu. Dand
+\
Σ
Ii(ous. Pua)s
D
DD]
(u.))ED
(u.))D
(1)
Lxes
3

---

where f(:--) & O with f(0, Pw,,) 0, g(., -,, and h(:;-) are functions with respect to and p. Lest and Leeg are prediction maccuracies of the estimator and regularizer, respectively. For all (u, i) pairs, they satisfy f( Pu, + (P.;):. 2 0 and hhou..,Pu.) 2 O. À >> 0 is scalar weight. The generalized estimator form LEs given in Eq. (T) covers the vast majority of existing estimators involving EIB [T]. IPS [6] DR [1]. More Robust DR (MRDR) [], Targeted DR (TDR) [15], MIS [6]. IPS/DR-SV [16.. and other IPS-based and DR-based methods. On the other hand, almost all existing regularization designs, including the Sample Variance (SV) [16), mean inverse square (MIS) [6]. Balancing- Meetiing--e------- Alean-Squrre Error (BMSE) [8] and so forth, can be transformed into the form Leeg given in (I) In previous works, the regularization technique plays critical role in variance reduction of estimators and improvement of the generalization performance to a certain extent. However, it still have some inevitable limitations described in the following box.
Core Results
For the general estimator with regularization Leat+Reg is impossible to reduce variance without sacrificing unbiasedness.
2) Regularization LReg cannot guarantee a bounded variance and generalization bound.
In what follows, provide detailed theoretical analysis to reveal the aforementioned limitations of regularization technique. Considering the variance Lest we have
Lest
VolLus] + 2ACov(LEss Luzz) + X Xo LLeg)
As mentioned in [ when the parameter is set as the optimal parameter Aopt Lou(Lapa Luce) No Long! the variance V Vo achieves its minimum and satisfies Vol S Vo in the sense that the regularization term ALReg enables the estimator Lest+Reg to reduce variance. However, the covariance Cov (Lest, LReg) needs to fulfill Cov(LEst, LReg) < 0 as À > 0. Otherwise, an inappropriate parameter will result in increased variance. The formal theoretical results provided by Theorems 3.1 and Corollary 3.2 which reveal the limitation 1) (see Appendix D for proofs). Corollary 3.2 the contrapositive Theorems 3.1
Theorem 3.1. Let LeauRog be defined in 7 and the estimator LEst be unbiased. If Leas Reg is unbiased, then the variance of LEsttReg is greater than the variance of the original estimator Lgst-
Corollary 3.2. If the variance of LEauReg is less than the variance of the original estimator Lest. then Log unbiased.
We further find that, if the variance of the original estimator is unbounded, the variances of estimators cannot be bounded by introducing a regularizer even if Pu,i = Po,i- The theoretical results are shown in Theorem 3.3 see Appendix [ for proofs).
Theorem 3.3. Let the bias of LEa+Res be bounded and the variance of LEst satisfy Pu.il cc. Then, there exists no regularizer Laeg that enables the
lim V variance and generalization bound of the estimator bounded even the learned imputed errors or propensities are accurate.
According to the previous works and the present Theorem regularizers enable variance reduction to certain extent while cannot enable estimators to possess bounded variances and generalization bounds. In other words, regularization techniques have limited impact improving the predictive performance of the model. In the next subsection, a novel perspective of dynamic estimator designs proposed, which not only achieves quantitative bias-variance joint optimization but also guarantees bounded variances and generalization bounds.
3.2 Dynamic Estimator Designs With Quantitative Optimization
Most of the existing estimators are based on IPS and DR methods, which are elaborately designed to reduce bias or variance. However, all these estimators are static estimators in the sense that they cannot achieve bias-variance joint optimization for each user-item pair (u, i). Even though some methods [5, [ can effectively reduce the variance of estimators, the estimators are biased and the corresponding variances are unbounded. this subsection, the core results are provided in the following box. Also, based on theses results, develop a fine-grained dynamic framework with

---

Figure I: The surfaces of determining factors and the objective function of dynamic estimators, and the optimal objective values: (a) g (b) h h (c) wi (2 + waht", (d) Objective.gt
quantitative optimization to guarantee the reduction and boundedness of variances and generalization bounds
Core Results
3) For the generalized estimator LEa. unbiasedness of the estimator will inevitably lead to the unbounded variance and generalization bound.
4) The core of the estimator design involves not merely simple bias- trade-off, but rather quantitative joint optimization of both bias and variance.
We find that the unbiased estimators with general form generally possess unbounded variances, which formally derived in Theorem 3.4, Its proofs are provided Appendix
Theorem 3.4 (Limitation of Static Estimator). Given prediction errors eu, imputed errors éu, and learned propensities for all user-item pairs (u,i), if for any Cu,i g(0, pu, * 0, LEst given in is unbiased, then the corresponding variance and generalization bound are unbounded.
According Theorem 3.4 the core objective of estimators is merely about eliminating bias, reducing variance, or simply achieving bias-variance trade-off but about quantitative joint op- timization between bias and variance. Therefore, as mentioned in Core Results, is necessary to develop a dynamic estimation framework to achieve the quantitative joint optimization.
Design Principle of Dynamic Estimators. works are designed as
The IPS-hased and DR-based dynamic learning frame-
LIaPS
1 [D]
Σ (u,))D
Ou,i foun (fu.s)
Cu.i, LINDR
D
Σ
ēu,
Ou,i frun (fu.s) Bus).
(2)
(u.))E
where f(-) is designed function and au,i [0. is optimizable parameters. When f( Pu. and va.,; = 1, D-IPS and D-DR are equivalent to the original IPS and DR estimators, respectively, which possess unbiasedness. When ((.,,) uu,; and Va,,, O, D-IPS and D-DR are equivalent to D Lnaive and EIB estimators, which have bounded variances and generalization bounds. The function (Pu..) in (2) actually mapping, which balances the bias and variance of estimators. The design principles of (((u..) are provided as follows:
(Isotonic Propensity) f(Du.s) with f(0) O, /(1) = 1, and l(Pr.s) > Pu, a monotoni- cally increasing function.
(Same Order) Pu. f(c.))
lim Pu.i
C, where C' is positive constant.
Some specific expressions of f(p..)) fulfilling the above design principles are summarized in Table I The corresponding biases, variances and tail bounds of D-IPS and D-DR estimators formally Tormulated in Lemmas D.I given in Appendix D From the biases and variances of the D-IPŠ and D-DR methods given as
Bias( Inaps)
Σ [u.i)E
h (p..i, Pu,i, auslens Bias(LnoR))
Σ (u.ileD
Bot (pn. (pu,isPanis Payss Ou,
[D]
[D]
Vo Lo(Laa
Σ
Bu (pu.s, Pu.si Jew.s Vo LoDKR]
1 (Du..s Pu.ty Cia )
[D]2
DDI
(R.))ED
(u.))e
5

---

Table 1: The specific expressions of fmu. (Pu.i) and their determining factors the bias and variance.
((,,,)
Puçi
sin(fa.s) sin(I)
log(pu,;+1) [og(2]
tanh(pe..) tann(ly
hB
Puud Pui
Pu.s sin (1)
Pu. log (2)
Pu.i tanh (1)
sim (fu.))
Tog" (Ba.+1)
tanh tanh"s:"(Pu.. (Pu.a)
hv
Pu.s (1 Puaa)
Pu.ill (1-11-1000 Pu.i) sin sinn (1)
Pa Pa..) 1'g 20 (2)
Pa.i (ill--pp Pa. tanh 2 (1)
sin 2 (Pu..)
log u0 (+),
tanh "un (Pu.s)
Pu,
where 13 (Pu,is Pu,ii an.i)
(Pu.) and (Du,i: Pu,i, On.)) Pu.il--pul functions h
faau. (Pu.e)
and hi determine the biases and variance, respectively. E and hi" corresponding the specific expressions of f(uu,) are given in Table The monotonicity of bias and variance provided in Appendix Proposiiion D.3 The surfaces of h and h5 are plotted in Figs. [a) and (b). It is observed that / is monotonically decreasing and h is monotonically increasing as the number of Qu,i increases.
Bias- Variance Quantitative Joint Optimization. According to Proposition D.3 given in Ap- pendix D the variance trade-off problem can be quantitatively formalized the tollowing joint optimization problem:
Objective = min (w0 + + (2VolL(au))) s.t. S and $1,
(3)
and
where wi and W2 are weights of the bias and variance.
According to determine factors h and h of bias and variance, respectively, the bias variinere joint optimization problem can be defined as
(ou.))) Objective min { ue (ig Edl () + wy FygFy (aur))))) s.t. 0 S aus S au.t
(4)
For each user-item pair (u, i), minimizing En (h())) and Ev (hb") given accurate propensity es- timations Pu.i leads to the bias and variance reduction, respectively. Therefore, the optimal pa- rameter au,i J0, 1] for each user-item pair (u, i) can achieve fine-graines bias-variance joint optimization. The function h under Ou,i E [0, 1). [(Pu.) 2 Pu.a and Pu,i = Pu,i satisfies Ess (Pu.iPPu. au.)) E [0, 1). On the other hand, he" under Qu,i E [0, 1] and Pu,i = Pa.i satis- fies ha (Pu, Pa,i; il E 20). Therefore, the objective function in (4) can be simplified as wi wr Est (aw,s) + Wyhh Est (..,). The curves of objective functions under different designed functions f(.) are given in Fig. Tc). It can be observed that for fixed propensity, there exists such that the objective function attains the minimum value. Besides, different measure metrics are also applicable In (h (aw,)))) and Est the analytical solution of the optimal parameter ag is given in Theorem 3.5 (see Appendix D for proofs).
for dynamic estimators, such as F((h = (1) (ou,,)) F. (o..s))
so on. In what follows, under the objective function h/ + wohh".
egt
Theorem 3.5
(The optimal parameter ).,)) Let the learned propensities be accurate, i.e., Pu,; = Pa,s- For weights wh and W2, the objective function w1 wyh + w22ks' under Cuui E [0, 1] achieves its minimum
In
200y
W1
Pu.a))
.0.1)
(5)
min
max
(n(f(pu.))
From the expression of the optimal parameter (5) the optimal solution of (4) under different weights depends on the weight ratio w2/ w/ Under different designed function f()), the schematic diagram of optimal objective values corresponding to the optimal parameter ox shown Fig. d.. Next, the generalization bounds of the developed dynamic estimator framework are further discussed. The formalized results are derived in Theorem 3.6 (see Appendix [D] for more details).
Theorem 3.6 (Generalization Bounds of D-IPS and D-DR). For any finite hypothesis space H of for Y and the optimal prediction matrix y- given eu,i and pu,; for all (u, E ', with probability 1 - p. the prediction inaccuracies LD-Ps(L- ,Y) and LD-DR - YY) under D-IPS and D-DR have

---

the following upper bounds
B (Pu,t, Pais, Oue a Pual
Est
Limes(Y ,ro+
Ri (ctt)):
Bar
[D]
(u,))
Vartance Term
Torm
1 (Pu,ts is Pu. Ore, . Dosl
LD.DR(Y ,
Σ
(8 NG (6,,s)
[D]
(w.))D
Varianer
Bus Term
where oni and Jū are the error and error deviation corresponding to y+ arg maxyeh { E(mLiEDD (Trattp)))) ) and Y+ arg maxvEH E(u,iEED
(Tais(por))
respectively, and the function i is formulated as h
logg 213d
Σ E(un)ED 2/D
F Tann (Bu.))
From Theorem 3.6, the bias-variance joint optimization is actually to mínimize generalization bounds, which include both the bias term and the variance term. Besides, the dynamic estimators with the optimal parameter opt make variances and generalization bounds bounded. The formal result is given in Theorem 3.7 (TTh corresponding proofs and bounds of variances are given in Appendix D)
Theorem 3.7 (Boundedness of Variance and Generalization Bound). Let a E [0, 1] be the optimal parameter of ( If the dynamic estimators adopt opt as the parameter, then the corresponding variance and generalization bounds are bounded.
4 Experiments
In this section, conduct extensive experiments to compare the performance of the present dynamic learning framework with existing SOTA approaches and to answer the following questions: Q1: Does the developed dynamic learning framework improve the prediction performance compared with the SOTA approaches? Q2: Do the present dynamic estimator designs reduce the variance and make performance more stable compared with the SOTA approaches? Q3: How do the performance and variance of the proposed method change under different optimization weights and estimator functions?
4.1 Experimental Setup
Dataset and Preprocessing. Three real-world datasets with MNAR and MAR samples are used conduct the experiments, namely COAT with 4,640 MAR and 6,960 MNAR ratings of 290 users to 300 coats, YAНОO!! R3 with 54,000 MAR and 1,04 MNAR ratings of 15,400 users to 1,000 songs, and KUAIREC with 4,676,570 video watching ratio records of 1,411 users 3,327 video. Similar to literature [8] 5), the rating scores in COAT and YAHOO! R3 are binarized as when it is greater than three, otherwise as O. For the KUAIREC dataset, the video watching ratios are binarized as O when it is less than two, otherwise as 1.
To avoid the uncertainty caused by the prediction and observe the performance of the prediction model, we take the matrix factorization (MF) (17) as the base model
Baselines and Experimental Details.
and compare the present dynamic learning framework with the following representative IPS-based and DR- BR-based approaches: naive MF [17. IPS (6), SNIPS (13) IPS-AT (18), CVIB [19. IPS- V2 [8) DR [T]. DR-JL M. MRDR-JL (S) Stable DR 121. Stable MRDR [21. TDR-CL [15]. TMRDR-CL 15). DR-V2 M. Here, we adopt the two common metrics used in recommender system, i.e., area under the ROC curve (AUC), and normalized discounted cumulative gain (NDCG), to evaluate the performance of prediction models. To guarantee the fair comparison, we set the same parameters for approaches. The learning rates are tuned in {0.001, 0.005, 0.01, 0.05) and weight decay is tuned in (1, le 1, le 2, le 3, le 4, le 5, le 6). Note that, for XX and D-XX approaches, their model structures and parameters are identical. Every approach preformed 10 times to record its mean and standard deviation.
7

---

4.2 Performance Comparison (for Q1 and Q2)
The MNAR records in datasets are used to train the prediction model and the MAR data is employed to
evaluate the present dynamic learning approaches and the existing SOTA approaches. The function in
proposed D-XX approaches is select (log(f..++1)) The performance of other functions provided (2)
in Table are discussed subsection 4.3 The performances of the developed dynamic estimators
and the SOTA approaches are shown in Table 2 where Gainauc = (AUCxx AUCD xx) AUCCxx
and Gainy = (NDCGxx- NDCGD xx) /NDDCxxxx e.g. Gain = (AUCip - AUCp IPS AUCips Gaing (NDCG1Ps NDCGD-IPs) / The weights in bias-variance joint optimization are set as w1 = 1 and W2 = 0.1. For almost all of metrics and datasets, the performances of IPS, IPS-AT, CVIB, IPS- DR, DR-JL, MRDR, DR-V2 outperform the naive method while the naive approach has smaller variance, which implies that unbiased estimators possess high variance. Besides, it can be observed that estimators with the dynamic learning mechanism greatly improve the performances and reduce the variances of various debiased approaches. such as IPS and D-IPS, DR and D-DR, DR-JL and D-DR-JL. Meanwhile, for SNIPS, MRDR-JL, the variance reduction of dynamic estimators do not seem obvious. One possible reason is that these approaches themselves can effectively reduce the variances of estimators by sacrificing unbiasedness. These experiment results further verify Theorem
3.4
Table 2: Performances of the proposed method and baselines (mean + standard deviation across 10 runs).
Coul
Yihou! R3
KuaRe:
Meladd
Guitucc
NDCG65
Geitare
NDCG65
ATC
NDCG0 50
IRANN
0.7421211 D8NG
0061115100011
0:469:5500000
0725622010012
056344100025
SNIPS
8 1342510 B4185
0A110 -00955
B.5127 (10005
0 000011000010
D/4015-
033061 (4.9977
17%-AT
0 07770 -C1953
RASTT Iugon2
107301-01010
D7715 (60061
CVVI
0 00/181
RATT9 IUD0ON
07237
1999
0,55562100022
0.434:000022
0:777:600016
0.7015:0
D-EPS (Ours)
0.777720 2015
3.16%
0 065544500047
1.5%%
0,57672000044
2.16%
0.663223.00 2M27
0.71%
07544:88.045
8.65%
0.7576220100 (009)
5.7
D- SNSES (hurs)
0.742944 9006
0.00
0.0030220003
822818-500012
4.55%
0.0995521002
3.70%
B3826-500017
0.137
0.004200.000
0.247%
TRR
0 07556-0 0052
0 6.655-00 0996
0 M133
0-701-60055
07518:00 0025
TRRII IT.
05774-0 R0NG
61201-000015
07806-000034
0
Statie DE
EL L.1502251.00 2013
05925
B28025500007
0.7026:0 1899)
Stakk MHUR
008442200013
0.7752201022
TMRRR-C
TO00-- 0916
04961-000011
0-5505--00006
O 08877-422199
07801-00001
05507-44 (0)
TIR.37
8 6.77/44 D004
D 06675-000997
C 0061310/0154
07879-00003
0.7973-0 (875)
D- (Durs)
0.75042111
0150902200002
158%
0.204322/042
4.55%
0.7997=221017
D.MIDR-JL (Durs)
0L7786200 DKGS
2.58%
0.0016:0:044
175%%
0.9911200027
0.96%
0.405%
029/8200051
2.3755
0.80052000000
AL.19%5
4.3 Ablation Studies (for Q3)
Effects of Different Weights and Functions in Dynamic Estimators. In subsection 3.2
provide four specific dynamic estimators given in Table [ We set the weights in bias-variance joint optimization (4) as w1 and Wi J0.02, 0.04, 0.06, 0.08, 1] for these four dynamic estimators to investigate the effects of weights on the performances and variances. From Eq. (5), the optimal parameter Cop is determined by the ratio wh and Coop determines the objective function. Therefore, we just focus on the effects of the weight ratios the performance and variances of estimators, which are given in Fig. [ It can be observed that, for D-IPS, D-IPS-AT, D-DR, D-DR-JI, and D-MRDR-I. -JI approaches, the performances increased at first and then decreased as the number of the weight ratio increases. Mcanwhile, the variances seem to achieve their minimums when the performances achieve their highest values. Since the smaller the weight ratio is, the smaller the bias of the dynamic estimator is, the experimental results given in Fig. reveal that the unbiasedness of estimators is not exactly equivalent the performances of estimators. Actually, from the generalization bounds given in Theorem 3.6 the bias- varianee joint optimization enable estimators to minimize the generalization
wi
Bo
66
Figure 2: Effects of different weight ratios functions (Pu,s)) (a) p (b) (si((p..)) sin(l)
wa on performances of dynamic estimators under different my
[oog(P.. +t (d) (tanh(f..)) tanh(1)
(c)
1og(2)
8

---

Table 3: Effects of different functions on performances of dynamic estimators.
siat9...
Mothods
AUC
Gainauc
NDCG@5
Gainy
AUC
Gainauc
NDCG@5
Gainy
D-IPS
0.7702:-000011
2.16%
0.6362:10.0043
-2.06%
0.7753+4.0017
2.84%
0.6475+0.0000
-0.32%
D DISNIPS
0.7413+-0.0045
0.13%
4.6146+000007
0.59%
0.73921-0.0038
-0.42%
116109-+0.0089
+0.02%
D-IPS-AT
0.7711+0000066
0.25%
16360+0000011
1.11%
0.7710++0.0022
0.23%
0.6346++0.005
0.89%
D-DR
0.7010::000114
2.28%
0.6384:1.00447
-0.64%
0.7763+400021
2.98%
0.6516:0.000:
1.42%
D-DR-JL
0.0093:00.0013
1.60%
0.6346:10.000
-2.31%
0.7748:000012
2.30%
0.6444
-0.80%
D.MRDR-II JL
0.701110.0016
1.59%
0.000
-2.11%
0.7751
2.12%
0.6470
0.0033
0.49%
(08(2)
Tombic))
Methods
AUC
Gainaue
NDCG@5
Gaina
AUC
Gainaue
NDCG@5
Gainy
D-IPS
0.7777=0.0015
3.16%
0.6884=0.0044
1.35%
0.7771=0000016
3.08%
0.6578=+0.000
1.26%
D-SNIPS
0.7429::00003
0.08%
0.6096:10.0062
-0.23%
0.7418:1..0000
-0.07%
0.6115=0000882
0.08%
D-IPS-AT
0..001.10.0012
0.17%
0.6367 10.00052
1.22%
0.771811.00111
0.34%
0.635710.0029
1.07%
D-DR
0.7804 0023
3.53%
0.6671:10.005 0001
3.83%
0.7792+000019
3,37%
0.6608:10:008
2.85%
D-DR-JL
0.7775:10.0016
2.65%
0.6677
1.25%
0.7782.10.0011
2.75%
0.6537:10.0039
0.63%
D D.MRDR-J. JL.
(.777661-0.0022
2.58%
0.6616-10.0044
1.75%
00.0071
2.49%
0.6576
1.14%
bounds and then further improve the generalization performance. Mcanwhile, we find that the variances of dynamic estimators is not decreasing when the weight ratio increases. This because, for different ratios, the global minimum of the objective function 4 cannot be reached within the interval a E [o, 1]. For SNIPS, the property of variance reduction might lead to the non-obvious performance and variance trends.
Under the identical weight ratio wi 0.1, we further discuss the effects of different functions fo (f(..)) on the prediction performance and variance. The experimental results are shown in Fig. 3. Nearly all dynamic estimators with different function expressions outperform the corresponding debiased approaches given Table further demonstrates that the proposed dynamic learning mechanism can greatly improve the performance of the original estimator. Besides, the prediction
performance of the dynamic estimator with (fu,)) = (l1((p..++))
outperforms other dynamic log(2)
estimators.
5
Related Work
Aiming at the prediction model bias caused by the MNAR data, EIB [10 and IPS (6] approaches are two classical unbiased estimators. To leverage the advantages of EIB and IPS, the DR method (TT) was designed make the unbiasedness of estimator doubly robust. Focusing on the unbiasedness of estimators, various estimation methods have been proposed to overcome mixed or even unknown biases in the data [9], to solve the sample selection bias problem in the multi-task learning 20 21 22 23 to improve the performance of the propensity model by different approaches [18 14],
and forth. A multiple robust estimator is developed in (4 by taking the advantage of multiple candidate imputation and propensity models, which is unbiased when any of the imputation or propensity models, or linear combination of these models is accurate. From novel function balancing perspective, Li et al. propose to approximate the balancing functions in reproducing kernel Hilbert space (25). Moreover, aimed at limitations of miscalibrated imputation and propensity models, Kweon and Yu 26 propose a doubly calibrated estimator and a tri-level joint learning framework to simultancously optimize calibration experts alongside prediction and imputation models. For the variance of estimators, an increasing body of works have emerged to reduce the variance. The most common estimator reducing variance is Self-NNormaliee IPS (SNIPS) [. Based on DR, literature S designed MRDR estimator to reduce the variance of the DR estimator by the present variance expression of DR. In [5), TDR estimator is elaborated reduce the bias and variance of DR simultaneously by the present cerimeeraett collaborative learning. Moreover, stable DR estimator
[2] achieves the bounded bias, variance, and generalization error bound simultaneously for arbitrarily small propensities by combining SNIPS and DR methods. Various regularization designs, such as SV [16) MIS [66], BMSE g and so forth, are also introduced into the estimator to achieve variance reduction.

---

6 Conclusions
To the best of our knowledge, this is the first work to reveal that the essence of estimator designs not merely to eliminate bias, to reduce variance, or to achieve a simple bias-variance trade-off but to quantitatively and simultaneously optimize bias and variance. Besides, the limitations of general regularization techniques and general static estimators are presented. Based on the general laws with respect to the relationship between bias and variance, we propose a systematic dynamic learning framework, which guarantees the bounded variances and generalization bounds by the present fine- sneeggained bias-variance joint optimization scheme. Extensive experiment results have verified the theoretical results and the performance of the present dynamic estimators. The search for optimal weights in the objective function and the functions in the dynamic estimation framework remains an open question.
References
[1] Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. In Proceed- ings of the twenty-first international conference on Machine learning, page 1 114, 2004.
[2] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts, In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930-1939, 2018
[3] Xiao Ma. Liqin Zhao, Guan Huang. Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1137-1140, 2018.
[4] Dongbo Xi. Zhen Chen, Peng Yan, Yinger Zhang, Yongchun Zhu, Fuzhen Zhuang. and Yu Chen. Modeling the sequential dependence among audience multi-step conversions with multi-task learning in targeted display advertising. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3745-3755 2021.
[5] Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development Information Retrieval, pages 275-284, 2021.
[6] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. international conference on machine learning, pages 1670-1679. PMLR, 2016.
[7] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Rui Zhang, and Jie Sun. generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 252-262, 2022.
[8] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In International Conference on Machine Learning, pages 20182-20194 PMLR, 2023.
[9] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. Autodebias: Learning to debias for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21-30, 2021.
[10] Harald Steck. Training and testing of recommender systems data missing not at random. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 713-722, 2010.
[11] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In International Conference on Machine Learning, pages PMLR, 2019.
10

---

[12] Haoxuan Li, Chunyuan Zheng. Xiao-Hur Zhou, and Peng Wu. Stabilized doubly robust learning for recommendation on data missing not at random. In Proceedings of the 11th International Conference Learning Representations, 2023.
[13] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems, 28, 2015.
[14] Wei Ma and George H Chen. Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under low nuclear norm assumption. Advances in neural information processing systems, 32, 2019.
[15] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. Tdr-cl: Targeted doubly robust collaborative learning for debiased recommendations. In Proceedings of the 11th International Conference on Learning Representations, 2023.
[16] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recom- mender systems with a few unbiased ratings. In Proceedings of the 14th ACM International Conference Web Search and Data Mining, pages 427-435, 2021.
[17] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[18] Yuta Saito. Asymmetric tri-training for debiasing missing-not-at-randde explicit feedback. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, pages 309-318, 2020.
[19] Zifeng Wang. Xi Chen, Rui Wen, Shao-Lun Huang, Ercan Kuruoglu, and Yefeng Zheng. Information theoretic counterfactual learning from missing-not-at-raann not-at-random feedback. Advances in Neural Information Processing Systems, 33:1854-1864, 2020.
[20] Wenhao Zhang, Wentian Bao, Yano Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In Proceedings of The Web Conference 2020, pages 2775-2755 7771-2781, 2020.
[21] Wujiang Xu, Qitian Wu, Runzhong Wang, Mingming Ha, Qiongxu Ma, Linxun Chen, Bing Han, and Junchi Yan. Rethinking cross-domain sequential recommendation under open-world assumptions. arXiv preprint arXiv:231 2..::331144. 11,0490, 2023.
[22] Wujiang Xu, Shaoshuai Li, Mingming Ha, Xiaobo Guo, Qiongxu Ma, Xiaolei Liu, Linxun Chen, and Zhenfeng Zhu. Neural node matching for multi-target cross domain recommendation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages $154-2166 IEEE, 2023.
[23] Wujiang Xu, Xuying Ning, Wenfang Lin, Mingming Ha, Qiongxu Ma, Linxun Chen, Bing Han, and Minnan Luo. Towards open-world cross-domain sequential recommendation: A model-agnostic contrastive denoising approach. arXiv preprint 2xXXi:2331.04700 2023.
[24] Haoxuan Li, Quanyu Dai, Yuru Li. Yan Lyu, Zhenhua Dong. Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation, In In Proceedings of the Thirry-Seventh AAAI Conference Artificial Intelligence, 2023.
[25] Haoxuan Li, Chunyuan Zheng, Yanghao Xiao, Peng Wu, Zhi Geng, Xu Chen, and Peng Cui. Debiased collaborative filtering with kernel-based causal balancing. In Proceedings of the 12th International Conference on Learning Representations, 2024.
[26] Wonbin Kweon and Hwanjo Yu. Doubly calibrated estimator for recommendation on data missing not at random. In In Proceedings of the ACM Web Conference 2024 (WWW '24), page 38 3810-3820, 2024.
11

---

A Derivation of bias and variance for naive, EIB, IPS, and DR estimators.
As mentioned in literature the bias of estimator is defined as
Bias L) Leaa Eo [4]]
(6)
According to the prediction inaccuracy expressions of Luaive and the definition of bias 66 naive estimator satisfies
the bias of
Bias [mniee
Cu.i |D|Eo =
[D]
Σ
D Pu.s
(7)
(W,))ED
(u,))
According the definition of variance for an estimation given in (5), the variance of the naive approach can be formulated as
V Vol Lmaive =Bo Bolvvive
Eo
[Lmiie]]
Bo
Σ Ou.itu,t
Σ (u.1)ED
Pu.:Cui
1012
1012
(u.))ED
Σ
Pa.i(1 - p),
O0
(u.)eD
The biases of EIB, IPS and DR methods have been given in Lemma 3.1 of (I]. The variance For the variance EIB, we obtain
formulations of IPS and DR estimators have been provided [.
Vo[LE =Eo [Tae]
[
[on, oun)eu.d) )
Bo
Σ
Pu, + (1 Pe.s) (u,i)ED
Σ
DDI2
1D12
(u.))D
Σ
Pu, Pu, :en, - En,s)
2
(M
Therefore, the bias and variance of naive, EIB, IPS, and DR estimators shown in Table 4 can be obtained.
B Unbounded Variance
According the definitions of variances of IPS and SR methods, if pu,i = Paçis when the propensity tends to zero, the variace IPS and DR satisfy
lim VolLus pu.. Pa.s] Pu.
lim Pu,
Σ
Pui e 20:
DD2
Pu,t
(u.))ED
lim Vol Prn. = Pais
lim 00
Σ
1 1-Pul
Pusk 82 00.
Pu,
D
Puçi
(u,
This demonstrates that the variances of IPS and DR are unbounded even the propensitics are accurate. If there exists propensity going zero then the variance tends to infinity
C Limitations of Regularization Techniques and Static Estimators
Theorem 3.1. Let LEa+Rey be defined in (1) and the estimator Les be unbiased, If LEatRey is unbiased, then the variance of ListReg greater than the one of the original estimator LEst-
Table 4: Bias and variance of naive, EIB, IPS, and DR estimators.
Model
naive
EIB
IPS
DR
Bias
J > (1 - Pus)ea.
Σ (1 - Padlu
(wojez
InsieD
(wojez
Variance
j012
Σ Pai(1 Poilets
Σ Pu.il Pas)
po
Σ
muil
Σ
R.
puil 62
(walen
12

---

Proof. The variance of Lest+Reg satisfies
Vo Vo Vo [Lx]] + 2ACov Laag) + 222 (LRoe)
To reduce the variance of Lest Vol Vo Regg satisfies Vol VolT-Bs-Rey S Vol [TEa], which implies that
2ACov (Lro, Laog) + X Vo [RR I O.
(8)
Therefore, the parameter À satisfies S
2Cov( 2Cov( FRa) and the optimal parameter is Aopt
Vol [oal
Cou(Lea lasssl On the other hand, since Lest and Rest are unbiased, we obtain Eo[LReg = 0. Then Cov( LESt Loeg) satisfies
Vol Voll
Cov(Les; IRRR)
=Bo Leu LRuz)
Eo (LBx)) (IRs))
=Bo LES
(9)
(LE (slo Dusseen + g(ou,is Pan. Pun)) (WIN)D
][
Eo
Σ
h(on, (uni)
DDF
(D.1)ED
To facilitate representation, f(ou,d Pu, g(ou, i Pu,, e is denoted as r(ou, Purdi Cu,s Suss) which satisfies r(Ou,ss Perke Ca,is e 2 O. Then the equation can be rewritten as
[
r(ou,s Perss (uuus:Purss Budt Bun Cani) ][
CoveLEEa Lneg)
Eo
Σ (uN))D
Σ
M(ows Muos)))
DDp
(u.J)D
1
Eo
Σ
r(ound Puris Pus (uns))
|Di2
(u.))ED
(u.4)ED
[D] [Pl
Eo
Mo, P,) (ox, (e ex)
|D|2
D [P]
D
Let us consider the term Bo Bolh(o,,Py)) p Ph, ck. 2x) in 10 which fulfills
Eo Bolh(o,.P (r)k( Pk, Ck2 2x) 2 0. Therefore, we obtain Cov( (Est, Leeg]
Eo(LEs 2
Corollary 3.2. If the variance of LEstRey is less than the variance of the original estimator Liss then LeatReg is not unbiased.
Proof. We use the method of proof by contradiction. Assume that when VolLEss S Vol WLLss
TaatReg is unbiased. According to the definition of Vo [TratReg] and Vol [TaatReg] < Vo [TE). we have
Vol -Vol + 2XCov(Lrss LRee) + X oo <VolLBul,
(10)
which implies that 2XCov(LEas LReg) + \oVo|LReg) 5 0. Therefore, the parameter À needs to
satisfy 0 A <
22oo(( IR.. Since Vol [Reg]] and X 2 O, Cov(LEa, LRey) & 0 needs to be
Vol[sa)
satisfied. On the other hand, as shown in A1, when TEattRe is unbiased, we have Cov (TrTTstt: IeReg)
Eo (Reg)) 2 0, which contradicts the condition Cov (Teaa TRegy) S O. Therefore, Corollary 3.2 holds.
L
Theorem 3.3.
Let the bias of Las=Reg be bounded and the variance of Les satisfy limp...->0 Vol (ooTns Po.i) co. Then, there exists regularizer LRey that enables the variance and generalization bound of the estimator bounded even the learned imputed errors or propensities are accurate.
13

---

Proof. According to the definition of variance, Vol satisfies
Vo BastReg
=Ioe ALRey) ) L MLeez)
(11)
=Bo(LEa + > 22 Lncpl - E (La + XLR
Since the bias of the estimator Res is bounded and limp..: E (Lea + ALReg] is also bounded, i.c. [ + ALRegl - B. Eq.
V L Pa,s Pu.il = oc,
satisfies
VoILESHREs]
=B0 [Lel + X2 [Bo [edg 21Bo L Ree TBo [Tal + To [eg 2AR Lea TReel
E Les + ALRez]
B²
which implies that limp Vo Lest Pu.. Pu.1] = c0.
Theorem 3.4. (Limitation of Static Estimator). Given prediction errors eu, imputed errors é,, and learned propensities Pu,; for all user-item pairs (u, i), for any éx,i - g(0, Pu,i)eu.i ((O,pu,))&, in is unbiased, then the corresponding variance and generalization bound unbounded.
& O, Lest given
Proof. According the formulation of the estimator ① and f(0, Pa.a) = 0, its bias is given as
(L)
Σ
Casi
[o[L]
D
(u.))ED
(12)
Σ
(1 - f(I, (us.).)) (g(L, Pu.s)o. + 9(0, P,. - p.. P..s))eurs)
D
(u.))ED
From 12
it can be observed that the unbiasedness of the estimator L implies that
|1- f(,, Pa.s) - (1( + pw.))) - pu.a) bu, O,
which is equivalent to
f(1, (u: [g(1, pu.))... 9(0, (..)).,)) pu, Cu. Cuii
(13)
and
fl,,ea)eeu. + 9(1, pu.)) Cu, g(0, (,, Duls)enct
(14)
Pu.i
Let us consider the variance of the estimator L. It satisfies
Vol [
D Bo
(/(onssPaa)eas + g(o Pus)) (M.))DD
Σ
(lssees) + (o(1, Puni)P + g(0, Pn. pun))
DD
Σ (UM)ED
Bo [ [f + g( Pn )en:
(15)
D
((:L.p.))Penee.. + [9(1, PuceDu.. + g(0, Pu. pu. Pend)
Σ Eo [S(oan + g(o plonss ) eii
because of 13
2
(u.))ED
Let us focus on the fist term in 15
Denote satisñis
flou,ts Pu.seeu + g(ou, Pu. as
r(Ou,s: Pa,is Cafis R. Then, the first term in 15
1 Bo (
Σ
r(ou,s, Cus,, ew.1)
1
Eo (3, Ex)
D
1m1
(16)
D
Eo ( (o..
because of Pmie, eme,enp
2
Σ
Eo
(fous,, + tll ((
(a.))DD
14

---

According to 15 and (16), the variance of the estimator L satisfies
[
Σ
Bo (J + g(ou.s Pu.s)eu.s))
D
(wN))D
[1(1, ses + Bu, + g(0, Pu. Puli)) Su.i) 12]
Bo [Jooont:Paweea + glou.s Pus) "
2
(17)
Dusdens + g(L, Pu..) Pu,t + (o. / pu.s) Eus
2
(u.))ED
1 Puit [ens - g(0, Pr., eu..s]
2
Pus
(R.))D
Therefore, when the propensity tends to zero, for any Ca,: pu, Su, * O, the limit of the variance satisfies Io Vo[L] = oc. Since the generalization bound contains the bias and variance terms, the generalization bound is also unbounded when the propensity tends to zero. The proof is completed.
D Proofs of Properties for Fine-Grained Dynamic Estimators
Lemma D.1 (Bias of D-IPS and D-DR). Given prediction errors Cu,i. imputed errors ēu,s and learned propensities Pu, for all user-item pairs (u, i), the biases of the D-IPS and D-DR methods are given as
Σ
(fu,i Pu, a) Biax(
Σ (u.))ED
1 (Pr, Pu.is 0r)ou,
D
D
(u.i)ED
(18)
where the function hg satisfies h (Pu,s Pu,is a:) =1
Po.f f (Pu.i)"
Proof. According the definition of bias 66). the biases of D-IPS and D-DR are formulated as
Bias(LD.iP))
D
Cu.i
Eo(LDIN)
D
Σ
(m(P))))
Pai
eu.
(4.)))
(UD))ED
(19)
Bias(Ln.R))
Σ
Cn,i Eol Lnnn
Σ
Puss
Bus
D
D
(a(
(u,1)E
(u.i)ED
Lemma D.2 (Variance of D-IPS and D-DR), Given Cu,o Cu,i, and Pu,i for all (u, E D, the variances of the D-IPS and D-DR methods are given
VolLous
Σ
( Pu,i Pu..,,Pu.,, aaee, VoLLD.R)
hV (Pu,i, Pa a)
D
(u.I)ED
(u.1)E
h where the function hy satisfies h (Pu,s, Pu,is a+) Pe. (Dn..)
Proof. Considering the definition of the variance, we obtain the variances of D-IPS and D-DR as Vo VolLDArS = Eo LöArsl
E3 LD.Ps)
Eo
Σ
pa(p)))) Cmis 91
Oud
Σ
Pu.s [a(por)
D
DD
(u.I)ED
(a.1)D
Σ
Pu,i(1 - Pans)
DDI2
p (fu.s)
enii
(u.))D
15

---

and
Vo LLDDD
=Eo [.. [o[Lem]
1
Bo
((
Σ
Ou,
Suit
)
21
1
Σ
Piit
Sun)
|D/2
for
(u.I)ED
(u,))
Pulel Perie
DF
l (Du.s)
(w.))D
Proposition D.3 (Monotonicity of Bias and Variance). For IPS-based and DR-based dynamic learning frameworks, and given En, Eu furifor all (u,i) i) E D, if ( 2 Pus and the parameter E [0, 1] increasing, then biases of D-IPS and D-DR are monotonically decreasing and their variances are monotonically increasing when learned propensities are accurate.
Proof. According the determining functions hB (Pu.i: Pu.cs a)) and hy (Pu,is (Puii Ok) in the bias and variance, the first derivative of hg and hy versus are derived as
Thg (P Pu.s
a) Pu,s Inf (fs.)) Öhy
2p Pus))ln(( (n((()
(20)
do
(
da
(2n()
For all Pugs E 1), we have In(s(fu))) < O. Therefore, Oh (bp.Pei.Puu..a
0 and
8ax D0 > result in the monotonically decreasing function hB and the monotonically increasing by for all user-item pairs (u, i) the number of E [0, 1] increases. Note the function (fu,s) satisfies [(Pu.s) 2 Pu,i, which implies that h 2 and liv when learned propensities are accurate. Therefore, biases of D-IPS and D-DR are monotonically decreasing and their variances are monotonically increasing when learned propensities are accurate.
Shy (Po
Lemma D.4 (Tail Bounds of D-IPS and D-DR). Given éu and Pu, for all (u. E D. for any prediction results, with probability 1 p. the deviation of D-IPS and D-DR estimators from their expectations satisfy
log(2) 2|D
Lours - Bo LD
Σ
Cu,i (pu.s)
(a.j)D
(21)
LAnaR Eol EO(LoNDR
log(
Σ
Su,t (a
2
(a,i))
Proof. Let XD-PPS and XD-DR be new random variables, which are defined as X and XD-DR êu,s + (a Su,s, respectively. Considering the independent observation indicators (ou,sl(u, D}, random variables (XD-IP i) E D) and {XDDDDR (u, i) D} independent of each other. Then, the probability distributions of X and XDDDRR can be obtained as follows: u,²
( Cu,i
Ou.
u,²
D.I-PS
Eu,i Pa,si Pr(x IMPS 0) Puçit
D.DR en,t + (mr(u..))
du,i Pass Pr(X DDRR
En,s) = 1 Pu.i
According to the Hoeffding's inequality, for any O, we have the following inequality
Pr Σ X Eo
Σ (u.))D
Xnni S 2 exp
2
(p 2n,z)
-2
(22)
(u.))D
(wi))D
where g( Ca,s) (Po.s))
for D-IPS and g(Pu.tr Swii)
Bu (P....))
for D-DR. Let ?
DI-
Therefore, (22 can be rewritten as
-2(7/D1)2
Pr *r([Toure Bo[Tnars] 2 27) s 2exp
Σ
g Bm.i)
(23)
Let Pr PrI((T(orai IDAPS Eo LD.
27) = According the inequality 23). the errors y for D-IPS
and D-DR can be solved as in 21
16

---

Theorem 3.5. (The optimal parameter or,)). Let the learned propensities be accurate, i.e., Pu.: = Pa. For weights w1 and w2, the objective function wiit h u2hhy under E [o, 1] achieves its minimum
= min
max
In
2w2
- pmni)
.0.1).
(24)
ln( (((rs))
Proof. The first derivative of the objective function h (alop) + wyhk ( (aoop)) versus a is derived
as
80bjective(Pu.s Pu.a,
Oh kit (Pu..s Pu.i,
aa W1
+ W2
da
da
Pu.,i In (p(r())))))
2p,,c (1 Pun)) (n))u((((pn
w1
(fris)
(au()us)
(25)
Let (Oljeciie(clP...=p.....
be zero. Then the optimal satisfies
da
In
2wz
Pu,s)
Copt
In(f
(26)
Note that needs fulfill 0 1. Therefore, the solution of the optimization problem with the constraint can be obtained.
Theorem 3.6. (Generalization Bounds of D-IPS and D-DR). For any finite hypothesis space H of Y and the optimal prediction matrix y- given é.,; and Pu,i for all (u, i) D, with probability 1 P. the prediction inaccuracies LD-PssY- Y) and LD-DR(Y Y) under D-IPS and D-DR have the following upper bounds
lhg (Pu,i: Pu.si ore
LI-Is(L 1
Σ
+ hG (eita);
D
(u
[ (Du, Puris 0r)
LoDaR(Y x0) + Σ
Bu (ott.):
D
(u.I)D
where ett and tth are the error and error deviation corresponding to ÿ arg max EQwiED tively, and the function had is formulated
and Y = arg max E(U,))DD ( (Te(e..)) )2 respec-
eut
y (Pu.s)
log( 21 n
ha (zit.)
Σ
(27)
2
(P
(,,
Proof.
According the definition of bias, the differences between (Y Y) and expectations of LIDIPs (Y- -Y and LOARR (8-, YYO) satisfy
L YY - Lo.Is( - Lowil Y) - Eo Lo ()] Bo Lo (8 Lo-1s(887)
(28)
LBias(Lo.ns(* + Bo[LD.s -D-
and
Treal ( Y) - Do =I =Ineal ( Y) Ro ( BolTD
(Y-) (Y T..DD
(29)
<Bias (8 (
+ Ro(TDDR (Y
respectively. From the inequalities 28 and 29). the expressions of Bias IPs (DD.I.(pP(-))) and
Bias( (DDLLDRRRYY have been given in Lemma 3.6 and, in what follows, terms Eo LDars (Y-))
LD-IPS (Y in 28 and [o(Lo.es LD-IPS (Y-) in 29 are discussed. Considering the finite
17

---

hypothesis space 7 .?..... and the Hoeffding's inequality, for any > O, the following inequalities be obtained
Pr Pr((Tons((*>) EooL
5T) =1 -1-Pt([Eomn((> EolLo.s (8 27)
21 Pr
max PrEN
[DDIS(P - Ios ( 27)
Pr ) EolLLuus EolLouss(P 27)
21
F=1
-2(y(D1)2
Σ 1=
2 exp
(30)
( ( c
(W,i)ED
2(7/DD)2
21 2/H exp
Σ
(M.t)ED
Pr P((]oooo(8*))- BOLLDOR (8 [
S {T) 21 2/H( exp
2(7|D1)2
Σ
ge (t,))
(u,)ED
Let 2/H exp
2(11DD2
be p. The errors "/uups and DD for D-IPS and D-DR can be
Σ
g2 :2...
solved as
IDMPP
log( 2|1
2/2
Σ
for (Bu.i)
Cuye
DD.R
logí
2/D
2 H
Σ
fa
6t
(31)
(M.EED
(W.))D
Therefore, LD.LDS (8 [] LDAIPS (8- in 28 and Iol D Y - LoWES (8-) in
290 fulfill
log( 2iHll
ctt
Bo ( LDUIS (8 (8)
Σ
2/
(Pa.i)
(u.D)D
(32)
log( (il 2|D
8w,s fo (pu,s)
Lo|LDDR (Y-)) LOADR (87)
Σ
(u,1)D
Combining 28 in Lemma I.
29 and
32 we can obtain the generalization bounds of D-IPS and D-DR given
Theorem 3.7. (Boundedness of Variance and Generalization Bounds). Let a op [0, be the optimal
opt parameter of (4). If the dynamic estimators adopt ou,i as the parameter, then the corresponding variance and generalization bounds bounded.
Proof. Considering the optimal parameter az, for each user-item pair (u, i) and the optimization problem obtain the corresponding optimal objective function
Objective w. (D (),)) + we Ev (h (a S w1 E2 (h)) (0)) (x
(33)
Since h (fu, Pu,is a) and h/ ( (pu,,s Pu a) > considering (33) we have
w2Ev (1 (a & Es (Iv (0)) + uy
(34)
w Ep (1 pu.s) + w2 Ev (p,,:(1 pu,)))
which implies that
(1, (1 Pu.s)
Ity (pu,ss Pu,ir a a.
2a (p..s)
< Ey
ER(1 ppas) + Ev - -P..i))
(35)
4F2
wi (1(
Ey
+ (0.25.))
w2
18

---

Therefore, the variance of dynamic estimators are bounded by Vo c= au.)
opt
20 hy (Pu.ds Pu.d, dg
D
(u.J)ED
Wi EB()
(0.25)
[D]
W2
(w,))
(36)
Vo LD.DR a = opt
Σ
By (Pr,ts Puiis af )0
[D]
(u,))D
wi En(1) Ev Ev (0.25))
F
D
W2
(u.))D
(Z7,)) in generalization bounds and the boundedness of h Pu,i og))) it is easy to obtain that under p * the generalization bounds of ID-IPS and LD-DR are bounded.
Considering the expression l
19

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: see Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see section 5 Conclusions
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: section 4 Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
see section 4 Experiments
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuiu wwwiii/gguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see subsection 4.1 Experimental Setup Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: see subsection 4.3 Ablation Studies
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: see section 4 Experiments
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::::///errss..oo httprp/......poooooo....ccccaaas
Answer: [Yes]
Justification: Every approach in Experiments is preformed261 10 times to record its mean and standard deviation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Introduction
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Three real-world public datasets are used in Experiments.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section 4 Experiments
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This problem is inapplicable to this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This problem inapplicable to this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
The datasets used in this paper are public.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "xHLDu14Trk", "reviewer": "Reviewer_AciS", "summary": "This paper first theoretically reveals the limitations of previous regularization techniques, such as unbounded variance and generalization bound. To address this problem, this paper defines the general form of the estimator with regularization. Then this paper develops a comprehensive dynamic learning framework, which can lead to reduced and bounded generalization bounds and variances. Experiments on two real-world datasets verify the theoretical results and the performance of the proposed method.", "questions": "I have some questions about the proofs: \\\nFirst, for the proof of Theorem 3.1, why Cov($L_{Est}$, $L_{Reg}$) $\\geq 0$? \\\nSecond, the description of Theorem 3.1 is vague. What is the meaning of \"greater than the one of the original estimator\"? In addition, in the current version, the contrapositive of Theorem 3.1 is \"less than any of the original estimators\", instead of \"less than one of the original estimator\" in Corollary 3.2. \\\nThird, for the proof of Theorem 3.3, I understand the $\\mathbb E_O[L^2_{Est}]$ tends to infinity, but why $\\mathbb E_O[L^2_{Est}] + \\lambda \\mathbb E_O[L^2_{Reg}] + 2 \\lambda^2 \\mathbb E_O[L_{Est} L_{Reg}]$ tends to infinity? In the extreme case, let $L_{Reg}  = -L_{Est}$ and $\\lambda = 1$, then $\\mathbb V_O[L_{Est+Reg}] = 0$.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "$\\bullet~$ The problem studied is important and relevant.\n\n$\\bullet~$ The idea is interesting and novel.\n\n$\\bullet~$ The theoretical results are sound.\n\n$\\bullet~$ The evaluations are solid and convincing.", "weaknesses": "$\\bullet~$ The analysis of Figure 1 is missing.\n\n$\\bullet~$ There is no $\\alpha_{u, i}$ in $lim$ in line 189. In addition, is the constant $C$ related to $\\alpha_{u, i}$?\n\n$\\bullet~$ Does the $w_1$ and $w_2$ in equation 3 be the hyper-parameter? How can we determine such hyper-parameters in practice and what is the tuning range for the experiments?\n\n$\\bullet~$ What is the unbiased condition for the proposed methods? Meanwhile, what is the bias when the optimal In addition, experiments according to the data sparsity should be conducted. If the observed data is not sparse, there may be no propensity score that will not tend to 0.\n\n$\\bullet~$ Some typos. For example, it should be $\\mathbb E_O[L^2_{Est}] + \\lambda^2 \\mathbb E_O[L^2_{Reg}] + 2 \\lambda \\mathbb E_O[L_{Est} L_{Reg}]$ in equation 10. Line 142 \"Theorems\" should be \"Theorem\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Weaknesses:**\n-  In the final version, we will add the following analysis of Figure 1 to lines 212 and 220, respectively.\n\n*''**Line 212:**.... The curves of objective functions under different designed functions $f(\\cdot)$ are given in Fig. 1(c). It can be oberved that for a fixed propensity, there exists an $\\alpha$ such that the objective function attains the minimum value. Besides, ...''*\n\n*''**Line 220:**.... Under different designed function $f(\\cdot)$, the schematic diagram of optimal objective values corresponding to the optimal parameter $\\alpha_{u,i}^{opt}$ is shown in Figure 1(d). Next, ...''*\n\n- The constant $C$ is not related to $\\alpha_{u,i}$. In line 189, what we want to convey is that for all $\\alpha_{u,i}\\in[0,1]$, the constant $C$ is the same.\n\n- Yes, $w_1$ and $w_2$ in Eq. (3) are the hyper-parameter. As shown in (5), $\\alpha_{u,i}^{opt}$ depends on the weight ratio $w_2/w_1$. The weight ratio space will leads to the Pareto frontier of prediction model performance. As disussed in Conclusions, various properties of the Pareto frontier and optimization methods for weight ratios are still an open question, and it is one of our future works. In practice, the tuning range of $w_2/w_1$ is usually set to be between 0 and 1.\n\n- In Theorem 3.5, when $\\frac{w_2}{w_1}=0$ and $f(\\hat p_{u,i})=\\hat p_{u,i}$, then the optimal parameter $\\alpha_{u,i}^{opt}$ is set as 1, and D-IPS and D-DR are equivalent to IPS and DR, which are unbiased. Therefore, the unbiased conditions are $\\frac{w_2}{w_1}=0$, $f(\\hat p_{u,i})=\\hat p_{u,i}$ when propensities or imputation errors are accurate. When $\\alpha$ is optimal, according to the Lemma D.1, the bias of the proposed estimator can be obtained. When $\\frac{w_2}{w_1}\\leq\\frac{f(p_{u,i})}{2(1-p_{u,i})}$, then $\\alpha_{u,i}^{opt}=1$; when  $\\frac{f(p_{u,i})}{2(1-p_{u,i})}\\leq\\frac{w_2}{w_1}\\leq\\frac{1}{2(1-p_{u,i})}$, then $\\alpha_{u,i}^{opt}=\\frac{\\ln\\Big(\\frac{2w_2}{w_1}(1-p_{u,i})\\Big)}{\\ln(f(p_{u,i}))}$; when  $\\frac{w_2}{w_1}\\ge\\frac{1}{2(1-p_{u,i})}$, then $\\alpha_{u,i}^{opt}=0$. Therefore, the dynamic estimators are applicable to different propensity distributions by adjusting the weight ratio $w _2/w _1$. Different weight ratios will lead to different distributions of instances on different estimators. Therefore, even if there is no propensity score that will not tend to 0, the developed fine-grained dynamic framework still can achieve quantitative optimization of bias and variance.\n\n- Equation (10) should be \n$$\\begin{align}\n\\mathbb{V} _O[L _\\text{Est+Reg}]\n=\\mathbb{E} _O[L^2 _\\text{Est}+\\lambda^2{L}^2 _\\text{Reg}+2\\lambda{L} _\\text{Est}L _\\text{Reg}]-\\mathbb{E}^2 _O[L _\\text{Est}+\\lambda{L} _\\text{Reg}].\n\\end{align}$$\n\n**Response for Questions:**\n\nA1. Considering (1), for all $(u,i)$ pairs, we have $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}\\ge0$ and $h(o _{u,i},\\hat{p} _{u,i})\\ge0$. To facilitate representation, $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}$ is denoted as $r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})$.  As given in the proof of Theorem 3.1, $\\text{Cov}(L _\\text{Est}, L _\\text{Reg})$ satisfies\n$$\\begin{align}\\text{Cov}(L _\\text{Est}, L _\\text{Reg})=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\mathbb{E} _O\\Bigg(\\bigg[\\sum _{(u,i)\\in\\mathcal{D}}r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})\\bigg]\\bigg[\\sum _{(u,i)\\in\\mathcal{D}}h(o _{u,i},\\hat{p} _{u,i})\\bigg]\\Bigg)\\nonumber\\\\\\\\=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\mathbb{E} _O\\Bigg(\\sum _{(u,i)\\in\\mathcal{D}}\\bigg[h(o _{u,i},\\hat{p} _{u,i})\\sum _{(u,i)\\in\\mathcal{D}}r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})\\bigg]\\Bigg)\\nonumber\\\\\\\\=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\mathbb{E} _O\\Bigg(\\sum _{j=1}^{\\vert{D}\\vert}\\sum _{k=1}^{\\vert{D}\\vert}h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)\\Bigg)\\nonumber\\\\\\\\=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{j=1}^{\\vert{D}\\vert}\\sum _{k=1}^{\\vert{D}\\vert}\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)].\\end{align}$$ \nSince $\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)]\\ge0$, we obtain $\\text{Cov}(L _{Est}, L _{Reg})=\\mathbb{E} _O(L _\\text{Est}L _\\text{Reg})\\ge0$.\n\nWe will revise this sentense in line 124 as follows, and add the above formula (1) to proof of Theorem 3.1.\n\n*... respectively. For all $(u,i)$ pairs, they satisfy $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}\\ge0$ and $h(o _{u,i},\\hat{p} _{u,i})\\ge0$. $\\lambda>0$ is a ...*\n\nA2. 'the one' in Theorem 3.1 and Corollary 3.2 is a pronoun that refers to the variance. What we want to convey in Theorem 3.1 and Corollary 3.2 are that\n\n*\"**Theorem 3.1** Let $L _\\text{Est+Reg}$ be defined in (1) and the estimator $L _\\text{Est}$ be unbiased. If $L _\\text{Est+Reg}$ is unbiased, then the variance of $L _\\text{Est+Reg}$ is greater than the variance of the original estimator $L _\\text{Est}$.\"*\n\n*\"**Corollary 3.2** If the variance of $L _\\text{Est+Reg}$ is less than the variance of the original estimator $L _\\text{Est}$, then $L _\\text{Est+Reg}$ is not unbiased.\"*\n\nIn the final version, we will modify Theorem 3.1 and Corollary 3.2 to the descriptions above to avoid any ambiguity.\n\nA3. According to the proof of Theorem 3.3, we can obtain\n$$\\begin{align}\\mathbb{V} _O[L _\\text{Est+Reg}]\\ge\\mathbb{E} _O[L^2 _\\text{Est}]+\\lambda^2\\mathbb{E} _O[{L}^2 _\\text{Reg}]+2\\lambda\\mathbb{E} _O[L _\\text{Est}L _\\text{Reg}]-\\bar{B}^2.\\nonumber\\end{align}$$\n\nBased on  A1., we can obtain $\\lambda^2\\mathbb{E} _O[{L}^2 _\\text{Reg}]\\ge0$ and $\\text{Cov}(L _{Est}, L _{Reg})\\ge0$. Since $\\mathbb{E} _O[L^2 _\\text{Est}]$ tends to infinity, $\\mathbb{E} _O[L^2 _\\text{Est}]+\\lambda^2\\mathbb{E} _O[{L}^2 _\\text{Reg}]+2\\lambda\\mathbb{E} _O[L _\\text{Est}L _\\text{Reg}]$ also tends infinity. For all estimators, ${L} _\\text{Est}\\ge0$ and ${L} _\\text{Reg}\\ge0$, ${L} _\\text{Est}=-{L} _\\text{Reg}$ is true only when ${L} _\\text{Est}={L} _\\text{Reg}=0$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LasdoVweVo", "reviewer": "Reviewer_h92D", "summary": "In recommender systems, there are many ratings that are missing not at random, which leads to additional bias in RS when trained using only observed data. This paper first gives a general form of the estimator with regularization, then reveals limitations of previous regularization techniques and the relationship between unbiasedness of the generalized estimator and its generalization bound. In addition, this paper finds an interesting phenomenon that Regularization $L_{Reg}$ cannot guarantee a bounded variance and generalization bound for previous estimators. Then this paper proposes a bias-variance quantitative joint optimization approach with theoretical guarantees that the proposed estimator has a bounded variance and generalization bound. Extensive experiments on two real-world datasets validate the effectiveness of the proposed methods.", "questions": "See the Weaknesses part above for the questions.", "limitations": "Yes, the authors adequately discussed and addressed the limitations of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Debias is a very important topic in the recommendation field. This paper is well-written with clear organization.\n\nS2: This paper summarizes the general form of previous estimators, and then proposes an dynamic estimators with sound theoretical results.\n\nS3: This paper provides a novel bias-variance quantitative joint optimization algorithm that can make the proposed estimator has bounded variance and generalization bound.\n\nS4: The experiments are sufficient and comprehensive. The results validate the effectiveness of the proposed methods. In addition, the experiments are conducted on public datasets which makes it easier to follow.", "weaknesses": "W1: The detailed proof of why Cov($L_{Est}$, $L_{Reg}$) $\\geq 0$ when $E_O[L_{Reg}] = 0$ is needed. In addition, the proof for Corollary \n3.2 is needed.\n\nW2: How to determine the explicit form of function $f^{\\alpha}(\\hat p_{u,i})$ in practice?\n\nW3: Authors should provide the detailed learning algorithm step-by-step to explain how the propensity model, prediction model and imputation model are learned.\n\nW4: Missing some recent references. There are many works focusing on the debiased recommendation. For example, [1] proposes a kernel balancing methods for learning propensity model, [2] proposes a doubly calibrated estimator that involves the calibration of both\nthe imputation and propensity models, and [3] proposes a multiple robust estimator combining multiple propensity and imputation models.\n\nW5: Some Typos. For example, it should be Appendix C in line 144 instead of Appendix D and some unexpected indents after the formula like line 203 should be removed.\n\n[1] Debiased Collaborative Filtering with Kernel-Based Causal Balancing. In ICLR, 2024.\n\n[2] Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random. In WWW, 2024\n\n[3] Multiple Robust Learning for Recommendation. In AAAI, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1. Considering (1), for all $(u,i)$ pairs, we have $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}\\ge0$ and $h(o _{u,i},\\hat{p} _{u,i})$. To facilitate representation, $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}$ is denoted as $r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})$. In the proof of Theorem 3.1, $\\text{Cov}(L _\\text{Est}, L _\\text{Reg})$ satisfies\n$$\\begin{align}\\text{Cov}(L _\\text{Est}, L _\\text{Reg})=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\mathbb{E} _O\\Bigg(\\sum _{(u,i)\\in\\mathcal{D}}\\bigg[h(o _{u,i},\\hat{p} _{u,i})\\sum _{(u,i)\\in\\mathcal{D}}r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})\\bigg]\\Bigg)\\nonumber\\\\\\\\=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\mathbb{E} _O\\Bigg(\\sum _{j=1}^{\\vert{D}\\vert}\\sum _{k=1}^{\\vert{D}\\vert}h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)\\Bigg)\\nonumber\\\\\\\\=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{j=1}^{\\vert{D}\\vert}\\sum _{k=1}^{\\vert{D}\\vert}\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)].\\end{align}$$ \nSince $\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)]\\ge0$, we obtain $\\text{Cov}(L _{Est}, L _{Reg})\\ge0$.\n\nWe will revise this sentense in line 124 as follows, and add the above formula (2) to proof of Theorem 3.1.\n\n*\"... respectively. For all $(u,i)$ pairs, they satisfy $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}\\ge0$ and $h(o _{u,i},\\hat{p} _{u,i})$. $\\lambda>0$ is a ...\"*\n\nSince Corollary 3.2 is the contrapositive of Theorem 3.1, the proof of Theorem 3.1 has been given. Therefore, Corollary 3.2 also holds. We further provide the proof of Corollary 3.2 as follows\n\n**proof of Corollary 3.2**\n\nWe use the method of proof by contradiction. Assume that when $\\mathbb{V} _O[L _\\text{Est+Reg}]\\leq\\mathbb{V} _O[L _\\text{Est}]$, $L _\\text{Est+Reg}$ is unbiased. \n\nAccording to $\\mathbb{V} _O[L _\\text{Est+Reg}]\\leq\\mathbb{V} _O[L _\\text{Est}]$, we have\n$$\\begin{align}\n\\mathbb{V} _O[L _\\text{Est+Reg}]=&\\mathbb{V} _O[L _\\text{Est}]+2\\lambda\\text{Cov}(L _\\text{Est}, L _\\text{Reg})+\\lambda^2\\mathbb{V} _O[L _\\text{Reg}]\\nonumber\\\\\\\\\n\\leq&\\mathbb{V} _O[L _\\text{Est}],\n\\end{align}$$\nwhich implies that $2\\lambda\\text{Cov}(L _\\text{Est}, L _\\text{Reg})+\\lambda^2\\mathbb{V} _O[L _\\text{Reg}]\\leq0$. Therefore, the parameter $\\lambda$ needs to satisfy $0\\leq\\lambda\\leq-\\frac{2\\text{Cov}(L _\\text{Est}, L _\\text{Reg})}{\\mathbb{V} _O[L _\\text{Reg}]}$, which implies $\\text{Cov}(L _\\text{Est}, L _\\text{Reg})\\leq0$. \n\nAs shown in A1, when $L _\\text{Est+Reg}$ is unbiased, we have $\\text{Cov}(L _{Est}, L _{Reg})\\ge0$, which contradicts the condition $\\text{Cov}(L _\\text{Est}, L _\\text{Reg})\\leq0$. Therefore, Corollary 3.2 holds.\n\nA2. We can obtain a family of functions. According to $w _1h _B^{Est}(\\alpha,p _{u,i})+w _2h _V^{Est}(\\alpha,p _{u,i})$ and Fig. (c), we can observe that different $f^{\\alpha _{u,i}}(\\hat{p} _{u,i})$ will lead to different objective function surface. For different datasets with different propensity distributions, we can determine the function form according to the minimum objective values of functions. Actually, there is further optimization potential for the selection of the function form. We can incorporate the selection of function forms into the bias-variance joint optimization problem (4), which is formulated as \n$${Objective}^{opt}=\\min _{\\alpha _{u,i}\\in[0,1], f\\in\\mathcal{F}}\\Big[w _1E _B(h^{Est} _B(f^{\\alpha _{u,i}}(\\hat{p} _{u,i})))+w _2E _V(h^{Est} _V(f^{\\alpha _{u,i}}(\\hat{p} _{u,i})))\\Big],{s.t.} 0\\leq\\alpha _{u,i}\\leq1$$\nThis problem is one of our future works.\n\nA3. This work focus on revealing the general rules of regularization techniques and unbiased estimators and developed a fine-grained dynamic framework to jointly optimize bias and variance. Therefore, almost all learning algorithms can be transformed into the fine-grained dynamic framework dynamic framework, such as joint learning of DR, double learning of MRDR, cycle learning of SDR, collaborative learning of TDR. In the final version, we will add the following core code of fine-grained dynamic estimators to Appendix E to explain how to transform a propensity-based debiased approach to the fine-grained dynamic estimator. \n\n**Core Code of Fine-Grained Dynamic Estimators**\n\n```\nw1 = 1\nw2 = 0.5\nrate = w2 / w1\nstar = 2 * rate * (1 - propensity)\n# corresponds to function 1 given in Table 1\nalpha = np.log(star) / np.log(propensity) \n# alpha = np.log(star) / np.log(np.sin(propensity)/np.sin(1)) \n# alpha = np.log(star) / np.log(np.log(propensity+1)/np.log(2)) \n# alpha = np.log(star) / np.log(np.tanh(propensity)/np.tanh(1)) \nlower_bound = np.zeros(np.size(propensity))\nupper_bound = np.ones(np.size(propensity))\nalpha = np.where(alpha > upper_bound, upper_bound, alpha)\nalpha = np.where(alpha < lower_bound, lower_bound, alpha)\npropensity = np.power(propensity, alpha) \n```\nA4. In the final version, we will add the following descriptions in line 313 to improve Related Work.\n\n*“...forth. To improve the robustness of estimators, a multiple robust estimator is developed in [3] by taking the advantage of multiple candidate imputation and propensity models, which is unbiased when any of the imputation or propensity models, or a linear combination of these models is accurate. From a novel function balancing perspective, Li et al. propose to approximate the balancing functions in reproducing kernel Hilbert space and design adaptively kernel balancing IPS/DR learning algorithms [1]. Moreover, aimed at limitations of miscalibrated imputation and propensity models, Kweon and Yu [2] propose a doubly calibrated estimator and a tri-level joint learning framework to simultaneously optimize calibration experts alongside prediction and imputation models. For the...”*\n\nA5. Thanks for your careful reading. We will correct typos, language issues, and unclear desciption, and further improve the presentation in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gV4bvkUnSZ", "reviewer": "Reviewer_aCF5", "summary": "This paper addresses the challenge of handling data missing-not-at-random, which is prevalent in applications like recommendation systems and display advertising. The authors highlight the limitations of current regularization techniques and unbiased estimators, which often result in high variance and unbounded generalization bounds. They propose a novel systematic fine-grained dynamic learning framework that jointly optimizes bias and variance. This framework adaptively selects the most appropriate estimator for each user-item pair according to a predefined objective function, ensuring reduced and bounded variances and generalization bounds with theoretical guarantees. Extensive experiments validate the theoretical findings and demonstrate the effectiveness of this dynamic learning approach in improving model stability and performance.", "questions": "1. In the experimental section, would other metrics such as precision, recall, or F1-score provide additional insights into the performance of your framework?\n2. The paper mentions that the proposed framework dynamically selects the most appropriate estimator for each user-item pair. Could you provide more details on how these selections are made in practice, and whether there are computational complexities associated with this dynamic selection?\n3. There is limited discussion on the practical implementation and scalability of your proposed framework. How does your approach handle large-scale datasets in terms of computational complexity and memory requirements?\n4. The ablation study explores the contributions of different components of your framework. Could you provide more insights into the most critical components and how they individually contribute to the overall performance?\n5. Your paper discusses the theoretical bounds for variance and generalization. Could you provide more practical insights or guidelines on how these bounds can be utilized or interpreted in real-world applications?\n6. There are several functions for the dynamic estimators listed in Table 1. How were these functions selected, and could you discuss the potential impact of choosing different functions on the performance of your framework?\n7. How sensitive is your method to the choice of hyperparameters, and what guidelines would you provide for selecting these parameters in practical applications?", "limitations": "The authors have made a commendable effort in addressing their work's limitations by thoroughly discussing the theoretical challenges and constraints of existing regularization techniques. However, additional detail would enhance the paper’s comprehensiveness, especially regarding the practical limitations of the proposed framework in terms of computational complexity and scalability. Understanding how the framework performs with large-scale datasets and the resources required for its implementation would benefit practitioners.\n\nRegarding the broader societal impact, the authors briefly mention potential applications but do not deeply explore societal implications. It would be valuable to discuss how their framework might affect users' privacy, data security, and potential biases in recommendation systems. Addressing potential negative consequences and proposing mitigation strategies would show a thorough consideration of ethical concerns and enhance the paper’s robustness.\n\nTo improve, the authors could include a dedicated section on the societal impacts of their work, examining both positive and negative aspects. This section should address how the framework could be used responsibly, ensuring it benefits society while minimizing potential harms. Additionally, incorporating feedback from stakeholders or conducting a preliminary ethical review could provide further insights and strengthen this discussion.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\nThe paper introduces a novel approach to handling data missing-not-at-random (MNAR) with a fine-grained dynamic framework for bias-variance joint optimization. This method moves beyond traditional bias or variance reduction by addressing both simultaneously through dynamic estimator selection tailored to each user-item pair. This innovative dual optimization strategy significantly enhances the robustness and accuracy of predictive models in various applications.\n\nQuality\nThe theoretical contributions are robust and well-grounded, with detailed mathematical derivations and proofs supporting the claims. The authors clearly identify the limitations of existing regularization techniques and unbiased estimators, providing a solid foundation for their proposed solution. Extensive experiments on real-world datasets validate the framework's effectiveness. Multiple performance metrics (AUC, NDCG) ensure a thorough assessment, demonstrating practical utility and reliability.\n\nClarity\nThe paper is well-structured and logically organized, making it relatively easy to follow despite the complexity. Each section builds on the previous one, creating a cohesive narrative from problem identification to theoretical underpinnings, proposed solution, and experimental validation. Clear headings, subheadings, mathematical formulations, and visual aids like tables and figures help in understanding key concepts and results. While some sections could be simplified, the overall presentation effectively communicates the core ideas.\n\nSignificance\nThe paper advances the state of the art in dealing with MNAR data, a common issue in many real-world applications like recommendation systems and online advertising. The framework dynamically optimizes bias and variance, addressing a critical gap in the literature and leading to more stable and accurate predictive models. Improved handling of MNAR data enhances the performance and reliability of machine learning systems across diverse domains. Theoretical insights into estimator design trade-offs can inspire future research and development.", "weaknesses": "One notable weakness is the experimental design. The experiments focus on AUC and NDCG metrics, which may not capture all aspects of model performance. Including metrics like precision, recall, or F1-score would offer a more comprehensive evaluation.\n\nThe paper also lacks a detailed comparison with state-of-the-art methods. While some comparisons are made, they are often superficial. The authors mention that their framework outperforms existing methods in bias and variance reduction but do not provide a detailed analysis of why this is the case. A thorough examination of the differences between their approach and other leading methods, focusing on specific scenarios, would strengthen the paper. A more rigorous ablation study to understand the contribution of each component of the proposed framework would also be beneficial.\n\nThe discussion on the practical implementation of the proposed framework is lacking. The authors provide little guidance on integrating their framework into existing systems or scaling it to handle large datasets. Practical considerations like computational complexity, memory requirements, and real-time applicability are not adequately addressed. More detailed information on these aspects, along with potential solutions or workarounds, would enhance the paper’s utility for practitioners.\n\nLastly, the paper could improve in presentation and readability. The writing is dense and technical, which may pose a barrier to understanding for readers unfamiliar with the subject matter. Breaking down complex ideas into simpler parts and including more visual aids like diagrams and flowcharts would make the paper more engaging and easier to follow. Clearer explanations and definitions of technical jargon would also help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1. Thanks for your valuable comments. In the final version, we will add more metrics. Besides, the baseline approaches and the dynamic estimators are conducted on the KuaiRec dataset to verify the performance of the developed dynamic fine-grained framework. The experimental results will be merged into Table 1.\n\nA2. Based on $\\alpha _{u,i}\\in[0,1]$, when $w _2/w _1$ satisfies $\\frac{w _2}{w _1}\\leq\\frac{f(p _{u,i})}{2(1-p _{u,i})}$, $\\alpha _{u,i}^{opt}=1$; when  $\\frac{f(p _{u,i})}{2(1-p _{u,i})}\\leq\\frac{w _2}{w _1}\\leq\\frac{1}{2(1-p _{u,i})}$, then $\\alpha _{u,i}^{opt}=\\frac{\\ln\\Big(\\frac{2w _2}{w _1}(1-p _{u,i})\\Big)}{\\ln(f(p _{u,i}))}$; when $\\frac{w _2}{w _1}\\ge\\frac{1}{2(1-p _{u,i})}$, then $\\alpha _{u,i}^{opt}=0$. Therefore, the daynamic estimator can be rewritten as\n$$\\begin{align}L _{D-IPS}=&\\frac{1}{\\vert\\mathcal{D}\\vert}\\sum _{(u,i)\\in\\mathcal{D}}\\mathbb{1} _{\\bigg[\\frac{w _2}{w _1}\\leq\\frac{f(p _{u,i})}{2(1-p _{u,i})}\\bigg]}\\frac{o _{u,i}}{f(\\hat{p} _{u,i})}e _{u,i}+\\mathbb{1} _{\\bigg[\\frac{f(p _{u,i})}{2(1-p _{u,i})}\\leq\\frac{w _2}{w _1}\\leq\\frac{1}{2(1-p _{u,i})}\\bigg]}\\frac{o _{u,i}}{f^{\\alpha^{opt} _{u,i}}(\\hat{p} _{u,i})}e _{u,i}\n+\\mathbb{1} _{\\bigg[\\frac{w _2}{w _1}\\ge\\frac{1}{2(1-p _{u,i})}\\bigg]}o _{u,i}e _{u,i},\\\\\\\\\nL _{D-DR}=&\\frac{1}{\\vert\\mathcal{D}\\vert}\\sum _{(u,i)\\in\\mathcal{D}}\\mathbb{1} _{\\bigg[\\frac{w _2}{w _1}\\leq\\frac{f(p _{u,i})}{2(1-p _{u,i})}\\bigg]}\\bigg(\\hat{e} _{u,i}+\\frac{o _{u,i}}{f(\\hat{p} _{u,i})}\\delta _{u,i}\\bigg)\n+\\mathbb{1} _{\\bigg[\\frac{f(p _{u,i})}{2(1-p _{u,i})}\\leq\\frac{w _2}{w _1}\\leq\\frac{1}{2(1-p _{u,i})}\\bigg]}\\bigg(\\hat{e} _{u,i}+\\frac{o _{u,i}}{f^{\\alpha^{opt} _{u,i}}(\\hat{p} _{u,i})}\\delta _{u,i}\\bigg)\n+\\mathbb{1} _{\\bigg[\\frac{w _2}{w _1}\\ge\\frac{1}{2(1-p _{u,i})}\\bigg]}\\bigg(\\hat{e} _{u,i}+o _{u,i}\\delta _{u,i}\\bigg).\n\\end{align}$$\nTherefore, the proposed framework dynamically selects the most appropriate estimator for each user-item pair.\n\n$f(\\hat{p} _{u,i})>\\hat{p} _{u,i}$ guarantees $h _B^{Est}\\ge0$ and $h _V^{Est}\\ge0$. With this operation, the joint optimization problem (3) can be simplified as $\\min _{\\alpha _{u,i}}\\{w _1h _B(\\alpha _{u,i})+w _2h _V(\\alpha _{u,i})\\}$. Then, we can obtain the analytical solution of $\\alpha _{u,i}^{opt}$ without increasing the computational complexity.\n\nA3. In (5), we have given the analytical solution of the optimal parameter. If an engineer have an IPS-based or DR-based estimators for large-scale datasets, then she/he can modify the IPS-based or DR-based estimators into a D-IPS-based or D-DR-based estimator by the following core code. \n```\nw1 = 1\nw2 = 0.5\nstar = 2 * w2 * (1 - propensity) / w1\nalpha = np.log(star) / np.log(propensity) \nlower_bound = np.zeros(np.size(propensity)) \nupper_bound = np.ones(np.size(propensity))\nalpha = np.where(alpha > upper_bound, upper_bound, alpha)\nalpha = np.where(alpha < lower_bound, lower_bound, alpha)\npropensity = np.power(propensity, alpha) \n```\n\nA4. In the proposed dynamic framework, the weight ratios and the function forms jointly determined the performance of estimators. As mentioned in A2, when $\\frac{w _2}{w _1}=0$ and $f(\\hat{p} _{u,i})=\\hat{p} _{u,i}$, then the optimal parameter $\\alpha _{u,i}^{opt}$ is set as 1, and the D-IPS and D-DR methods are equivalent to IPS and DR methods, which are unbiased approaches. According to Fig. 2, we can obeserved the performance of dynamic estimators under different weight ratio, which includes the case of $\\frac{w _2}{w _1}=0$. On the other hand, under the identical weight ratio $\\frac{w _2}{w _1}=0.1$, effects of different functions on performances of dynamic estimators are shown in Table 3. From $Gain _{AUC}$ and $Gain _{N}$ in Table 3, we can observe the individual contribution of different function forms.\n\nA5. The definition of the variance reveals the preformance stability of the estimator in whole propensity space. For a new dataset, we cannot know in advance the range of the propensities in this dataset. An unknown biased dataset may disrupt the stability of the prediction model, which brings significant risks to practical applications. Therefore, it is necessary to consider the boundedness of variance and generalization to ensure the stability and performance of prediction models. In real-world applications, the theoretical bounds for variance and generalization indrectly reflect the stability and generalization performance of estimators, respectively.\n\nA6. We find that the function $f(\\hat{p} _{u,i})$ is not unique. According to the functon design principles, we can obtain a family of functions. According to Fig. (c), for a fixed propensity, different functions correspond to different minimum objective values. For different datasets with different propensity distributions, we can determine the function form according to the minimum objective values of functions under the propensity distribution. Actually, there is further optimization potential for the selection of the function form. We can incorporate the selection of function forms into the bias-variance joint optimization problem (4). The corresponding optimization problem can be rewritten as \n$${Objective}^{opt}=\\min _{\\alpha _{u,i}\\in[0,1], f\\in\\mathcal{F}}\\Big[w _1E _B(h^{Est} _B(f^{\\alpha _{u,i}}(\\hat{p} _{u,i})))+w _2E _V(h^{Est} _V(f^{\\alpha _{u,i}}(\\hat{p} _{u,i})))\\Big],{s.t.} 0\\leq\\alpha _{u,i}\\leq1,$$\nwhere $\\mathcal{F}$ is the candidate function set. This problem is one of our future works.\n\nA7. $w _1$ and $w _2$ in Eq. (3) are the hyper-parameter. $\\alpha _{u,i}^{opt}$ only depends on $w _2/w _1$. $w _2/w _1$ will leads to the Pareto frontier of the estimation performance. As of now, optimization problem for weight ratios is still an open question because of unknown properties of its Pareto frontier. In practice, for a new dataset, the tuning range of $w _2/w _1$ is usually set to be between 0 and 1. Then several $w _2/w _1$ are used to test the estimation performance and the optimal weight ratio is selected as the final parameter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XahZOVYNB3", "reviewer": "Reviewer_rrSW", "summary": "This paper analyzes the relationship between bias, variance, and generalization bound of proposed general estimator with data missing not at random, and proposes a quantitative bias-variance joint optimization method to achieve bounded variance.", "questions": "- Q1: In the proof of Theorem 3.4, the second equality at line 757 is not evidently clear and requires further elaboration. Specifically, I am uncertain whether the order of summation and squaring operations is commutative as suggested in the proof.\n- Q2: The formal definitions of symbols $E_{B}$ and $E_{V}$ are absent, which are important in core Theorem 3.7. Furthermore, their associated properties such as reversibility and boundedness are under-discussed.\n- Q3: The function $f^\\alpha(\\hat p_{u,i})$ is manually designed and selected, but the advantages of two design criteria Isotonic Propensity and Same Order, have not been discussed in theory or experimentation.", "limitations": "Please see the \"Weakness\" and \"Questions\".", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- S1: The research question is good, and investigating debiasing methods when data is biased can be beneficial in real-world applications.\n- S2: The authors provide some theoretical insights and proofs.\n- S3: The authors conduct statistical significance test on the experimental results.", "weaknesses": "- W1: The organization and expression of the article need improvement. The current layout of figures, tables, theorems, and formulas is somewhat crowded, which hinders readability.\n- W2: The discussion on the limitations of previous methods is not accurately described, and some expressions lack precision. For example, in lines 99-100, the boundedness of the variances of IPS and DR estimators is not related to the accuracy of the estimated propensities; instead, it is related to whether there are extremely small values among these estimates. As shown in the Table 4, if the estimated propensities are accurate and there are no extremely small values, the variances of both IPS and DR are bounded.\n- W3: The proof of Theorem 3.1 requires further discussion, and the validity of the final inequality in the proof at line 739 is questionable, as it depends on the specific forms of functions $f$, $g$, and $h$, which significantly influence the final conclusion. For instance, assuming $f_{1}$, $g_{1}$, and $h_{1}$ satisfy $cov(L_{Est}, L_{Reg}) \\geq 0$ , then setting $h_{2} = -h_{1}$ results in $cov(L_{Est}, L_{Reg}) \\leq 0$. Therefore, it remains to be discussed whether this conclusion holds for any arbitrary function. If I have misunderstood, please correct me.\n- W4: The authors validate the effectiveness of their method using only two real-world datasets, which lacks sufficient persuasiveness. For experiments on real-world datasets, it is recommended that the authors further validate their findings using the KuaiRec dataset [1].\n- W5: Typos, for example, 'is' in line 11 and 'click, conversion' in line 24.\n\n**References**\n\n[1] KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems. CIKM 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Weaknesses**\nA1. For the discussion of figures, the descriptions of theoretical results and their proofs, and experimants, according to comments from reviewers, we will add more explanation to avoid any ambiguity and improve the readability. \n\nA2. Sorry for the unprecise descriptions. We will modify the descriptions in lines 99-100 to the following descriptions. \n\n*''..., then IPS and DR estimators are unbiased. For a new dataset, we cannot know in advance the range of the propensities in this dataset. Therefore, a new dataset may introduce extremely small propensities to lead to unbounded variances of IPS and DR, which will disrupt the stability of estimators, especially for larger datasets. It is unacceptable for real industrial scenarios.''*\n\nA3. Considering (1), for all $(u,i)$, we have $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}\\ge0$ and $h(o _{u,i},\\hat{p} _{u,i})\\ge0$. Let $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}$ be denoted as $r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})$. As given in the proof of Theorem 3.1, we have\n$$\\begin{align}\\text{Cov}(L _\\text{Est}, L _\\text{Reg})\\=\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{j=1}^{\\vert{D}\\vert}\\sum _{k=1}^{\\vert{D}\\vert}\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)].\\end{align}$$ \nSince $\\mathbb{E} _O[h(o _j,\\hat{p} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)]\\ge0$, we obtain $\\text{Cov}(L _{Est}, L _{Reg})=\\mathbb{E} _O(L _\\text{Est}L _\\text{Reg})\\ge0$.\n\nWe will add the above formula to proof of Theorem 3.1.\n\nA4. The baseline approaches and the dynamic estimators are conducted on the KuaiRec dataset, where the experiment setting is same as the setting of other datasets. The corresponding results are given as follows:\n\n| Methods | AUC | Gain (AUC) |NDCG@50 | Gain (NDCG@50)|\n|:-:|:-:|:-:|:-:|:-:|\n|  naive |0.7498±0.0010 |- |0.7356±0.0012 |- |\n|  IPS |0.7314±0.0023 |- |0.7450±0.0015 |- |\n|  SNIPS |0.8015±0.0020 |- |0.8082±0.0007 |-| \n|  IPS-AT |0.7733±0.0063 |- |0.8003±0.0037 |-|  \n|CVIB |0.7727±0.0064 |- |0.7852±0.0065 |- |\n|IPS-V2 |0.7787±0.0016 |- |0.7905±0.0029 |- |\n|D-IPS (Ours) |0.7947±0.0005 |8.65% |0.7876±0.0009 |5.71% |\n|D-SNIPS (Ours) |0.8026±0.0017 |0.137% |0.8084±0.0005 |0.247% |\n|D-IPS-AT (Ours) |0.7882±0.0042 |1.93% |0.8143±0.0023 |1.75% |\n|DR |0.7701±0.0058 |- |0.7818±0.0029 |- |\n|DR-JL |0.7808±0.0034 |- |0.7930±0.0033 |- |\n|MRDR-JL |0.7735±0.0008 |- |0.8121±0.0013 |- |\n|Stable DR |0.7812±0.0007 |- |0.7928±0.0040 |- |\n|Stable MRDR |0.7844±0.0013 |- |0.7752±0.0021 |- |\n|TDR-CL |0.7858±0.0016 |- |0.7776±0.0015 |- |\n|TMRDR-CL |0.7801±0.0017 |- |0.8047±0.0013 |- |\n|DR-V2 |0.7839±0.0027 |- |0.7923±0.0056 |- |\n|D-DR |0.7956±0.0064 |3.31% |0.7835±0.0040 |2.17% |\n|D-DR-JL |0.7742±0.0011 |-0.845% |0.7897±0.0017 |-0.416% |\n|D-MRDR-JL |0.7918±0.0011 |2.37% |0.8105±0.0010 |-0.197% |\n\nIn the final version, we will merge this table into Table 2.\n\n\nA5. Thanks for your careful reading. We will correct the language issues and further improve the presentation in the final version.\n\n**Response for Questions:**\n\nA1. Sorry for the incorrect formula in line 757 and thanks very much for your careful reading. $'='$ should be $'\\ge'$ in this formula.\n\nDenote $f(o _{u,i},\\hat{p} _{u,i})e _{u,i}+g(o _{u,i},\\hat{p} _{u,i})\\hat{e} _{u,i}$ as $r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})$. Then, the term in line 757 satisfies\n$$\n\\begin{align}\n\\frac{1}{\\vert\\mathcal{D}\\vert^2}&\\mathbb{E} _O\\Bigg[\\bigg(\\sum _{(u,i)\\in\\mathcal{D}}r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})\\bigg)^2\\Bigg]\\\\\\\\\n=&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{j=1}^{\\vert\\mathcal{D}\\vert}\\sum _{k=1}^{\\vert\\mathcal{D}\\vert}\\mathbb{E} _O\\Big[r(o _j,\\hat{p} _j,e _j,\\hat{e} _j)r(o _k,\\hat{p} _k,e _k,\\hat{e} _k)\\Big]\\\\\\\\\n\\ge&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{j=1}^{\\vert\\mathcal{D}\\vert}\\mathbb{E} _O\\big[r^2(o _j,\\hat{p} _j,e _j,\\hat{e} _j)\\big]\\quad\\quad\\quad\\text{becase of } r(o _{u,i},\\hat{p} _{u,i},e _{u,i},\\hat{e} _{u,i})\\ge0\n\\end{align}$$\nTherefore, $\\mathbb{V}_O[L]$ satisfies\n$$\n\\begin{align}\n\\mathbb{V} _O[L]\\ge&\\frac{1}{\\vert\\mathcal{D}\\vert^2}\\sum _{(u,i)\\in\\mathcal{D}}\\frac{1-p _{u,i}}{p _{u,i}}\\big[e _{u,i}-g(0,\\hat{p} _{u,i})\\hat{e} _{u,i}\\big]^2.\n\\end{align}\n$$\nTherefore, for any $e _{u,i}-g(0,\\hat{p} _{u,i})\\hat{e} _{u,i}\\ne0$, $\\lim _{p _{u,i}\\to0}\\mathbb{V} _O[L]=\\infty$. \n\nWe will use the above proof to replace the formulas in lines 757 and 758.\n\nA2. In the optimization problem (4), $E _B(\\cdot)$ and $E _V(\\cdot)$ are the measure metrics of bias and variance, respectively. As mentioned in **Bias-Variance Quantitative Joint Optimization**, (4) can be simplified as $w _1h _B^{Est}(\\alpha _{u,i})+w _2h _V^{Est}(\\alpha _{u,i})$. We can obtain the analytical solution of the optimal parameter $\\alpha _{u,i}^{opt}$ given in Theorem 3.5.\n\nA3. Considering the principle **Isotonic Propensity**, the function $f(\\hat{p} _{u,i})$ is a probability mapping. Therefore, $f(\\hat{p} _{u,i})$ should be a monotonically increasing function with $f(0)=0$ and $f(1)=1$. $f(\\hat{p} _{u,i})>\\hat{p} _{u,i}$ guarantees $h _B^{Est}\\ge0$ and $h _V^{Est}\\ge0$. With this operation, (3) can be simplified as $\\min _{\\alpha _{u,i}}\\{w _1h _B(\\alpha _{u,i})+w _2h _V(\\alpha _{u,i})\\}$. Then, we can obtain the analytical solution of $\\alpha _{u,i}^{opt}$ without increasing the computational complexity. Next, $\\lim\\limits _{\\hat{p} _{u,i}\\to0}\\frac{\\hat{p} _{u,i}}{f(\\hat{p} _{u,i})}=C,\\forall\\alpha _{u,i}\\in[0,1]$ ensures\n$$\n\\begin{align}\n\\lim _{\\hat{p} _{u,i}\\to0}h^{\\text{Est}} _B(\\hat{p} _{u,i},p _{u,i},\\alpha _{u,i})=1-Cf^{1-\\alpha _{u,i}}(0)\n\\end{align}\n$$\nand\n$$\n\\begin{align}\n\\lim _{\\hat{p} _{u,i}\\to0,\\alpha _{u,i}\\leq0.5}h^{\\text{Est}} _V(\\hat{p} _{u,i},p _{u,i},\\alpha _{u,i})=\\lim _{\\hat{p} _{u,i}\\to0,\\alpha _{u,i}\\leq0.5}C(1-p _{u,i})f^{1-2\\alpha _{u,i}}(\\hat{p} _{u,i})=0\n\\end{align}\n$$\nIn final version, we will add the above discussion to explain these two function design principles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper highlights the limitations of existing regularization techniques in debiased recommendations and introduces a general bias-variance joint optimization approach to overcome these challenges. The reviewers found the authors' responses to be very helpful in addressing their concerns, with one reviewer even upgrading their score. Overall, all three reviewers provided positive evaluations.
Decision: Accept (poster)