Unbiased Decisions Reduce Regret: Adversarial Optimism for the Bank Loan Problem
Authors: Elena Gal
Shaun Singh
Aldo Pacchiano
Benjamin Walker
Terry Lyons
Jakob Nicolaus Foerster
Venue: Submitted to ICLR 2023 (2023)
Keywords: adversarial domain adaptation
online learning
adversarial de-biasing
neural optimism

Abstract: In many real world settings binary classification decisions are made based on limited data in near real-time, e.g. when assessing a loan application. We focus on a class of these problems that share a common feature that the true label is only observed when a data point is assigned a positive label by a learner, e.g. we only learn of an outcome of \emph{accepted} loan applications. In this setting, sometimes referred to as the Bank Loan Problem (BLP) in the literature, the labelled training set suffers from accumulating bias since it is created by learners past decisions. 
Prior work mitigates the consequences of this bias by injecting optimism into the model to allow the learner to correct self-reinforcing false rejections. This reduces long term regret but comes at the cost of a higher false acceptance rate. 
We introduce \emph{adversarial optimism} (AdOpt) to directly address the bias in the training set using \emph{adversarial domain adaptation}. The goal of AdOpt is to learn an unbiased but informative representation of past data, by reducing the distributional shift between the set of \textit{accepted} data points and all data points seen thus far.  AdOpt integrates classification made using this debiased representation of the data with the recently proposed \emph{pseudo-label optimism}(PLOT) method to increase the rate of correct decisions at every timestep.
AdOpt significantly exceeds state-of-the-art performance on a set of challenging BLP benchmark problems.

Under review as a conference paper at ICLR 2023
UNBIASED DECISIONS REDUCE rEGRET: ADVERSARIAL OPTIMISM FOR THE BANK LOAN PROBLEM
Anonymous authors Paper under double-blind review
ABSTRACT
In many real world settings binary classification decisions are made based on lim- ited data in near real-time, e.g. when assessing a loan application. We focus on a class of these problems that share a common feature: the true label is only ob- served when a data point is assigned a positive label by the principal, e.g. we only find out if an applicant defaults if we accept their loan application in the first place. this setting, sometimes referred as the Bank Loan Problem (BLP) the literature, the labelled training can accumulate bias since it is influenced by the past decisions. Prior work mitigates the consequences of this bias by inject- ing optimism into the model to allow the learner correct self-reinfe self-reinforcing false rejections. This reduces long term regret but comes at the cost of a higher false acceptance rate. We introduce adversarial optimism (AdOpt) to directly address bias in the training set using adversarial domain adaptation. The goal of AdOpt is to learn an unbiased but informative representation of past data, by reducing the distributional shift between the set of accepted data points and all data points seen thus far. We integrate classification made using this "debiased" representation of the data with the recently proposed pseudo-laber optimism (PI.TT) method to in- crease the rate of correct decisions at every timestep. AdOpt significantly exceeds Mlateoffthe-es performance set of challenging BLP benchmark problems.
INTRODUCTION
In variety of online decision making problems, principals have to make acceptance or rejection decision for a given instance based on observing data points an online fashion. Across a broad range of these, the true label is only revealed for those data points which the principal accepted, creating a biased labelled dataset.
In this work concentrate addressing this issue for the specifie class of binary classification tasks, also known as the "Bank Loan Pooleem"(BLPP motivated by the characteristic example of a lender deciding on outcomes of loan applications. The lender's objective is to maximize profit, i.e. accept as many credible applicants as possible, while denying those who would ultimately default. The caveat that the lender doesn't learn whether rejected applicants would have actually repaid the loan. Hence a decision policy that relies solely on the past experience lacks the opportunity to correct for erroneous rejection decisions. These "false rejects" are relf-reinforcing since the correct label is never revealed for rejected candidates.
The dynamic nature of the data collection mechanism in the BLP offers a very simple and clearly defined example of accumulating bias of the kind we indicated above. As time progresses, the growing pool of accepted applicants (the models training set), created by the models decisions, forms an increasingly biased dataset, whose distribution is different from that of the general applicant population. This distributional shift affects the accuracy of the predictions of model trained on the set of accepted points for any future applicants.
common approach for mitigating the consequences of a biased model is to inject optimism into the decision making strategy. Of particular note to is the recently proposed Pseudo-Label Opti- mism (Pacchiano et al. 202T, PLOT) that provides a simple and computationally efficient way to introduce optimism which can be used in combination with deep neural networks (DNNs). This op-

---

Under review as a conference paper at ICLR 2023
New batch
Make unbiased predictions on batch
Train de-blased classifier
Accepted
Confirm unblased predictions with optimistic classifier
applicants
Train optimistic classifier
Figure 1: The AdOpt algorithm
timism translates self-fulfilling false rejects into self-correcting false accepts, which strictly reduces long term regret. However, the cost of optimism an increased false acceptance rate in particular early on in the learning process.
We propose and evaluate a novel approach to the BLP that is motivated by methods from the do- main of learning in the presence of distributional shift. Specifically, utilize adversarial domain adaptation to learn a de-biased representation of the training data that minimizes this distributional difference, while preserving the informative features. Although very natural in this setting, this is to our knowledge the first attempt to utilise adversarial domain adaptation to tackle bias in the online context. Our experiments show that the de-biased classifier can achieve increased recall on the new queries while maintaining sufficient precision.
However, by itself this adversarially de-biased classifier suffers from fundamental flaw: It needs to trade-off between truly informative features and reducing bias. Clearly, both cannot be accomplished the same time, which leads this method performing poorly in some settings.
To overcome these issues we introduce adversarial optimism method (AdOpt), which combines the de-biased classification approach with PLOT. When presented with a new query each step, AdOpt uses de- b-abiassd representation of the existing training data and trains a de-biased classifier to assess the probability being a true positive. If the de-biased classifier recommends accept the point, verify its suggestion using the pseudo-label optimism of PLOT: we add the data point to the original labelled dataset with a positive label and train the optimistic classifier on this mixed dataset. Finally, we use the classification of the optimistic classifier to accept reject the data point (see Figure [
Compared with the PLOT, AdOpt has the advantage of utilizing the de-biased classifier for identifi- cation of candidates for the pseudo-labe optimism routine. In Pacchiano et al. (2021) the pseudo- label optimism was combined with the e-greedy approach for pseudo-label candidates selection to mitigate the issue of high false-positive rate. However the major weakness of this approach that the proportion of selected candidates for exploration stays constant from batch batch, and that their selection is random and not in any way driven by the data. As a result re-running and carefully analyzing the experiments the PLOT paper with several dataset sampling strategies shows that doesn't It consistently beat SOTA on 2 out of 3 datasets from Pacchiano et al. (202T Figure and section 55.. The upshot is that better way choosing pseudo-label candidates is needed achieve optimal performance. Since the de- classifier is able to catch more positives with better ac- curacy, by combining it with PLOT we converge to better accuracy for the dataset after seeing a smaller number of candidates and with fewer misclassified queries.
We evaluate AdOpt against a number of established approaches from literature, namely PLOT, "greedy", e-greedy (with decaying schedule) and NeuralUCB Zhou et al.) (2020) algorithms. AdOpt outperforms the state-of-the-ar methods on 4 out of benchmark datasets. Conducting high number of experiments across several sampling methods allows us t-test confirm the statistical significance of our results with high accuracy (Figures
Our ablation study on the effectiveness of PLOT in the AdOpt algorithm demonstrates that the ad- dition of PLOT significantly reduces the wide standard deviation that constitutes the main weakness of standalone adversarially de-biased classifier (Figure 2 In addition, the Standalone Adversarial
2

---

Under review as a conference paper at ICLR 2023
approach requires hyperparameter tuning to achieve balance between recall and precision, whereas AdOpt performance is significantly less affected by this balance in our experiments.
The of the de-biased classification routine has potential to reduce the risk of accumulating bias in online decision making which could have profound ramifications for a variety of socially relevant issues, such access funding bail decisions. We have conducted preliminary evaluation and summarised the results in Figure 5 of the Appendix that support this conjecture. This potential property of the adversarial approach is especially attractive since the increase fairness occurs as direct consequence of attempting to improve classification performance and not due to introducing any kind of artificial constraint, e.g. in the form of "protected" characteristic. We delay the thorough investigation of this direction to future work.
RELATED WORK
Contextual bandits and function approximation The Bank Loan problem (BLP) is specific instance of a contextual bandit problem with two possible actions and a reward function that is if the accepted applicant returns the loan,-1 if he doesn't and 0 if the lender decides to reject the applicant. There exist a multitude of methods for this setting ranging from adversarial bandit ap- proaches such as EXP4 Auer al. (2022) to methods that rely on simple parametric assumptions on the data (for example linear responses) such as OFUL Abbasi- Yadkori et al. (2011) or GLM Filippi al. (2010). These methods' applicability to the BLP depends on the data satisfying restrictive parametric assumptions. Specifically, these methods are not compatible with the use of deep neural network (DNN) function approximators.
Neural solutions to the bank loan problem More recently, deep neural networks have been successfully applied to this class of problems, with several proposed strategies (Riquelme et al. (2018) et al.] (2020) Xu et al.] (2020).. For example, in |Riquelme et al.] (2018) the authors conduct an extensive empirical evaluation of existing bandit algorithms on public datasets. A more recent line of work has focused on devising ways to add optimism to the predictions of neural net- work models. Methods such as Neural UCB and Shallow Neural UCB Zhou et al. 2020): Xu et al. (220)) are designed to add an optimistic bonus to the model predictions that is a function of the representation layers of the model. Their theoretical analysis is inspired by insights gained from Neural Tangent Kernel theory. Other recent works in the contextual bandit literature such as Foster & Rakhlin (2020) have started to pose the question of how to extend theoretically valid guarantees into the function approximation scenario, so far with limited success Sen et al. (202T).
Although neural bandit methods such as Neural UCB can be directly applied to the BLP through its reduction to contextual bandit problem, the current state of the art on these problems is achieved by the PLOT algorithm from Pacchiano et al. (202I). PLOT relies on a data filtering mechanism that is combined with an optimistic labelling strategy to choose which points to assign the optimistic pseudo-label. However, this approach suffers from increased false acceptance rate in particular early on in the learning process. In Pacchiano et al. (202I) the authors mitigate this problem by assigning pseudo-labele with fixed (small) probability e, however this approach is rather arbitrary and doesn't ' take into account the intrinsic properties of the dataset candidate.
Selective labels The bank loan problem also known as theeseeeciv labels" problem (Lakkaraju et al. 2017). but the majority of the literature in this area so far focuses on the offline setup. The recent paper of Wei (202T) considers online scenario, but in contrast with our setting their online mechanism requires the data distribution to be stationary. Additionally although the authors mention their methods can be extended to the setting of NNs, their experimental evaluation is conducted mainly over linear models.
Reject inference This line of methods attempts correct bias in credit scoring by modelling the missing outcomes of the rejected applications. While statistical and machine learning methods are most commonly used, more recently deep learning methods have also been applied, e.g. Mancisidor ct al. (2020). However, like in the majority of selective labels literature, the online scenario has not been considered.
3

---

Under review as a conference paper at ICLR 2023
Fairness Fairness is central concern in algorithmic decision- and existing literature of- fers numerous approaches and perspectives. Although systematically assessing faimesss aspects is not the goal of our present work we feel compelled to mention selection of papers. Fairness in the contextual bandits setting is explored by e.g. ]Joseph et al. (2016), [Schumann et al. (2019). Domain adaptation techniques for censoring of protected features were proposed and evaluated by Edwards & Storkey and Zhang et al. (2018) among others and [Cosoon et al. (2019), |Singh et al. (202I) address fairness in the context of distributional shift. The above works do not consider the online setting, or more specifically the BLP. In the setup of "selective labels" labels"[ "Kilbertus et al. (2020) considers learning under fairnsss constraints under assumption of stationary data distribution (not required by our method) and Coston et al. (2021) proposes a method for selecting fair models from existing ones.
Repeated Loss Minimization Previous work Hashimoto et al. (0188) have studied the problem of repeated classification settings where the underlying distribution is influenced by the model itself. The works MMller et al. (2021); [Perdomo [Pedoo:Peroommo et al.] (2020) and HHardt et al. (2016) extend this idea to introduce the problem domains of performative prediction and strategic classification. Unlike the above setting, in our case only the labelled training set is affected by the model, which results in a distributional shift between this set and the set of points presented to the model. This shift is the reason for using domain adaptation in our setting.
3 BACKGROUND
3.1 THE BANK LOAN PROBLEM
We are focusing on the class of sequential learning tasks, where every timestep the learner is presented with a query x, E IRd and needs to predict its binary label y/: E {0. 1). Acceptance of the query carries an unknown reward of 2y, 1 and rejection a fixed reward of 0. Using the bank loan analogy, the lender receives a unit gain for accepting a customer who repays his loan, experiences a unit loss for accepting a customer who defaults and receives nothing if they reject an application. The learners objective is to maximize their reward, which is equivalent to maximizing the number of correct decisions made during training (Kilbertus et al. 2020).
We assume that the distribution of the labels is given by probability function pa.:
{
with probability po. (xx)
y (x+))
(1)
o.w.
where Pa. : R2 + R is a function parameterized by 0. E O. We assume that the conditional distribution of the responses is fixed for all 1, although in principal the distribution of the query points P(x) may be different for different l's: e.g. the bank may extend it's loan provision at a certain time point to and start targeting a different applicant demographic.
We assume that the learners decision at time t is parameterized by 0₂ and is given by the rule:
1
If po. (a::) 2 common measure of the learners performance regret:
2 accept
R(t)= Σ max(0, 2p.. 1) a(xx) (xx) 1)
Em1
where we denote by a(xe) E {0, }} the indicator of whether the learner has decided to accept (1) or reject (0) the data point Xy.
Minimizing regret is standard objective in the online learning and bandits literature (see Lattimore & Szepesvári 2020)).
Using the set of accepted data points, i.e. the greedy approach, Po, can be obtained by minimising binary cross entropy loss:
L(O At) =
Σ
-x(x) (2) - (1 - v(x)) log (1 - -x()))
(2)
XEA.
4

---

Under review as a conference paper at ICLR 2023
where A, = (xelae = 1 and & - 1 1} denotes the dataset of accepted points up the time 1. In what follows we call such pe. the biased model.
3.2 THE SREUDO-LABEL OPTIMISM APPROACH TO BLP (PLOT)
Learners that derive their strategy solely from training accepted data points are prone acquiring rejection bias, and thus they accumulate infinite regret. However, the e-greedy strategy of randomly accepting candidates risks accepting too many false-positives, leading to the same outcome. In Pacchiano et al. 2 the authors propose pseudo-label optimism (PLOT),, the simple yet effective approach to optimism. The PL.OT approach can summarized as follows: to each new query Xo rejected by the greedy model it optimistically assigns positive pseudo-label, retrains the model on the dataset that includes this data point with positive label and uses the resulting pseudo- model to confirm or ignore the optimistic prediction. Formally, PLOT adds term to the loss function
C (0) (o) U (xx. 1)})
= Σ
-y(x) log (pe (x)) - (1 - y(x)) log (1 po (x) - log (xo))
XEAr
pecudo-label loss
3.3 ADVERSARIAL DOMAIN ADAPTATION
Domain adaptation studies strategies for learning in the presence of distributional shift between the labelled data used for training and the unlabelled data domain containing test samples. Adversarial domain adaptation, first proposed in Ganin et al. (2016), is based on the theory of distance between domains introduced in Ben- David et al. (2006) 2010) and the GAN approach of Goodfellow et al. (214)) The main idea of this method is constructing a representation of the data that both discrim- inative for the given classification task and domain invariant. Further work in adversarial domain adaptation has generalised the method to different adversarial loss functions and generative tasks, e.g. (Liu & Tuzel 2016 Tzeng al. 2017 Zhang et al. 2018), and successfully applied it to wide array of challenges, c.g. (Wang et al. 2019 Edwards & Storkey). Our approach in the present work
similar the DANN algorithm of Ganin et (0166), that we found to give stable results for the
duration of the training.
Given a labelled source dataset S C Rd and a target dataset T C Rd whose labels we need to predict, the adversarial domain adaptation approach is to simultaneously train generator Go Rdd - Rdd that encodes the data, a classifier C. Ti'd - {0. 1}, and discriminator Do Rd4 - (0, 1) that discriminates between the encodings of samples from S and T, so that C.p (Ga() (.)) has high classification accuracy on S, and Go is trained adversarially with D& to minimise the possibility of distinguishing between samples from GS(S) and GT) Following Zhang et al. (2018) we will refer the representation of the data by Gø as "de- biased" and to C. the "de-biased classifier" "de- biased" in our context refers to a representation that doesn't 't distinguish between test and train domains.
The training of generator, classifier, and discriminator can be summarised by the training objective:
min min max E log(Do(Go(x)) + E log(1 ()(G((x))) + L.ee(Cy y(x)), (4) xES
à XET
where L the binary cross entropy loss and y S + {0, 1} provides the labels.
4 ADVERSARIAL OPTIMISM
In this section we describe our main contribution, Adversarial Optimism (AdOpt) in detail. AdOpt uses two classifiers. The first one is "biased" classifier trained on all the accepted data thus far. The second one our adversarially de- bibsaedd classifier. AdOpt then proceeds as follows:
If data point accepted by the biased classifier, we accept and add the dataset with the true label.
If a data point is instead rejected by the biased classifier, we use de- biased classifier to decide whether to add it to the pseudo-label dataset.
5

---

Under review as a conference paper at ICLR 2023
As in PLOT, retrain on the pseudo-label candidates with optimistic labels to decide final acceptance.
4.1 ADVERSARIAL DE- BE-BIASING IN THE BLP CONTEXT
Let us recap the task hand: in the BLP setting 3.1 time we possess labelled set of previously accepted applicants A, and we want to utilize this information to label a new applicant. The obstacle to overcome is that the distribution of the data points in the general applicant pool A, is different from the distribution of data points in As. that was influenced by the previous choices of the classifier. This will compromise the performance of a classifier trained on A,. We address this issue by using the domain adaptation method of Section 3.3 to find the de-biased representation Go of the data in Ar and train the de- biased classifier Co to predict a label for the candidate x sampled from A.
We expect Co to give optimistic predictions for the labels of x E A. in the following sense: as shown by Edwards & Storkey, adversarial domain adaptation produces a classifier and generator that converge to approximating the demographic parity property
P(Ci(GG = 1 x = P(Cu(G(()) = T
(5)
Equation 5 states that the percentage of positive predictions on the transformed labelled set (S the same as on (T). In the BLP setting, as the training progresses and the classifier becomes better at accepting the right candidates, the distribution of true positive labels in A, diverges from
that of the general population: for > 0
(6) Note that the above is simply a consequence of the classifier being successful in its job and is independent of the accumulation of bias due to erroneous rejections. Since in the adversarial adaptation approach the classifier Cp is trained to preserve the accuracy on the labelled set Ar, P(C,, (G)(x)) = 1) is going be close ((ytrae(x) = 1|x As) both when E A, and E A. In the BLP setting, the adversarial training regime creates de-biased classifier that optimistically overpredicts positive labels for the incoming queries.
P(y(x)= = 1/x A.) P(y(x) 1/x A).
To verify empirically that our adversarial training regime in the context of BLP produces de-biased classifier with properties above, we create performance metrics log across the experiments comparing the performance of the de-biased classifier and the classifier trained on the original biased data. The mean and standard deviation for the de-biased classifier that is trained from scratch each step are in Table I
The last two columns empirically evaluate Equation 5
As can be seen from Table D increased recall of the adversarially de-biased classifier comes at the expense of lower precision. This trade off can be regulated by the length of adversarial training. We empirically found that for minimizing regret best performance achieved when training for small number of epochs at each step without resetting the weights of the adversarial triad. This approach also reduces the training duration.
4.2 THE ADOPT METHOD
A known drawback of the adversarial training is its instability. It precludes the standalone adversarial classifier from reliably achieving the optimal results. Our experiments show that the standalone adversarial classifier is highly sensitive to the number of training epochs, hence hyperparameter tuning is required for optimal performance. To mitigate this, we combine the adversarially de- biased classifier with the PLOT approach as illustrated in Figure ! If the point is rejected by the biased classifier and recommended for acceptance by the de-biased classifier we evaluate this recommendation against the models prior knowledge by adding it to the original dataset with positive pseudo-label and retraining an optimistic classifier to make a final decision. Incorporating prior model knowledge via PLOT helps to further reduce regret incurred by our algorithm.
Using the de-biased classifier for assigning pseudo-labels enables AdOpt to explore faster. As ex- plained in 1.. the de-biased classifier possesses increased recall. While this comes at expense of lower precision, its acceptance suggestions are superior to randomly picking pseudo-label candi- dates as is done in Pacchiano et al. (202T). Since lowering the threshold in the biased classifier also has the effect of trading precision and recall we also compare AdOpt with pseudo-labelling by

---

Under review as a conference paper at ICLR 2023
Table 1: Mean and STD values for performance metrics of biased and de-biased classifiers on the "Adult" dataset. We find that adversarial de- biasing trades off precision and recall. All columns except the last contain metrics for the incoming batch, or "Target dataset" T. The last column contains the proportion of predicted positives for the "Source dataset" S to illustrate equation 5
Recall
Precision
Predicted Positive
True Pos.
Bias
No Bias
Bias
No Bias
Bias
No Bias
No Bias S
Mean
0.24
0.59
0.83
0.78
0.31
0.18
0.66
0.69
STD
0.07
0.2
0.16
0.19
0.12
0.07
0.11
0.11
changing threshold of the biased classifier in Figure 6 approaches with minimal hyperparameter tuning.
AdOpt shows superior performance both
During the exposition in we considered the setting where only one point is acted upon in each time-step. In practice however it is more computationally effective to consider the scenario, where each time-step learner receives a batch of data points B; = {x..1..... Xt and sees labels for accepted points (Vr,1 Vr, B²) Bt E 10. where B the size of the batch, and B is the number of accepted points. All the notions from 3.1 and 3.2 extend naturally to this setting. We train a de-biase bia-d classifier adversarially for every timestep on the pair (A,, B,) and use it to obtain optimistic predictions for the labels of the data points in B₁. We then make of the pseudo-label filtering from 3.2 where the loss iis adjusted to be computed over the batch:
(O),,, = (() U {(x, 1) for x E B.})
Σ -ylog (po(x)) - (1 y) log(1 -o(()) )
Σ (x)()
pseudo-labue
where B. C Be denotes the subset of points recommended for acceptance by the de-biased classifier.
Algorithm AdOpt
Initialize accepted dataset AA
for / /= 1,...,.............. , do
1. Observe batch B, (xx() 1=1-
Run adversarial domain adaptation algorithm with starting weights initialized from training
B._1 on (Ae, B.) obtain generator and classifier pair (G, C).
3. Train the biased model Pe, by minimizing the loss L. (O)AA))
Compute the pseudo-label filtered batch B, (xx{') (1) 1) pe, (x(") 1/2 and C(G(z!))
1}.
5. Train the pscudo-label model by minimizing the optimistic pseudo-label loss, 2 (0) A Bi) if Pop (xt) 2 2 or Po. (x,) 2
6. For (() E B, compute acceptance decision via at
0 o.W.
Update Act1 + Ar ((x(2) se B)
end
During the very first time-step 1), the algorithm accepts all the points in Bi. subsequent time-steps the adversarial domain adaptation algorithm is trained on the pair (Ac, B.) to produce the pair consisting of de-biased classifier and the generator of de-biased data representation (C, G). Next we identify a subset of the current batch composed of those points that are both currently being predicted as rejects by the biased model and have been selected as potential positives by the de- biasee classifier. They comprise the pseudo-label batch of potential candidates for acceptance. We train the pseudo-label model on the combined dataset and accept those data points, whose labels were confirmed by the pseudo-label model predictions.
7

---

Under review as a conference paper at ICLR 2023
A notable feature of PLOT is that the size of the dataset of accepted points grows, the predictions of the pseudo-label model differ less and less from the predictions of the biased model trained on the set of the accepted points. Ideologically once the dataset is sufficiently large and accurate information can be inferred about the true labels, the inclusion of B, into the pseudo-label loss has vanishing effect on the model's predictions. The latter has the beneficial effect of making false positive rate decrease with t. This balances the opposite trend in the predictions of the de-biased adversarial classifier, that tends to predicting more positives as the training progresses as result of the Equations [ 5n
We provide the details of the adversarial training algorithm in the Appendix A.1
EXPERIMENTS
Datasets and methods To assess the performance of AdOpt, we consider five benchmark datasets: the UCI datasets Adult, Portuguese Bank, Communities and Crime, and German Credit Data [Lih-- man et al. (2033) as well as MNIST converted to the binary reward format of the Bank Loan problem by defining the positive class to be images of the digit 5. If point is accepted, it receives reward if its true label is positive, and -1 otherwise. If point is rejected, reward is always O.
We compare against four baseline methods: PLOT, NeuralUCB, Greedy (no exploration), and de- cayed c-greedy method. For e-greedy, we follow (Kveton et al., (019)) and use decayed schedule, dropping to 0.1%% % exploration by T=2500. We combine PLOTT with e-greedy selection of pseudo- label candidates as in Pacchiano et al. (202T) In addition we evaluate the performance of the stan- dalone adversarially de-biased classifier as an ablation study.
Since sampling of batches presented during training well as stochastic nature of optimisation has a significant effect on the final regret, to obtain the results in this paper we average the performance of each method across 5 different batch sampling methods and 5 separate runs of 2500 timesteps each, so altogether 25 experiments for each method. We note that this is different from the approach in Pacchiano et al. (2021) where only one specific way of sampling examples from the datasets was examined. This explains the difference in our evaluation of performance of PLOTT on Bank and MNIST where it tied with NeuralUCB (as evidenced by average t-values of respectively 11.4792 and 0.4754 obtained in our experiments). We also note that, as opposed to Pacchiano et al. (2021), we report absolute values of regret at every step, rather than regret with respect to a baseline model, which allows us to achieve more transparent results compared to Pacchiano et al. (22TT) The high number of experiments allows us to conduct t-test to confirm statistical significance of our results, with t- t-values of > 2 indicating less than 5 percent chance of null- null-hypothesis.
The number of training epochs for the adversarially di-bid classifier is a parameter that can in principle be tuned to optimise the results for each dataset. The number of epochs also influences the runtime of the AdOpt algorithm. For the shortest possible duration of one epoch per batch the runtime is comparable to that of PLOT. However compute has relatively low priority for our purposes here: compute is cheap while mistakes are costly. We illustrate the effect of increasing the number of epochs per batch Crime and German datasets in Figures and 4 Further particulars on experiment details and hyperparameters can be found in Appendix A.2
Main results We report the following main results. First, in Table I we compare the performance metrics for the adversarially de-biased classifier (trained with resetting of weights for 30 epochs for every batch) and the greedy classifier trained on the original biased data in a training run on the Adult dataset. The de-biased classifier achieves higher recall while maintaining predictive ability evidenced by its precision value. Our code generates a full log of performance metrics for the biased and de-biased classifiers for every run of the algorithm.
we report cumulative regret for each method. AdOpt outperforms all other methods in our experiments by a significant margin on Adult and MNIST even when the triad of generator, discriminator and classifier is trained for I epoch only for every batch (without resetting the weights). The respective t-values are 7.6778 and 3.8547. When training with epoch AdOpt ties with PLOT on Bank, Crime and German datasets with t-values of -0.7147, 0.3842 and -0 -0.6844. However increasing the length of adversarial training at each batch improves AdOpt performance on these datasets (see Figures 3 and 4). Specifically with training for 10 epochs at each batch
Second, in Figures and
8

---

Under review as a conference paper at ICLR 2023
Bank
VINST
10-
Transaps
Figure 2: Cumulative regret of AdOpt vs. PLOT, NeuralUCB, Greedy, Epsilon greedy and Stan- dalone Adversarial classifier. The adversarially de-biased component of AdOpt trained for epoch at each batch. Cumulative regret is reported as a function of timestep, for 2500 timesteps. The shaded region represents one standard deviation from the mean, computed across 25 experi- ments, 5 runs over different sampling methods. We find that AdOpt outperforms other methods by a large margin on Adult and MNIST (t-values of 7.6778 and 3.8547 at 2500 steps) and matches PLOT and NeuralUCB (respeeeeeetive t-values of -0.7147, 0.6602). t-values of > indicate less than 5 percent chance of null-hypothesis
AdOpt shows statistically significant advantage over PLOT on Crime (t-value of 2.4451) and gains advantage over PLOTT on German with t-value of 1.6827 which indicates trend of approaching statistical significance.
We also evaluate and plot the performance of the standalone adversarially de-biased classifier trained for the same number of epochs as for AdOpt at every batch a an ablation study.
Third, our initial evaluation of bias with respect to gender on the Adult dataset shown in Figure in the Appendix indicates that the adversarially de-biased classifier exhibits significantly less bias in its predictions for different subgroups within those categories than the other methods that we evaluated. Thus it significantly reduces the discrimination of candidates with various characteristics within those categories, even though it was not specifically trained to be unbiased with respect to them. We plan further investigate this encouraging observation in the sequel to present work.
6 CONCLUSION
We apply adversarial domain adaptation to online learning in the Bank Loan Problem setting to address the distribution shift between the labelled training dataset and the true distribution present due past acceptance rejection decisions. We present AdOpt, novel algorithm that experi- mentally establishes a new state-of-the-art across a variety of instances of the BLP, significantly outperforming current methods, even with minimal hyperparameter tuning.
In general, we expect the performance of domain adaptation-basee methods like AdOpt to have the biggest impact datasets with difficult exploration problems. In future work three key steps towards leveraging domain adaptation for increasingly difficult exploration problems:
Harder datasets for the bank loan problem, which can include non-stationary or adversarial datasets.
Generalization of AdOpt to higher dimensional actions, enabling application to new, more complex problem settings.
Supported by work on the first two items, consideration of the general Reinforcement Learning regime, with sequential states and actions.
Our preliminary experimental results indicate that the adversarially de-biased classifier has the po- tential to greatly reduce the risk of seiffreinn rejection-logy for the class of decision making problems that we considered. This might result in fairer access bank loans for underrepresented minorities and overall more unbiased algorithmic decision making. This finding makes adversarial- based methods promising candidates for further exploration in this direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yasin Abhasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24, 2011.
Peter Auer, Nicolo Cesa-Bianchi Yoav Freund, and Robert E Schapire. The nonstochastic multi- armed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoff- man (eds.), Advances in Neural Information Processing Systems, volume 19. MIT
Press, 2006. URL c2b0422caab0ce111222611444444222
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning, 79:151-175, 2010.
URL htte:://www..prrneernnnn com/ wccmm//coneeeet/6666200888811771
Amanda Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R Varshney, Skyler Speak- man, Zairah Mustahsan, and Supriyo Chakraborty. Fair transfer learning with missing protected attributes. In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp- 91-98, 2019.
Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing fairness over the set of good models under selective labels. In International Conference on Machine Learning, pp. 2144-2155 PMLR, 2021.
Harrison Edwards and Amos J. Storkey. Censoring representations with an adversary. In ICLR 2016.
Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 586-594, 2010.
Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pp. 3199-3210. PMLR, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. J. Mach. Learn. Res., 17(1):096-2030,, jan 2016. ISSN 1532-4435.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classifica- tion. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pp. 111-122, 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning. pp. 1929-1938. PMLR, 2018.
Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. Advances in neural information processing systems, 29, 2016.
Niki Kilbertus, Manuel Gomez Rodriguez, Bernhard Schölkopf, Krikamol Muandet, and Isabel Valera. Fair decisions despite imperfect predictions. In International Conference on Artificial Intelligence and Statistics, pp. 277-287. PMLR, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference Learning Representations, 12 2014.
10

---

Under review as a conference paper at ICLR 2023
Branislav Kveton, Csaba Szepesvari, Sharan Vaswani, Zheng Wen, Tor Lattimore, and Mohammad Ghavamzadeh. Garbage in, reward out: Bootstrapping exploration in multi-armed bandits. In International Conference Machine Learning, pp. 3601-3610. PMLR, 2019.
Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining. pp. 275-2222200 284, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Moshe Lichman al. Uci machine learning repository, 2013.
Ming-Yi Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 469-477, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Rogelio A Mancisidor, Michael Kampffmeyer, Kjersti Aas, and Robert Jenssen. Deep generative models for reject inference in credit scoring. Bhowledge-Basede Systems, 196:105758, 2020.
John Miller, Juan C Perdomo, and Tijana Zmic. Outside the echo chamber: Optimizing the perfor- mative risk. arXiv preprini arXiv: 21.22102...87777 2021.
Aldo Pacchiano, Shaun Singh, Edward Chou, Alex Berg, and Jakob Foerster. Neural pseudo-label optimism for the bank loan problem. Advances Neural Information Processing Systems, 34, 2021.
Juan Perdomo, Tijana Zmic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pp. 7599-7609, PMLR, 2020.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empiri- cal comparison of bayesian deep networks for thompson sampling, 2018.
Candice Schumann, Zhi Lang, Nicholas Mattei, and John P Dickerson. Group fairness in bandit arm selection. arXiv preprint 191iv: 2019.
Rajat Sen, Alexander Rakhlin, Lexing Ying, Rahul Kidambi, Dean Foster, Daniel Hill, and Inderjit Dhillon. Top-k extreme contextual bandits with arm hierarchy, arXiv preprint arXiv:2102. 078000 2021.
Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairess violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Ac- countability, and Transparency, pp. 3-13. 2021.
E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2962-2971, Los Alamitos, CA, USA, jul 2017, IEEE Computer Society. doi: 10. 10.1109/CVP..2017.331 URL https: //doi.
doi.
org/ 10. 1ttp:///11.110//.222020100
Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, and Hongning Wang. Adver- sarial domain adaptation for machine reading comprehension. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJNLP)), pp. 2510-2520, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19. 1254. URL httpt:ttts//acccc.l aawttaaaaanaaaaa http:p////ao.aanoooooooooooo//// wgg..orgg/11-113200/
Dennis Wei. Decision-making under selective labels: Optimal finite-domain policies and beyond. In International Conference on Machine Learning, pp. 1 11035-1106. PMLR, 2021.
Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. arXív preprint arXin:20/2.01788) 2020.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In Proceedings of the 2018 AAAI/ACM Conference on Al, Ethics, and Society, pp. 335-340, 2018.

---

Under review as a conference paper at ICLR 2023
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb- exploration. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JorR. 2020.
12

---

Under review as a conference paper at ICLR 2023
Cims
Trandags
Figure 3: Cumulative regret of AdOpt PLOT, NeuralUCB, Greedy, Epsilon greedy and Stan- dalone Adversarial classifier on Crime datasets when training adversarially de-biased classifier for 1, 2 and 10 epochs at every batch. Cumulative regret is reported as function of timestep, for 2500 timesteps. The shaded region represents one standard deviation from the mean, computed across 25 experiments. We find that increasing the length of adversarial training at cach batch im- proves the performance AdOpt. At 10 batches per epochs AdOpt outperforms the next leading method, PLOT with t-value of 2.4451. t-values of 2 indicating less than 5 percent chance of null-hypothesis
Eedto Steed
Aibeur
25
200
1000 2003
2000
2.30
366
1000 10:00 2003
2305
300
2000 10.00
2003
250
Figure 4: Cumulative regret of AdOpt vs. PLOT, NeuralUCB, Greedy, Epsilon greedy and Stan- dalone Adversarial classifier on the Crime datasets when training adversarially de-biased classifier and epochs at every batch. Cumulative regret is reported as a function of timestep, for 2500 timesteps. The shaded region represents one standard deviation from the mean, computed across 25 experiments. We find that increasing the length of adversarial training at each batch im- proves the performance of AdOpt. At 10 batches per epochs AdOpt outperforms the next leading method, PLOT with t-value 1.6826. t-values of > indicating less than 5 percent chance of null-hypothesis
A APPENDIX
A.1 ADVERSARIAL TRAINING ALGORITHM
Given the datasets S and T of labelled and unlabelled data the algorithm 1ashmmmm@rraa trains a classifier, gener- ator and discriminator triad in 3.3 Before beginning of the optimisation, S and by adding y-label to each data point. In S the value of the y-label is the known true label for each data point and in set the ylbel to be zero (the values of y on We also add a label to all the data points indicating whether the data point originated from S or T.
augmented
used for optimization).
At each step of the optimisation, batch of size B is sampled from the union of the augmented datasets. The classifiers predicted y-label and discriminators predicted z-label are computed for each element in the batch. If there are samples that originated from S, then the binary cross entropy loss of the predicted y label is calculated for those elements, else it is set to zero. The negative binary cross entropy loss of the predicted æ-label is calculated for all the elements of the batch. These losses are then combined, using the hyperparameter to control the relative weighting of the classifier and discriminator's loss. Finally, the gradient of the combined loss is used update the
13

---

Under review as a conference paper at ICLR 2023
Acoge
Advensr
0.7
04-
mat rocm
034
0.3
000
6.
39:00
582
1990
10:00
2003
2005
900
1000
1000
2000
29.33
Gready
Spelion Gready
Neural.Ce UCB
0.5
0.5
24-
for
mode-
male
1000
1500 2500 380 3000 2009 2502
300 1000
2000
2000
2530
Figure 5: The recall and false positives ratio (fpr) values per batch for Females and Males subgroups of the Adult dataset for the different methods evaluated in the paper. The adversarially de-biased classifier is the only one among evaluated methods that shows similar values of recall and fpr for these groups, thus coming close to satisfying the equality of odds definition of fairness. This remark- able property of the adversarial classifier is potentially very interesting subject of future study.
ensyoutl Ieroc
33
530
1000
1303
2300
2030
300
1000 1300
2003
7500
360 1009 2000
250
Figure 6: Cumulative regret of AdOpt vs PLOT combined with pseudo-labelling using biased classi- fier at thresholds 0.1 and 0.25 ("GreedyPLOTT") While lowering the threshold of the biased classifier also has an effect of trading recall and precision, adversarial domain adaptation more successful at finding the optimal threshold for minimizing cumulative regret and does so with no hyperpa- rameter tuning.
14

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Adversarial Domain Adaptation
Input:
Unlabelled data T = { J==1 labelled data S {(x01)} batch size B, number of batches NB, and adversarial weight X. Output: The generator Gp and classifier Cip-
Let T = {(x()) 0,
x E T) and 8 {(x01, ((xx))())))) x E
Initialise the generator Go, classifier Cib, and discriminator D& with parameters 8, t/ and respectively.
for 1,... NB do
Sample batch {(x, yı, 2)) from TUS. 2. Compute
y {ūr ú: = C,(GGGx))))
and
2 {&s & Do (x:))ee)))
Ba Σ i=1
3. Compute B'
2i).
4. if B' = 0 then
Le = 0
else
Lo
Σ
z; ly log(ū:) (1 yı) lo)(1111 k;)],
B'
end
5. Compute the discriminator loss,
1 Σ B
Ln
(2) + +1 z.) log(1 2.).
Compute the combined loss, L La + ALD.
7. Update 0, ab, and á using optimiser and VoL, V L, and - V respectively.
end
generator and classifier's parameters, and the negative of the gradient of the combined loss used to update the discriminator's parameters.
A.2 EXPERIMENT DETAILS AND HYPERPARAMETERS
Our experiments use the optimal hyperparameters for each method reported in previous papers. NeuralUCB uses an alpha of 0.4, and a discount factor of 0.9. PLOT uses a two-layer, 40-node, multi-layer perceptron. AdOpt is trained using an encoded dimension 100, as well as a hidden layer size of 100. Lambda is set to 1, and we train the adversarial model with Binary Cross Entropy loss. Each experiment runs on a single GPU, and our results were collected by trivially distributing 25 experiments in parallel on 25 different GPU cores.
For the adversarial training, the generator, classifier, and discriminator are multi-layer perceptrons, where the dimension of the input, hidden layer(s), and output are [Data Dimension, 100, 100, 100), [100, 100, 1], and [100, 100, 100, 1] respectively. We used Adam optimizer, with (31, 32) (0, 0.99) and learning rate of 0.0005 for the discriminator and 0.0001 for the generator and classi- fier Kingma & Ba (2044).
As the historical dataset of accepted points grows, there is large disparity in size between the sample (new batch) and target (historical) datasets for adversarial domain adaptation. To handle this data size discrepancy, restrict the target dataset for adversarial training randomly sampled
15

---

Under review as a conference paper at ICLR 2023
batch of size 32 * 100 from the historical data, once the data reaches this size. As we use batch size of 32, this occurs after 100 time steps.
16


Reviews:
[{"review_id": "o17s1pPTIP", "reviewer": "Reviewer_ktkm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a class of problems known as the Bank Loan Problem (BLP), where the learner only observes whether a customer will repay a loan if the loan is accepted. The labeled training data in this problem is biased since it is affected by previous decisions. The authors propose adversarial optimism (AdOpt) to address this problem utilizing adversarial domain adaptation. First, they train a debiased classifier using adversarial training and incorporate this classifier with the pseudo-label optimism (PLOT) method to increase the rate of correct decisions. The numerical results demonstrate the advantages of AdOpt in the standard BLP benchmark.\n", "summary_of_the_review": "Overall, I find the paper interesting, and I appreciate this approach to solving an important but often neglected problem in ML.", "strength_and_weaknesses": "Strengths:\n-The paper concentrates on an interesting and significant problem characterized as the Bank Loan Problem.\n-The paper is easy to follow.\n\nWeaknesses:\n1. The use of adversarial domain adaptation in the paper is not conceivable. The adversarial domain adaptation aims to equalize the probability of 1s from both domains (Equation 5). However, it inherently contradicts the PLOT method (Step 5, Algorithm 1).\n2. It is unclear how the debiased classifier's performance impacts the learners' final performance compared to the previous method (PLOT). It would be valuable if the authors could provide a visualization of the proposed method (see Figure 2 in the PLOT paper).\n3. The proposed approach requires training an additional debiased classifier in comparison to PLOT. However, I am not convinced that the experimental results in Figure 2 represent a noticeable improvement in the learners' performance over PLOT (see results in Bank, Crime, and German datasets).\n4. The paper argues that using the debiased classifier for assigning pseudo-labels enables AdOpt to explore faster. Why do the authors not consider other RL methods(such as Double Deep-Q) as the baselines?\n5. Why is MNIST selected as the comparative dataset while the paper examines the Bank Loan Problem?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cKHJbSJLRO", "reviewer": "Reviewer_4jYB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "PLOT [Pacchiano 2021] leverages a adversarial-training-like scheme to train NN for online binary classification problem: at each time step, retrain the model M while pretending that the new data are of the positive class, and give prediction on the new data using the retrained model M.\n\nThis paper proposes to add a second classifier C to this training process. C is trained at each time step using adversarial domain adaptation; it classifies the embedding of the data. Now, only those that are rejected by M and accepted by C enters the PLOT training for this time step.\n\n", "summary_of_the_review": "The paper is well-written, the contribution is non-trivial, and should be of interest to a broad online learning / strategic learning / domain adaptation audience. However, due to missing important ablation studies, I recommend reject.", "strength_and_weaknesses": "=== Strength ===\n\nAccording to the authors and the best of my knowledge, this paper is the first attempt to utilize adversarial domain adaptation to tackle bias in online learning. The paper is well-written and easy to follow. \n\n=== Weaknesses ===\n\nThe paper refers to G as a de-biased generator and C a de-biased classifier, but there is no discussion in 3.3 on what \"de-biased\" means, or how exactly adversarial domain adaptation achieves it. \n\nIf the role of the de-biased classifier, as expected, turns out to be trading off precision and recall (compared to the biased classifier), why don't we just do thresholding on the biased classifier? That is, go with PLOT, but simply choose a good threshold (<.5) in the biased classifier to select nodes to assign pseudo-labels. In any case, a precision-recall-curve comparison is necessary to establish that the de-biased classifier is trading off precision and recall at a larger area-under-the-precision-recall-curve (compared to the biased classifier) in the threshold region of interest, but this is missing from the paper.\n\nThe paper also lacks an evaluation of the de-biased classifier: how unbiased is it actually, over the time steps?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "STj3s34zYc", "reviewer": "Reviewer_cz4a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the bank-loan problem: a binary classification setting where the learner only receives feedback when it assigns an example the positive label. In this setting, training on a dataset consisting only of the accepted examples naturally leads to bias, since the learner has no ability to correct mislabelled rejections. A recent work addresses this by adding rejected samples to the dataset with an optimistic (positive) label. This naturally leads to a higher false acceptance rate. To help alleviate this issue, this paper proposes modifying this approach by using a second classifier with high recall to decide which points to add to the dataset, so that fewer false acceptances make their way into the dataset. This high-recall classifier is trained using an existing adversarial domain adaptation technique.", "summary_of_the_review": "Overall the paper proposes a reasonable solution to an issue with a state-of-the-art method for this setting, but fails to provide convincing evidence of improvement over the algorithm it is based on (in theory nor experimentally), and so do not recommend this paper for acceptance.", "strength_and_weaknesses": "**Strengths** The core idea seems like a reasonable way to to address a shortcoming of an existing method\n\n**Weaknesses** I found the paper hard to follow in general. It felt like the relevant background explaining PLOT and where the domain adaptation technique fits into the picture was scattered around the paper in an unintuitive way;  I had to read some of the sections several times and out of order to understand what was being proposed. Essentially, the difference between the proposed algorithm (AdaOpt) and PLOT is just that AdaOpt uses a different mechanism to decide whether to include a rejected sample in the dataset. \n\nThe idea of using a high-recall classifier to decide which examples to optimistically add to the dataset to help mitigate high false acceptance rate of PLOT seems reasonable. However, as far as I can tell there are no concrete performance guarantees here, so the only way we can evaluate the proposed approach is via the experimental results. Unfortunately, I don't think that the results manage to demonstrate the claims of the paper. Since AdaOpt is a modification of PLOT, the main baseline to outperform is PLOT, yet Figure 2 implies that there is no statistically significant difference between PLOT and AdaOpt on any of the datasets except for \"Adult\", wherein AdaOpt finishes with slightly lower regret in the end. \n\n**Misc** I think Perdomo et. al. (2020) is an relevant citation to discuss in this paper, as it studies risk minimization in settings where the decision-making policy effects the training data distribution, as it does in the bank loan problem.\n\n\nReferences\n---\n- Perdomo, J., Zrnic, T., Mendler-D\\\"unner, Celestine, & Hardt, M. (2020).\n  Performative Prediction. In H. D. III, & A. Singh, Proceedings of the 37th\n  International Conference on Machine Learning (pp. 7599–7609). : PMLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject