Neural Pseudo-Label Optimism for the Bank Loan Problem
Authors: Aldo Pacchiano
Shaun Singh
Edward Chou
Alexander C. Berg
Jakob Nicolaus Foerster
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural bandits
function approximation
classification
bank loan problem

Abstract: We study a class of classification problems best exemplified by the \emph{bank loan} problem, where a lender decides whether or not to issue a loan. The lender only observes whether a customer will repay a loan if the loan is issued to begin with, and thus modeled decisions affect what data is available to the lender for future decisions. As a result, it is possible for the lender's algorithm to ``get stuck'' with a self-fulfilling model. This model never corrects its false negatives, since it never sees the true label for rejected data, thus accumulating infinite regret. In the case of linear models, this issue can be addressed by adding optimism directly into the model predictions. However, there are few methods that extend to the function approximation case using Deep Neural Networks. We present Pseudo-Label Optimism (PLOT), a conceptually and computationally simple method for this setting applicable to DNNs. \PLOT{} adds an optimistic label to the subset of decision points the current model is deciding on, trains the model on all data so far (including these points along with their optimistic labels), and finally uses the resulting \emph{optimistic} model for decision making. \PLOT{} achieves competitive performance on a set of three challenging benchmark problems, requiring minimal hyperparameter tuning. We also show that \PLOT{} satisfies a logarithmic regret guarantee, under a Lipschitz and logistic mean label model, and under a separability condition on the data. 

Neural Pseudo-Label Optimism for the Bank Loan Problem
Aldo Pacchiano" Microsoft Research
Shaun Singh" FAIR
Edward Chou FAIR
Alexander C. Berg FAIR
Jakob Foerster FAIR
Abstract
We study a class of classification problems best exemplified by the bank loan problem, where lender decides whether or observes whether a customer will repay a loan if the loan is issued to begin with, and thus modeled decisions affect what data is available to the lender for future decisions. As result, it is possible for the lender's 's algorithm to "get stuck" with a self-fulfilling model. This model never corrects its false negatives, since it never sees the true label for rejected data, thus accumulating infinite regret. In the case of linear models, this issue can be addressed by adding optimism directly into the model predictions. However, there are few methods that extend to the function approximation case using Deep Neural Networks. We present Pseudo- Label Optimism (PLOT), conceptually and computationally simple method for this setting applicable to DNNs. PLOT adds an optimistic label to the subset decision points the current model is deciding on, trains the model on all data far (including these points along with their optimistic labels), and finally uses the resulting optimistic model for decision making. PLOT achieves competitive performance on set of three challenging benchmark problems, requiring minimal hyperparameter tuning. We also show that PLOT satisties logarithmic regret guarantee, under a Lipschitz and logistic mean label model, and under a separability condition the data.
issue a loan. The lender only
Introduction
Binary classification models are used for online decision-making a wide variety of practical settings.
These settings include bank lending 44 22 42), criminal recidivism prediction 43 45 5l credit card fraud 7
39 self-driving motion planning [35 and recommendation In many of these tasks, the model only receives the true labels for examples accepted by the learner. We refer this class of online learning problem as the bank loan problem (BLP), motivated by the following prototypical task. A learner interacts with an online sequence of loan applicants.
41
spam detection
systems 36 10
18
the beginning of each time-step the learner receives features describing loan applicant, from which the learner decides whether or not to issue the requested loan. If the learner decides to issue the loan, the learner, at some point in the future, observes whether or not the applicant repays the loan. If the loan is not issued, further information is observed by the learner. The process then repeated in the subsequent time-steps. The learner's objective is to maximize its reward by handing out loans to many applicants that can repay them and deny loans to applicants unable to pay them back.
The BLP can be viewed as a contextual bandit problem with two actions: accepting/rejecting loan application. Rejection carries fixed, known, reward of O: with no loan issued, there is no dependency on the applicant. If in contrast the learner accepts a loan application, it receives reward of 1 in case the loan is repaid and suffers a loss of if the loan is not repaid. The probabilities of repayment
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

associated to any individual are not known in advance, thus the learner is required to build an accurate predictive model of these while ensuring not too many loans are handed out to individuals who can't t repay them and not too many loans are denied to individuals that can repay. This task can become tricky since the samples used to train the model, which govern the model's future decisions, can suffer from bias as they are the result of past predictions from potentially incompletely trained model. In the BLP setting, a model can get stuck in a self-fulfilling false rejection loop, in which the very samples that could correct the erroneous model never enter the training data in the first place because they are being rejected by the model.
Existing contextual bandit approaches typically assume a known parametric form on the reward function. With restrictions on the reward function, a variety of methods ((13) () introduce strong theoretical guarantees and empirical results, both in the linear and generalized linear model settings Pl These methods often make use of end-to-end optimism, incorporating uncertainty about the reward function in both the reward model and decision criteria.
In practice, however, deep neural networks (DNNs) are often used to learn the binary classification model [38] 47), presenting to us a scenario that is vastly richer than the linear and generalized linear model assumptions of many contextual bandit works. In the static setting these methods have achieved effective practical performance, and in the case of [47), theoretical guarantees. A large class of these methods use two components: a feature extractor, and post-hoo host exploration strategy fed by the feature extractor. These methods leave the feature extractor itself open to bias, with the limited post-hoc exploration strategy. Another class of methods incorporate uncertainty into the neural network feature extractor 32 building on the vast literature on uncertainty estimation in neural networks.
We introduce an algorithm, PLOT (see [. which explicitly trains DNNs to make optimistic online decisions for the BLP, by incorporating optimism in both representation learning and decision making. The intuition behind PLOT's optimistic optimization procedure is as follows:
"If I trained a model with the query point having positive label, would predict it to be positive?"
If the answer to this question is yes, PLOT would accept this point. To achieve this, at each time step, the PLOT algorithm re-trains its base model, treating the new candidate points as they had already been accepted and temporarily adds them to the existing dataset with positive geeudolabells, PLOT's accept and reject decisions are based on the predictions from this optimistically retrained base model. The addition of the fake pseudo-labeles points prevents the emergence of self- fulfilingg false negatives. In contrast to false rejections, any false accepts introduced by the pseudo-labels are self-correcting. Once the model has (optimistically) accepted enough similar data points and obtained their true, negative label, these will overrule the impact of the optimistic label and result in a reject for novel queries.
While conceptually and computationally simple, we empirically show that PLOT obtains competitive performance across a set of 3 different benchmark problems in the BLP domain. With minimal hyperparameter tuning, matches or outperforms greedy, e-greedy (with a decaying schedule) and state-of-the the-art methods from the literature such as (euralUCB(47), Furthermore, our analysis shows that PLOT is 3-5 times more likely to accept a data point that the current model rejects if the data point is indeed a true accept, compared to a true reject.
1.1 Related Work
Contextual Bandits As we formally describe in Section 2 the BLP can be formulated as a contextual bandit. Perhaps the most related setting the BLP in the contextual bandits literature is the work of [30]. In their paper the authors study the loan problem in the presence of offline data. In contrast with our definition for the BIP,, their formulation is not concerned with the online decision making component of the BIP,, the main point of interest in our formulation. Furthermore, their setting is also concerned with studying ways to satisfy faimess constraints, an aspect of the loan problem that we consider completely orthogonal to our work. Other recent works have forayed into the analysis and deployment of bandit algorithms in the setting of function approximation. Most notably the
in a linear model the expected response of point x satisties y = xTe. 0. whereas in a generalized linear model the expected response satisfies y = u(x () for some non -oneariyyyyyy typically assumed to be the logistic function.

---

authors of [38] conduct an extensive empirical evaluation of existing bandit algorithms on public datasets. A more recent line of work has focused on devising ways add optimism to the predictions of neural network models. Methods such as Neural UCB and Shallow Neural UCB 47 46 are designed to add an optimistic bonus to the model predictions, that a function of the representation layers of the model. Their theoretical analysis is inspired by insights gained from Neural Tangent Kernel (NTK) theory. Other recent works in the contextual bandit literature such as [14] have started to pose the question of how to extend theoretically valid guarantees into the function approximation scenario, so far with limited success 40). A fundamental component of our work that is markedly different from previous approaches to is to explicitly encourage optimism throughout representation learning, rather than post-hoc exploration on top of a non-optimistic representation.
Learning with Abstention The literature on learning with abstention shares many features with our setting. In this literature, an online learner can choose to abstain from prediction for a fixed cost, rather than incurring arbitrary regret In our setting, rejected point always receives a constant reward, similar to learning with abstention. However, here, regret in the BLP is measured against the potential reward, rather than against a fixed cost. Although the BLP itself does not naturally admit abstention, the extension of PL.OT to abstention setting is an interesting future problem.
Repeated Loss Minimization A closely related problem setting to the BLP (see Section 2 is Repeated Loss Minimization. Previous works [17] have studied the problem of repeated classification settings where the acceptance or rejection decision produces change in the underlying distribution of individuals faced by the decision maker. In their work, the distributional shift induced by the learner's 's decisions is assumed to be intrinsic to the dynamics of the world. This line of work has recently garnered a flurry of attention and inspired the formalization of different problem domains such as strategic classification [15] and performative prediction [37 31 A common theme in these works is the necessity of thinking strategically about the learner's 's actions and how these may affect its future decisions of problems encountered by decision makers when faced with the BLP in the presence of a reward function. We do not treat the world as a strategic decision maker, instead we treat the distribution of data points presented to the learner as fixed, and focus on understanding the effects that making online decisions can have the future accuracy and reward experienced by agent engaged in repeated classification. The main goal in this setting is to devise methods that allow the learner to get trapped in false rejection false acceptance cycles that may compromise its reward. Thus, the learner's 'ss task is not contingent on the arbitrariness of the world, but on a precise understanding of its own knowledge of the world.
result of the world reacting to them. In this paper we focus
different set
Learning with partial feedback In 39 the authors study the one-sided feedback setting for the application of email spam filtering and show that the approach of [99 was less effective than simple greedy strategy. The one-sided feedback setting has in common with our definition of the BLP the assumption that an estimator of the instantaneous regret is only available in the presence of an accept decision. The main difference between the setting of [19] and ours is that the BLP is defined in the presence of possibly noisy labels. Moreover our algorithm PLOT can be used with powerful function approximators, setting that goes beyond the simple label generating functions studied in [19]. In related work [4] considers the problem of one-sided learning in the group- fairness context with the goal of satisfying equal opportunity [16] at every time-step. They consider convex combinations over a finite set of classifiers and arrive at solution which is a randomized mixture of at most two of these classifiers. Moving beyond the single player one-sided feedback problem (6) studies setting which generalizes the one-sided feedback, called partial monitoring, through considering repeated two-player games in which the player receives a feedback generated by the combined choice of the player and the environment, proposing a randomized solution. (2) provides a classification of such two-player games in terms of the regret rates attained and information.
3 study variant of the problem with side
2 Setting
We formally define the bank loan problem (BLP) as a sequential contextual binary decision problem with conditional labels, where the labels are only observed when datapoints are accepted. In this setting, a decision maker and data generator interact through a series of time-steps, which we index
3

---

by time t. At the beginning of every time-step f. the decision maker receives a data point X, E R and has to decide whether accept or reject this point. The label y/ E {0. 1), is only observed if the data point is accepted. If the datapoint is rejected the learner collects a reward of zero. If instead the datapoint accepted the learner collects a reward of 1 if y/+ our notation and discussion on the setting where only one point is acted upon in each time-step. All definitions below can be extended naturally to the batch scenario, where a learner receives a batch of data points Xt, Xt, B, and sees labels for accepted points (V),1 "",,,....,,,,"" ,3ttn' ) 3/ {0, 1 in each time-step, where B is the size of the batch, and B' is the number of accepted points.
and -1 otherwise. Here, we focus
In the BLP, contexts (unlabeled batches of query points) do not have to be IID their distributions, P(x), may change adversarially. As a simple example, the bank may only see applicants for loans under $1000, until some time point where the bank sees applicants for larger loans. Although contexts do not have to be IID, we assume that the reward function itself is always stationary - i.e. the conditional distribution of the responses, Ply x), is fixed for all t. Finally, in the BLP, it is common for rewards to be delayed c.g. the learner does not observed the reward for its decision at time t until some later time, t'. In this work, we assume that rewards are received immediately, as a wide body of work exists for adapting on-line learning algorithms to delayed rewards [29]
Reward The learner's objective is to maximize its cumulative accuracy, or the number of correct decisions has made during training, as in (2 In our model, if the learner accepts the datapoint, Tt, it receives reward of 2y, 1, where y/, is binary random variable, while a rejection leads to a reward of O. Concretely, in the loan scenario, this reward model corresponds to a lender that makes a unit gain for repaid loan, incurs a unit loss for an unpaid loan and collects zero reward whenever no loan is given.
Contextual Bandit Reduction The BLP can be easily modeled as a specific type of contextual bandit problem (24), where at the beginning of every time-step f the learner receives a context x,, and has the choice of selecting one of two actions {Accept, Reject). is easy to see this characterization implies an immediate reduction of our problem into a two-action contextual bandit problem, where the payout of one of the actions known by the learner (re (Reject,,, x4) 0). To distinguish the problem we study from a more general contextual bandits setting, we refer to our problem setting as the bank-loan problem (BLP).
3 Pseudo-Label Optimism
Overview In this section, we describe PLOT in detail, and provide theoretical guarantees for the method. Recall the discussion from Section [ where we described the basic principle behind the PLOT Algorithm. At the beginning of each time-step f, PLOT retrains its base model by adding the candidate points with positive pseudo-labels into the existing data buffer. The learner then decides whether to accept or reject the candidate points by following the predictions from this optimistically trained model. Although the implementation details of PLOT (see Algorithm I are bit more involved than this, the basic operating principle behind the algorithm remains rooted in this very simple idea.
Primarily, PLOT aims to provide similar guarantees as the existing contextual bandit literature, generalized to the function approximation regime. To do so, we rely on the following realizability assumption for the underlying neural model and the distribution of the labels:
Assumption (Labels generation). We assume the labels ye E {0, 1} are generated according to the following model:
[1 with probability p(fo. (x+))
yt
0 O.W.
For some function fo. IRd + R. parameterized by 0, E O and where puz) = expix) is the logistic TToxp(z)
link function. We denote the function class parameterized by 0 E O Fe (So s.1. 0 E O).
For simplicity, we discuss PLOT under the assumption that Fe is a parametric class. Our main theoretical results of Theorem Toold when this parameterization rich enough to encompass the set of all constant functions.

---

In PLOT the learner's decision at time t is parameterized by parameters, 0, and function fo, and takes the form:
If fe. (2;) 2 0 accept
We call the function fe: the learner's model. We denote by E {0, }} the indicator of whether the learner has decided to accept (1) or reject (0) data point x,. We measure the performance of a given decision making procedure by its pseudo-regref"
R(t)= Σ max 2u -1) -ae(22(Jo, 1)
For all denote the pseudo-reward received at time / a, = (xx)) - 1). The optimal pseudo-reward at time equuls if fo. (æx) I 0 and (,((), - otherwise. Minimizing regret standard objective in the online learning and bandits literature (see (25)). As consequence of Assumption 1
the optimal reward maximizing decision rule equals the true model fe.-
In order to show thcoretical guarantees for our setting will work with the following realizability assumption.
Assumption (Neural Realizability). There exists an L L- Liipschitz fo. Rd - IR such that for all X, Idd:
V1 {
with probability u(fo. (x:))
O.w.
Recall that in our setting the learner interacts with the environment in series of time-steps t 1....T, T, observing points {x, l and labels y/2 only during those timesteps when at = 1. The learner also receives expected reward of (2u(fe. (xx) (x.)) 1), quantity for which the learner only has access to an unbiased estimator. Whenever Assumption Seiimpttionhhohdde hoolds, a natural way of computing an estimator O, of 0, is via 1 and & S 1) the dataset of accepted points up the learners decision at time & 1, the regularized log- leeliikelihod (or negative cross- cross-entropy loss) can be written as,
estimation. If we denote by D, = {(x, ye) s.t.a; =
(0)D), = Σ yloo ()(fo(x)) - (1 - y) log (1 ((f((x)) + 2 (x,y)ED;
The Realizable Linear Setting If fo()) x TT 0, the BLP can be reduced to generalized linear contextual bandit problem (see (13) via the following reduction. At time t, the learner observes Reject decision can be
context of the form (0, ). In this case, the payoff corresponding realized by action O for all models in Fe.
Unfortunately, this reduction does not immediately apply to the neural realizable setting. In the neural setting, there may not exist a vector Id with which to model the payoff of the Reject action known priori to satisfy fe, (z) 0. Stated differently, there may not exist easy way to represent the bank loan problem as natural instance of a two action contextual bandit problem with the payoffs fully specified by the neural function class at hand. We get around this issue here, because in the BLP it is enough to compare the model's prediction with the neutral probability 1/2. Although make Assumption 2 for the purpose of analyzing and explaining our algorithms, in practice it is not necessary that this assumption holds.
Just in the case of generalized linear contextual bandits, utilizing the model given by Sa, may lead to catastrophic under estimation of the true response p(fe, (xx) for any query point x,. The core of PLOT is a method to avoid the emergence of self-fulilling negative predictions. We do so using a form of implicit optimism, resulting directly from the optimization of new loss function, which call the optimistic pseudo-label loss.
'The prefix pseudo in the naming of the useudo-regre is common moniker to indicate the reward considered conditional expectation. It has relation to the pseudo-label optimism of methods.
5

---

Definition 1 (Optimistic Pseudo-Label Loss). Let D = {(xe, y1) be a dataset consisting of labeled datapoints E RRd and responses Ve E {0, 1] and let B = { unlabeled data points. We define the optimistic pseudo-label loss of (D, B) as.
j== C R& be a dataset of
(0D, = (0 U {(x, for B})
Σ
- -ylog (u(So(x))) - log(1 11 m(fo(x))) +
(1- -1
(x.))D(R.B)
W log(m((o(x)) 21
presde-fabei loss
Where W > O is a weighting factor, > is the 'focus' radius and D B) = =(xx y) E D s.r. Ex' E B with |xx vix-xx]
x'I R).
Let's take a closer look at the optimistic pseudo-label loss. Given any pair of labeled-unllbeeldd datasets (D, B). optimizing for the optimistic loss L (0) B, W) corresponds to minimizing the cross-entropy of dataset of pairs (x, y) of the form (x, y) E B or (x, 1) such that x E B. other
words, the minimizer of L (0 D, B, W) aims to satisfy two objectives:
Minimize error on the labeled data
2. Maximize the likelihood of positive label for the unlabeled points in B.
The model @c resulting from minimizing _ will therefore strive be optimistic over B while keeping low loss value, and consequently high accuracy (when Assumption Soropptionnhollo holds) over the true labels of the points DD We note that is much larger than WB (the weighted size of B), optimizing f will favor models that are accurate over 'D instead of optimistic over B. Whenever B), the opposite true.
[D]
PLOT Algorithm Based on these insights, we design Pseudo-Labels for Optimism (PLOT), an algorithm that utilizes the optimistic pseudo-label loss to inject the appropriate amount of optimism into the learner's decisions: high for points that have not been seen much during training, and low for those points whose acceptance may cause a catastrophic loss increase over the points accepted by the learner so far.
Algorithm 1 Pscudo-Labels for Optimism (PLOT)
Input: E greedy exploration parameter, weight schedule (W,) radius Initialize accepted dataset D for 1.... do
0
1. Observe batch B, [x{"), _ and sample Q {[g} such that Øj sid Ber(s) Build the MILE estimator,
i = min (,))),
1) s.t. fā, (xx") < 0 and qi = 1}. 4. Calculate the minimizer of the empirical optimistic pseudo-label loss,
3. Compute the pseudo-label filtered batch B, ((x(",
0
min C (0),, B3, W.,
S1 if for (x(()) 2 0
3. For all x B, compute acceptance decision via af
0 o.w.
4. Update Dr+1 + DA P (((() (
vi (
end
During the very first time-step 1), PLOT accepts all the points in B, In subsequent time-steps PLOT makes use of dual e-greedy and MLE-greed filtering subroutine to find a subset of the

---

current batch composed of those points that are both currently being predicted as rejects by the MLE estimator and have been selected by the e-greedy schedule (see step of PLOT).
This dual filtering mechanism ensures that only a small proportion (based on the <) of the datapoints ever considered to be included into the empirical optimistic pseudo-label loss. The MLE filtering mechanism further ensures that not all the points selected by e-greedy are further investigated, but only those that are currently being rejected by the MLE model. This has the effect of preventing the pseudo-label filtered batch from growing too large.
As we have mentioned above, the relative sizes of the labeled and unlabeled batches has effect on the degree of optimism the algorithm will inject into its predictions. As the size of the collected dataset grows, the inclusion of B,, has less and less effect on 0 In the limit, once the dataset is sufficiently large and accurate information be inferred about the true labels, the inclusion of B, into the pseudo-label loss has vanishing effect. The later has the beneficial effect of making false positive rate decrease with t.
The following guarantee shows that in the case of separable data satisfying Assumption 2 the PLOT Algorithm initialized with the right parameters R, ) satisfies logarithmic regret guarantee.
Theorem 1. Let P be distribution over data point, label pairs (x. y) satisfying
1. All x E supp(P) are bounded |x|| S. B.
2. The conditional distributions of the labels y satisfy the data generating Assumption 2 with fuinsciitzfunctions functions containing all constant functions.
F class of I I- Lipschitz
3. fo, (x) 2 > 0 holds for all x supp((P).
Let the marginal distribution of P over points x be Px and let's assume the PLOT algorithm will be used in the presence of i.i.d. data such that X; ~ Px independently for all E N. Define A, Ei= {xe B(xi, R}} and Dr Et= yel {xe E B (x1, R)} where B(x, R) corresponds to the || ll2 ball centered at x and radius R. Let 8' E (0,1). If R 128E.
W. = max (4/tlm/ ())))) (2),)+ and 1, the PLOT Algorithm with batch size 1 "m2
satisfies for all L N simultaneously,
R(1) S δ
In
TPX
With probability at least 1 - 8', where px is a parameter that only depends on the geometry Px and O hides logarithmic factors in T and px.
The proof can be found in Appendix ??. Although the guarantees of Theorem require knowledge of in practice this requirement can easily be alleviated by using any of variety of Model Selection approaches such as in 34 11 27 (3], at the price of a slightly worse regret rate. In the following section conduct extensive empirical studies of PLOT and demonstrate competitive finite time regret on variety of public classification datasets.
Experimental Results
Experiment Design and Methods We evaluate the performance of PLOT on three binary classifi- cation problems adapted to the BLP setting. In time-step f, the algorithm observes context E Rd and classifies the point, a.k.a accepts/rejects the point. If the point accepted, reward one given if that point is from the target class, and minus one otherwise. If the point is rejected, the reward zero. We focus on two datasets from the UCI Collection [12] the Adult dataset and the Bank dataset. Additionally a binary classification problem, where the positive class has income > S50k. The Bank dataset is also binary, with the positive class a successful marketing conversion. On MNIST, convert the multi-class problem to binary classification by taking the positive class to be images of the digit 5,
make of MNIST 26 (d=784). The Adult dataset is defined as
Google Colab: shorturl.aUpzDDY7
7

---

and treating all other images as negatives, Our main metric of interest is regret, measured against a baseline model trained on the entire dataset. The baseline model is used instead of the true label, as many methods cannot achieve oracle accuracy on real-world problems even with access to the entire dataset.
We focus on comparing our method to other neural algorithms, as prior papers [38 23
generally find neural models to have the best performance on these datasets. In particular, we focus on NeuralUCB 47] as a strong benchmark method. We perform a grid search over few values of the hyperparameter of NeuralUCB, considering [0.1, 1, 4, 10] and report results from the best value. We also consider greedy and e-greedy methods. For e-greedy, we follow [23]. and give the method an unfair advantage, i.e. we use decayed schedule, dropping to 0.1 1% exploration by T=2000. Otherwise, the performance is too poor to plot.
In our experiments, we the PLOT weight parameter 1, equivalent to simply adding the pseudo- label point to the dataset. We set the PLOTT radius parameter to co, thus including all prior observed points in the training dataset. Although our regret guarantees require problem-dependedd settings of these two parameters, PLOT achieves strong performance with these simple and intuitive settings, without sweeping.
For computational efficiency, we run our method on batches of data, with batch size n = 32. We average results over runs, running for horizon of t 2000 time-steps. Our dataset consists of the points accepted by the algorithm, for which we have the true labels. We report results for a two-layer, 40-node, fully-connected neural network. At each timestep, we train this neural network on the above data, for fixed number of steps. Then, a second neural network is cloned from those weights. A new dataset with pseudo-label data and historical data is constructed, and the second neural network is trained on that dataset for the same number of steps. This allows us to keep a continuously trained model which only sees true labels. The pseudo-label model only ever sees one batch of pseudo-labels. Each experiment runs on a single Nvidia Pascal GPU, and replicated experiments, distinct datasets, and methods can be run in parallel, depending on GPU availability.
Analysis of Results In the top row of Figure D we provide cumulative regret plots for the above datasets and methods. Our method's cumulative regret is consistently competitive with other methods, and outperforms on MNIST. In addition, the variance of our method is much lower than that of NeuralUCB and Greedy, showing very consistent performance across the five experiments.
The bottom row of Figure provides breakdown of the decisions made by model. As described in Section on average the pseudo-label model only acts on e-percent of points classified as negative by the base model. We provide the cumulative probability of acceptance of true positive and true negative points acted on by the pseudo-label model. As the base model improves, the pseudo-label model receives fewer false positives, and becomes more confident in supporting rejections from the base model. To differentiate this decaying process from the pseudo-label learning, we highlight the significant gap between the probability of accepting positives and the probability of accepting negatives in our method. This shows that the PLOT method is not simply performing decayed-e strategy, but rather learning for which datapoints to inject optimism into the base classifier.
PLOT in action. We illustrate the workings of the PLOT algorithm by testing it on simple XOR dataset. In Figure 2 we illustrate the evolution of the model's decision boundary in the presence of laeudo-label optimism. On the top left panel of Figure 2 we plot 300 samples from the XOR dataset. There are four clusters in the XOR dataset. Each of these is produced by sampling a multivariate normal with isotropic covariance with a diagonal value of 0.5. The cluster centers are set at (0, 5), (0, 0), (5. -2), and 5). All points sampled from the red clusters are classified as 0 and all points sampled from the black clusters are classified as Although there are no overlaps in this picture, there is non-zero probability that black point may be sampled from deep inside a black region and vice versa.
5 Conclusion
We propose PLOT, a novel algorithm that provides end-to-end optimism for the bank loan problem with neural networks. Rather than post-hoe optimism separate from the neural net training, optimism is directly incorporated into the neural net loss function through the addition of optimistic pscudo-
8

---

waey
Canalatuu leggnt
Canalatte Ieght
0.4
MELT
Adult
Bank
Figure 1: Comparison of PLOT to <-greedy (EG), greedy, and NeuralUCB[4] [4 Reward and pseudo- label accuracy are reported function of the timestep. One standard deviation from the mean, computed across the five experiments, is shaded.
Label
Vodal Label
Poed Pendhaaaeeddooo
-3
Accuracion XOR
Model Betere Prestoo Lates TTaring
Vedel After Pocudo Label Trining
0.5
PLOT-005
repet
Nova se repet Fanado
15
30
125
150
175
100
Frnado
Hodia defore Loeaa traising
MedelAAar Labe triining
20-
legaha
Vagatin
25
05
20
SUMARASECEE BRLES
baiches
Figure 2: PLOT with parameter é .5 and batch size equals 3. Top model boundary for batch The model's test accuracy after training equals 50% Having seen very little data the model is very sensitive pseudo-label optimism. Middle model boundary for batch 81. The model's accuracy equals 97.8%. The model boundary is pulled towards the pseudo labeled point. Bottom model boundary for batch 105. The model's test accuracy equals 99.9%. The model training has stabilized. The extra optimism does not change the model boundary.

---

labels. We provide regret guarantees for PLOT, and demonstrate its performance on real-world problems, where its performance illustrates the value of end-to-end optimism.
Our current analysis and deployment of PLOT focused on the bank loan problem, with binary actions, where pseudo-label semanties most clear. Due to the deep connections with active learning and binary contextual bandits, extending this work to larger action spaces is an interesting future direction.
Although the BLP is naturally modeled with delayed feedback, PLOT assumes that rewards are received immediately, delayed rewards [29]. Contexts (unlabeled batches of query points) do not have to be IID their distributions, (xx), may change adversarially. Handling this type of shift is key component of PLOOT's s approach. Optimism is essential to avoiding feedback loops in online learning algorithms, with significant implications for the fairness literature. We presented regret analyses here, which hope can lay the foundation for future work on the analysis of optimism in the fairness literature.
wide body of work exists for adapting on-line learning algorithms to
Statement of Broader Impact
Explicitly incorporating optimism into neural representation learning key ensuring optimal exploration in the bank loan problem. Other methods for exploration run the risk of feature blindness, where a neural network loses its uncertainty over certain features. When representation learning method falls victim to this, additional optimism is insufficient to ensure exploration. This has ramifications for fairness and safe decision making. believe that explicit optimism a key step forward for safe and fair decision making.
However, do want to provide caution around method's limitations. Our regret guarantees and empirical results assume L.ID data, and may not prevent representation collapse in non -tatioati and adversarial settings. Additionally, although our empirical results show strong performance in the non-separable setting, our regret guarantees only hold uniformly in the separable setting.
10

---

References
[1] Y. Abbasisiiiee Abbasi-Yadkori, A. Pacchiano, and M. Phan. Regret balancing for bandit and rl model selection. arXiv preprint arXiv:2006.0549, 2020.
[2] A. Antos, G. Bartók, D. Pál, and C. Szepesvári. Toward classification of finite partial- monitoring games. Theoretical Computer Science, 473:77-99, 2013..
[3] G. Bartók and C. Szepesvári. Partial monitoring with side information. International Conference Algorithmic Learning Theory, pages 305-319. Springer, 2012.
[4] Y. Bechavod, K. Ligett, A. Roth, B. Waggoner, and S. Z. Wu. Equal opportunity in online classification with partial feedback. In Advances in Neural Information Processing Systems, pages 8972-8982, 2019.
[5] R. Berk. An impact assessment of machine learning risk forecasts on parole board decisions and recidivism. Journal of Experimental Criminology, 13(2):193-22 2017.
[6] N. Cesa-Biancio G. Lugosi, and G. Stoltz. Regret minimization under partial monitoring. Mathematics of Operations Research, 31 31(3):562-555 580, 2006.
[7] P. K. Chan, W. Fan, A. L. Prodromidis, and S. J. Stolfo. Distributed data mining in credit card fraud detection. IEEE Intelligent Systems and Their Applications, 14(6)667774, 1999.
W. Chu, M. Zinkevich, L. Li, Thomas, and B. Tseng. Unbiased online active learning in data streams. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 195-203, 2011.
[9] C. Cortes, G. DeSalvo, C. Gentile, M. Mohri, and S. Yang. Online learning with abstention. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1059-1067. PMLR, 10-15 Jul 2018.
[10] Covington, J. Adams, and E. Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference recommender systems, pages 191-198, 2016.
[11] A. Cutkosky, C. Dann, A. Das, C. Gentile, A. Pacchiano, and M. Purohit. Dynamic balancing for model selection in bandits and rl. In International Conference on Machine Learning, pages PMLR, 2021.
[12] D. Dua and C. Graff. UCI machine learning repository, 2017.
[13] S. Filippi, O. Cappe, A. Garivier, and C. Szepesvári. Parametric bandits: The generalized linear case. In NIPS, volume 23, pages 586-594, 2010.
[14] D. Foster and A. Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199-3210. PMLR, 2020.
[15] M. Hardt, N. Megiddo, C. Papadimitriou, and M. Wootters. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pages 111-122, 2016.
[16] M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pages 3315-3323, 2016.
[17] T. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. Faimess without demographics in repeated loss minimization. In International Conference on Machine Learning, pages
1929-1938. PMLR, 2018.
[18] X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pages 1-9, 2014.
[19] D. P. Helmbold, N. Littlestone, and P. M. Long. Apple tasting. Information and Computation, 161(()885-139) 2000.
[20] N. Jindal and B. Liu. Review spam detection. In Proceedings of the 16th international conference on World Wide Web, pages 1 1189-1190, 2007.
[21] N. Kilbertus, M. G. Rodriguez, B. Schölkopf, K. Muandet, and I. Valera. Fair decisions despite imperfect predictions. In International Conference on Artificial Intelligence and Statistics, pages 277-287. PMLR, 2020.
11

---

[22] G. Kou, Y. Peng, and C. Lu. Mcdm approach to evaluating bank loan default models. Techno- logical and Economic Development of Economy, 20(2):292-311, 2014.
[23] B. Kveton, C. Szepesvari, S. Vaswani, Z. Wen, T. Lattimore, and M. Ghavamzadeh. Garbage in, reward out: Bootstrapping exploration in multi-armed bandits. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 3601-3610, PMLR, 09-15 Jun 2019.
[24] J. Langford and T. Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008
[25] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[26] Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227) 2324, 1998.
[27] Lee,, A. Pacchiano, V. Muthukumar, W. Kong, and Brunskill. Online model selection for reinforcement learning with function approximation. In International Conference on Artificial Intelligence and Statistics, pages 3340-3348. PMLR, 2021.
[28) U. Lee, S. Yoon, H. Shim, P. Vasseur, and C. Demonceaux. Local path planning in a complex environment for self-driving car. In The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent, pages 445-450. IEEE, 2014.
[29] C. Mesterharm. On-line learning with delayed label feedback. In S. Jain, H. U. Simon, and E. Tomita, editors, Algorithmic Learning Theory, pages 199-413, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.
[30] B. Metevier, S. Giguere, S. Brockman, A. Kobren, Y. Brun, E. Brunskill, and P. Thomas. Offline contextual bandits with high probability fairness guarantees. Advances in neural information processing systems, 32, 2019.
[31] J. Miller, J. C. Perdomo, and T. Zmnc. Outside the echo chamber: Optimizing the performative risk. arXiv preprint arXiv:2102.080 02.05570, 2021.
[32] I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning, 2018.
[33] A. Pacchiano, C. Dann, C. Gentile, and P. Bartlett. Regret bound balancing and elimination for model selection in bandits and rl. arXiv preprint 20ki::0012.1044: 2020.
[34] A. Pacchiano, M. Phan, Y. Abbasi- Yadkori, A. Rao, J. Zimmert, T. Lattimore, and C. Szepesvari. Model selection contextual stochastic bandit problems. arXiv preprint arXiv:2003.0m 01 704, 2020.
[35] B. Paden, M. Čáp. S. Z. Yong, D. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles, 1(1) 1(1):33-55, 2016.
[36] M. J. Pazzani and D. Billsus. Content-based recommendation systems. In The adaptive web, pages 325-341. Springer, 2007.
[37] J. C. Perdomo, T. Zrnic, C. Mendler-Dünne and M. Hardt. Performative prediction. arXiv preprint arXiv:2002.066: 06663, 2020.
[38] C. Riquelme, G. Tucker, and J. Snoek. Deep bayesian bandits showdown: An empirical
comparison of bayesian deep networks for thompson sampling, 2018
[39] D. Sculley. Practical learning from one-sided feedback. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 609-6 618. 2007.
[40] R. Sen, A. Rakhlin, L. Ying, R. Kidambi, D. Foster, D. Hill, and I. Dhillon. Top-k extreme contextual bandits with arm hierarchy. arXiv preprint arXiv:2102.0788 2021.
[41] A. Srivastava, A. Kundu, S. Sural, and A. Majumdar. Credit card fraud detection using hidden markov model. HEEE Transactions on dependable and secure computing, 5(1):37-48, 2008.
[42] A. K. Tiwari. Machine learning application in loan default prediction. Machine Learning, 4(5),
2018.
12

---

[43] N. Tollenaar and P. Van der Heijden. Which method predicts recidivism best?: comparison of statistical, machine learning and data mining predictive models. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(2):565-55)) 2013.
[44] C.-F. Tsai and M.-L. Chen. Credit rating by hybrid machine learning techniques. Applied soft computing, 10(2):74-380, 2010.
[45] Wang, R. Mathieu, Ke, and H. Cai. Predicting criminal recidivism with support vector machine. In 2010 International Conference on Management and Service Science, pages 1 1-9. IEEE, 2010.
[46] Xu, Wen, H. Zhao, and Q Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:20/2.01785 2020.
[47] D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] Yes, in our societal impact section we discuss limitations of our work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have added section on the broader impact of our method.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] The appendix contains detailed proof's of our theoretical claims
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Our plots include error bars for 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We described the types of GPUs used, as well as the parallelism used in our method.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the MNIST and Adult dataset creators.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We have included our code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] The datasets we used are public and previously published (e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [N/A] The datasets used are public and previously published, and do not have Pllfoffensive content
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No crowdsoureing or human subjects.
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "ay0c_rbYc5q", "reviewer": "Reviewer_7avZ", "summary": "This paper proposes a new approach to improve the performance of non-linear predictors (DNN models) on binary labels when the observations are endogenously selected. The issue arising in traditional setups is that the algorithm will only select observations with high positive outcome (reward). This may lead to sub-optimal and unfair models generating a large proportion of false negatives. The solution proposed in this paper is to add optimistic labels (with positive rewards) to a subset of points that the model would normally reject. Doing so optimistically bias the model, and stimulate the exploration of the more uncertain regions. The suggested algorithm and paper focus is on online learning methods. Several experiments highlight the competitiveness of this approach compared to similar ones.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The main contribution of the paper appears to be described in Section 5, which is on pages 6-7. The previous three sections (setting, related work, and background) occupy a lot of space without necessary conveying crucial information about this work. The paper would benefit from some restructuring.\n* There is a large literature on self-selection bias beyond the online learning literature which is nowhere discussed.\n* Being familiar but not a specialist in online learning (OL), I have the following questions about the approach:\n  * OL is typically applied to problems with high data velocity, when model re-training should be made online because a batch approach would be too costly and/or slow. The examples discussed do not convey the necessity of online updates. Similarly, there is no adversarial game-like sequential interaction. Why would one prefer OL as opposed to, for example, an offline approach that would look at the entire historical sample to quantify model uncertainty and then setup an exploration strategy for the next $n$ observations.\n  * It seems that the DNN is fully recalibrated after a new data batch is created (e.g. Algorithm 2). Please comment.\n* The numerical experiments in section 6 illustrate that the approach is in line with the selected OL benchmarks, but do not highlight a noticeable performance gain or computational advantage.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The approach limitations regarding the data independence and separability should be further discussed. Observations and labels in a bank-loan problem would typically be exposed to correlated and time-varying factors depending, for example, on the current health of the economy.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "VHB229tjMc", "reviewer": "Reviewer_XPXH", "summary": "The paper studies class of problems, called bank loan problem (BLP), where the learner only observes whether a customer will repay a loan if the loan is issued to begin with.\nTo solve this problem, the paper proposes a simple and efficient algorithm, Pseudo-Labels for Optimism (PLOT), which adds an optimistic label to a subset of data points rejected by the model and trains the model on all the data seen so far including the optimistically added points. Empirical evidence shows that PLOT achieves competitive performance in comparison to known storng baselines such as NeuralUCB.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n---------------\n\n- The proposed algorithm PLOT, for bank loan problem, is a simple and intuitive algorithm to enable optimism in the bank loan problem.\n\n- Theorem 1, provides regret analysis for the linear logistic models.\n\n- Theorem 2, states that under some assumptions, PLOT converges to a classifier with perfect accuracy.\n\n\nWeakness:\n---------------\n\n- Some of the assumptions are hard to justify in the Theorem 2 (see Assumptions 4 and 5 in the appendix). I would appreciate any intuition as to why the authors feel this connectedness assumption would be satisfied in the neural setting.\n\n- Proof of the Theorem 2 in the appendix feels more like sketch than any rigorous mathematical argument. \n\nQuestions for Authors:\n---------------\n- What is Algorithm 3 on line 270?\n\n- Your proof of Theorem 2, requires additional assumptions (4, and 5 listed in the appendix). Is there a reason why these assumptions would be satisfied in a Neural setting?\n\n- The bank loan problem is also related to the Selective Classification / Learning with abstention literature, where a network has an extra option to either predict on a new example or reject/abstain the example (implying that the learner is not confident in its prediction on this new example). In this literature, a very simple and strong baseline is thresholding based on the entropy of the prediction ( entropy of the prediction is something your counterfactual loss anyways incorporates). This could be a simple and strong baseline in this problem as well.\n\n- Can you confirm if the dependence on d in the Theorem 1 is linear or \\sqrt{d}? The proof in the appendix seems to suggest otherwise.\n\nWriting Clarity:\n---------------\nPaper is easy to follow except for some of the typos listed below and the proof of Theorem 2 in the appendix.\n\nLine 57: model bey training..\nLine 92: the learner has decided to accept or not datapoint xt....\nLine 515 (appendix): lemmaconfidenceboundrussacappendix ..\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NIYioDG_Nwh", "reviewer": "Reviewer_9rUK", "summary": "Authors propose neural pseudo-label optimism for solving the bank loan problem. This is an interesting setting and I judge the work in the paper (theory & experiments) to be mostly correct. I consider this paper to be sufficiently novel, although there are certainly a large number of papers on the bank loan problem. I am not an expert on the bank loan problem in particular, so I hope another review can provide more clarity as to this paper’s novelty. I am tentatively voting for a rating of 7, but I would like the authors to address some of my concerns below.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Interesting topic & sufficient novelty (as far as I am aware)\n- Complementary experiments & theory\n- Acceptable writing quality\n\nCons:\n- Some parts of the theory are slightly contrived (but this could be fixed)\n- Experiments could use additional details and polish\n\nI strongly suggest the authors rewrite the “neural realizability” aspect of their theoretical framework (or at least provide more details). I understand the motivation behind it, but it seems like a bit of a stretch. For example, consider Assumption 2. Based on the universal approximation property of DNNs, I don’t doubt Assumption 2, but in practice we have no way of knowing whether or not the architecture of choice actually meets this Assumption (at least not before training on a similar dataset). The authors could go a similar route using something like NTK theory which is the limit of wide-networks and is theoretically sound in terms of training and generalization. I understand there is work show a wider class of overparameterized networks can interpolate data due to local convexity, but still this thread of research is not mentioned either. \n\nRelated to the above, I am confused if Theorem 2 relies on the assumption of a convex loss function or also convexity of the loss with respect to the model parameters. The authors should clarify these details or at least include a reference to the supplement in which this is discussed.\n\nEXPERIMENTS:\n - Most details are present but the description could use a bit more clarity. The code is provided in the supplement. I did not attempt to run it.\n- Need more details concerning the DNN architectures (unless these are in the supplement and I just missed it)\n- For the Bank dataset (which is the most interesting for this topic) does not seem to have major improvement between the proposed method and the baselines. The authors might rescale the y-axis so that we can see the differentiation between the curves for the bank dataset\n- Error bars/regions are barely visible\n- 3 datasets are ok but MNIST in particular does not seem like a great candidate for this topic. I like the use of “Adult” and “Bank” and encourage the authors to move MNIST to the supplement in the place of 1 or 2 more “realistic” datasets.\n\n\nMinor:\n-typo in line 57: “bey”\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sure", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "F5GHVxRXqyO", "reviewer": "Reviewer_aBf3", "summary": "This paper first motivates the \"bank loan problem\", which is a scenario where the learner only observes whether a customer will repay a loan if the loan is issued to begin with. The paper moves on to discussions of this problem and introduces contextual linear bandit formulations. This is extended to the case where the model is a deep neural network, and a Theorem shows theoretical guarantees under the case where data is separable. Experiments show improvement over baseline methods, including the recent NeuralUCB method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The paper focuses on an important problem characterized as the \"bank loan problem\" and provides a simple way to use neural networks for this problem.\n- Experiments show improvement over baselines.\n- Code is provided in the supplementary materials and should be helpful for the research community to reproduce results.\n\n### Weaknesses and questions\n\n- Some of the equations seem to be weird. For example, in the equations after line 225 and 232, the first term has $\\log f(x)$, but should this be $\\log \\mu (f(x))$ since $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$?\n- Notations are confusing. $\\mathcal{L}^\\lambda(\\bf{\\theta}|\\mathcal{D}_t)$ are used as the ordinary cross entropy loss in Appendix C but is used as the negative cross entropy loss after line 225.\n- Line 271 points to Appendix E but I couldn't find Appendix E in the supplementary material. \n- In the code, I can see that the neural network is a fully-connected 1 hidden layer neural network. There may be a space issue, but I believe this is basic info and should be included in the main paper.\n- In Theorem 1: Is $1-2\\delta$ a valid probability when $\\delta \\in (0,1)$?\n\n#### Minor suggestions\n- It would be interesting to see experiments where the separability is violated and how it affects the results.\n- line 277: Should $R^d$ be $\\mathbb{R}^d$?\n- Eq. after line 161: Are the parenthesis located in wrong places?\n\n---\n\nThank you for the response. I found Appendix C corresponds to the proof of Theorem 2, instead of Appendix E. The author responses in the other reviews about the details of this proof was also helpful. Since my concerns were addressed, I would like to raise my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The scores from four reviews are 4, 5, 5, and 7. The authors have a persuasive reply to the review with the lowest score, so this score is too harsh. As the authors say, the criticisms in one of the reviews with a 5 score are rather superficial. Overall, the paper is on an important topic and is correct and novel, so I lean towards publication.

One issue not mentioned in the paper is delayed feedback. When a bank grants a loan, a lot of time passes before the true label becomes available, and during this time the function to be learned p(y|x) may change. A related issue mentioned by a reviewer is correlation between labels of different examples, as caused for example by a recession. The last section should discuss these issues briefly, and also discuss fairness, as opposed to merely mentioning it. Lack of exploration can lead to never giving loans to members of a group that has historically faced prejudice. Of course, the three issues just mentioned apply more broadly than just specifically to granting of credit.
Decision: Accept (Poster)