Neural Pseudo-Label Optimism for the Bank Loan Problem

1 Introduction . Binary classification models are used for online decision-making in a wide variety of practical settings . These settings include bank lending [ 44 , 22 , 42 ] , criminal recidivism prediction [ 43 , 45 , 5 ] , credit card fraud [ 7 , 41 ] , spam detection [ 20 , 39 ] , self-driving motion planning [ 35 , 28 ] , and recommendation systems [ 36 , 10 , 18 ] . In many of these tasks , the model only receives the true labels for examples accepted by the learner . We refer to this class of online learning problem as the bank loan problem ( BLP ) , motivated by the following prototypical task . A learner interacts with an online sequence of loan applicants . At the beginning of each time-step the learner receives features describing a loan applicant , from which the learner decides whether or not to issue the requested loan . If the learner decides to issue the loan , the learner , at some point in the future , observes whether or not the applicant repays the loan . If the loan is not issued , no further information is observed by the learner . The process is then repeated in the subsequent time-steps . The learner ’ s objective is to maximize its reward by handing out loans to as many applicants that can repay them and deny loans to applicants unable to pay them back . The BLP can be viewed as a contextual bandit problem with two actions : accepting/rejecting a loan application . Rejection carries a fixed , known , reward of 0 : with no loan issued , there is no dependency on the applicant . If in contrast the learner accepts a loan application , it receives a reward of 1 in case the loan is repaid and suffers a loss of −1 if the loan is not repaid . The probabilities of repayment ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . associated to any individual are not known in advance , thus the learner is required to build an accurate predictive model of these while ensuring not too many loans are handed out to individuals who can ’ t repay them and not too many loans are denied to individuals that can repay . This task can become tricky since the samples used to train the model , which govern the model ’ s future decisions , can suffer from bias as they are the result of past predictions from a potentially incompletely trained model . In the BLP setting , a model can get stuck in a self-fulfilling false rejection loop , in which the very samples that could correct the erroneous model never enter the training data in the first place because they are being rejected by the model . Existing contextual bandit approaches typically assume a known parametric form on the reward function . With restrictions on the reward function , a variety of methods ( [ 13 , 8 ] ) introduce strong theoretical guarantees and empirical results , both in the linear and generalized linear model settings2 . These methods often make use of end-to-end optimism , incorporating uncertainty about the reward function in both the reward model and decision criteria . In practice , however , deep neural networks ( DNNs ) are often used to learn the binary classification model [ 38 , 47 ] , presenting to us a scenario that is vastly richer than the linear and generalized linear model assumptions of many contextual bandit works . In the static setting these methods have achieved effective practical performance , and in the case of [ 47 ] , theoretical guarantees . A large class of these methods use two components : a feature extractor , and a post-hoc exploration strategy fed by the feature extractor . These methods leave the feature extractor itself open to bias , with the limited post-hoc exploration strategy . Another class of methods incorporate uncertainty into the neural network feature extractor ( [ 32 ] ) , building on the vast literature on uncertainty estimation in neural networks . We introduce an algorithm , PLOT ( see Algorithm 1 ) , which explicitly trains DNNs to make optimistic online decisions for the BLP , by incorporating optimism in both representation learning and decision making . The intuition behind PLOT ’ s optimistic optimization procedure is as follows : “ If I trained a model with the query point having a positive label , would it predict it to be positive ? '' If the answer to this question is yes , PLOT would accept this point . To achieve this , at each time step , the PLOT algorithm re-trains its base model , treating the new candidate points as if they had already been accepted and temporarily adds them to the existing dataset with positive pseudo-labels . PLOT ’ s accept and reject decisions are based on the predictions from this optimistically retrained base model . The addition of the fake pseudo-labeled points prevents the emergence of self-fulfilling false negatives . In contrast to false rejections , any false accepts introduced by the pseudo-labels are self-correcting . Once the model has ( optimistically ) accepted enough similar data points and obtained their true , negative label , these will overrule the impact of the optimistic label and result in a reject for novel queries . While conceptually and computationally simple , we empirically show that PLOT obtains competitive performance across a set of 3 different benchmark problems in the BLP domain . With minimal hyperparameter tuning , it matches or outperforms greedy , -greedy ( with a decaying schedule ) and state-of-the-art methods from the literature such as NeuralUCB [ 47 ] . Furthermore , our analysis shows that PLOT is 3-5 times more likely to accept a data point that the current model rejects if the data point is indeed a true accept , compared to a true reject . 1.1 Related Work . Contextual Bandits As we formally describe in Section 2 , the BLP can be formulated as a contextual bandit . Perhaps the most related setting the BLP in the contextual bandits literature is the work of [ 30 ] . In their paper the authors study the loan problem in the presence of offline data . In contrast with our definition for the BLP , their formulation is not concerned with the online decision making component of the BLP , the main point of interest in our formulation . Furthermore , their setting is also concerned with studying ways to satisfy fairness constraints , an aspect of the loan problem that we consider completely orthogonal to our work . Other recent works have forayed into the analysis and deployment of bandit algorithms in the setting of function approximation . Most notably the 2In a linear model the expected response of a point x satisfies ȳ = x > θ , whereas in a generalized linear model the expected response satisfies ȳ = µ ( x > θ ) for some non-linearity µ , typically assumed to be the logistic function . authors of [ 38 ] conduct an extensive empirical evaluation of existing bandit algorithms on public datasets . A more recent line of work has focused on devising ways to add optimism to the predictions of neural network models . Methods such as Neural UCB and Shallow Neural UCB [ 47 , 46 ] are designed to add an optimistic bonus to the model predictions , that is a function of the representation layers of the model . Their theoretical analysis is inspired by insights gained from Neural Tangent Kernel ( NTK ) theory . Other recent works in the contextual bandit literature such as [ 14 ] have started to pose the question of how to extend theoretically valid guarantees into the function approximation scenario , so far with limited success [ 40 ] . A fundamental component of our work that is markedly different from previous approaches to is to explicitly encourage optimism throughout representation learning , rather than post-hoc exploration on top of a non-optimistic representation . Learning with Abstention The literature on learning with abstention shares many features with our setting . In this literature , an online learner can choose to abstain from prediction for a fixed cost , rather than incurring arbitrary regret [ 9 ] . In our setting , a rejected point always receives a constant reward , similar to learning with abstention . However , here , regret in the BLP is measured against the potential reward , rather than against a fixed cost . Although the BLP itself does not naturally admit abstention , the extension of PLOT to abstention setting is an interesting future problem . Repeated Loss Minimization A closely related problem setting to the BLP ( see Section 2 ) is Repeated Loss Minimization . Previous works [ 17 ] have studied the problem of repeated classification settings where the acceptance or rejection decision produces a change in the underlying distribution of individuals faced by the decision maker . In their work , the distributional shift induced by the learner ’ s decisions is assumed to be intrinsic to the dynamics of the world . This line of work has recently garnered a flurry of attention and inspired the formalization of different problem domains such as strategic classification [ 15 ] and performative prediction [ 37 , 31 ] . A common theme in these works is the necessity of thinking strategically about the learner ’ s actions and how these may affect its future decisions as a result of the world reacting to them . In this paper we focus on a different set of problems encountered by decision makers when faced with the BLP in the presence of a reward function . We do not treat the world as a strategic decision maker , instead we treat the distribution of data points presented to the learner as fixed , and focus on understanding the effects that making online decisions can have on the future accuracy and reward experienced by an agent engaged in repeated classification . The main goal in this setting is to devise methods that allow the learner to get trapped in false rejection or false acceptance cycles that may compromise its reward . Thus , the learner ’ s task is not contingent on the arbitrariness of the world , but on a precise understanding of its own knowledge of the world . Learning with partial feedback In [ 39 ] , the authors study the one-sided feedback setting for the application of email spam filtering and show that the approach of [ 19 ] was less effective than a simple greedy strategy . The one-sided feedback setting has in common with our definition of the BLP the assumption that an estimator of the instantaneous regret is only available in the presence of an accept decision . The main difference between the setting of [ 19 ] and ours is that the BLP is defined in the presence of possibly noisy labels . Moreover our algorithm PLOT can be used with powerful function approximators , a setting that goes beyond the simple label generating functions studied in [ 19 ] . In a related work [ 4 ] considers the problem of one-sided learning in the group-based fairness context with the goal of satisfying equal opportunity [ 16 ] at every time-step . They consider convex combinations over a finite set of classifiers and arrive at a solution which is a randomized mixture of at most two of these classifiers . Moving beyond the single player one-sided feedback problem [ 6 ] studies a setting which generalizes the one-sided feedback , called partial monitoring , through considering repeated two-player games in which the player receives a feedback generated by the combined choice of the player and the environment , proposing a randomized solution . [ 2 ] provides a classification of such two-player games in terms of the regret rates attained and [ 3 ] study a variant of the problem with side information .

Summary: This paper first motivates the "bank loan problem", which is a scenario where the learner only observes whether a customer will repay a loan if the loan is issued to begin with. The paper moves on to discussions of this problem and introduces contextual linear bandit formulations. This is extended to the case where the model is a deep neural network, and a Theorem shows theoretical guarantees under the case where data is separable. Experiments show improvement over baseline methods, including the recent NeuralUCB method.