ROI Maximization in Stochastic Online Decision-Making

{ 1/ ( N∆2 ) , N−1/3 } , where N is the number of innovations and ∆ is the suboptimality gap in Π . A significant hurdle of our formulation , which sets it aside from other online learning problems such as bandits , is that running a policy does not provide an unbiased estimate of its performance . 1 Introduction . Often , companies have to make yes/no decisions , such as whether to adopt a new technology or retire an old product . However , finding out the best option in all circumstances could mean spending too much time or money in the evaluation process . If the decisions to make are many , one could be better off making more of them quickly and inexpensively , provided that these decisions have an overall positive effect . In this paper , we investigate the problem of determining a decision policy to balance the reward over cost ratio optimally ( i.e. , to maximize the return on investment ) . A motivating example . Consider a technology company that keeps testing innovations to increase some chosen metric ( e.g. , benefits , gross revenue , revenue excluding the traffic acquisition cost ) . Before deploying an innovation , the company wants to figure out whether it is profitable . As long as each innovation can be tested on i.i.d . samples of users , the company can perform randomized tests and make statistically sound decisions . However , there is an incentive to make these tests run as quickly as possible because , for example , the testing process is expensive . Another reason could be that keeping a team on a project that has negative , neutral , or even borderline positive potential prevents it from testing other ideas that might lead to a significantly better improvement . In other words , it is crucial to learn when to drop barely positive innovations in favor of highly positive ones , so to increase the overall flow of improvement over time ( i.e. , the ROI of the tests ) . More generally , our framework describes problems where an agent faces a sequence of decision tasks consisting of either accepting or rejecting an innovation . Before making each decision , the agent can invest resources into reducing the uncertainty on the value brought by the innovation . The global objective is to maximize the total ROI . Namely , the ratio between the total value accumulated by accepting innovations and the total cost . For an in-depth discussion on alternative goals , we refer the reader to the Supplementary Material ( Section D ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The model . Each task n in the sequence is associated with a pair ( µn , Dn ) that the learner can never directly observe . • µn is a random variable representing the ( possibly negative ) true value of the n-th innovation . • Dn is a probability distribution over the real numbers with expectation µn , modeling the feedback on the n-th innovation that the learner can gather from testing ( see below ) . During the n-th task , the learner can draw arbitrarily many i.i.d . samples Xn,1 , Xn,2 , . . . from Dn , accumulating information on the unknown value µn of the innovation currently being tested . After stopping drawing samples , the learner can decide to either accept the innovation , earning µn as a reward , or reject it and gain nothing instead . We measure the agent performance during N tasks as the ( expected ) total amount of value accumulated by accepting innovations µn divided by the ( expected ) total number of samples requested throughout all tasks . In formulas , the ROI of a strategy is then∑N n=1 E [ µn · I { µn is accepted } ] ∑N n=1 E [ number of samples requested during the n-th task ] In Section 3 we present this setting in more detail and introduce the relevant notation . I.I.D . assumption . We assume that the value µn of the n-th innovation is drawn i.i.d . from an unknown and fixed distribution . This assumption is meaningful if past decisions do not influence future innovations whose global quality remains stable over time . In particular , it applies whenever innovations can progress in many orthogonal directions , each yielding a similar added value ( e.g. , when different teams within the same company test improvements relative to individual aspects of the company ) . If both the state of the agent and that of the environment evolve , but the ratio of good versus bad innovations remains essentially the same , then this i.i.d . assumption is still justified . In other words , it is not necessarily the absolute quality of the innovations that has to remain stationary , but rather the relative added value of the innovations given the current state of the system . This case is frequent in practice , especially when a system is close to its technological limit . Last but not least , algorithms designed under stochastic assumptions often perform surprisingly well in practice , even if i.i.d . assumptions are not fully satisfied or simply hard to check . A baseline strategy and policy classes . A natural , yet suboptimal , approach for deciding if an innovation is worth accepting is to gather samples sequentially , stopping as soon as the absolute value of their running average surpasses a threshold , and then accepting the innovation if and only if the average is positive . The major drawback of this approach is that the value µn of an innovation n could be arbitrarily close to zero . In this case , the number of samples needed to reliably determine its sign ( which is of order 1/µ2n ) would become prohibitively large . This would result is a massive time investment for an innovation whose return is negligible at best . In hindsight , it would have been better to reject the innovation early and move on to the next task . For this reason , testing processes in practice needs hard termination rules of the form : if after drawing a certain number of samples no confident decision can be taken , then terminate the testing process rejecting the current innovation . Denote by τ this capped early stopping rule and by accept the accept/reject decision rule that comes with it . We say that the pair π = ( τ , accept ) is a policy because it fully characterize the decision-making process for an innovation . Policies defined by capped early stopping rules ( see ( 4 ) for a concrete example ) are of great practical importance ( Johari et al. , 2017 ; Kohavi et al. , 2013 ) . However , policies can be defined more generally by any reasonable stopping rule and decision function . Given a ( possibly infinite ) set of policies , and assuming that µ1 , µ2 , . . . are drawn i.i.d . from some unknown but fixed distribution , the goal is to learn efficiently , at the lowest cost , the best policy π ? in the set with respect to a sensible metric . Competing against fixed policy classes is a common modeling choice that allows to express the intrinsic constraints that are imposed by the nature of the decision-making problem . For example , even if some policies outside of the class could theoretically yield better performance , they might not be implementable because of time , budget , fairness , or technology constraints . Challenges . One of the biggest challenges arising in our framework is that running a decisionmaking policy generates a collection of samples that —in general— can not be used to form an unbiased estimate of the policy reward ( see the impossibility result in Section E of the Supplementary Material ) . The presence of this bias is a significant departure from settings like multiarmed and firing bandits ( Auer et al. , 2002 ; Jain and Jamieson , 2018 ) , where the learner observes an unbiased sample of the target quantity at the end of every round ( see the next section for additional details ) . Moreover , contrary to standard online learning problems , the relevant performance measure is neither additive in the number of innovations nor in the number of samples per innovation . Therefore , algorithms have to be analyzed globally , and bandit-like techniques —in which the regret is additive over rounds— can not be directly applied . We argue that these technical difficulties are a worthy price to pay in order to define a plausible setting , applicable to real-life scenarios . Main contributions . The first contribution of this paper is providing a mathematical formalization of our ROI maximization setting for repeated decision making ( Section 3 ) . We then design an algorithm called Capped Policy Elimination ( Algorithm 1 , CAPE ) that applies to finite policy classes ( Section 4 ) . We prove that CAPE converges to the optimal policy at rate 1/ ( ∆2N ) , where N is the number of tasks and ∆ is the unknown gap between the performance of the two best policies , and at rate N−1/3 when ∆ is small ( Theorem 1 ) . In Section 5 we tackle the challenging problem of infinitely large policy classes . For this setting , we design a preprocessing step ( Algorithm 2 , ESC ) that leads to the ESC-CAPE algorithm . We prove that this algorithm converges to the optimal policy in an infinite set at a rate of N−1/3 ( Theorem 4 ) . Limitations . Although we do not investigate lower bounds in this paper , we conjecture that our N−1/3 convergence rate it is optimal due to similarities with bandits with weakly observable feedback graphs ( see Section 4 , “ Divided we fall ” ) . Another limitation of our theory is that it only applies to i.i.d . sequences of values µn . It would be interesting to extend our analysis to distributions of µn that evolve over time . These two intriguing problems are left open for future research . 2 Related Work . Return on Investment ( ROI ) was developed and popularized by Donaldson Brown in the early Nineties ( Flesher and Previts , 2013 ) and it is still considered an extremely valuable metric by the overwhelming majority of marketing managers ( Farris et al. , 2010 ) . Beyond economics , mathematics , and computer science , ROI finds applications in other fields , such as cognitive science and psychology ( Chabris et al. , 2009 ) . Despite this , to the best of our knowledge , no theoretical online learning framework has been developed specifically for ROI maximization . However , our novel formalization of this sequential decision problem does share some similarities with other known online learning settings . In this section , we review the relevant literature regarding these settings and stress the differences with ours . Prophet inequalities and Pandora ’ s box . In prophet inequalities ( Lucier , 2017 ; Correa et al. , 2019 ; Alaei et al. , 2012 ) , an agent observes sequentially ( usually non-negative ) random variables Z1 , . . . , Zn and decides to stop at some time τ ; the reward is then Zτ . Variants include the possibility of choosing more than one random variable ( in which case the reward is some function of the selected random variables ) , and the possibility to go back in time ( to some extent ) . The Pandora ’ s box problem is slightly different ( Weitzman , 1979 ; Kleinberg et al. , 2016 ; Esfandiari et al. , 2019 ) ; in its original formulation , the agent can pay a cost cn ≥ 0 to observe any Zn . After stopping exploring , the agent ’ s final utility is the maximum of the observed Zn ’ s minus the cumulative cost ( or , in other variants , some function of these ) . Similarly to the ( general ) prophet inequality , the agent in our sequential problem faces random variables ( Zn = µn in our notation ) and sequentially selects any number of them ( possibly with negative values ) without the possibility to go back in time and change past decisions . The significant difference is that the agent in our setting never observes the value of µn . In Pandora ’ s box , the agent can see this value by paying some price ( that approximately scales as 1/ε2 where ε is the required precision ) . Finally , the global reward is the cumulative sum ( as in prophets ) and not the maximum ( as in Pandora ’ s box ) of the selected variables , normalized by the total cost ( as in Pandora ’ s box , but our normalization is multiplicative instead of additive , as it represents a ROI ) . Multi-armed bandits . If we think of the set of all policies used by the agent to determine whether or not to accept innovations as arms , our setting becomes somewhat reminiscent of multi-armed bandits ( Slivkins , 2019 ; Bubeck and Cesa-Bianchi , 2012 ; Rosenberg et al. , 2007 ) . However , there are several notable differences between these two problems . In stochastic bandits , the agent observes an unbiased estimate of the expected reward of each pulled arm . In our setting , the agent not only does not see it directly , but it is mathematically impossible to define such an estimator solely with the feedback received ( see the impossibility result in Section E of the Supplementary Material ) . Hence , off-the-shelf bandit algorithms can not be run to solve our problem . In addition , the objective in bandits is to maximize the cumulative reward , which is additive over time , while the ROI is not . Thus , it is unclear how formal guarantees for bandit algorithms would translate to our problem . We could also see firing bandits ( Jain and Jamieson , 2018 ) as a variant of our problem , where µn belongs to [ 0 , 1 ] , Dn are Bernoulli distribution with parameter µn , and policies have a specific form that allows to easily define unbiased estimates of their rewards ( which , we reiterate , is not possible in our setting in general ) . Furthermore , in firing bandits , it is possible to go back and forth in time , sampling from any of the past distributions Dn and gathering any number of samples from it . This is a reasonable assumption for the original motivations of firing bandits because the authors thought of µn as the value of a project in a crowdfunding platform , and , in their setting , drawing samples from Dn corresponds to displaying projects on web pages . However , in our setting , µn represents the theoretical increment ( or decrement ) of a company ’ s profit through a given innovation , and it is unlikely that a company would show new interest in investing in a technology that has been tested before and did not prove to be useful ( a killed project is seldom re-launched ) . Hence , when the sampling of Dn stops , an irrevocable decision is made . After that , the learner can not draw any more samples in the future . Finally , as in multi-armed bandits , the performance criterion in firing bandits is the cumulative reward and not the global ROI . Another online problem that shares some similarities with ours is bandits with knapsacks ( Badanidiyuru et al. , 2018 ) . In this problem , playing an arm consumes one unit of time together with some other resources , and the learner receives an unbiased estimate of its reward as feedback . The process ends as soon as time or any one of the other resources is exhausted . As usual , the goal is to maximize the cumulative regret . As it turns out , we can also think of our problem as a budgeted problem . In this restatement , there is a budget of T samples . The repeated decision-making process proceeds as before , but it stops as soon as the learner has drawn a total of T samples across all decision tasks . The goal is again to maximize the total expected reward of accepted innovations divided by T ( see Section D of the Supplementary Material for more details on the reduction ) . As per the other bandit problems , there are two crucial differences . First , running a policy does not reveal an unbiased estimate of its reward . Second , our objective is different , and regret bounds do not directly imply convergence to optimal ROI . For other examples of budget-constrained bandit settings , see also ( Cayci et al. , 2020 , 2019 ) . Repeated A/B testing . We can view our problem as a framework for repeated A/B testing ( Tukey , 1953 ; Genovese et al. , 2006 ; Foster and Stine , 2008 ; Heesen and Janssen , 2016 ; Javanmard et al. , 2018 ; Azevedo et al. , 2018 ; Li and Barber , 2019 ; Schmit et al. , 2019 ; Xu et al. , 2021 ; Johari et al. , 2021 ) , in which assessing the value of an innovation corresponds to performing an A/B test , and the goal is maximizing the ROI . A popular metric to optimize sequential A/B tests is the so-called false discovery rate ( FDR ) —see ( Ramdas et al. , 2017 ; Yang et al. , 2017 ) and references therein . Roughly speaking , the FDR is the ratio of accepted µn that are negative over the total number of accepted µn ( or more generally , the number of incorrectly accepted tests over the total number if the metric used at each test changes with time ) . This , unfortunately , disregards the relative values of tests µn that must be taken into account when optimizing a single metric ( Chen and Kasiviswanathan , 2020 ; Robertson and Wason , 2018 ) . Indeed , the effect of many even slightly negative accepted tests could be overcome by a few largely positive ones . For instance , assume that the samples Xn , i of any distribution Dn belong to { −1 , 1 } , and that their expected value µn is uniformly distributed on { −ε , ε } . To control the FDR , each A/B test should be run for approximately 1/ε2 times , yielding a ratio of the average value of an accepted test to the number of samples of order ε3 . A better strategy , using just one sample from each A/B test , is simply to accept µn if and only if the first sample is positive . Direct computations show that this policy , which fits our setting , achieves a significantly better performance of order ε . Some other A/B testing settings are more closely related to ours , but make stronger additional assumptions or suppose preliminary knowledge : for example , smoothness assumptions can be made on both Dn and the distributions of µn ( Azevedo et al. , 2018 ) , or the distribution of µn is known , and the distribution of samples belongs to a single parameter exponential family , also known beforehand ( Schmit et al. , 2019 ) . A related topic that sits in between multi-armed bandits and repeated A/B testing is best-arm identification—see , e.g. , the recent paper ( Garivier and Kaufmann , 2021 ) and references therein— where the learner has to minimize the number of observations needed to select a ( near ) -optimal arm . Rational metareasoning . Our setting is loosely related to the AI field of meta-reasoning ( Griffiths et al. , 2019 ; Hay et al. , 2012 ) . In a metalevel decision problem , determining the utility ( or reward ) of a given action is computationally intractable . Instead , the learner can run a simulation , investing a computational cost to gather information about this hidden value . The high-level idea is then to learn which actions to simulate . After running some simulations , the learner picks an action to play , gains the corresponding ( hidden ) reward , and the state of the system changes . In rational meta-reasoning , the performance measure is the value of computation ( VOC ) : the difference between the increment in expected utility gained by executing a simulation and the cost incurred by doing so . This setting is not directly comparable to ours for two reasons . First , the performance measure is different , and the additive nature of the difference that defines the VOC gives no guarantees on our multiplicative notion of ROI . Second , in this problem , one can pick which actions to simulate , while in our settings , innovations come independently of the learner , who has to evaluate them in that order .

Summary: This paper considers a sequential decision process that models that of ideation. At any time an agent has access to a reward distribution and they can either take a sample from the distribution, accept the distribution and receive a reward equivalent to the mean of the distribution, or reject the distribution and receive the reward of 0. The goal of the agent is to find a policy that maximizes their expected ROI, that is the ratio of the total reward of their policies over the total cost (expected number of samples a policy takes).