ROI Maximization in Stochastic Online Decision-Making

{ 1/ ( N∆2 ) , N−1/3 } , where N is the number of innovations and ∆ is the suboptimality gap in Π . A significant hurdle of our formulation , which sets it aside from other online learning problems such as bandits , is that running a policy does not provide an unbiased estimate of its performance . 1 Introduction . Often , companies have to make yes/no decisions , such as whether to adopt a new technology or retire an old product . However , finding out the best option in all circumstances could mean spending too much time or money in the evaluation process . If the decisions to make are many , one could be better off making more of them quickly and inexpensively , provided that these decisions have an overall positive effect . In this paper , we investigate the problem of determining a decision policy to balance the reward over cost ratio optimally ( i.e. , to maximize the return on investment ) . A motivating example . Consider a technology company that keeps testing innovations to increase some chosen metric ( e.g. , benefits , gross revenue , revenue excluding the traffic acquisition cost ) . Before deploying an innovation , the company wants to figure out whether it is profitable . As long as each innovation can be tested on i.i.d . samples of users , the company can perform randomized tests and make statistically sound decisions . However , there is an incentive to make these tests run as quickly as possible because , for example , the testing process is expensive . Another reason could be that keeping a team on a project that has negative , neutral , or even borderline positive potential prevents it from testing other ideas that might lead to a significantly better improvement . In other words , it is crucial to learn when to drop barely positive innovations in favor of highly positive ones , so to increase the overall flow of improvement over time ( i.e. , the ROI of the tests ) . More generally , our framework describes problems where an agent faces a sequence of decision tasks consisting of either accepting or rejecting an innovation . Before making each decision , the agent can invest resources into reducing the uncertainty on the value brought by the innovation . The global objective is to maximize the total ROI . Namely , the ratio between the total value accumulated by accepting innovations and the total cost . For an in-depth discussion on alternative goals , we refer the reader to the Supplementary Material ( Section D ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The model . Each task n in the sequence is associated with a pair ( µn , Dn ) that the learner can never directly observe . • µn is a random variable representing the ( possibly negative ) true value of the n-th innovation . • Dn is a probability distribution over the real numbers with expectation µn , modeling the feedback on the n-th innovation that the learner can gather from testing ( see below ) . During the n-th task , the learner can draw arbitrarily many i.i.d . samples Xn,1 , Xn,2 , . . . from Dn , accumulating information on the unknown value µn of the innovation currently being tested . After stopping drawing samples , the learner can decide to either accept the innovation , earning µn as a reward , or reject it and gain nothing instead . We measure the agent performance during N tasks as the ( expected ) total amount of value accumulated by accepting innovations µn divided by the ( expected ) total number of samples requested throughout all tasks . In formulas , the ROI of a strategy is then∑N n=1 E [ µn · I { µn is accepted } ] ∑N n=1 E [ number of samples requested during the n-th task ] In Section 3 we present this setting in more detail and introduce the relevant notation . I.I.D . assumption . We assume that the value µn of the n-th innovation is drawn i.i.d . from an unknown and fixed distribution . This assumption is meaningful if past decisions do not influence future innovations whose global quality remains stable over time . In particular , it applies whenever innovations can progress in many orthogonal directions , each yielding a similar added value ( e.g. , when different teams within the same company test improvements relative to individual aspects of the company ) . If both the state of the agent and that of the environment evolve , but the ratio of good versus bad innovations remains essentially the same , then this i.i.d . assumption is still justified . In other words , it is not necessarily the absolute quality of the innovations that has to remain stationary , but rather the relative added value of the innovations given the current state of the system . This case is frequent in practice , especially when a system is close to its technological limit . Last but not least , algorithms designed under stochastic assumptions often perform surprisingly well in practice , even if i.i.d . assumptions are not fully satisfied or simply hard to check . A baseline strategy and policy classes . A natural , yet suboptimal , approach for deciding if an innovation is worth accepting is to gather samples sequentially , stopping as soon as the absolute value of their running average surpasses a threshold , and then accepting the innovation if and only if the average is positive . The major drawback of this approach is that the value µn of an innovation n could be arbitrarily close to zero . In this case , the number of samples needed to reliably determine its sign ( which is of order 1/µ2n ) would become prohibitively large . This would result is a massive time investment for an innovation whose return is negligible at best . In hindsight , it would have been better to reject the innovation early and move on to the next task . For this reason , testing processes in practice needs hard termination rules of the form : if after drawing a certain number of samples no confident decision can be taken , then terminate the testing process rejecting the current innovation . Denote by τ this capped early stopping rule and by accept the accept/reject decision rule that comes with it . We say that the pair π = ( τ , accept ) is a policy because it fully characterize the decision-making process for an innovation . Policies defined by capped early stopping rules ( see ( 4 ) for a concrete example ) are of great practical importance ( Johari et al. , 2017 ; Kohavi et al. , 2013 ) . However , policies can be defined more generally by any reasonable stopping rule and decision function . Given a ( possibly infinite ) set of policies , and assuming that µ1 , µ2 , . . . are drawn i.i.d . from some unknown but fixed distribution , the goal is to learn efficiently , at the lowest cost , the best policy π ? in the set with respect to a sensible metric . Competing against fixed policy classes is a common modeling choice that allows to express the intrinsic constraints that are imposed by the nature of the decision-making problem . For example , even if some policies outside of the class could theoretically yield better performance , they might not be implementable because of time , budget , fairness , or technology constraints . Challenges . One of the biggest challenges arising in our framework is that running a decisionmaking policy generates a collection of samples that —in general— can not be used to form an unbiased estimate of the policy reward ( see the impossibility result in Section E of the Supplementary Material ) . The presence of this bias is a significant departure from settings like multiarmed and firing bandits ( Auer et al. , 2002 ; Jain and Jamieson , 2018 ) , where the learner observes an unbiased sample of the target quantity at the end of every round ( see the next section for additional details ) . Moreover , contrary to standard online learning problems , the relevant performance measure is neither additive in the number of innovations nor in the number of samples per innovation . Therefore , algorithms have to be analyzed globally , and bandit-like techniques —in which the regret is additive over rounds— can not be directly applied . We argue that these technical difficulties are a worthy price to pay in order to define a plausible setting , applicable to real-life scenarios . Main contributions . The first contribution of this paper is providing a mathematical formalization of our ROI maximization setting for repeated decision making ( Section 3 ) . We then design an algorithm called Capped Policy Elimination ( Algorithm 1 , CAPE ) that applies to finite policy classes ( Section 4 ) . We prove that CAPE converges to the optimal policy at rate 1/ ( ∆2N ) , where N is the number of tasks and ∆ is the unknown gap between the performance of the two best policies , and at rate N−1/3 when ∆ is small ( Theorem 1 ) . In Section 5 we tackle the challenging problem of infinitely large policy classes . For this setting , we design a preprocessing step ( Algorithm 2 , ESC ) that leads to the ESC-CAPE algorithm . We prove that this algorithm converges to the optimal policy in an infinite set at a rate of N−1/3 ( Theorem 4 ) . Limitations . Although we do not investigate lower bounds in this paper , we conjecture that our N−1/3 convergence rate it is optimal due to similarities with bandits with weakly observable feedback graphs ( see Section 4 , “ Divided we fall ” ) . Another limitation of our theory is that it only applies to i.i.d . sequences of values µn . It would be interesting to extend our analysis to distributions of µn that evolve over time . These two intriguing problems are left open for future research . 2 Related Work . Return on Investment ( ROI ) was developed and popularized by Donaldson Brown in the early Nineties ( Flesher and Previts , 2013 ) and it is still considered an extremely valuable metric by the overwhelming majority of marketing managers ( Farris et al. , 2010 ) . Beyond economics , mathematics , and computer science , ROI finds applications in other fields , such as cognitive science and psychology ( Chabris et al. , 2009 ) . Despite this , to the best of our knowledge , no theoretical online learning framework has been developed specifically for ROI maximization . However , our novel formalization of this sequential decision problem does share some similarities with other known online learning settings . In this section , we review the relevant literature regarding these settings and stress the differences with ours . Prophet inequalities and Pandora ’ s box . In prophet inequalities ( Lucier , 2017 ; Correa et al. , 2019 ; Alaei et al. , 2012 ) , an agent observes sequentially ( usually non-negative ) random variables Z1 , . . . , Zn and decides to stop at some time τ ; the reward is then Zτ . Variants include the possibility of choosing more than one random variable ( in which case the reward is some function of the selected random variables ) , and the possibility to go back in time ( to some extent ) . The Pandora ’ s box problem is slightly different ( Weitzman , 1979 ; Kleinberg et al. , 2016 ; Esfandiari et al. , 2019 ) ; in its original formulation , the agent can pay a cost cn ≥ 0 to observe any Zn . After stopping exploring , the agent ’ s final utility is the maximum of the observed Zn ’ s minus the cumulative cost ( or , in other variants , some function of these ) . Similarly to the ( general ) prophet inequality , the agent in our sequential problem faces random variables ( Zn = µn in our notation ) and sequentially selects any number of them ( possibly with negative values ) without the possibility to go back in time and change past decisions . The significant difference is that the agent in our setting never observes the value of µn . In Pandora ’ s box , the agent can see this value by paying some price ( that approximately scales as 1/ε2 where ε is the required precision ) . Finally , the global reward is the cumulative sum ( as in prophets ) and not the maximum ( as in Pandora ’ s box ) of the selected variables , normalized by the total cost ( as in Pandora ’ s box , but our normalization is multiplicative instead of additive , as it represents a ROI ) . Multi-armed bandits . If we think of the set of all policies used by the agent to determine whether or not to accept innovations as arms , our setting becomes somewhat reminiscent of multi-armed bandits ( Slivkins , 2019 ; Bubeck and Cesa-Bianchi , 2012 ; Rosenberg et al. , 2007 ) . However , there are several notable differences between these two problems . In stochastic bandits , the agent observes an unbiased estimate of the expected reward of each pulled arm . In our setting , the agent not only does not see it directly , but it is mathematically impossible to define such an estimator solely with the feedback received ( see the impossibility result in Section E of the Supplementary Material ) . Hence , off-the-shelf bandit algorithms can not be run to solve our problem . In addition , the objective in bandits is to maximize the cumulative reward , which is additive over time , while the ROI is not . Thus , it is unclear how formal guarantees for bandit algorithms would translate to our problem . We could also see firing bandits ( Jain and Jamieson , 2018 ) as a variant of our problem , where µn belongs to [ 0 , 1 ] , Dn are Bernoulli distribution with parameter µn , and policies have a specific form that allows to easily define unbiased estimates of their rewards ( which , we reiterate , is not possible in our setting in general ) . Furthermore , in firing bandits , it is possible to go back and forth in time , sampling from any of the past distributions Dn and gathering any number of samples from it . This is a reasonable assumption for the original motivations of firing bandits because the authors thought of µn as the value of a project in a crowdfunding platform , and , in their setting , drawing samples from Dn corresponds to displaying projects on web pages . However , in our setting , µn represents the theoretical increment ( or decrement ) of a company ’ s profit through a given innovation , and it is unlikely that a company would show new interest in investing in a technology that has been tested before and did not prove to be useful ( a killed project is seldom re-launched ) . Hence , when the sampling of Dn stops , an irrevocable decision is made . After that , the learner can not draw any more samples in the future . Finally , as in multi-armed bandits , the performance criterion in firing bandits is the cumulative reward and not the global ROI . Another online problem that shares some similarities with ours is bandits with knapsacks ( Badanidiyuru et al. , 2018 ) . In this problem , playing an arm consumes one unit of time together with some other resources , and the learner receives an unbiased estimate of its reward as feedback . The process ends as soon as time or any one of the other resources is exhausted . As usual , the goal is to maximize the cumulative regret . As it turns out , we can also think of our problem as a budgeted problem . In this restatement , there is a budget of T samples . The repeated decision-making process proceeds as before , but it stops as soon as the learner has drawn a total of T samples across all decision tasks . The goal is again to maximize the total expected reward of accepted innovations divided by T ( see Section D of the Supplementary Material for more details on the reduction ) . As per the other bandit problems , there are two crucial differences . First , running a policy does not reveal an unbiased estimate of its reward . Second , our objective is different , and regret bounds do not directly imply convergence to optimal ROI . For other examples of budget-constrained bandit settings , see also ( Cayci et al. , 2020 , 2019 ) . Repeated A/B testing . We can view our problem as a framework for repeated A/B testing ( Tukey , 1953 ; Genovese et al. , 2006 ; Foster and Stine , 2008 ; Heesen and Janssen , 2016 ; Javanmard et al. , 2018 ; Azevedo et al. , 2018 ; Li and Barber , 2019 ; Schmit et al. , 2019 ; Xu et al. , 2021 ; Johari et al. , 2021 ) , in which assessing the value of an innovation corresponds to performing an A/B test , and the goal is maximizing the ROI . A popular metric to optimize sequential A/B tests is the so-called false discovery rate ( FDR ) —see ( Ramdas et al. , 2017 ; Yang et al. , 2017 ) and references therein . Roughly speaking , the FDR is the ratio of accepted µn that are negative over the total number of accepted µn ( or more generally , the number of incorrectly accepted tests over the total number if the metric used at each test changes with time ) . This , unfortunately , disregards the relative values of tests µn that must be taken into account when optimizing a single metric ( Chen and Kasiviswanathan , 2020 ; Robertson and Wason , 2018 ) . Indeed , the effect of many even slightly negative accepted tests could be overcome by a few largely positive ones . For instance , assume that the samples Xn , i of any distribution Dn belong to { −1 , 1 } , and that their expected value µn is uniformly distributed on { −ε , ε } . To control the FDR , each A/B test should be run for approximately 1/ε2 times , yielding a ratio of the average value of an accepted test to the number of samples of order ε3 . A better strategy , using just one sample from each A/B test , is simply to accept µn if and only if the first sample is positive . Direct computations show that this policy , which fits our setting , achieves a significantly better performance of order ε . Some other A/B testing settings are more closely related to ours , but make stronger additional assumptions or suppose preliminary knowledge : for example , smoothness assumptions can be made on both Dn and the distributions of µn ( Azevedo et al. , 2018 ) , or the distribution of µn is known , and the distribution of samples belongs to a single parameter exponential family , also known beforehand ( Schmit et al. , 2019 ) . A related topic that sits in between multi-armed bandits and repeated A/B testing is best-arm identification—see , e.g. , the recent paper ( Garivier and Kaufmann , 2021 ) and references therein— where the learner has to minimize the number of observations needed to select a ( near ) -optimal arm . Rational metareasoning . Our setting is loosely related to the AI field of meta-reasoning ( Griffiths et al. , 2019 ; Hay et al. , 2012 ) . In a metalevel decision problem , determining the utility ( or reward ) of a given action is computationally intractable . Instead , the learner can run a simulation , investing a computational cost to gather information about this hidden value . The high-level idea is then to learn which actions to simulate . After running some simulations , the learner picks an action to play , gains the corresponding ( hidden ) reward , and the state of the system changes . In rational meta-reasoning , the performance measure is the value of computation ( VOC ) : the difference between the increment in expected utility gained by executing a simulation and the cost incurred by doing so . This setting is not directly comparable to ours for two reasons . First , the performance measure is different , and the additive nature of the difference that defines the VOC gives no guarantees on our multiplicative notion of ROI . Second , in this problem , one can pick which actions to simulate , while in our settings , innovations come independently of the learner , who has to evaluate them in that order .

Summary: This paper studies the problem of Return On Investment (ROI) maximization in repeated decision-making. The main contributions are as follows.  The authors first propose a framework for ROI maximization in Sec. 3. The main contribution here is to formulate an objective/regret in Eq. (2), which is a ratio of reward over cost. The problem is then to maximize the average reward-cost ratio of $N$ tasks, where each task's value is an i.i.d. sample from a true $\mu$.  When the policy class is finite with $K > 0$ policies, the authors propose the Capped Policy Elimination (CAPE) in Algorithm 1, and show that its convergence rate is $\tilde{O}(1/N^{1/3})$ in Theorem 1. The main idea is to make a decision first, then use oversampling to get i.i.d. samples of the reward and cost, and then use concentration inequality to estimate the performance gap $\Delta$ of policies to do policy elimination.  When the policy class is countably infinite, the authors propose the : Extension to Countable (ESC) in Algorithm 2 to reduce the policy class to finite. The main idea is to estimate the upper bound of the optimal policy's cost, such that any optimal policy would take smaller number of observations. Then CAPE is used on the reduced policy class, and a similar $\tilde{O}(1/N^{1/3})$ convergence rate upper bound is shown in Theorem 4.