A unified framework for bandit multiple testing

1 Introduction to bandit multiple hypothesis testing . Scientific experimentation is often a sequential process . To test a single null hypothesis — with “ null ” capturing the setting of no scientific interest , and the alternative being scientifically interesting — scientists typically collect an increasing amount of experimental data in order to gather sufficient evidence such that they can potentially reject the null hypothesis ( i.e . make a scientific discovery ) with a high degree of statistical confidence . As long as the collected evidence remains thin , they do not reject the null hypothesis and do not proclaim a discovery . Since executing each additional unit of data ( stemming from an experiment or trial ) has an associated cost ( in the form of time , money , resources ) , the scientist would like to stop as soon as possible . This becomes increasingly prevalent when the scientist is testing multiple hypotheses at the same time , and investing resources into testing one means divesting it from another . For example , consider the case of a scientist at a pharmaceutical company who wants to discover which of several drug candidates under consideration are truly effective ( i.e . testing a hypothesis of whether each candidate has greater than baseline effect ) through an adaptive sequential assignment of drug candidates to participants . Performing follow up studies on each discovery is expensive , so the scientist does not want to make many “ false discoveries ” i.e . drugs that did not have an actual effect , but were proclaimed to have one by the scientist . To achieve these goals , one could imagine 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the scientist collecting more data for candidates whose efficacy is unclear but appear promising ( e.g . drugs with nontrivial but inconclusive evidence ) , and stop sampling candidates that have relatively clear results already ( e.g . drugs that have a clear and large effect , or seemingly no effect ) . Past work . This problem combines the challenges of multiple hypothesis testing with multi-arm bandits ( MABs ) . In a “ doubly-sequential ” version of the problem studied by Yang et al . [ 46 ] , one encounters a sequence of MAB problems over time . Each MAB was used to test a single special placebo arm against several treatment arms , and if at least one treatment dominated the placebo , then they aimed to return the best treatment . Thus each MAB was itself a single adaptive sequential hypothesis test , and the authors aimed not to make too many false discoveries over the sequence of MAB instances . This paper instead considers the formulation of Jamieson and Jain [ 19 ] , henceforth called JJ , but our techniques apply equally well to the above setup . To avoid confusions , note that our setup is very different from the active classification work of the same authors [ 17 ] . To recap , JJ consider a single MAB instance without a placebo arm ( or rather , leaving it implicit ) , and try to identify as many treatments that work better than chance as possible , without too many false identifications . To clarify , we associate each arm with one ( potentially composite ) null hypothesis — for example , the hypothesis that corresponding drug has no ( significant ) effect . A single observed reward when pulling an arm corresponds to a statistic that summarizes the results of one experiment with the corresponding drug , and the average reward across many experiments could correspond to an estimate of the average treatment effect , which would be ( at most ) zero for null arms and positive for non-nulls . Thus , a strategy for quickly finding the arms with positive means corresponds to a strategy for allocating trial patients to drug candidates that allows the scientists to rapidly find the effective drugs . However , the above corresponds to only the simplest problem setting . In more complex settings , it may be possible to pull multiple arms in each round , and observe correlated rewards . Further , the arms may have some combinatorial structure that allows only certain subsets of arms to be pulled . There could be multiple agents ( eg : hospitals ) pulling the same set of arms and seeing independent rewards ( eg : different patients ) or dependent rewards ( eg : patient overlap or interference ) . Further , if some set of experiments by one scientist yielded suggestive but inconclusive evidence , another may want to follow up , but not start from scratch , instead picking up from where the first left off . Last , the MAB may be stopped for a variety of reasons that may or may not be in the control of the scientist ( eg : a faster usage of funding than expected , or additional funding is secured ) . We dive in the details of these scenarios in Appendix D.3 . Our contribution . We introduce a modular meta-algorithm for bandit multiple testing with provable FDR control that utilizes “ e-values ” — or , more appropriately , their sequential analog , “ e-processes ” — a recently introduced alternative to p-values ( or p-processes ) by Ramdas et al . [ 32 ] for various testing problems , that are inherently related to martingales , gambling and betting [ 33 , 14 , 15 , 44 ] . This work is the first to carefully study e-processes in general MAB settings , building on prior work that studied a special case [ 44 ] . We also are the first to extend the bandit multiple testing problem to the combinatorial bandit setting — JJ had previously only analyzed the problem in the single-arm , independent reward setting . Utilizing e-processes provide our meta-algorithm with several benefits . ( a ) For composite nulls , it is typically easier to construct e-processes than p-processes ; the same holds when data from a single source is dependent . When combining evidence from disparate ( independent or dependent ) sources , it is also more straightforward to combine e-values than p-values ( see Appendix D.3 ) . ( b ) The same multiple testing step applies in all bandit multiple testing problems , regardless of all the various details of the problem setup mentioned in the previous paragraph . Consequently , FDR control in our meta-algorithm is agnostic to much of problem setup and can be proved in a vast array of settings . This is not true when working for p-values . In particular , the techniques for proving FDR control in JJ are highly reliant on the specific bandit setup in their paper . ( c ) The exploration step can be — but does not have to be — decoupled from the multiple testing ( combining evidence ) step . This results in a modular procedure that can be easily ported to new problem settings to yield transparent guarantees on FDR control . By virtue of being a meta-algorithm , we do not ( and can not ) provide “ generic ” sample complexity guarantees : these will depend on all of the finer problem details mentioned above , on the exploration algorithm employed , on which e-processes are constructed . Our emphasis is on the flexibility with which FDR control can be guaranteed in a vast variety of problem setups . Further research can pick up one problem at a time and design sensible exploration strategies and stopping rules , developing sampling complexity bounds for each , and these bounds will be inherited by the meta-algorithm . However , we do formulate some generic exploration algorithms in Appendix C based on best arm identification algorithms [ 1 , 22 , 11 , 18 , 23 , 10 , 20 ] . When instantiated to the particular problem setup studied by JJ ( independent , sub-Gaussian rewards , one arm in each round , etc . ) , we get a slightly different algorithm from them — the exploration strategy can be inherited to stay the same , but the multiple testing part differs . JJ use p-processes for each arm to determine whether that arm should be added to the rejection set , and correct for testing multiple hypotheses by using the BH procedure [ 6 ] to ensure that the false discovery rate ( FDR ) , i.e . the proportion of rejections that are false discoveries in expectation , is controlled at some fixed level 𝛿. Adaptive sampling induces a peculiar form of dependence amongst the p-values , for which the BH procedure provides error control at an inflated level ; in other words , one has to use BH at a more stringent level of approximately 𝛿/log ( 16/𝛿 ) to ensure that the FDR is less than 𝛿 . On the other hand , we use the e-BH procedure [ 44 ] , an analogous procedure for e-values , which can ensure the FDR is less than 𝛿 without any inflation , regardless of the dependence structure between the e-values of each arm . Our algorithm has improved sample efficiency in simulations and the same sample complexity in theory . Formal problem setup . We define the bandit as having 𝑘 arms , and 𝜈𝑖 as the ( unknown ) reward distribution for arm 𝑖 ∈ [ 𝑘 ] = { 1 , ... , 𝑘 } . Every arm 𝑖 is associated with a null hypothesis , which is represented by a known , prespecified set of distributions 𝒫𝑖 . If |𝒫𝑖|=1 , it is a ‘ point null hypothesis ’ , and otherwise it is a ‘ composite null hypothesis ’ . Examples of the latter include “ all [ 0,1 ] -bounded distributions with mean≤0.5 ” or “ all 1-sub-Gaussian distributions with mean≤0 ” or “ all distributions that are symmetric around 0 ” or “ all distributions with median ≤ 0 ” . While we assume by default that all rewards from an arm are i.i.d. , we also formulate tests for hypotheses on reward distributions that may violate this assumption in Appendix G. If 𝜈𝑖∈𝒫𝑖 , then we say that the 𝑖-th null hypothesis is true and we call 𝑖 a null arm ; else , we say 𝑖-th null hypothesis is false and we call it a non-null arm . Thus , the set of arms are partitioned into two disjoint sets : nulls ℋ0⊆ [ 𝑘 ] and non-nulls ℋ1 : = [ 𝑘 ] ∖ℋ0 . Let 𝒦 ⊆ 2 [ 𝑘 ] denote the subsets of arms that can be jointly queried in each round . At each time 𝑡 , the algorithm chooses a subset of arms ℐ𝑡 ∈ 𝒦 to sample jointly from . The special choice of 𝒦 = { { 1 } , { 2 } , ... , { 𝑘 } } recovers the standard bandit setup , but otherwise this setting is known as combinatorial bandits with semi-bandit feedback [ 12 ] . We also consider the special case of full-bandit feedback ( the algorithm sees all rewards at each time step ) in Appendix D.1 . We denote the reward sampled at time 𝑡 from arm 𝑖∈ℐ𝑡 as 𝑋𝑖 , 𝑡 . Let 𝑇𝑖 ( 𝑡 ) denote the number of times arm 𝑖 has been sampled by time 𝑡 , and 𝑡𝑖 ( 𝑗 ) be the time of the 𝑗th sample from arm 𝑖 . We now define a canonical “ filtration ” for our bandit problem . A filtration ( ℱ𝑡 ) 𝑡≥0 is a series of nested sigma-algebras that encapsulates what information is known at time 𝑡 . ( We drop the subscript and just write ( ℱ𝑡 ) for brevity , and drop the parentheses when just referring to a single sigma-algebra at time 𝑡 . ) Define the canonical filtration as follows for 𝑡∈N : ℱ𝑡 : =𝜎 ( 𝑈∪ { ( 𝑖 , 𝑠 , 𝑋𝑖 , 𝑗 ) : 𝑠≤ 𝑡 , 𝑖∈ℐ𝑠 } ) and we let ℱ0 : =𝜎 ( 𝑈 ) where 𝑈 is uniformly distributed on [ 0,1 ] and its bits capture all private randomness used by the bandit algorithm that are independent of all observed rewards . Let ( 𝜆𝑡 ) be a sequence of random variables indexed by 𝑡∈N . ( 𝜆𝑡 ) is said to be predictable w.r.t . ( ℱ𝑡 ) if 𝜆𝑡 is measurable w.r.t . ℱ𝑡−1 i.e . 𝜆𝑡 is fully specified given the information in ℱ𝑡−1 . An N-valued random variable 𝜏 is a stopping time ( or stopping rule ) w.r.t . to ( ℱ𝑡 ) if { 𝜏 = 𝑡 } ∈ℱ𝑡 — in other words , at each time 𝑡 , we know whether or not to stop collecting data . Let 𝒯 denote the set of all possible stopping times/rules w.r.t . ( ℱ𝑡 ) , potentially infinite . Technically , the algorithm must not just specify a strategy to select ℐ𝑡 , but also specify when sampling will stop . This is denoted by the stopping rule or stopping time 𝜏 * ∈𝒯 . Once the algorithm halts at some time 𝜏 , it produces a rejection set 𝒮𝜏 ⊆ [ 𝑘 ] . We consider two metrics w.r.t . 𝒮 : the FDR as discussed prior , and true positive rate ( TPR ) , which is the proportion of non-nulls that are discovered in expectation . These two metrics are defined as follows : FDR ( 𝒮𝜏 ) : = E [ ︁ |ℋ0∩𝒮𝜏 | |𝒮𝜏 |∨1 ] ︁ , TPR ( 𝒮𝜏 ) : = E [ ︁ |ℋ1∩𝒮𝜏 | |ℋ1| ] ︁ . We consider algorithms that always satisfy FDR ( 𝒮𝜏 ) ≤𝛿 for any number and configuration of nulls ℋ0 and any choice of null and non-null distributions . In fact , our algorithm will produce a sequence of candidate rejection sets ( 𝒮𝑡 ) that satisfies sup𝜏∈𝒯 FDR ( 𝒮𝜏 ) ≤𝛿 . This is a much stronger guarantee than the typical setting considered in the multiple testing literature . On the other hand , TPR is a measurement of the power of the algorithm i.e . how many of the non-null hypotheses does the algorithm discover . Our implicit goal in the multiple testing problem is to maximize the number of true discoveries while not making too many mistakes i.e . keep the FDR controlled . In hypothesis testing , the set of null distributions 𝒫𝑖 for each arm 𝑖 is known , because the user defines the null hypothesis they are interested in testing . When the null hypothesis is false , the non-null distribution can be arbitrary . Consequently , we can prove results about FDR , but we can not prove guarantees about TPR without several further assumptions on the non-null distributions , dependence across arms , etc . For a particular setting where we make such a set of assumptions , we demonstrate in Section 4 that we can prove TPR guarantees for algorithms within our framework . Hence , our FDR controlling framework is not vacuous as it includes powerful algorithms i.e . algorithms which make many true discoveries . However , our focus is primarily to show that the FDR control of our framework is robust to a wide range of conditions . Finally , note that in bandit multiple testing , one does not care about regret . The problem is more akin to pure exploration , where we aim to find a 𝒮 with FDR ( 𝒮𝜏 * ) ≤𝛿 and large TPR as quickly as possible . Now that we have specified the problem we are interested in , we can introduce our main technical tools for ensuring FDR control at stopping times : e-processes and p-processes .

Summary: The paper proposes a general framework for multiple hypothesis testing with data generated from bandit algorithms. In particular, the paper proposes a new e-value-based multiple testing method that 1) has a tighter FDR bound and 2) applies to more general settings, compared with previous p-valued-based methods (Jamieson and Jain, 2018). In a special case, the paper investigates the efficiency of its proposed algorithm and shows that the sample complexity needed for the TPR to be above the pre-specified threshold matches that of previous work (Jamieson and Jain, 2018). Additionally, the paper improves the FDR bound of the p-value-based method (Jamieson and Jain, 2018) by leveraging results from Su (2018).