A unified framework for bandit multiple testing

1 Introduction to bandit multiple hypothesis testing . Scientific experimentation is often a sequential process . To test a single null hypothesis — with “ null ” capturing the setting of no scientific interest , and the alternative being scientifically interesting — scientists typically collect an increasing amount of experimental data in order to gather sufficient evidence such that they can potentially reject the null hypothesis ( i.e . make a scientific discovery ) with a high degree of statistical confidence . As long as the collected evidence remains thin , they do not reject the null hypothesis and do not proclaim a discovery . Since executing each additional unit of data ( stemming from an experiment or trial ) has an associated cost ( in the form of time , money , resources ) , the scientist would like to stop as soon as possible . This becomes increasingly prevalent when the scientist is testing multiple hypotheses at the same time , and investing resources into testing one means divesting it from another . For example , consider the case of a scientist at a pharmaceutical company who wants to discover which of several drug candidates under consideration are truly effective ( i.e . testing a hypothesis of whether each candidate has greater than baseline effect ) through an adaptive sequential assignment of drug candidates to participants . Performing follow up studies on each discovery is expensive , so the scientist does not want to make many “ false discoveries ” i.e . drugs that did not have an actual effect , but were proclaimed to have one by the scientist . To achieve these goals , one could imagine 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the scientist collecting more data for candidates whose efficacy is unclear but appear promising ( e.g . drugs with nontrivial but inconclusive evidence ) , and stop sampling candidates that have relatively clear results already ( e.g . drugs that have a clear and large effect , or seemingly no effect ) . Past work . This problem combines the challenges of multiple hypothesis testing with multi-arm bandits ( MABs ) . In a “ doubly-sequential ” version of the problem studied by Yang et al . [ 46 ] , one encounters a sequence of MAB problems over time . Each MAB was used to test a single special placebo arm against several treatment arms , and if at least one treatment dominated the placebo , then they aimed to return the best treatment . Thus each MAB was itself a single adaptive sequential hypothesis test , and the authors aimed not to make too many false discoveries over the sequence of MAB instances . This paper instead considers the formulation of Jamieson and Jain [ 19 ] , henceforth called JJ , but our techniques apply equally well to the above setup . To avoid confusions , note that our setup is very different from the active classification work of the same authors [ 17 ] . To recap , JJ consider a single MAB instance without a placebo arm ( or rather , leaving it implicit ) , and try to identify as many treatments that work better than chance as possible , without too many false identifications . To clarify , we associate each arm with one ( potentially composite ) null hypothesis — for example , the hypothesis that corresponding drug has no ( significant ) effect . A single observed reward when pulling an arm corresponds to a statistic that summarizes the results of one experiment with the corresponding drug , and the average reward across many experiments could correspond to an estimate of the average treatment effect , which would be ( at most ) zero for null arms and positive for non-nulls . Thus , a strategy for quickly finding the arms with positive means corresponds to a strategy for allocating trial patients to drug candidates that allows the scientists to rapidly find the effective drugs . However , the above corresponds to only the simplest problem setting . In more complex settings , it may be possible to pull multiple arms in each round , and observe correlated rewards . Further , the arms may have some combinatorial structure that allows only certain subsets of arms to be pulled . There could be multiple agents ( eg : hospitals ) pulling the same set of arms and seeing independent rewards ( eg : different patients ) or dependent rewards ( eg : patient overlap or interference ) . Further , if some set of experiments by one scientist yielded suggestive but inconclusive evidence , another may want to follow up , but not start from scratch , instead picking up from where the first left off . Last , the MAB may be stopped for a variety of reasons that may or may not be in the control of the scientist ( eg : a faster usage of funding than expected , or additional funding is secured ) . We dive in the details of these scenarios in Appendix D.3 . Our contribution . We introduce a modular meta-algorithm for bandit multiple testing with provable FDR control that utilizes “ e-values ” — or , more appropriately , their sequential analog , “ e-processes ” — a recently introduced alternative to p-values ( or p-processes ) by Ramdas et al . [ 32 ] for various testing problems , that are inherently related to martingales , gambling and betting [ 33 , 14 , 15 , 44 ] . This work is the first to carefully study e-processes in general MAB settings , building on prior work that studied a special case [ 44 ] . We also are the first to extend the bandit multiple testing problem to the combinatorial bandit setting — JJ had previously only analyzed the problem in the single-arm , independent reward setting . Utilizing e-processes provide our meta-algorithm with several benefits . ( a ) For composite nulls , it is typically easier to construct e-processes than p-processes ; the same holds when data from a single source is dependent . When combining evidence from disparate ( independent or dependent ) sources , it is also more straightforward to combine e-values than p-values ( see Appendix D.3 ) . ( b ) The same multiple testing step applies in all bandit multiple testing problems , regardless of all the various details of the problem setup mentioned in the previous paragraph . Consequently , FDR control in our meta-algorithm is agnostic to much of problem setup and can be proved in a vast array of settings . This is not true when working for p-values . In particular , the techniques for proving FDR control in JJ are highly reliant on the specific bandit setup in their paper . ( c ) The exploration step can be — but does not have to be — decoupled from the multiple testing ( combining evidence ) step . This results in a modular procedure that can be easily ported to new problem settings to yield transparent guarantees on FDR control . By virtue of being a meta-algorithm , we do not ( and can not ) provide “ generic ” sample complexity guarantees : these will depend on all of the finer problem details mentioned above , on the exploration algorithm employed , on which e-processes are constructed . Our emphasis is on the flexibility with which FDR control can be guaranteed in a vast variety of problem setups . Further research can pick up one problem at a time and design sensible exploration strategies and stopping rules , developing sampling complexity bounds for each , and these bounds will be inherited by the meta-algorithm . However , we do formulate some generic exploration algorithms in Appendix C based on best arm identification algorithms [ 1 , 22 , 11 , 18 , 23 , 10 , 20 ] . When instantiated to the particular problem setup studied by JJ ( independent , sub-Gaussian rewards , one arm in each round , etc . ) , we get a slightly different algorithm from them — the exploration strategy can be inherited to stay the same , but the multiple testing part differs . JJ use p-processes for each arm to determine whether that arm should be added to the rejection set , and correct for testing multiple hypotheses by using the BH procedure [ 6 ] to ensure that the false discovery rate ( FDR ) , i.e . the proportion of rejections that are false discoveries in expectation , is controlled at some fixed level 𝛿. Adaptive sampling induces a peculiar form of dependence amongst the p-values , for which the BH procedure provides error control at an inflated level ; in other words , one has to use BH at a more stringent level of approximately 𝛿/log ( 16/𝛿 ) to ensure that the FDR is less than 𝛿 . On the other hand , we use the e-BH procedure [ 44 ] , an analogous procedure for e-values , which can ensure the FDR is less than 𝛿 without any inflation , regardless of the dependence structure between the e-values of each arm . Our algorithm has improved sample efficiency in simulations and the same sample complexity in theory . Formal problem setup . We define the bandit as having 𝑘 arms , and 𝜈𝑖 as the ( unknown ) reward distribution for arm 𝑖 ∈ [ 𝑘 ] = { 1 , ... , 𝑘 } . Every arm 𝑖 is associated with a null hypothesis , which is represented by a known , prespecified set of distributions 𝒫𝑖 . If |𝒫𝑖|=1 , it is a ‘ point null hypothesis ’ , and otherwise it is a ‘ composite null hypothesis ’ . Examples of the latter include “ all [ 0,1 ] -bounded distributions with mean≤0.5 ” or “ all 1-sub-Gaussian distributions with mean≤0 ” or “ all distributions that are symmetric around 0 ” or “ all distributions with median ≤ 0 ” . While we assume by default that all rewards from an arm are i.i.d. , we also formulate tests for hypotheses on reward distributions that may violate this assumption in Appendix G. If 𝜈𝑖∈𝒫𝑖 , then we say that the 𝑖-th null hypothesis is true and we call 𝑖 a null arm ; else , we say 𝑖-th null hypothesis is false and we call it a non-null arm . Thus , the set of arms are partitioned into two disjoint sets : nulls ℋ0⊆ [ 𝑘 ] and non-nulls ℋ1 : = [ 𝑘 ] ∖ℋ0 . Let 𝒦 ⊆ 2 [ 𝑘 ] denote the subsets of arms that can be jointly queried in each round . At each time 𝑡 , the algorithm chooses a subset of arms ℐ𝑡 ∈ 𝒦 to sample jointly from . The special choice of 𝒦 = { { 1 } , { 2 } , ... , { 𝑘 } } recovers the standard bandit setup , but otherwise this setting is known as combinatorial bandits with semi-bandit feedback [ 12 ] . We also consider the special case of full-bandit feedback ( the algorithm sees all rewards at each time step ) in Appendix D.1 . We denote the reward sampled at time 𝑡 from arm 𝑖∈ℐ𝑡 as 𝑋𝑖 , 𝑡 . Let 𝑇𝑖 ( 𝑡 ) denote the number of times arm 𝑖 has been sampled by time 𝑡 , and 𝑡𝑖 ( 𝑗 ) be the time of the 𝑗th sample from arm 𝑖 . We now define a canonical “ filtration ” for our bandit problem . A filtration ( ℱ𝑡 ) 𝑡≥0 is a series of nested sigma-algebras that encapsulates what information is known at time 𝑡 . ( We drop the subscript and just write ( ℱ𝑡 ) for brevity , and drop the parentheses when just referring to a single sigma-algebra at time 𝑡 . ) Define the canonical filtration as follows for 𝑡∈N : ℱ𝑡 : =𝜎 ( 𝑈∪ { ( 𝑖 , 𝑠 , 𝑋𝑖 , 𝑗 ) : 𝑠≤ 𝑡 , 𝑖∈ℐ𝑠 } ) and we let ℱ0 : =𝜎 ( 𝑈 ) where 𝑈 is uniformly distributed on [ 0,1 ] and its bits capture all private randomness used by the bandit algorithm that are independent of all observed rewards . Let ( 𝜆𝑡 ) be a sequence of random variables indexed by 𝑡∈N . ( 𝜆𝑡 ) is said to be predictable w.r.t . ( ℱ𝑡 ) if 𝜆𝑡 is measurable w.r.t . ℱ𝑡−1 i.e . 𝜆𝑡 is fully specified given the information in ℱ𝑡−1 . An N-valued random variable 𝜏 is a stopping time ( or stopping rule ) w.r.t . to ( ℱ𝑡 ) if { 𝜏 = 𝑡 } ∈ℱ𝑡 — in other words , at each time 𝑡 , we know whether or not to stop collecting data . Let 𝒯 denote the set of all possible stopping times/rules w.r.t . ( ℱ𝑡 ) , potentially infinite . Technically , the algorithm must not just specify a strategy to select ℐ𝑡 , but also specify when sampling will stop . This is denoted by the stopping rule or stopping time 𝜏 * ∈𝒯 . Once the algorithm halts at some time 𝜏 , it produces a rejection set 𝒮𝜏 ⊆ [ 𝑘 ] . We consider two metrics w.r.t . 𝒮 : the FDR as discussed prior , and true positive rate ( TPR ) , which is the proportion of non-nulls that are discovered in expectation . These two metrics are defined as follows : FDR ( 𝒮𝜏 ) : = E [ ︁ |ℋ0∩𝒮𝜏 | |𝒮𝜏 |∨1 ] ︁ , TPR ( 𝒮𝜏 ) : = E [ ︁ |ℋ1∩𝒮𝜏 | |ℋ1| ] ︁ . We consider algorithms that always satisfy FDR ( 𝒮𝜏 ) ≤𝛿 for any number and configuration of nulls ℋ0 and any choice of null and non-null distributions . In fact , our algorithm will produce a sequence of candidate rejection sets ( 𝒮𝑡 ) that satisfies sup𝜏∈𝒯 FDR ( 𝒮𝜏 ) ≤𝛿 . This is a much stronger guarantee than the typical setting considered in the multiple testing literature . On the other hand , TPR is a measurement of the power of the algorithm i.e . how many of the non-null hypotheses does the algorithm discover . Our implicit goal in the multiple testing problem is to maximize the number of true discoveries while not making too many mistakes i.e . keep the FDR controlled . In hypothesis testing , the set of null distributions 𝒫𝑖 for each arm 𝑖 is known , because the user defines the null hypothesis they are interested in testing . When the null hypothesis is false , the non-null distribution can be arbitrary . Consequently , we can prove results about FDR , but we can not prove guarantees about TPR without several further assumptions on the non-null distributions , dependence across arms , etc . For a particular setting where we make such a set of assumptions , we demonstrate in Section 4 that we can prove TPR guarantees for algorithms within our framework . Hence , our FDR controlling framework is not vacuous as it includes powerful algorithms i.e . algorithms which make many true discoveries . However , our focus is primarily to show that the FDR control of our framework is robust to a wide range of conditions . Finally , note that in bandit multiple testing , one does not care about regret . The problem is more akin to pure exploration , where we aim to find a 𝒮 with FDR ( 𝒮𝜏 * ) ≤𝛿 and large TPR as quickly as possible . Now that we have specified the problem we are interested in , we can introduce our main technical tools for ensuring FDR control at stopping times : e-processes and p-processes .

Summary: The authors give a unified framework for bandit multiple testing in terms of e-values. This framework is more flexible than prior work, allowing for dependencies between random variables and adaptive algorithm without corrections required in prior work, most notably JJ. The authors apply this result in a setting studied previouly where each arm is sub-gaussian and independent, and a single arm is queried at each step. They give sample complexity bounds matching prior state-of-the-art and show comparable performance in experiments.