On Optimal Robustness to Adversarial Corruption in Online Decision Problems

algorithms with second-order regret bounds , achieve O ( logN∆ + √ C logN ∆ ) -regret , where N , ∆ , and C represent the number of experts , the gap parameter , and the corruption level , respectively . We further provide a matching lower bound , which means that this regret bound is tight up to a constant factor . For the multi-armed bandit problem , we also provide a nearly-tight lower bound up to a logarithmic factor . 1 Introduction . In this work , we consider two fundamental sequential decision-making problems , the problem of prediction with expert advice ( expert problem ) and the multi-armed bandit ( MAB ) problem . In both , a player chooses probability vectors pt over a given action set [ N ] = { 1 , 2 , . . . , N } in a sequential manner . More precisely , in each round t , a player chooses a probability vector pt ∈ [ 0 , 1 ] N over the action set , and then an environment chooses a loss vector ` t ∈ [ 0 , 1 ] N . The player chooses pt , and then observes ` t in the expert problem . In the MAB problem , the player picks action it ∈ [ N ] following pt and then observes ` tit . The goal of the player is to minimize the ( pseudo- ) regret R̄T defined as RTi∗ = T∑ t=1 ` > t pt − T∑ t=1 ` ti∗ , R̄Ti∗ = E [ RTi∗ ] , R̄T = max i∗∈ [ N ] R̄Ti∗ . ( 1 ) For such decision-making problems , two main types of environments have been studied : stochastic environments and adversarial environments . In stochastic environments , the loss vectors are assumed to follow an unknown distribution , i.i.d . for all rounds . It is known that the difficulty of the problems can be characterized by the suboptimality gap parameter ∆ > 0 , which denotes the minimum gap between the expected losses for the optimal action and for suboptimal actions . Given the parameter ∆ , mini-max optimal regret bounds can be expressed as Θ ( logN∆ ) in the expert problem [ Degenne and Perchet , 2016 , Mourtada and Gaïffas , 2019 ] and Θ ( N log T∆ ) in the MAB problem [ Auer et al. , 2002a , Lai and Robbins , 1985 , Lai , 1987 ] . In contrast to the stochastic model , the adversarial model does not assume any generative models for loss vectors , but the loss at each round may behave adversarially depending on the choices of the player up until that round . The mini-max optimal regret bounds for the adversarial model are Θ ( √ T logN ) in the expert problem [ Cesa-Bianchi and Lugosi , 2006 , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Freund and Schapire , 1997 ] and Θ ( √ TN ) in the MAB problem [ Audibert and Bubeck , 2009 , Auer et al. , 2002b ] . As can be seen in the regret bounds , achievable performance differs greatly between the stochastic and adversarial regimes , which implies that the choice of models and algorithms will matter in many practical applications . One promising solution to this challenge is to develop best-of-both-worlds ( BOBW ) algorithms , which perform ( nearly ) optimally in both stochastic and adversarial regimes . For the expert problem , Gaillard et al . [ 2014 ] provide an algorithm with a BOBW property , and Mourtada and Gaïffas [ 2019 ] have shown that the well-known Hedge algorithm with decreasing learning rate ( decreasing Hedge ) enjoys a BOBW property as well . For the MAB problem , the Tsallis-INF algorithm by Zimmert and Seldin [ 2021 ] has a BOBW property , i.e. , achieves O ( N log T∆ ) -regret in the stochastic regime and O ( √ NT ) -regret in the adversarial regime . One limitation of BOBW guarantees is , however , that they do not necessarily provide nontrivial regret bounds for a situation in which the stochastic and the adversarial regimes are mixed , i.e. , an intermediate situation . To overcome this BOBW-property limitation , our work focuses on an intermediate ( and comprehensive ) regime between the stochastic and adversarial settings . More specifically , we consider the adversarial regime with a self-bounding constraint introduced by Zimmert and Seldin [ 2021 ] . As shown by them , this regime includes the stochastic regime with adversarial corruption [ Lykouris et al. , 2018 , Amir et al. , 2020 ] as a special case in which an adversary modifies the i.i.d . losses to the extent that the total amount of changes does not exceed C , which is an unknown parameter referred to as the corruption level . For expert problems in the stochastic regime with adversarial corruption , Amir et al . [ 2020 ] have shown that the decreasing Hedge algorithm achieves O ( logN∆ +C ) -regret . For the MAB problem , Zimmert and Seldin [ 2021 ] have shown that Tsallis-INF achieves O ( N log T∆ + √ CN log T ∆ ) -regret in adversarial regimes with self-bounding constraints . To be more precise , they have proved a regret upper bound of O ( ∑ i6=i∗ log T ∆i + √ C ∑ i6=i∗ log T ∆i ) , where i∗ ∈ [ N ] is the optimal action and ∆i represents the suboptimality gap for each action i ∈ [ N ] \ { i∗ } . Ito [ 2021b ] has shown that similar regret bounds hold even when there are multiple optimal actions , i.e. , even if the number of actions i with ∆i = 0 is greater than 1 . In such cases , the terms of ∑ i 6=i∗ 1/∆i in regret bounds are replaced with ∑ i : ∆i > 0 1 ∆i . In addition to this , Masoudian and Seldin [ 2021 ] have improved the analysis to obtain a refined regret bound ofO ( ( ∑ i6=i∗ 1 ∆i ) log+ ( ( N−1 ) T ( ∑ i6=i∗ 1 ∆i ) 2 ) + √ C ( ∑ i 6=i∗ 1 ∆i ) log+ ( ( N−1 ) T C ∑ i6=i∗ 1 ∆i ) ) , where log+ ( x ) = max { 1 , log x } . The contributions of this work are summarized in Table 1 , alongside previously reported results . As shown in Theorems 3 and 4 , this paper provides an improved regret upper bound of O ( logN∆ +√ C logN ∆ ) for the expert problem in the adversarial regime with self-bounding constraints . This regret upper bound is tight up to a constant factor . In fact , we provide a matching lower bound in Theorem 5 . In addition to this , we show an Ω ( N∆ + √ CN ∆ ) -lower bound for MAB , which implies that 1Note that Amir et al . [ 2020 ] adopt a different definition of regret than in this paper . Details and notes for comparison are discussed in Remark 1 . Tsallis-INF by Zimmert and Seldin [ 2021 ] achieves a nearly optimal regret bound up to an O ( log T ) factor in the adversarial regime with self-bounding constraints . The regret bounds in Theorems 3 and 4 are smaller than the regret bound shown by Amir et al . [ 2020 ] for the stochastic regime with adversarial corruption , especially when C = Ω ( logN∆ ) , and they can be applied to more general problem settings in the adversarial regime with self-bounding constraints . Note here that this study and their study consider slightly different definitions of regret . More precisely , they define regret using losses without corruption , while this study uses losses after corruption to define regret . In practice , appropriate definitions would vary depending on individual situations . For example , if each expert ’ s prediction is itself corrupted , the latter definition is suitable . In contrast , if only the observation of the player is corrupted , the former definition seems appropriate . However , even after taking this difference in definitions into account , we can see that the regret bound in our work is , in a sense , stronger than theirs , as is discussed in Remark 1 of this paper . In particular , we would like to emphasize that the new bound of O ( logN∆ + √ C logN ∆ ) provides the first theoretical evidence implying that the corresponding algorithms are more robust against adversarial corruption than the naive Follow-the-Leader algorithm . Note also that the regret bound by Amir et al . [ 2020 ] is tight as long as the former regret definition is used . This work shows the tight regret upper bounds for two types of known algorithms . The first , ( Theorem 3 ) , is the decreasing Hedge algorithm , which has been analyzed by Amir et al . [ 2020 ] and Mourtada and Gaïffas [ 2019 ] as well . The second ( Theorem 4 ) represents algorithms with second-order regret bounds [ Cesa-Bianchi et al. , 2007 , Gaillard et al. , 2014 , Hazan and Kale , 2010 , Steinhardt and Liang , 2014 , Luo and Schapire , 2015 ] . It is worth mentioning that Gaillard et al . [ 2014 ] have shown that a kind of second-order regret bounds impliesO ( logN∆ ) -regret in the stochastic regime . Theorem 4 in this work extends their analysis to a broader setting of the adversarial regime with self-bounding constraints . In the proof of Theorems 3 and 4 , we follow a proof technique given by Zimmert and Seldin [ 2021 ] to exploit self-bounding constraints . To show regret lower bounds in Theorems 5 and 6 , we construct specific environments with corruption that provide insight into effective attacks which would make learning fail . Our approach to corruption is to modify the losses so that the optimality gaps decrease . This approach achieves a mini-max lower bound in the expert problem ( Theorem 5 ) and a nearly-tight lower bound in MAB up to a logarithmic factor in T ( Theorem 6 ) . We conjecture that there is room for improvement in this lower bound for MAB under assumptions regarding consistent policies [ Lai and Robbins , 1985 ] , and that the upper bound by Zimmert and Seldin [ 2021 ] is tight up to a constant factor . 2 Related work . In the context of the expert problem , studies on stochastic settings seem to be more limited than those on adversarial settings . De Rooij et al . [ 2014 ] have focused on the fact that the Follow-theLeader ( FTL ) algorithm works well for a stochastic setting , and they have provided an algorithm that combines FTL and Hedge algorithms to achieve the best of both worlds . Gaillard et al . [ 2014 ] have provided an algorithm with a second-order regret bound depending on VTi∗ = ∑T t=1 ( ` > t pt− ` ti∗ ) 2 in place of T and have shown that such an algorithm achieves O ( logN∆ ) -regret in the stochastic regime . Mourtada and Gaïffas [ 2019 ] have shown that a simpler Hedge algorithm with decreasing learning rates of ηt = Θ ( √ logN t ) enjoys a tight regret bound in the stochastic regime as well . This simple decreasing Hedge algorithm has been shown by Amir et al . [ 2020 ] to achieve O ( logN∆ +C ) -regret in the stochastic regime with adversarial corruption . For online linear optimization , a generalization of the expert problem , Huang et al . [ 2016 ] have shown that FTL achieves smaller regret in the stochastic setting and provides best-of-both-worlds algorithms via techniques reported by Sani et al . [ 2014 ] . For MAB , there are a number of studies on best-of-both-worlds algorithms [ Bubeck and Slivkins , 2012 , Zimmert and Seldin , 2021 , Seldin and Slivkins , 2014 , Seldin and Lugosi , 2017 , Pogodin and Lattimore , 2020 , Auer and Chiang , 2016 , Wei and Luo , 2018 , Zimmert et al. , 2019 , Lee et al. , 2021 , Ito , 2021b ] . Among these , studies by Wei and Luo [ 2018 ] , Zimmert and Seldin [ 2021 ] , Zimmert et al . [ 2019 ] are closely related to this work . In their studies , gap-dependent regret bounds in the stochastic regime are derived via { pt } -dependent regret bounds in the adversarial regime , similarly to that seen in this work and in previous studies by Gaillard et al . [ 2014 ] , Amir et al . [ 2020 ] . Studies on online optimization algorithms robust against adversarial corruption have been extended to a variety of models , including those for the multi-armed bandit [ Lykouris et al. , 2018 , Gupta et al. , 2019 , Zimmert and Seldin , 2021 , Hajiesmaili et al. , 2020 ] , Gaussian process bandits [ Bogunovic et al. , 2020 ] , Markov decision processes [ Lykouris et al. , 2021 , Chen et al. , 2021 , Jin et al. , 2021 ] , the problem of prediction with expert advice [ Amir et al. , 2020 ] , online linear optimization [ Li et al. , 2019 ] , and linear bandits [ Bogunovic et al. , 2021 , Lee et al. , 2021 ] . Literature can also be found regarding effective attacks on bandit algorithms [ Jun et al. , 2018 , Liu and Shroff , 2019 ] . As summarized by Hajiesmaili et al . [ 2020 ] , there can be found studies on two different models of adversarial corruption : the oblivious corruption model and the targeted corruption model . In the former ( e.g. , in studies by Lykouris et al . [ 2018 ] , Gupta et al . [ 2019 ] , Bogunovic et al . [ 2020 ] ) , the attacker may corrupt the losses ` t after observing ( ` t , pt ) without knowing the chosen action it while , in the latter ( e.g. , in studies by Jun et al . [ 2018 ] , Hajiesmaili et al . [ 2020 ] , Liu and Shroff [ 2019 ] , Bogunovic et al . [ 2021 ] , Ito [ 2021a ] , Erez and Koren [ 2021 ] ) , the attacker can choose corruption depending on ( ` t , pt , it ) . We discuss differences between these models in Section 3 . This work mainly focuses on the oblivious corruption model for MAB problems . It is worth mentioning that Tsallis-INF [ Zimmert and Seldin , 2021 ] works well in the oblivious corruption models , as is shown in Table 1 , as well as achieving best-of-both-worlds .

Summary: The paper considered expert problem and multi-armed bandits in stochastic environment with adversarial corruption. In this setting, an adversary modifies stochastic feedback up to total corruption level $C$. The authors analyzed Hedge algorithm with decreasing learning rate and algorithms with second-order regret bounds, and showed that the regret has a $\sqrt{C}$ dependency to the corruption level. Matching lower bound in the expert problem and near-tight lower bound up to logarithmic factor in the multi-armed bandit problem, suggesting the analyzed algorithms achieve optimal robustness.