Title: Online Sign Identification: Minimization of the Number of Errors in Thresholding Bandits
Authors: Reda Ouhamma
Rémy Degenne
Pierre Gaillard
Vianney Perchet
Venue: NeurIPS 2021
Keywords: Multi-armed bandit
thresholding bandits

Abstract: In the fixed budget thresholding bandit problem, an algorithm sequentially allocates a budgeted number of samples to different distributions. It then predicts whether the mean of each distribution is larger or lower than a given threshold. We introduce a large family of algorithms (containing most existing relevant ones), inspired by the Frank-Wolfe algorithm, and provide a thorough yet generic analysis of their performance. This allowed us to construct new explicit algorithms, for a broad class of problems, whose losses are within a small constant factor of the non-adaptive oracle ones. Quite interestingly, we observed that adaptive methods
empirically greatly out-perform non-adaptive oracles, an uncommon behavior in standard online learning settings, such as regret minimization. We explain this surprising phenomenon on an insightful toy problem.

TL;DR: We generalize the thresholding bandit setting and devise a generic algorithm as well as a proof methodology that also applies to existing algorithms in literature.

Review discussion:
Generally, I think this is a good paper with a nice literature review, presentation, and technique depth. The developed new algorithm, which aims to solve a generic goal, is found to have a better regret bound than the previous LSA algorithm when considering the goal to minimize aggregate regret.
Also, it is interesting to see the new rule to decide the signs of the arms.
I do have some minor comments detailed in the following:
Line 36: “As as consequence” -> As a consequence
Line 49: “the performances of an algorithm”: performances -> performance
Lemma 1: what is \tilde{\Delta}_k^2?
Line 179: “that each depend only on the rewards”: depend -> depends
Assumption 1: it is better to specify the parameters of F i.e., F(n, x; a) for clearer presentation
Line 278: in the example introduced, the sign is not defined. I guess the sign should be the actual parameter x?
Figure 3: In the caption of the left subfigure, it says Median, 0.25, and 0.75 empirical quantiles are presented.
However, for each algorithm, only one line is presented. I am wondering which one you are referring to?
Also for the right subfigure, the ratio of the averaged errors should be at most 1, and why it appears to be greater than 1 in the figure?
Regarding the experiments, I think it may be better to add one more figure presenting the performance of algorithms LSA, APT, and FWT regarding random input instances. The current one \mu_k = (-1)^k (k/K)^2 looks artificial to me.
No
We thank the reviewer for the nice remarks about the novelty and generality of our work.
We now address specific comments:
"Lemma 1: what is $\tilde{\Delta}_k^2$?"
It's a typo and should be replaced by $\Delta_k^2$, the squared gap.
"In the caption of the left subfigure, it says Median, 0.25, and 0.75 empirical quantiles are presented.
However, for each algorithm, only one line is presented."
The line is the median, and the quantiles are represented by the shaded area of the same color.
"Also for the right subfigure, the ratio of the averaged errors should be at most 1, and why it appears to be greater than 1 in the figure?"
The ratio can be higher than 1 if the algorithm makes more mistakes than the non-adaptive oracle. It is the case for APT (which is not designed for that task) and for the uniform allocation, which is not the optimal non-adaptive allocation for that example.
In contrast, the adaptive algorithms LSA and FWT which are designed for the correct task have average error lower than the non-adaptive oracle.
Note also that the two sub-figures of figure 3 correspond to experiment with different arm means. The error proportions of all algorithms are lower than that of the oracle on the left experiment (in which APT happens to be a sensible algorithm), but not on the right.
The problem studied in this paper is well-known in the community. The algorithms are not very novel according to my knowledge.
However, the authors managed to get better bounds by a better analysis.
Another main contribution of this paper is to make an abstraction of the existing algorithms and make a unified and thorough analysis.
The results are very close to the optimal, i.e., there are only a gap of 4 in the exponential term.
The paper is well written and well structured.
The main concern of this paper is the significance.
In fact, this paper does not propose novel algorithms or mathematical techniques, but more like a deeper study and summary of existing works.
But due to its quality and clarity, I tend to give a score between 6 and 7. To be more conservative in case I missed anything, I will vote a 6 for now and wait for the author responses.
After the rebuttal, I am raising the score from 6 to 7.
No
We thank the reviewer for the positive comments on our analysis of the algorithms.
The main concern expressed in the review is the significance of our results, which we address in two parts.
First, we object to the affirmation that "this paper does not propose novel algorithms or mathematical techniques".
Second, our paper brings more hindsight about the problem, beyond loss bounds.
Out of the three algorithms that we analyse, one is new (FWT). That algorithm enjoys the tightest upper bound of the three.
Although it is based on the idea of the well known Frank-Wolfe method, the design of FWT requires additional ingredients to obtain an index based algorithm with the desired properties.
Furthermore, as pointed out in the review, the way we analyse all three methods is based on a new abstraction using generic index based algorithms. This, together with the more minor innovation of returning the sign of the arm at the time at which the index is maximal (and not at the final time), allows us to derive tight bounds for all algorithms with one unified proof.
Section 3.3 shows that the FWT algorithm and the analysis are general and extend to more tasks, like the sum-of-gaps objective.
Beyond proving error bounds for fixed budget problems, we also highlight in section 4 the benefits of adaptivity. Adaptive algorithms beat the oracle empirically and on a theoretical toy example that we introduce. Since the state of the art for upper bounds don't even show that adaptive algorithms match the oracle, this is an intriguing observation about fixed budget identification.
Thank you for the response.
Also, I have read the reviews of other reviewers. I am convinced that the results of this paper are of reasonable significance so I will raise my score to 7.
Originality: their proof based on the Frank-Wolfe method for any index function that satisfies their criteria is novel. It encompasses current methods but they also provide a better algorithm (for the constant in the exponent).
Clarity: The paper is well written. This reviewer found a few typos (see below) but it is clear otherwise.
Significance: the results are useful because they give a general proof method that may be applied to other algorithms. It provides a structure to construct good index functions for thee bandits based on their bounds. I think that this paper will help other build on the results provided.
Critiques: the authors give an example of when adaptivity helps but they also show cases where it hurts (non-oracle algorithms cannot ignore certain arms like the oracle). This is an interesting point and a discussion about what types of problems fall into each category would have enhanced the paper.
No
We thank the reviewer for the time spent on the review and the positive feedback. We agree that characterizing the tasks on which adaptivity helps is a very interesting research direction. In this work, we managed to exhibit experimentally examples of tasks in which it helps, as well as a theoretical toy example. We are not yet in a position to give a more general characterization of when an adaptive algorithm can beat the non-adaptive oracle. A next step could be to obtain upper bounds for the adaptive algorithms that actually match (or even beat) the performance of the oracle, by extending the proof method of the toy example of section 4: an adaptive algorithm can "gain samples" by using the observations to make early decisions when the random deviations are favorable.

Meta-review:
role: PC
data: comment: The reviewers came to consensus that this paper makes a good progress on the thresholding bandit problem from the perspectives of upper and lower bounds as well as an interesting discussion on the benefit of adaptivity. I agree with these opinions and please polish the manuscript so that the minor concerns raised by the reviewers become clear in the final version.

Decision: Accept (Spotlight)