Title: Meta-Learning for Contextual Bandit Exploration
Authors: Amr Sharaf
Hal Daumé III
Venue: ICLR 2019
Keywords: meta-learning
bandits
exploration
imitation learning

Abstract: We describe MÊLÉE, a meta-learning algorithm for learning a good exploration policy in the interactive contextual bandit setting. Here, an algorithm must take actions based on contexts, and learn based only on a reward signal from the action taken, thereby generating an exploration/exploitation trade-off. MÊLÉE addresses this trade-off by learning a good exploration strategy based on offline synthetic tasks, on which it can simulate the contextual bandit setting. Based on these simulations, MÊLÉE uses an imitation learning strategy to learn a good exploration policy that can then be applied to true contextual bandit tasks at test time. We compare MÊLÉE to seven strong baseline contextual bandit algorithms on a set of three hundred real-world datasets, on which it outperforms alternatives in most settings, especially when differences in rewards are large. Finally, we demonstrate the importance of having a rich feature representation for learning how to explore.


TL;DR: We present a meta-learning algorithm, MÊLÉE, for learning a good exploration function in the interactive contextual bandit setting.

Review discussion:
This paper investigates a meta-learning approach for the contextual bandit problem. The goal is to learn a generic exploration policy from datasets, and then to apply the exploration policy to contextual bandit tasks. The authors have adapted an algorithm proposed for imitation learning (Ross & Bagnell 2014) to their setting. Some theoretical guarantees straightforwardly extracted from (Ross & Bagnell 2014) and from (Kakade et al 2008) are presented. Experiments are done on 300 supervised datasets.
Major concerns:
1 This paper investigates a problem that does not correspond to the real problem: how to take advantage of a plenty of logs generated by a known stochastic policy (or worst unknown deterministic policy) for the same (or a close) contextual bandit task?
Most of companies have this problem. I do not know a single use case, in which we have some full information datasets, which are representative of contextual bandit tasks to be performed. If the full information datasets does not correspond to the contextual bandit tasks, it is not possible to learn something useful for the contextual bandit task.
2 The experimental validation is not convincing.
The experiments are done on datasets, which are mostly binary classification datasets. In this case, the exploration task is easy. May be it is the reason why the exploration parameter \mu or \epsilon = 0 provides the best results for MELEE or \epsilon-greedy?
The baselines are not strong. The only tested contextual bandit algorithm is LinUCB.
However a diagonal approximation of the covariance matrix is used when the dimension exceeds 150. In this case LinUCB is not efficient. There are a lot of contextual bandit algorithms that scale with the dimension.
3 The theoretical guarantees are not convincing.
The result of Theorem 1 is a weak result. A linear regret against the expected reward of the best policy is usually considered as a loosely result. Theorem 2 shows that there is no theoretical gain of the use of the proposed algorithm: the upper bound of the expected number of mistakes obtained when Banditron is used in MELEE is upper than the one of Banditron alone.
Minor concerns:
The algorithms are not well written. POLOPT function has sometimes one parameter, sometimes two and sometimes three parameters. The algorithm 1 is described in section 2, while one of the inputs of the algorithm 1 (feature extractor function) is described in section 3.1. The algorithm 1 seems to return all the N exploration policies. The choice of the returned policy has to be described.
In contextual bandits, the exploration policy is not handcrafted. The contextual bandit algorithms are designed to be optimal or near optimal in worst case: they are generic algorithms.
We thank the reviewer for the provided feedback. Please find our response below:
1) Relevance to Real Problems:
We believe that there is a fundamental misunderstanding in this point of the review regarding the experimental setup we study on our paper.
We want to stress that we don’t assume access to full information datasets that are representative of the contextual bandit task to be performed. As mentioned on the abstract & section 3.1 of the paper, MELEE uses offline synthetic datasets during the training phase.
Contrary to the assertion in the review, these synthetic full information datasets are quite different from the task dependent contextual bandit dataset.
These synthetic datasets are very diverse and broad in their complexity, and exploration strategies learned on these datasets does indeed generalize to real contextual bandit datasets as we have verified both empirically and theoretically. The context vectors for these synthetic datasets are quite different in structure from the real contextual bandit task at hand, for which we don’t assume access to any sort of full information data. We learn a dataset independent exploration policy from these synthetic datasets, and use meta-features that can generalize across different datasets to learn how to explore in realistic contextual bandit settings.
We describe how we generate these synthetic datasets in appendix B. We generate 2D datasets by first sampling a random variable representing the Bayes classification error. The Bayes error is sampled uniformly from the interval 0.0 to 0.5. This Bayes error controls for the amount of noise in the dataset.
2) Experimental Validation:
It’s not true that we only compare to the LinUCB exploration algorithm. We compare to seven other contextual bandit exploration algorithm these algorithms are (Section 3.3): Epsilon greedy, Exponentiated Gradient Epsilon Greedy, Tau-first exploration, LinUCB, Cover, and Cover-Nu. Many of these algorithms does indeed use data in devising an exploration strategy.
For example LinUCB, Cover, and Cover-NU all leverage information from the observed data to balance exploration and exploitation.
3) Theorem 1 and sublinear Regret:
This is a really good observation. The regret bounded by Theorem 1 is dependent on the term epsilon-hat-class (i.e. the average regression regret for each policy π-n). Sublinear regret is still achievable whenever this term decreases over the time horizon T. For any reasonable underlying learning algorithm, we expect this term to be decreasing at a rate of T^-a (e.g. a:½), putting this together, the sublinear regret will still be achievable.
4) Theorem 2 and expected number of mistakes:
The theoretical gain is still guaranteed because it’s never the case that the upper bound of the expected number of mistakes obtained when Banditron is used in MELEE is larger than the one of Banditron alone. This follows directly from the edge assumption we make, as Eγt ≥ 0, and Γ ≤ 1.
5) Minor concerns:
We thank the reviewer for highlighting these concerns. The authors appreciate the reviewer’s suggestions for improving the overall exposure of the paper. In order to make it easier for reviewers’ to track the changes we kept the structure largely consistent with the original submission, but we’ll take all of these comments into account in the final version.
6) Hand-crafted Exploration:
With “hand-crafted” exploration algorithms we meant “not learned”, we agree that this terminology is not accurate and we will remove it in the final version.
7) Returned Policy:
Theoretically Algorithm 1 averages between the set of learned N policies.
In practice, it’s typical that the final policy leads to a better performance empirically. In our experiments, Algorithm 1 returns the final N-th policy. We’ll describe the return policy explicitly in the paper and fix the notation for POLOPT.
This paper proposes a new method (Melee) to explore on contextual bandits. It uses a supervised full information data set to evaluate (using counterfactual estimation) and select (using imitation learning) a proper exploration strategy. This exploration strategy is then used to augment an e-greedy contextual bandit algorithm.
The novelty is that the exploration strategy is learned from the data, as opposed to being engineered to minimize regret. The edge of Melee stems from the expected improvement for choosing an action against the standard bandit optimization recommendation.
Pros:
- using data to learn exploration strategy in tis manner is a novel idea for bandits
- good experimental results
- well written paper
Cons:
- Practical impact may be minimal. This setting is seldom encountered in reality.
- No comparison with Thompson sampling bandits, which also use data in devising an exploration strategy. I suggest authors compare to better suited bandits and exploration strategies, beyond basic e-greedy and UCB.
- Article assumes knowledge of imitation learning. which is not a given in bandit literature. I suggest a simple explanation or sketch of the imitation algorithm.
- Theoretical guarantees questionable. Theorem 1 talks about "no-regret algorithm". you then extend this notion and claim "if we can achieve low regret ....
then ....". It is unclear to me how this theorem allows you to make such claim. A low regret is > no-regret, and hence a bound on no-regret may not generalize to low regret.
- May want to add noise to augmentation data, to judge robustness of method.
Overall, given the novelty of the idea and the good results, I am inclined to accept, with major modifications. Improvements of the method and analysis are likely to follow. Given the flaws though, I am not fighting for this paper.
Minor comments:
sec 2.1: you may want to explain why you require reward to be [0,1]
Alg 1: explain Val and rho in algorithm.
sec 2.3: what is "ergo".
Also, you may want to refer to f as "function" and to pi as "policy". referring to f as policy may be confusing (even though it is a policy).
For example: "(line 8) on which it trains a new policy"
End of 2.4: "as discussed in 2.4" should be "in 2.3"
sec 3.3: why is epsilon=0 the best? is it because synthetic data has no noise? This result surprises me.
We thank the reviewer for the detailed review and insightful comments. We clarify several points below.
1) Practical Impact:
We request a clarification from AnonReviewer3 for why they think the practical impact may be minimal. The setting we study in the paper is the standard contextual bandit setting encountered in reality. The algorithm learns an exploration policy for balancing exploration with exploitation in contextual bandits, a fundamental issue addressed by any contextual bandit algorithm. We stress that our algorithms doesn’t assume access to fully supervised datasets at runtime, we rely on synthetic fully supervised datasets only for offline training. These datasets are generated synthetically and doesn’t require labelling effort from annotators at runtime.
2) Comparison with Thompson Sampling:
We compared MELEE to seven other exploration algorithms. Many of these algorithms does indeed use data in devising an exploration strategy.
For example LinUCB, Cover, and Cover-NU all leverage information from the observed data to balance exploration and exploitation. Exponentiated Gradient epsilon-greedy as well uses the observed data to select the best epsilon for exploration. For completeness, we will add Thompson Sampling in our comparison.
3) Knowledge of imitation learning:
We thank the reviewer for highlighting this issue. We will include a more detailed introduction for imitation learning in the final version for this work.
4) Theoretical guarantees no-regret vs low-regret:
What we mean by low regret in this statement is the low average regret epsilon-class-hat: the average regression regret for each policy π-n), not the no-regret LEARN procedure in Alg 1 - line 16. We’ll rephrase this to make this distinction clear in the final draft for this paper.
5) Noise in augmentation data:
We include noise in the augmentation datasets used for training MELEE. These datasets are generated synthetically and the details for the data generation process is highlighted in Appendix B. We generate 2D datasets by first sampling a random variable representing the Bayes classification error. The Bayes error is sampled uniformly from the interval 0.0 to 0.5. This Bayes error controls for the amount of noise in the dataset.
6) Minor comments:
The authors thank the reviewer for highlighting these issues. We’ll take all of these comments into account in the final version.
7) Why we require reward to be [0,1] in Alg 1:
We’ll add a clarification for why we require bounded rewards. Theoretically, this is required to ensure the no-regret bound in theorem 1. Empirically,  for multi-class contextual bandit classification problems, we use a reward of one for the correct action, and the reward of zero for all other incorrect actions.
8) Why is epsilon=0 the best?
Empirically, MELEE doesn’t require the added extra exploration on top of the learned exploration strategy, and at runtime the best performance was achieved when we set the additional exploration parameter \mu to 0 . At training time the synthetic datasets we used are not noise-free. As described in point (5) of this response, we control the amount of noise in the training dataset via the Bayes error parameter. Bietti et. al. observed a similar behavior for the same datasets we used in our experiments. For epsilon-greedy exploration, the best performance was achieved when setting epsilon to zero. They attribute this to the diversity of the context vectors in these datasets.
9) Major Modifications:
We assume the “major modifications” are the issues highlighted in the “cons” section of the review. We kindly request a clarification about any other major modifications the reviewer thinks should be necessary.
References:
Alberto Bietti, Alekh Agarwal, and John Langford. A Contextual Bandit Bake-off. working paper or preprint, May 2018. URL https://hal.inria.fr/hal-01708310.
The paper proposes to train exploration policies for contextual bandit problems through imitation learning on synthetic data-sets. An exploration policy takes the decision of choosing an action on each time-step (balancing explore/exploit) based on the history, the confidences of taking different actions suggested by a policy optimizer (bet expert policy given the history). The idea in this paper is to generate many multi-class supervised learning data-sets and sun an imitation learning algorithm for training a good exploration policy. I think this is a novel idea and I have not seen this before.
Moreover some intuitive features for training the exploration policy, like the historical counts of the arms, the time-step, arms rewards variances are used on top the the confidence scores from the policy optimizer. It is shown empirically that these extra features add value.
Overall I think this is a well-written paper with very thorough experimentation. The results are also promising. It would be interesting to gain some insights from the learnt policy, in order to improve hand-designed policies.
For example, in a few data-sets it would be interesting to see whether the learnt policy is similar to epsilon greedy in the early stages and switches to greedy after a point, or which of the hand-designed strategies like bagging/cover is the learnt policy most similar to in terms of choice of actions, however I am not sure how such an analysis can be done.  It would also be fair to discuss the offline training time and online run-time of the algorithm with respect to others.
Also, I think the paper should provide a brief introduction to imitation learning, as it is commonly not known in the bandit community.
We thank the reviewer for recognizing the contribution of our method. We answer each of the improvement points below.
1) Analysis for Learnt Policy:
We agree that it’d be interesting to analyze and gain more insight from the learnt policy to design better exploration algorithms for contextual bandits, however, it’s not clear how to perform this analysis. One possibility is to track the exploitation / exploration decisions made by the learnt policy over-time. We can also compute feature importance estimates or perform an ablation study for the features used by MELEE. Similarity between MELEE and other exploration algorithms in terms of the selected action could also be analyzed.
However, these results could be highly dependent on the underlying dataset properties.
2) Offline training time and online runtime:
In our experiments, the online runtime for MELEE was similar to epsilon-greedy & exponentiated gradient epsilon-greedy. MELEE was faster than both Cover (which requires a bag of policies) and LinUCB (which requires an inversion for the estimated covariance matrix. Offline training for MELEE requires more time for generating the synthetic data and running the imitation learning algorithm. We trained the model used in our experiments for approximately one day. We will provide exact statistics about the training time and the online runtime performance for MELEE in the final version for this work.
3) Introduction to Imitation Learning:
We thank the reviewer for highlighting this issue. We will include a more detailed introduction for imitation learning in the final version for this work.

Meta-review:
role: AC
data: metareview: This paper provides an interesting strategy for learning to explore, by first training on fully supervised data before deploying that policy to an online setting. There are some concerns, however, on the realism and utility of this setting that should be further discussed. If the offline data is not related to the contextual bandit problem, it would be surprising for this to have much benefit, and this should be better motivated and discussed. Because there are no theoretical guarantees for exploration, a discussion is needed and as suggested by a reviewer the learned exploration policies could be qualitatively examined. For example, the paper says "While these approaches are effective if the distribution of tasks is very similar and the state space is shared among different tasks, they fail to generalize when the tasks are different. Our approach targets an easier problem than exploration in full reinforcement learning environments, and can generalize well across a wide range of different tasks with completely unrelated features spaces." This is a pretty surprising statement, that your idea would not work well in an RL setting, but does work well in a contextual bandit setting. 

There should also be a bit more discussion comparing to previous approach to learn how to explore, including in active learning. It is true that active learning is a different setting, but in both a goal is to become optimal as quickly as possible. Similarly, the ideas used for RL could be used here as well, essentially by setting gamma to 0. 

Overall, the ideas here are interesting and well-written, but need a bit more development on previous work, and motivation for why this approach will be effective. 

Decision: Reject