Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: •	Summary
    This paper introduced a meta-learning algorithm for the contextual bandit problem, MELEE, which learns an exploration policy based on simulated and synthetic contextual bandit tasks. The training is mainly divided into two steps. In step one, they proposed to train a policy optimizer, which maps features and actions to rewards. This policy optimizer could be used to reveal the most valuable action to take according to the modeled reward. All possible actions and their corresponding values are revealed to the policy optimizer because of the existing ground-truth labels in the synthetic dataset. The policy optimizer would then suggest which action to take. The algorithm takes the action in an  greedy fashion, i.e. with probability  it will follow the suggestion and with probability  it will sample it uniformly at random. The policy optimizer, historical actions and the taken actions are appended to the training set for training the exploration policy  in the next step. The procedure in step one is proposed to be done in  rounds. In step two, the training set is used for training an exploration policy  . During testing, the contexts are drawn from the real world, the policy optimizer will first evaluate the whole history and the exploration policy will generate actions with the input from the policy optimizer and the context. The algorithm suggests the action to explore in an  greedy fashion. The proposed algorithm is evaluated on a dataset for learning to rank, and 300 synthetic datasets. It shows better performances in most cases.
    I am critical about the paper  because 1) the experiments show that all more recently published exploration methods are worse than weak baselines, for which it lacks enough justification and convincing explanations. 2) the experiments are difficult to understand with only citation to publications. Detailed information of the datasets, tasks, procedures is missing.
•	Main arguments
    The paper is in general hard to follow because of too many citations of the previous works without simple explanations. It refers to the imitation algorithm, AggreVate, which is an instantiate of meta-learning for contextual bandits. Meanwhile, they failed to clarify the difference between the proposed algorithm and the existing one, making the training algorithm part confusing. The major concern lies in the experiment section, I can not see big performance difference between the proposed method and the  greedy based methods in Figure 1 (left) as 1) the variances are large 2) there are overlapping . It is surprising to see all recently published method are worse than the classical  greedy method. These results may require deeper investigations. In addiction, the used datasets in experiments are mentioned without any details and task definitions, which makes the experiments part unclear and the result not that convincing. Below are some other inconsistencies in the paper:
1.	I had a hard time to understand what the function  means. In Section 2,  is used to map user feature to predicted rewards for actions, i.e.  is a function with user features and actions as input and reward as output. However, in equation 1, it only takes user feature as input. In Section 5.1,  is called a classifier (I regard it as a variant of function  ). Both are not consistent with the mapping definition when first formally defined. This point confuses me so that I cannot fully understand what the POLOPT does as the function  is the output of it.
2.	By the end of section 2, the paper claims that they used direct method for it simplicity and unbiased property. However, as verified in [1], the direct method is biased with low variance whereas IPS is unbiased with high variance.
[1] Dudık, Miroslav, John Langford, and Lihong Li. “Doubly Robust Policy Evaluation and Learning.”


•	Things to improve that did not impact the score
a.	clear definition of the introduced notations, including the ones in algorithms 1.
•	Questions:
a.	What is  ?
b.	What is POLOPT?
c.	Why do you choose AggreVate?
d.	Will the performance change if using any other methods instead of direct method for policy optimizer?
e.	What is roll-out policy in line 8 of Algorithm 1? (In text only the roll-out value is defined.)
f.	What will happen if the roll-in action is different from the behavior in test time?
g.	How is exploration policy trained?

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a meta-learning algorithm to solve the problem of exploration in a contextual bandit task using prior knowledge. This is analogous to how exploration strategies have been learned from past data in the meta-learning for RL (for example, [1]). Their algorithm simulates contextual bandit problems from fully labeled data and uses it to learn an exploration policy that works well on tasks with bandit feedback. The training step for the exploration policy builds upon the AggreVaTe algorithm, where policy optimization is performed on the history augmented data by using a separate roll-out policy to estimate the advantage of a particular action for a particular context, from the point of view of regret minimization.  In terms of theory, they show that by using specific algorithms (example, Banditron) as the inner policy optimization procedure, a no-regret algorithm can be obtained.   

Some of my questions are:

1. The method seems a bit hacky to me, for example, it requires calibration of f, requires access to test time examples, and it is unclear why this should be needed if an algorithm were provably good. Can this be elaborated upon?

2. Section 3.2 and Algorithm 1 are very unclear, and it requires multiple passes to be able to understand in the current draft. I encourage the authors to revise these sections for increasing clarity. 

3. In Algorithm 1, which is the exploitation policy? The theory section says " In particular, we first relate the
regret of the learner in line 16 to the overall regret of $pi$", but line 16 in the algorithm refers to "end for". Can the authors clearly point me to the location where the exploitation policy is mentioned in the algorithm? What is the precise definition of $\pi_n$ (Line 15), and how does it relate to the average exploration policy and the policy $\bar{\pi}$ used in Theorem 1. These definitions have not been made clear, making this hard to follow. What is the optimal policy $\pi^*$, in particular, what is the formal meaning of "which effectively cheats” at training time by looking at the true labels."

4. The paper lacks discussion about how the roll-out and the roll-in policy affect regret bounds, and what assumption about these is used to derive no-regret guarantees. Can this be described? 
 
Overall, my recommendation currently is reject. I feel that the paper tries to theoretically analyse an interesting problem, however, at this point the paper is very hard to follow and not complete. I encourage the authors to revise the details and improve clarity. Also, it would be good if the authors can explain the significance of the results at the cost of added complexity.

References:
[1] Meta-exploration of structured exploration strategies, Gupta et.al. NeuRIPS 2018
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper presents an approach to learning exploration strategies for contextual bandits by meta-learning the exploration strategy across synthetic tasks. The approach is shown to outperform alternative exploration strategies on a learning to rank task as well as simulated bandit problems derived from classification tasks.

Applying meta-learning to this setting is a novel and interesting approach and appears to have nice results.

The main issue with this paper is that the presentation of the algorithm is vague and it is difficult to determine what the algorithm actually does. Here are a few of my main questions:

1. The algorithm is dependent on a function class F and PolOpt which finds an f in F with low expected regret. However, F is not really defined. In equation 1, it appears to directly output an action. Below that, it says "For example, F may be... mapping user features x in X to predicted rewards for actoins a in [K]." In Section 5.1, the policy is using a probability distribution over actions from f, the entropy of the predicted probability distribution, and a one hot encoding for the predicted action f(a). What is the class F? Is it predicting a probability distribution and an action? What's the one hot encoding of the predicted action, is that an action sampled from the probability distribution of F?

2. Section 3.1 describes the test time behavior of MELEE. They state the goal is to learn an exploration policy pi, and specifically state that pi can "not depend directly on the context x" so it "only learns to explore and not also to solve the underlying task-dependent classification problem". However, in the equations, pi is a function of x_t. The algorithm is motivated by the exploration exploitation trade-off problem, however there's no policy doing any exploiting here.  When does exploitation happen? Is it a separate policy? How is it trained and when is it used? The decision of when to explore and when to exploit is the key question here and it's unclear how that is happening in this algorithm. It's unclear to me how the algorithm achieves good rewards on the task without any attempt at exploitation.

3. What are the rollouts and expert cost-to-go in this setting? In section 3.2, you say the algorithm asks what would the "long time reward look like if I were to take this action." So are you looking at how that action would affect future exploitation? You say there's an optimal reference policy* which effectively cheats by looking at the true labels. Is this policy taking the action with the highest reward at each state? Or is it looking at the effects of exploration on the learned policy? The start of page 4 says that the "cost-to-go Q of the expert" is observed, why would the expert in a bandit have a cost-to-go? In the section on roll-out values, you say that you evaluate the value of taking this action and then assuming all future actions were taken by the expert. But this then precludes there being any effect of this action on the policy and any exploration effect. Are you solely optimizing for the one step reward? In the algorithm, the rollouts computed on line 8 don't appear to be used anywhere.

4. How are the synthetic tasks setup? There are almost no details other than that they are two dimensional and have uniformly distributed class conditional distributions. How are they related to the learning to rank task you perform at the end? How are they related to the other tasks you evaluate on? It seems like the choice of synthetic tasks would be important to what the algorithm learns, so it's important to be clear on what these tasks were and how closely they match the target tasks. It would be even better to show results with different sets of synthetic tasks that are more or less similar to the target tasks.

5. In the results, you say you limit the number of labels to two extremes, 0 and 4. I assume you mean you limit it to just those two labels. What do you do with the other data? Do you drop it or do you map it to these two labels?
review_title: Official Blind Review #1

Meta-review: This paper introduces MELEE, a meta-learning procedure for contextual bandits. In particular, MELEE learns how to explore by training on datasets with full-information about what every reward each action would obtain (e.g., using classification datasets). The idea is strongly related to imitation learning, and a regret bound is demonstrated for the procedure that comes from that literature. Experiments are performed. 

Perhaps due to the generality in which the algorithm was presented, reviewers found some parts of the work unintuitive and difficult to follow. The work may greatly benefit from having an explicit running example for F and pi and how it evolves during training. Some reviewers were not impressed by the experimental results relative to epsilon-greedy. Yes, epsilon-greedy is a strong baseline, but MELEE introduces significant technical debt and data infrastructure so it seems fair to expect a sizable bump over epsilon-greedy or else why is it worth it?

Perhaps with revisions and experiments within a domain that justify its complexity, this paper may be suitable at another venue. But it is not deemed acceptable at this time, Reject.  
Decision: Reject