Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview

The authors propose a reinforcement learning approach to learn a general active query policy from multiple heterogeneous datasets. The reinforcement learning part is based on a policy network, which selects the data instance to be labeled next. They use meta-learning on feature histograms to embed heterogeneous datasets into a fixed dimensional representation. The authors argue that policy-based reinforcement learning allows learning the criteria of active learning non-myopically. The experiments show the proposed approach is effective on 14 UCI datasets.

strength

* The paper is mostly clear and easy to follow.
* The overall idea is interesting and has many potentials.
* The experimental results are promising on multiple datasets.
* There are thorough discussion with related works.

weakness

* The graph in p.3 don't show the architecture of the network clearly.
* The motivation of using feature histograms as embedding is not clear.
* The description of the 2-D histogram on p.4 is not clear. The term "posterior value" sounds ambiguous.
* The experiment sets a fixed budget of only 20 instances, which seems to be rather few in some active learning scenarios, especially for non-linear learners. Also, the experiments takes a fixed 20K iterations for training, and the convergence status (e.g. whether the accumulated gradient has stabilized the policy) is not clear.
* Are there particular reasons in using policy learning instead of other reinforcement learning approaches?
* The term A(Z) in the objective function can be more clearly described.
* While many loosely-related works were surveyed, it is not clear why literally none of them were compared. There is thus no evidence on whether a myopic bandit learner (say, Chu and Lin's work) is really worse than the RL policy. There is also no evidence on whether adaptive learning on the fly is needed or not.
* In Equation 2, should there be a balancing parameter for the reconstruction loss?
* Some typos
    - page 4: some duplicate words in discriminative embedding session
    - page 4: auxliary -> auxiliary
    - page 7: tescting -> testing


review_title: nice idea, needing more experiments

final_score: rating: 7: Good paper, accept
review_content: This reviewer has found the proposed approach quite compelling, but the empirical validation requires significant improvements:
1) you should include in your comparison Query-by- Bagging & Boosting, which are two of the best out-of-the-box active learning strategies
2) in your empirical validation you have (arbitrarily) split the 14 datasets in 7 training and testing ones, but many questions are still unanswered:
 -  would any 7-7 split work just as well (ie, cross-validate over the 14 domains)
 - do you what happens if you train on 1, 2, 3, 8, 10, or 13 domains? are the results significantly different? 

OTHER COMMENTS:
- p3: both images in Figure 1 are labeled Figure 1.a
- p3: typo "theis" --> "this" 

Abe & Mamitsuksa (ICML-1998). Query Learning Strategies Using Boosting and Bagging.
review_title: This is an exciting paper on an important topic (active learning), but it suffers from a weak empirical evaluation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The approach solves an important problem as getting labelled data is hard. The focus is on the key aspect, which is generalisation across heteregeneous data. The novel idea is the dataset embedding so that their RL policy can be trained to work across diverse datasets.

Pros: 
1. The approach performs well against all the baselines, and also achieves good cross-task generalisation in the tasks they evaluated on. 
2. In particular, they alsoevaluated on test datasets with fairly different statistics from the training datasets, which isnt very common in most meta-learning papers today, so it’s encouraging that the method works in that regime.

Cons: 
1. The embedding strategy, especially the representative and discriminative histograms, is complicated. It is unclear if the strategy is general enough to work on harder problems / larger datasets, or with higher dimensional data like images. More evidence in the paper for why it would work on harder problems would be great. 
2. The policy network would have to output a probability for each datapoint in the dataset U, which could be fairly large, thus the method is computationally much more expensive than random sampling. A section devoted to showing what practical problems could be potentially solved by this method would be useful.
3. It is unclear to me if the results in table 3 and 4 are achieved by retraining from scratch with an RBF SVM, or by freezing the policy network trained on a linear SVM and directly evaluating it with a RBF SVM base learner.

Significance/Conclusion: The idea of meta-learning or learning to learn is fairly common now. While they do show good performance, it’s unclear if the specific embedding strategy suggested in this paper will generalise to harder tasks. 

Comments: There’s lots of typos, please proof read to improve the paper.

Revision: I thank the authors for the updates and addressing some of my concerns. I agree the computational budget makes sense for cross data transfer, however the embedding strategy and lack of larger experiments makes it unclear if it'll generalise to harder tasks. I update my review to 6. 
review_title: A novel meta-learning way to do active learning, slightly complicated embedding strategy, needs more evidence to show if it'll generalise to more challenging problems. 

Meta-review: In general, this seems like a sensible idea, but in my opinion the empirical results do not show a very compelling margin between using *entropy* as an active learning selection criterion vs the proposed methods. The difference is small enough that in practice it is very hard for me to believe that many researchers would choose to use the meta-learning via deep RL method (given that they'd need to train on multiple datasets and tune REINFORCE which is not going to be obviously easy). For that reason I am inclined to reject the paper.

In a follow-up version, I would heed the advice of Reviewer 1 and do more ablation analyses to understand the value of myopic vs non-myopic, cross-dataset vs. not, bandits vs RL, on the fly vs not (these are all intermingled issues). The relative lack of such analyses in the paper does not help in terms of it passing the bar.
Decision: Reject