Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an approach to learning active learning policies that rely on imitating the behaviour of known heuristics. The resulting learnt policies can successfully transfer between the related datasets. 

Strong points:
- This paper focuses on the important problem and attempts to provide a simple solution that leverages the prior knowledge about the problem (existing heuristic solutions) while being data-driven.
- The experimental studies are quite informative and detailed and provide analysis of the behaviour of the proposed solution.
- The proposed method outperformed the presented baselines in most cases.
- The experiments with policy transfer between different datasets seem very interesting and focus on the realistic and important setting.
- The experiments with ablations by excluding various experts are informative and insightful.

Weak points:
- The novelty of the proposed method is not very strong: The main idea is to select the best behaviour among several heuristics, that was explored before (Hsu&Lin, 2015, 1, 2, 3), the learning mechanism relies on imitation learning, that was also explored in previous works for AL (Liu et al, 2018), and the implementation in terms of state parametrization seem to be similar to previous works too (Contardo et al, 2017, Konyushkova et al., 2017, Liu et al, 2018).
- This paper puts an emphasis on the batch nature of the proposed method (for example, when contrasting against the related work). However, I did not understand how the policy training is particularly targeted to the batch setting.
- The nature of the proposed method is the most similar to the algorithms that select a particular heuristic to use at each step of the training. While the authors mention some of such algorithms (Hsu&Lin, 2015), I believe a few others works deserve some attention [1, 2, 3]. Importantly, there is no comparison to the methods from this group.
- The authors claim that the imitation learning is done by the DAGGER method. However, as far as I understand, DAGGER makes an assumption that an expert can provide an optimal action from any state as supervision. If I understand the method correctly, such an assumption cannot be satisfied in the setting of the paper. To my mind, it sounds that the method is closer to supervised learning with epsilon-greedy data collection.
- Most experiments are performed on rather low dimensional data in MNIST, FMINST and KMNIST. The transfer to more complex datasets such as CIFAR is shown in the appendix, but the gains by the proposed method seem to be marginal.


I am leaning towards the rejection of this paper in the light of limited novelty of the presented method and limited gains especially on more complex realistic datasets.

Questions:

- Section 4.4 contains a curious and counter-intuitive observation: "there is an upper limit to the size of D_{such} after which performance degrades again". However, the authors did not provide any explanation of this phenomena. Why is this happening?
- Could the authors elaborate a bit more on the connection between the proposed learning mechanism and DAGGER?
- I think the most significant part of the method is the ways how the training data is collected with supervision of the experts. While this procedure is covered in section 3.3, I think it deserves a bit more attention. For example, I am still not sure if and how *batches* of data are selected by heuristics strategies.
- Why do the authors say that the method is closer to the baselines "towards the end obviously"? Is the most of the data labelled at that point?
- The learning curves in many experiments seem to be quite noisy. How many experiments are averaged?
- How do the baselines compare to the proposed method with varying batch size (as some methods are particularly targeted to batch size=1 or large batch size>100)?

Additional comments:

- The language in the paper often uses unusual word ordering and constructions that makes it difficult to read. For example, "as ideally we label", "besides the above mentioned", "do still not work on batches".
- A lot of experimental results are presented by training and testing the policy on the same dataset. As this is a rather unrealistic setting and serves mostly as sanity check, less space in the paper could be dedicated to it.


[1] Online choice of active learning algorithms. Y. Baram, R. El-Yaniv, and K. Luz. JMLR, 2004.
[2] Can active learning experience be transferred? H.-M. Chu and H.-T. Lin. ICDM, 2016.
[3] Active learning by learning. W.-N. Hsu and H.-T. Lin. In American Association for Artificial Intelligence Conference, 2015

===Post rebuttal===

I have read the authors response and would like to thank the authors for the clarifications. I am still inclined to keep my original score.

review_title: Solving active learning by learning to imitate the heuristics

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a batch-mode active learning approach via imitation learning to derive a sample selection strategy for active learning. The proposed method makes use of a pool of existing heuristics, learning a sample selection strategy that rely on different heuristics in the pool depending on the different stage of the active learning process. The use of imitation learning makes sense, as it helps to alleviate the high sample complexity problem of deploying a reinforcement learning strategy. Furthermore, a batch-mode active learning strategy is desirable compared to the previous work (Liu et al. 2018) that uses a single sample. Experimental results on MNIST,  FMNIST, and KMNIST demonstrates the utility of the proposed method. I have the following concerns:

1. While I think the idea of framing the active learning problem as an imitation learning problem has merits, it is unclear to me how the proposed framework is different from that of (Liu et al. 2018). The authors point out that (Liu et al. 2018) achieves single-sample mode active learning but the proposed framework achieves batch-mode active learning. However, the way that batch-mode is achieved in this paper seems trivial by computing and using the top-k instances according to certain metrics.

2. It is also unclear how the pool of experts used in the paper generalizes beyond the setting in (Liu et al. 2018), and why this might be desirable. Given this concern and the previous one, it would be great if the authors can clarify the relationships between the two works so as to better highlight the specific contributions made in this work.

3. In terms of empirical validation, is it possible to also compare the proposed method with (Liu et al. 2018)? Furthermore, the authors demonstrate the utility of the proposed method in three related datasets (MNIST,  FMNIST, and KMNIST). I am not sure if such empirical validation is sufficient. How about other datasets that contain digits such as SVHN? Is it possible to imitate on CIFAR 10 and transfer on CIFAR 100? How about NLP tasks that are reported in (Liu et al. 2018)? On the other hand, I think the ablation studies provided by the authors do help to demonstrate how different components of the framework will play and influence the eventual outcome.

Miscellaneous:
1. There is a missing comma in the last sentence of the paragraph right before Section 3.1?

review_title: The proposed framework makes sense, but exact contributions need to be clarified.

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Paper summary:

In this work, an imitation learning (AL) approach is proposed to imitate multiple active learning algorithms, in order to take their advantages to learn a better active learning algorithm. The main idea is to treat the active learning algorithms as experts and utilize the DAGGER algorithm for imitation learning. The proposed approach is evaluated on MNIST, fashion-MNIST, and Kuzushiji-MNIST, showing that the learned active learner outperforms baseline active learners, meanwhile is transferrable to other datasets.

#### Advantages:

- The idea of using imitation learning is interesting.

- The survey of existing active learning approaches is quite thorough.

#### Disadvantages:

- The idea of using DAGGER to combine multiple active learners was explored in Liu et. al. https://www.aclweb.org/anthology/P18-1174. The technical novelty is thus limited. 

- The experiments are not sufficient for the following perspectives:

  - The performance gain is somehow not very significant, the datasets are restricted to images with the same number of classes. 

  - The policy utilizes the features of training instances as part of the states. While I am in doubt of why such features can be transferrable to new datasets? 

  - There are no comparisons to the work of Liu et. al., which also utilizes DAGGER to imitate active learners.

Overall evaluation:

I think the paper proposes a reasonable solution, while the contributions and novelty are somehow below the standard of ICLR.

Additional:

- The methodology for the proposed approach is similar to the "learning to teach" approach https://openreview.net/pdf?id=HJewuJWCZ, which learns a teaching policy for selecting the smallest set of labeled instances to accelerate learning. Adding some discussions to their work can be useful.

- The "test set" in algorithm 1 should be "validation set".

#### Update after rebuttal:
Thanks for the detailed feedback. While I still find the technical novelty is limited comparing to [Liu et. al.]. For this reason, my score remains unchanged.

review_title: good idea for using imitation learning to combine the advantages of multiple active learners, while the approach and the experimental evaluation are not sufficiently sound

Meta-review: In the discussion, all reviewers acknowledge the novelty of this paper, such as learning from a wide range of AL heuristics, and the ability to transfer the to tasks with arbitrary number of classes. They also think that the additional experiments provided by the authors improve the paper's empirical validity. 

However, a major issue raised by the reviewers is that the novelty (especially when compared with Liu et al) may not be enough for ICLR this time. One research direction (implicitly suggested by Reviewer 2) was to learn active learning strategies that go beyond selecting top-k scoring examples to explicitly account for batch diversity. We encourage the authors to address this in the next version.
Decision: Reject