Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper evaluates Bayesian Neural Networks for active learning on CIFAR10 dataset. It investigates incremental vs full-batch training for network updates. It uses more complex dataset (CIFAR10 over MNIST) than a prior work that did comparison only on MNIST (Gal et. al., 2016). 

Pros: 
-Simple and clear presentation
-It presents episode-based active learning setting, which is closer to application scenarios in robotics

Cons:
-It ignores comparing against different acquisition functions and classifiers, which are important for evaluating good active learning techniques, and instead only compares simple and heuristic ways to pick incremental or full data to train on
-Improvements are small. In addition, it’s reasonable to show accuracy on 70% randomly selected data, etc. 
-It has limited novelty, since Gal et. al., 2016 already applied BNN for active learning. They also have updated paper with new results (https://arxiv.org/abs/1703.02910)

review_title: Limited novelty and significance

final_score: rating: 4: Ok but not good enough - rejection
review_content: The submission investigates episodic active learning - meaning the additional samples arrive sequentially in batches and groundtruth labels can be acquired from an oracle in an active learning fashion.

more specifically, the submission investigates bayesian neural networks. the choice is not well motivated and a bit unclear. no comparison is provided.

Different strategies for step-wise training and label acquisition are evaluated.
The evaluation is diffuse as there are multiple competing goals, e.g.:
- queries to the oracle
- accuracy of the final model
- accuracy of intermediate models
- computation time (?)

for the first three plots are shown - but the results are presented in a way that makes the strategies difficult to compare.

main points criticism:
- it is strange that the fully supervised case performs slightly worse than two of the incremental approaches. it might be noise or there might be a problem with the supervised baseline. there is no satisfying explanation to this observation in the submission
- the submission seems too much out of context. no direct related work is cited to the problem; the theme of sequentially retrieving labels is common to most active learning and experimental design papers; performance is typically plotted over the number of samples.
- no baselines are computed - only the 5 strategies the authors came up with 
- no prior strategies were drawn from the related work

the authors make a point in the conclusion that their setting doesn't allow to re-observe  samples. but from reading the setup, it seems this is only true for the active learning scheme that is only allowed to pick from the current batch/pool (training is still performed on all selected samples by some strategies). but if this is an important point - the submission fails to highlight it's importance in the experiments. a baseline should be shown - where the active learning is picking from all the whole set. but the suspicion remains that there is not too much of a difference. therefore i would be important to show those numbers.
review_title: review

Decision: Reject