Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel training framework which adaptively selects informative samples that are fed to
the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model.
The idea is intuitive and easy to follow. Experimental results demonstrate the efficacy of the proposed method.
I have two questions about this work:
1. The informative samples are fed to the training process, what about the rest "non-informative" ones?
2. What is the characteristics of the selected informative samples? i.e., for a class of images, which images should be informative?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes a method to efficiently select hard samples during the training of a neural network. This is achieved via a variational auto-encoder (VAE) that encodes the samples into a latent space. The VAE is trained in a preparation stage using the images only and fixed at later stage. During training of a DNN framework, samples are selected in the latent space and then decoded via the Decoder in VAE to generate the input for DNN framework. The advantage of such a framework is that now it is able to calculate the gradient w.r.t. the input samples of DNN. This gradient is used to determine the sampling strategy in the next iteration to select harder samples. Two different sampling methods are explored, including nearest neighbor and interpolation (with annotation tool step). The experiments are conducted on small-scale datasets like MNIST CIFAR-10, and IVUS MSE with satisfactory gain over the baselines. Overall, the paper is very well-written and easy to follow. Although the experiment results are not super exciting mainly because of small-scale datasets and not enough gain in the numbers, some of the analysis in Figure 4 are quite insightful to validate the assumption and motivation of this work. So I propose to accept this work for its novelty. I think this work will benefit future research in this direction. 

However, I do have some concerns that I wish the authors could clarify if possible. First, the approach is very similar to online hard negative mining (OHNM) that is purely based on the loss to repeatedly select the samples that generate a larger loss. The major difference is that this work can model the sample distribution and thus select samples based on the gradient w.r.t. the samples in the latent space. This is very novel to me. However, I am wondering if the authors could compare with this sample baseline of OHNM. My concern is that the baselines in this work is too simple and it is not surprise that there is advantage over a simple baseline that is trained without any hard sample mining. 

Second, the experiments are all conducted on small-scale and simple datasets like MNIST and CIFAR10. I am concerned how effective this approach could work for large-scale dataset. In the experiment, even for CIFAR10, a vanilla VAE will not work to reconstruct the input. So the authors have used alpha-GAN to help image reconstruction. If that is the case for CIFAR10 with only 10 classes, how could we extend this work to even larger dataset with more complicated background like ImageNet? I would think the preparation step itself is a very challenging task. This is my major concern that will question the effectiveness of the approach in real applications. 

Third, a related question to the above one. As the input to the DNN is the reconstructed image from the pre-trained decoder, there will be some information loss during the reconstruction process. This is the major challenge, I think, for large-scale applications. Is that possible to use the original image as the input to DNN while still being able to find hard samples using the latent space and the image space correlation? 

Fourth, I really like the visualization of Figure 4 that shows the trajectory of the sampling process that follows the boundaries between classes. The authors also mentioned that some trajectories explore towards outside util there is no real samples, which should be avoided. Could the authors comment on how to avoid such cases? In my understand, as the input is randomly sampled at the beginning, it cannot avoid such cases unless some evaluation is done during training to stop the sampling for these trajectories. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper proposes a method for sequential and adaptive selection of training examples to be
presented to the training algorithm. The selection happens in a latent space, based on choosing
samples which are in the direction of gradient of the loss in the latent space. Two selection
strategies are investigated: nearest neighbor and interpolation followed by generation. Results on
shown on MNIST, CIFAR10 and IVUS (Intravascular Ultrasound) datasets.
                                                                
                                        
Detailed comments:                      
The proposed method works in two stages. First a VAE is trained using unannotated samples. In the
second stage, hard examples are found, in every iteration, in the latent space of the VAE and used
for sequential training. The sampling is done using the gradient of the objective function in the
latent space. The method makes sense, however the choice of space in which the sample selection is
being done is not well motivated or validated. The space could have been the original image space
(although given the high dimension, it would probably not work), or could have been any intermediate
feature space. Why was the space chosen to be the VAE latent space? Would it be possible to 
demonstrate some benefits of doing so, theoretically and empirically? 
                                        
The experiment section is relatively weak. The datasets used are relatively small and in two out of
the three datasets, the method does not improve. The dimension of the latent space is also
surprisingly small (2). While the main body of the paper describes the method with VAEs, the
experiments for the CIFAR10 dataset (where the results were in favor) were done \alpha-GAN. 
review_title: Official Blind Review #1

Meta-review: VAE-based sample selection for training NNs.  A well-written experimental paper that is demonstrated through a number of experiments, all of which are minimal and from which generalization is not per se expected.  The absence of an underlying theory, and the absence of rigorous experimentation makes me request to extend either or, better, both.  
Decision: Reject