Efficient Deep Representation Learning by Adaptive Latent Space Sampling
Authors: Yuanhan Mo
Shuo Wang
Chengliang Dai
Rui Zhou
Zhongzhao Teng
Wenjia Bai
Yike Guo
Venue:  (2020)
Keywords: Deep learning
Data efficiency

Abstract: Supervised deep learning requires a large amount of training samples with annotations (e.g. label class for classification task, pixel- or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain. During the training of a deep neural network, the annotated samples are fed into the network in a mini-batch way, where they are often regarded of equal importance. However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples. In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation. To address the challenges of expensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed to the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model. To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and a medical image dataset IVUS for biophysical simulation task. On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of our framework.

Under review as a conference paper at ICLR 2020
EFFICIENT DEEP REPRESENTATION LEARNING BY ADAPTIVE LATENT SPACE SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
Supervised deep learning requires a large amount of training samples with anno- tations (e.g. label class for classification task, pixel- voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain. Dur- ing the training of deep neural network, the annotated samples are fed into the network in mini-batch way, where they are often regarded of equal importance. However, some of the samples may become less informative during training, as the magnitude of the gradient start to vanish for these samples. In the meantime, other samples of higher utility or hardness may be more demanded for the training process to proceed and require more exploitation. To address the challenges of ex- pensive annotations and loss of sample informativeness, here we propose a novel training framework which adaptively selects informative samples that are fed the training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model. To evaluate the proposed training framework, we perform experiments on three different datasets, including MNIST and CIFAR-10 for image classification task and medical image dataset IVUS for biophysical simulation task. On all three datasets, the proposed framework outperforms a random sampling method, which demonstrates the effectiveness of proposed framework.
INTRODUCTION
Recent advances in deep learning have been successful in delivering the state-of-the-art (SOTA) performance in a variety of areas including computer vision, nature language processing, etc. Not only do advanced network architecture designs and better optimization techniques contribute to the success, but the availability
large annotated datasets (e.g. ImageNet (Deng et al. 2009), MS COCO (Lin et al. 2014), Cityscapes (Cordts et al. 2016)) also plays an important role. However, it is never an easy task to curate such datasets. Collecting unlabeled data and the subsequent annotating process are both expensive and coneuconnum In particular, for some applications such as medical imaging, the annotation is limited by the available resources of expert analysts and data protection issues, which makes it even challenging for curating large datasets. For example, takes hours for an experienced radiologist to segment the brain tumors on medical images for even just one case.
On the contrary supervised deep learning, human beings are capable of learning a new behaviour or concept through the most typical cases rather than accumulative learning for a lot of cases. Intu- itively, we may ask: Is it really necessary train deep neural network with massive samples? Are we able to select a subset of most representative samples for network training which can save the annotation cost, improve data efficiency and lead to an at least equivalent or even better model? To the best of our knowledge, this is less explored domain in deep learning and relevant applications, where a lot of efforts have been put into optimizing the network designs. Rather than improving the performance of neural network given curated training set, here annotated samples can be more efficiently utilized to reach level of performance. We consider such property "data efficiency'. namely how efficient learning paradigm utilizes annotated samples to achieve a pre-defined performance measure.
more interested in how
In this paper, propose model state-aware framework for data-efficient deep representation learn ing, illustrated in Figure [ The main idea is to mine 'harder' training samples progressively on the data manifold according the current parameter state of network until certain criteria is fulfilled

---

Under review as a conference paper at ICLR 2020
Samples
Encoder
Decoder
Samples
Latent space
T no
7
Ground truth
Labeling tool
Loss (ly).
Decoder
Samples
Adaptive sampling
Prediction
Deep
Backpmpagation
Figure The general pipeline of proposed framework. The preparation stage is located at the top left corner which represents the training of a variational auto-encoder (VAE) using unannotated samples. The main stage is located within the dashed rectangle, where the decoder (generator) as well as its latent space are used for mining hard training samples according to the error information propagated backward via the target model and decoder (generator). Each proposed sample will be annotated by the labeling tool.
(e.g. size of training dataset or performance on validation dataset). The harder samples with respect given network state defined those yielding higher loss, which are estimated through back- propagation (Hinton et al. 2006). To be able to select plausible harder samples, generative model employed for embedding data into low-dimensional latent space with better compactness and smoothness. In particular, we investigate two sampling strategies in the latent space, namely sam- pling by nearest neighbor (SNN) and sampling by interpolation (SI) for different applications. The data efficiency of our framework is evaluated on three datasets, including MNIST and CIFAR- 11 for image classification tasks, as well as a medical image set IVUS for biophysical simulation task.
There are three major contributions of this work:
A general and novel framework is proposed for model state-aware sampling and data- efficient deep representation learning, which be used in variety of scenarios with high annotating cost.
2. Unlike previous studies (Sener & Savarese 2017 Peiyun et al.) 2019), generative model is introduced propose informative training samples. Two latent space sampling strategies are investigated and compared.
3. The framework is only applicable for sampling on an existing dataset, but it also al- lows suggestive annotation and synthesizing new samples. We demonstrate the latter in biophysical simulation task, where artificial samples are synthesized from the latent space.
RELATED WORK
2.1 FEW-SHOT LEARNING
In recent years, few-shot learning (FSL) has received a lot of attention, which is relevant to this work in terms of improving data efficiency in training. Few-shot learning was firstly proposed by Fei-Fei et al. (2006) and Fink (2005). Fei-Fei et al. (2006) proposed Bayesian implementation which leverages knowledge from previously learned categories for one-shot image classification categories may contain only training sample. Instead of directly learning for the few-shot tasks, similarity learning aims to learn the relevance of given two objects from a limited
task, where
number of samples (Koch et al. 2015
Fink 2005
Snell et al. 2017 Kulis et al. 2013 Vinyals
2

---

Under review as a conference paper at ICLR 2020
et al., 2016; Kim et al. 2019). For example, Koch al. (2015) proposed the Siamese network for the few-shot classification by calculating the similarity between labeled image and a target image. [nell et al. (2017) proposed neural network that maps the samples into metric space. Then the classification can be performed by computing distances to the prototype representations of each class. |Vnyyal al. (20T6) proposed to learn deep neural network that maps small laheled support set and an unseen example to its label, avoiding fine-tuning to adapt to new class types. It is noted that most FSL methods focus on the classification task. Some recent methods start address other tasks such as object detection etc (Shrivastava et al. [016)] [Schwartz et al., 2018: Zhang et al.) 2019 [chonfeld et al. 2019). Schwartz et al. (2018) proposed jointly learn an embedding space and the data distribution of given categories in a single training process which brought new task into few-shot learning, namely few-shot object detection.
2.2 HARDNESS- LEARNING
This work is also related to the field of hardness-aware learning. The concept of mining hard exam- ples has been employed in different machine learning paradigms for improving training efficiency and performance (Schroff al., 2015 Huang al. 2016] Yuan et al. 2017; Malisiewicz et al., 2011 Wang & Gupta] 2015 Zheng et al. 2 Harwood et al. 2017). The main idea is to select those training samples that contribute the most the training process. For example, Schroof et al. (2015) proposed the FaceNet which employed a triplet training approach. A triplet loss was used to minimize the distance between an anchor and a positive points and maximize the distance between the anchor and a negative points. The pairs of anchor and positive points are randomly selected, pre-set criterion. To ad-
while hard pairs of anchor and negative points are selected according dress the issue of gradients being close to zero, HHawwood et al. (2017) combined the triplet model and the embedding space by using smart sampling methods. The smart sampling method is able to identify those examples which produce larger gradients for the training process.
2.3 ACTIVE LEARNING
Our work is also highly related to active learning which has been widely studied (Settles) 2009).
Sener & Savarese (2017) proposed to formulate active learning core-set selection problem. The core-set contains those points such that a model trained with them is competitive to a model trained with the rest points. [Peiyun al. (2019) were also addressing the similar problem in this paper, investigating whether it is possible to actively select examples to annotate. They proposed an novel active learning paradigm that is able assign label sample by asking series of hierarchical questions.
METHODOLOGY
Before we introduce the proposed framework in details, it is worth noting that all the quantities and notations used in the following sections are listed in Appendix A.I A.I
3.1 OVERVIEW
The proposed training framework consists of two stages as shown in Figure I The first stage can be considered preparation phase where a VAE-based generative model is trained using unan- notated samples. We obtain the generator (decoder) G and the encoder E with trained and fixed parameters as well as the n-dimensional latent space IR... the second stage, 'hard' samples are mined iteratively in the latent space and fed into the network for training. The sampling strategies are based on the error information (normalized gradients) of the neural network model F backpropa- gated through the generator G. In general, the hard samples can be either chosen from the candidates in the unannotated dataset or synthesized using the generator G. Accordingly, we investigate two sampling strategies named 'sampling by nearest neighbor (SNN)' and 'sampling by interpolation which will be introduced in the following sections. The sampling and used in the framework accordingly.
(SI)* as shown in Figure 2
Herr we VAE as example demonstrate the idea. Other kinds of generative models can be also
3

---

Under review as a conference paper at ICLR 2020
SNN
SI
Figure 2: The figure demonstrates two sampling methods proposed in this paper: SNN (left) and SI (right).
training process repeated until the stop criterion (the amount of training samples reaches pre-set value) is satisfied.
In the following sections, the core processes of proposed training framework will be outlined. Ex- tensive details to facilitate replication are given in Appendix A As the training of a generative model not a focus of this paper, details of the preparation phase also given in Appendix A
3.2 PROGRESSIVE TRAINING
As samples are mined and added to the train progressively, we denote the original dataset of all samples as DD and the mined set of hard samples used training the network T(1, where denotes an index round.
In the first round where t 1, the first incremental subset ( is constructed by randomly selecting pre-set size of samples from ID and sent to the labeling tools, resulting the initial train set T()) {(x1 31), (x2, y2). .... (æ., (4)) For f = 2, .... the neural network model F is first trained
for a couple of epochs with the current train set T()) using a given loss function L. Then
A=1
an incremental subset T()) with a pre-set size is constructed following the sampling strategies to propose more informative samples with respect to the current model state. The incremental subset t-1
T(t) will be added to the train set U (T)) forming - (T)) for the next round of training.
k=1
k=1
3.3 ESTIMATING DIRECTIONS TOWARD HARDER SAMPLES
Our method is designed to mine the most informative samples for training the model. To achieve t-]
this, we randomly select a few annotated samples {(11,)) y1), (x2, U2)...., (0......,,,,,, (æ yu)) from U T())
k=1
and identify the embedding position p for each æ in latent spaces using E. Then, we evaluate the given loss function L with ((F(G(p)). )), (F) (F(G(p))) 22), (((G(())).,.)) (F.CC))))))
as:
yk)
where (p))) the prediction via the generator G.(.) and the target model Po(-), and T. is any given metric. This would enable us to calculate the gradient ac for each sampling point p in op
the latent space R2 using the well-known back pock-propagation algorithm.
3.4 SAMPLING BY NEAREST NEIGHBOR
For each point p in latent space R", now there a corresponding gradient ac direction to increase the loss L. The normalized gradients denoted as d for each p. A new position p' that represents 'harder' sample for the model in the latent space IR" can be identified using Eq.
Op which is actually a
D p ad.
4

---

Under review as a conference paper at ICLR 2020
Algorithm The Proposed Framework Using Sampling by Nearest Neighbor
1:
f-6
2: Randomly Initialise training subset T(1) = { 31), (222, y2), (I)) C
3: while Condition is not fulfilled do
4: 8+ +++1
5: Train F with T() given loss function L
6:
< ((11,)) 91). (22, y2). (er,yy)) C T())
b Randomly select a subset
7:
p = (x)) for each (a, y) E
8:
Evaluate C with [(G(p)) yı), (G(p2); y2.). (G(p)), 9r)
9:
Calculate #c for each p using backpropagation
8p
10:
Identify harder point p' for each p by p' + -p-- + Identify the nearest neighbor p for each p' The new incremental set T {(G(p)), v1), (G(p2), (.) (G(p)))
0c
8p
12:
13:
return F
where is step size. For some scenarios where the labeling tool not feasible annotate synthesized sample G(p), a nearest neighbor from the embedding E(D) will be found for p. Euclidean distance used identify the nearest neighbors.
Finally, the nearest neighbors can be used for retrieving their corresponding training pairs in ID in order to construct a new incremental set T(t) <- {[G(p)), yı), (G(p2): y2)..., (G(p)), y))) for the next round training. The pseudo code using sampling by nearest neighbor can be found in Algorithm 1.
3.5 SAMPLING BY INTERPOLATION
For scenarios where the labeling tool is capable to annotate arbitrary reasonable inputs (e.g. an equation solver), propose alternative sampling strategy called 'sampling by interpolation'. Synthesized samples are produced by the generator G following the sampling strategy and used for training the neural network model.
At the first round = 1, set of variables {#1, x)) in latent space R" is drawn i.dd from a multivariate normal Gaussian distribution. For each 2, the generator adopted for generating the synthesized training samples which are annotated by the labeling tool obtaining the first incremen- tal set (1) [('1. 91), (x'g, 12),,.. (w',, y,)) Here x denotes the synthesized data generated by G and y denotes the corresponding ground truth provided by S. Then, the first incremental set will be considered as the initial train set for training. For 2,3, ..., n, the elements T()) are the synthesized sample rather than existing samples. The incremental subset T(t) will be added the
previous train set U (k) for training of next round.
Then, the process described Section would be followed such that the gradients obtained from loss function C using the back propagation. Once the gradients derived for each point in latent space, we can use Eq. 2 again to identify such that the generator can produce more informative sample G(z') for constructing of proposed framework with sampling by interpolation is shown in Algorithm
0z can be
incremental set. The pseudo code
4 EXPERIMENTS
4.1 DATASETS
We evaluated the proposed framework on three different datasets, namely the handwritten digits classification - MNIST (Deng) 2012). image classification CIFAR-10 (Krizhevsky et al. 2009) and biophysical simulation IVUS dataset. As MNIST and CIFAR- 10 are well-known to the ma- chine learning community, will only introduce the IVUS dataset here. Intravascular ultrasound
5

---

Under review as a conference paper at ICLR 2020
Algorithm 2 The Proposed Framework Using Sampling by Interpolation
1: [-]
2: Draw a set of latent space variables Z = [21. 22, 2))(0) i.id from a normal Gaussian.
3: Randomly Initialise training subset T(1) (G( (G(z2), y2), (G(z)).,)))
where is provided by a labeling tool y (())))
4: while Condition is not fulfilled do
5: t
6: Train F with U (()) given loss function &
k=1
7: - (=1, 22, 25)
Ü z(e)
>> Randomly select subset
k=1
8:
Evaluate L. with ((G(z)). 9n), (G())) (1)))
Calculate ac for each z using backpropagation äs 10: Identify harder point for each 2 by z' + 2 + 8g (G(E))
9:
0c
11: The new incremental set T(() < [(((z)) vn),
return F
(IVUS) is a catheter- based medical imaging modality to identify the morphology and composition of atherosclerotic plaques the blood vessels. The IVUS-derived structural parameters (c.g. plaque burden and minimum lumen area) are predictive in clinical diagnosis and prognosis. Imaged-based biophysical simulation of coronary plaques is used for assessing the structural stress within the ves- sel wall, which relies time- consuminn finite element analysis (FEA) (Teng et al., (2014). Here, aim to train deep neural network to approximate the FEA, which takes medical image as input and predicts a structural stress map. Our IVUS dataset consist of 1, 1,200 slices of 2D gray-scale images of coronary plaques and corresponding vessel wall segmentation from the vendor. An in-hous, Python package named 'VasFEM' serves labeling tool by solving partial differential equations (PDEs) on segmentation masks. The details of relevant PDEs and FEA implementation are described in Appendix A.2
4.2 EXPERIMENTAL PROTOCOL
The main purpose of experiments is demonstrate the effectiveness of our proposed framework which can achieve the same performance using less training data. Thus, the performance of given target model was evaluated using the same experimental setting (i.e. same optimizer, number of training iteration, batch size and size of training data).
The baseline method employed in this paper randomly selects a subset of training samples form real data until certain condition is fulfilled, namely predefined size of training samples. Instead of randomly choose training samplings, the proposed framework adopt aforementioned sampling strategies which progressively select generate harder samples until predefined condition is ful- filled.
An independent evaluation was carried out the test sets of three dataset. For MNIST and CIFAR- 10, the default test datasets which contain 10,000 testing samples respectively were used for evalu- ation the data-efficiency. For the IVUS dataset, random held-out test dataset which contains 210 cases was evaluated using mean square еттог performance metric. Two of them (MNIST, CIFAR) were performed with sampling by nearest neighbor since there is not labeling system for the tasks, while experiments of IVUS was performed with sampling by interpolation where VasFEM was used for providing ground truth during training.
4.3 IMPLEMENTATION
was trained using samples in train set without annotations. According to preliminary experiments it is hard to reconstruct and generate the image for CIFAR-10 with a vanilla VAE. Thus, a-GAN (Appendix A.3). proposed in Rosca et al. (2017), was employed which consists of four parts: 1) encoder which map the image the latent space, 2) a generator which reconstructs the image from the latent representation, 3) a code discriminator which can distinguish the learned latent distribution and the Gaussian distribution,
For MNIST and IVUS datasets, a vanilla VAE (Appendix AA.3)

---

Under review as a conference paper at ICLR 2020
CIFAR-10
0.7
65P
-50
-700
2.065
1.01
3,56
Numbor of somples
Nambor
Narr somaloo
Figure 3: Comparison of the baseline method and the proposed method using different number of training samples on MNIST, CIFAR-10 and IVUS
4) image discriminator which can distinguish the reconstructed image and the real image. The dimensions of latent spaces for all three datasets are set to 3.
During training, convolutional neural networks (Appendix A.3 are employed for the classification tasks (MNIST and CIFAR-10) 10) using cross entropy loss. The biophysical simulation on image-to mnage-to-image prediction task, for which deconvolution layers employed in
IVUS dataset the target model and the mean square error was used as the loss function. Adam optimizer was used for all three tasks.
4.4 QUANTITATIVE RESULTS
For the experiment, progressively increased the size of train set and reported the accuracy and mean square eтror (MSE) on the independent test sets respectively. Each experiment was repeated for five times for plotting the mean and variance (Figure 3). Not only value of performance each size (indexed by t) is reported (Table []. the curves of accuracy and MSE are also included in Figure 3 after each increment of training samples.
observed that the target model (denoted as 'Ours' in Table I trained under the proposed framework yields a better performance than the baseline (denoted as 'Rand' in Table method until the target model reaches the bottleneck where the performance can hardly be improved by increasing size of the training samples. We also observed that the proposed framework yields relatively low marginal performance improvement on MNIST and IVUS due to the simplicity of given tasks.
Table 1: A comparison of performance over different sizes of train set. For MNIST and CIFAR-10, higher accuracy represents better results. For IVUS, smaller MSE represents better results.
2
4
6
8
10
MNIST
Rand
91.0
94.1
95.6
96.4
96.9
97.1
97.2
97.4
97.7
97.8
Accuracy (%)
Ours
91.2
95.1
96.2
96.7
97.2
97.5
97.6
97.7
97.8
97.8
CIFAR-10
Rand
35.6
40.4
44.6
51.4
54.9
54.3
56.3
56.5
60.6
60.5
Accuracy (%)
Ours
37.0
50.0
56.7
58.0
62.7
64.2
65.3
66.5
66.9
66.9
IVUS
Rand
12.4
12.1
11.8
11.3
9.7
7.5
6.4
6.2
5.5
5.6
MSE
Ours
12.2
11.7
10.7
9.4
7.5
6.7
5.9
5.8
5,5
5.2
5 DISCUSSION
In our framework, an annotating system (i.e. labeling tool or original dataset) is integrated into the training process and used in an active manner. Based on the current model state, more informative samples proposed by a generator are annotated online and appended to the current train set for further training. This closed-loop design makes the most use of the annotating system, which would be very useful in scenarios with high annotation cost, e.g. medical image segmentation. From the performance curves in Figure 3] observed immediate drop when fresh samples were fed into the neural work. But the performance rebounded to a higher level as the neural network learned the information carried by these samples. Compared the random sampling, hardness-awan

---

Under review as a conference paper at ICLR 2020
sampling resulted in deeper drop followed by higher rebound, indicating that more informative sample were mined.
2D latent space
Sampling snapshots
Figure 4: Latent space sampling on MNIST
The framework is able to mine new samples along the gradients of loss function. The samples mined in this way would bring more utility to training process than existing samples in the train set. To intuitively demonstrate this, a 2D latent space for MNIST embeddings is visualized on the left side in Figure We choose couple of initial sampling point and evolve them using rules of sampling by nearest neighbor without random re-selection, resulting collection trajectories. Six typical trajectories are selected to be visualized in the 2D MNIST latent space. It can be observed that trajectories like a.b.c and are the most desired exploration strategy in the latent space as they actually walk around the boundary between classes, where massive ambiguous samples are located. This provides us with a visual evidence that our framework indeed encourages to explore the areas of high uncertainty. The snapshots for every point in the corresponding trajectories on the right side Figure 4 which also provide strong support to our statement. There are two undesired trajectories denoted
also visualized
and We can that and both have been initialized closed to outer boundary and they keep on exploring toward outside until there is no any nearest neighbor around. Such trajectories should be avoided by periodically re-selecting existing training samples for further exploring harder samples since they are not be able to provide informative samples any more.
set of points within
Another crucial part is the choice of dimensionality of latent space. It is important for generator capable to create plausible and diverse samples as they are fact used for estimating more informative sample. sigher-dimensional latent space would encourage the diversity of synthesized samples, however, would make the sampling points distributed the latent space too sparse so that it will be difficult to identify the nearest neighbor for a given point in the latent space. Two sampling strategies along with the proposed framework were proposed for addressing different applications: 1) Sampling by nearest neighbor aims to handle the situation where there not external labeling tool and an existing annotated dataset is available. It is able to select the harder training samples from real dataset according current training state. 2) Sampling by interpolation works when there an external labeling tool (a FEM solver in our paper) available, which is able to provide ground truth for those synthesized training samples in a real-time manner. Synthesized training samples more accurate in terms of the degree of difficulty since they are produced using the actual location in latent space instead of the nearest neighbors.
6 CONCLUSION
We proposed a model state-aware framework for efficient annotating and learning. Hard samples from the data manifold mined progressively in the low-dimensional latent space. It is obvious that the proposed framework can not be only generalized to existing machine learning applications, but also those realistic scenarios as long as labeling tool is available.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 245. IEEE, 2009.
Li Deng. The mnist database of bandwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-12,, 2012.
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE transactions pattern analysis and machine intelligence, 28(4):594-611, 2006.
Michael Fink. Object classification from single example utilizing class relevance metries. In Advances in neural information processing systems, pp. 449-456, 2005.
Ben Harwood, BG Kumar, Gustavo Carneiro, Ian Reid, Tom Drummond, et al. Smart mining for deep metric learning. Proceedings of the IEEE International Conference on Computer Vision, pp. 2821 2821-2 2017.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1 18:751115 2006.
Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-aware deep feature embedding. In Advances neural information processing systems, pp. 1262-1270, 2016.
Junsik Kim, Tae- Mhyu Oh, Seokju Lee, Fei Pan, and In So Kweon. Variational prototyping-encone One-shot learning with prototypical images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9462-9470, 2019.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In /CMI. deep learning workshop, volume 2, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Brian Kulis al. Metric learning: A survey, Foundations and Trends in Machine Learning, 5(4): 287-364, 2013.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tomasz Malisiewicz, Abhinav Gupta, and Alexei Efros. Ensemble of exemplar-svms for object detection and beyond. 2011.
Hu Peiyun, C. Lipton Zachary, Anandkumar Anima, and Ramanan Deva. Active learning with partial feedback. 2019.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint 2017.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata, Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8247-8255, 2019.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 813,823, 2015.

---

Under review as a conference paper at ICLR 2020
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Sharathchandra Pankanti, Rogerio Feris, Abhishek Kumar, Raja Giries, and Alex Bronstein. Repmet: Representative-basee metric learning for classification and one-shot object detection. arXiv preprint 1r06vv1806.044 04728, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: (at::7708.04500 2017.
Burr Settles. Active learning literature survey. Technical report, University of Visconsin-Madinn Department of Computer Sciences, 2009.
Abhinav Shrivastava, Abhinav Gupta, and Ross B. Girshick. Training region-based object detectors with online hard example mining. CoRR, abs//1604.03540, 2016. JRLLattpp:/arxxvv.rrr
abs/1604. 03540
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087,2017
Zhongzhao Teng, Adam Brown, Patrick A Calvert, Richard A Parker, Daniel R Obaid, Yuan Huang, Stephen Hoole, Nick EJ West, Jonathan Gillard, and Martin Bennett. Coronary plaque structural stress is associated with plaque composition and subtype and higher in acute coronary syndrome: the beacon (biomechanical evaluation of atheromatous coronary arteries) study. Circulation: Cardiovascular Imaging, 7(3):461-4777 2014.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEFE International Conference on Computer Vision, pp. 2794-2802, 2015.
Yuhui Yuan, Kuiyuan Yang, and Chao Zhang. Hard-aware deeply cascaded embedding. Proceed- ings of the IEEE international conference computer vision, pp. 814 024--223,44 2017.
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Blass-agnostic segmen- tation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5217-5226, 2019.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou Hardness- aware deep metric learning- In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 72-872--1 22, 2019.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 NOTATIONS
Symbols and Notations
B
A The original training dataset
incremental set that consist of training sample and its annotation
8
temporary which consist of randomly selected train- ing samples.
R"
A n-dimsional latent space
given loss function
The trained generator derived from a VAE-based model.
E
The trained encoder derived from VAB-based model.
F
The target model that will be optimzed during training
An external labeling tool providing synthesized training sample with ground truth
t
Round index
A training sample in its original space
x
A synthesized training sample by generator
p
point in the given latent space corresponds real train-
ing sample
p'
point in the given latent space after displacement
2
A interpolated point the given latent space
z'
A interpolated point in the given latent space after displace- ment
A.2 LABELING TOOL FOR IVUS DATASET
Imaged-based biomechanical analysis of coronay plaques were performed following the procedure described in [Tegg et al. 2014). The workflow wrapped into Phython package named 'VasFEM' as labeling tool, which is available upon request. The input to the labeling system is a segmentation mask of plaque and the output is the corresponding structural stress map with the same resolution.
The material of plaque is assumed to be incompressible and non-linear which is described by the Mooney-Rivlin
modified Mooney- Rivlin strain energy density function:
W cT 3) + Dí Pathi - - 1]+ xt 1)
(3)
where 7, 2/ with I being the first invariant of the unimodular component of the left Cauchy-Green deformation tensor. J = FFt and F is the deformation gradient. is the La- grangian multiplier for the incompressibility. с₁ 0.138 kPa, D1 3.833 kPa and D3 18.803 are material parameters of the blood vessels derived from previous experimental work (Teng et al.) 2014). The finite element method used solve the governing equations of plane-strain problem:
puitt
Tej
(i.j=1, = 1,2)
where [vx] and [o.y] are the displacement vector and stress tensor, respectively, is the material density and stands for time.
A template pulsatile blood pressure waveform is applied on the lumen border. The structural stress map at the systole time point is extracted for analysis. takes about ten mins to perform 2D finite element analysis segmentation mask with 512x512 pixel IVUS image. As we focus the data

---

Under review as a conference paper at ICLR 2020
efficiency of our proposed framework, simplified the simulation by resampling the segmentation mask into 64x64 pixel image size and ignore the different components within the plaque. This reduced the simulation time to two mins. An example of the input image and output stress map is shown in Fig ST
000
Input mask
Finite element mesh
Structural stress map
Figure S1: An example of the input and output the labeling tool for IVUS dataset.
A.3
STRUCTURE OF NETS
Table S1: The structure of the residual block for o-GAN
Operation
Kernel
Strides
Padding
IChannel OChannel
2D Convolution
3
IChannel
IChannel
Batch Normalization
Leaky-Relu RRlu Activation
2D Convolution
3
IChannel
IChannel
Batch Normalization
Leaky-Relu Activation
Table S2: The structure of the encoder for a-GAN
Operation
Kernel
Strides
Padding
IChannel
OChannel
Convolution
64
Average Pooling
2
2
Relu Activation
Residual Block
64
64
Average Pooling
2
2
Residual Block
64
64
Average Pooling
2
2
Residual Block
64
64
Operation
Input
Output
Fully Connected Layer
65536
12

---

Under review as a conference paper at ICLR 2020
Table S3: The structure of the generator for a-GAN
Operation
Input
Output
Fully Connected Layer
3
65536
Relu Activation
Operation
Kernel
Strides
Padding
IChannel 64
OChannel
Residual Block
64
Average Pooling
2
Residual Block
64
64
Average Pooling
2
2
Residual Block
64
64
Average Pooling
2
2
Residual Block
64
64
2D Convolution
64
Tanh Activation
Table S4: The structure of the image discriminator for a-GAN
Operation
Kernel
Strides
Padding
IChannel
OChannel
2D Convolution
5
2
3
64
Average Pooling
2
2
Relu Activation
Residual Block
64
Average Pooling
2
2
Residual Block
64
64
Average Pooling
2
Residual Block
64
64
Operation
Input
Output
Fully Connected Layer
65536
Table S5: The structure of the code discriminator for a-GAN
Operation
Input
Output
Fully Connected Layer
700
Leaky Rely Activation
Fully Connected Layer
700
700
Leaky Rely Activation
Fully Connected Layer
700
Sigmoid Activation
13

---

Under review as a conference paper at ICLR 2020
Table S6: The structure of the target model for IVUS
Operation
Kernel
Strides
Padding
IChannel
OChannel
2D Convolution
2
8
Batch Normalization
Relu Activation
Max Pooling
2
2
2D Convolution
2
16
Batch Normalization
Relu Activation
Max Pooling
2
2
2D Convolution
2
16
32
Batch Normalization
Relu Activation
Max Pooling
2
2
2D Convolution
2
32
64
Batch Normalization
Relu Activation
Max Pooling
2
2
2D Convolution
2
64
64
Batch Normalization
Relu Activation
2D Transpose Convolution
2
64
64
Batch Normalization
Relu Activation
Up-sampling
2
2D Transpose Convolution
2
64
32
Batch Normalization
Relu Activation
Up-sampling
2
2D Transpose Convolution
2
16
Batch Normalization
Relu Activation
Up-sampling
2
2D Transpose Convolution
2
16
8
Batch Normalization
Relu Activation
Up-sampling
2
2D Transpose Convolution
2
Sigmoid Activation
Table S7: The structure the target model for MNIST
Operation
Kernel
Strides
Padding
IChannel
OChannel
2D Convolution
5
0
20
Relu Activation
Max Pooling
2
2
2D Convolution
5
0
20
50
Relu Activation
Max Pooling
2
2
Operation
Input
Output
Fully Connected Layer
800
500
Relu Activation
Fully Connected Laye)(Mean)
500
10
14

---

Under review as a conference paper at ICLR 2020
Table S8: The structure of the target model for CIFAR-10 (VGG 16)
Operation
Kernel
Strides
Padding
IChannel
OChannel
2D Convolution
3
3
64
Batch Normalization
Relu Activation
2D Convolution
3
64
64
Batch Normalization
Relu Activation
Max Pooling
2
2D Convolution
3
128
128
Batch Normalization
Relu Activation
2D Convolution
3
128
128
Batch Normalization
Relu Activation
Max Pooling
2
2D Convolution
3
256
256
Batch Normalization
Relu Activation
2D Convolution
3
256
256
Batch Normalization
Relu Activation
2D Convolution
3
256
256
Batch Normalization
Relu Activation
Max Pooling
2D Convolution
3
512
512
Batch Normalization
Relu Activation
2D Convolution
3
512
512
Batch Normalization
Relu Activation
2D Convolution
512
512
Batch Normalization
Relu Activation
Max Pooling
2
2D Convolution
3
512
512
Batch Normalization
Relu Activation
2D Convolution
3
512
512
Batch Normalization
Relu Activation
2D Convolution
3
512
512
Batch Normalization
Relu Activation
Max Pooling
2
Operation
Input
Output
Fully Connected Layer
512
10
15

---

Under review as a conference paper at ICLR 2020
Table S9: The encoder of VAE for MNIST
Operation
Input
Output
Fully Connected Layer
784
400
Relu Activation
Fully Connected Layer
400
400
Relu Activation
Fully Connected Meynr(Mean)
400
Fully Connected Layer( Variance)
400
Table $10: The decoder of VAE for MNIST
Operation
Input
Output
Fully Connected Layer
400
Relu Activation
Fully Connected Layer
400
400
Relu Activation
Fully Connected Layer
400
784
Table $11: The encoder of VAE for IVUS
Operation
Input
Output
Fully Connected Layer
4096
400
Relu Activation
Fully Connected Layer
400
400
Relu Activation
Fully Connected Layer(Mean)
400
Fully Connected Layer((aarance
400
Table S12: The decoder of VAE for IVUS
Operation
Input
Output
Fully Connected Layer
3
400
Relu Activation
Fully Connected Layer
400
400
Relu Activation
Fully Connected Layer
400
4096
16


Reviews:
[{"review_id": "rJxqsKr6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a method for sequential and adaptive selection of training examples to be\npresented to the training algorithm. The selection happens in a latent space, based on choosing\nsamples which are in the direction of gradient of the loss in the latent space. Two selection\nstrategies are investigated: nearest neighbor and interpolation followed by generation. Results on\nshown on MNIST, CIFAR10 and IVUS (Intravascular Ultrasound) datasets.\n                                                                \n                                        \nDetailed comments:                      \nThe proposed method works in two stages. First a VAE is trained using unannotated samples. In the\nsecond stage, hard examples are found, in every iteration, in the latent space of the VAE and used\nfor sequential training. The sampling is done using the gradient of the objective function in the\nlatent space. The method makes sense, however the choice of space in which the sample selection is\nbeing done is not well motivated or validated. The space could have been the original image space\n(although given the high dimension, it would probably not work), or could have been any intermediate\nfeature space. Why was the space chosen to be the VAE latent space? Would it be possible to \ndemonstrate some benefits of doing so, theoretically and empirically? \n                                        \nThe experiment section is relatively weak. The datasets used are relatively small and in two out of\nthe three datasets, the method does not improve. The dimension of the latent space is also\nsurprisingly small (2). While the main body of the paper describes the method with VAEs, the\nexperiments for the CIFAR10 dataset (where the results were in favor) were done \\alpha-GAN. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJldn3zH9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel training framework which adaptively selects informative samples that are fed to\nthe training process. The adaptive selection or sampling is performed based on a hardness-aware strategy in the latent space constructed by a generative model.\nThe idea is intuitive and easy to follow. Experimental results demonstrate the efficacy of the proposed method.\nI have two questions about this work:\n1. The informative samples are fed to the training process, what about the rest \"non-informative\" ones?\n2. What is the characteristics of the selected informative samples? i.e., for a class of images, which images should be informative?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygRX1NCtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to efficiently select hard samples during the training of a neural network. This is achieved via a variational auto-encoder (VAE) that encodes the samples into a latent space. The VAE is trained in a preparation stage using the images only and fixed at later stage. During training of a DNN framework, samples are selected in the latent space and then decoded via the Decoder in VAE to generate the input for DNN framework. The advantage of such a framework is that now it is able to calculate the gradient w.r.t. the input samples of DNN. This gradient is used to determine the sampling strategy in the next iteration to select harder samples. Two different sampling methods are explored, including nearest neighbor and interpolation (with annotation tool step). The experiments are conducted on small-scale datasets like MNIST CIFAR-10, and IVUS MSE with satisfactory gain over the baselines. Overall, the paper is very well-written and easy to follow. Although the experiment results are not super exciting mainly because of small-scale datasets and not enough gain in the numbers, some of the analysis in Figure 4 are quite insightful to validate the assumption and motivation of this work. So I propose to accept this work for its novelty. I think this work will benefit future research in this direction. \n\nHowever, I do have some concerns that I wish the authors could clarify if possible. First, the approach is very similar to online hard negative mining (OHNM) that is purely based on the loss to repeatedly select the samples that generate a larger loss. The major difference is that this work can model the sample distribution and thus select samples based on the gradient w.r.t. the samples in the latent space. This is very novel to me. However, I am wondering if the authors could compare with this sample baseline of OHNM. My concern is that the baselines in this work is too simple and it is not surprise that there is advantage over a simple baseline that is trained without any hard sample mining. \n\nSecond, the experiments are all conducted on small-scale and simple datasets like MNIST and CIFAR10. I am concerned how effective this approach could work for large-scale dataset. In the experiment, even for CIFAR10, a vanilla VAE will not work to reconstruct the input. So the authors have used alpha-GAN to help image reconstruction. If that is the case for CIFAR10 with only 10 classes, how could we extend this work to even larger dataset with more complicated background like ImageNet? I would think the preparation step itself is a very challenging task. This is my major concern that will question the effectiveness of the approach in real applications. \n\nThird, a related question to the above one. As the input to the DNN is the reconstructed image from the pre-trained decoder, there will be some information loss during the reconstruction process. This is the major challenge, I think, for large-scale applications. Is that possible to use the original image as the input to DNN while still being able to find hard samples using the latent space and the image space correlation? \n\nFourth, I really like the visualization of Figure 4 that shows the trajectory of the sampling process that follows the boundaries between classes. The authors also mentioned that some trajectories explore towards outside util there is no real samples, which should be avoided. Could the authors comment on how to avoid such cases? In my understand, as the input is randomly sampled at the beginning, it cannot avoid such cases unless some evaluation is done during training to stop the sampling for these trajectories. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: VAE-based sample selection for training NNs.  A well-written experimental paper that is demonstrated through a number of experiments, all of which are minimal and from which generalization is not per se expected.  The absence of an underlying theory, and the absence of rigorous experimentation makes me request to extend either or, better, both.  
Decision: Reject