Title: INTRODUCING ACTIVE LEARNING FOR CNN UNDER THE LIGHT OF VARIATIONAL INFERENCE

Abstract: One main concern of the deep learning community is to increase the capacity of representation of deep networks by increasing their depth. This requires to scale up the size of the training database accordingly. Indeed a major intuition lies in the fact that the depth of the network and the size of the training set are strongly correlated. However recent works tend to show that deep learning may be handled with smaller dataset as long as the training samples are carefully selected (let us mention for instance curriculum learning). In this context we introduce a scalable and efficient active learning method that can be applied to most neural networks, especially Convolutional Neural Networks (CNN). To the best of our knowledge, this paper is the first of its kind to design an active learning selection scheme based on a variational inference for neural networks. We also deduced a formulation of the posterior and prior distributions of the weights using statistical knowledge on the Maximum Likelihood Estimator. We describe our strategy to come up with our active learning criterion. We assess its consistency by checking the accuracy obtained by successive active learning steps on two benchmark datasets MNIST and USPS. We also demonstrate its scalability towards increasing training set size.

1. INTRODUCTION:

A daunting challenge in many contexts is to gather annotated data. This can be a long and tedious process, which often slows down the development of a framework and may jeopardize its economic prospects.

We refer to active learning Cohn (1994) as the field of machine learning which targets building iteratively the annotated training set with the help of an oracle.

In this setting and in a context of pool-based active learning1 , a model is trained on a small amount of data (i.e. the initial training set) and a scoring function discriminates samples which should be labeled by the oracle from the ones which do not hold new information for the model. The queried samples are then submitted to an oracle (which can be another decision algorithm for instance in co-training context, or a human expert in interactive learning context) to be labeled. They are then added to the current training set. Finally the model is retrained from scratch. This process is repeated recursively to grow the training set.

Although active learning and deep learning represent two important pillars of machine learning, they have mainly coexisted into independent stream of works owing to the complexity of combining them. The main issues are the scalability and the adaptability of common active learning schemes when considering architectures with a huge number of parameters such as deep networks. Another issue lies in the overall number of training iterations since training a deep architecture remains a computationally expensive process despite all the optimizations through GPU processing. This specificity has prevented deep learning from being prevalent within active learning. Indeed seminal active learning frameworks Cohn (1994) have mainly focused on adding one sample at-a-time. When it comes to selecting a batch of queries, the most intuitive solution is to select top scoring samples. Such a solution is immediate in the process but fails to model the correlations between samples. Labeling one sample at-a-time may therefore lead to the labeling of another sample totally useless.

In our work, batches of actively selected samples are added at each training iteration. We propose a batch active learning framework designed for deep architectures, especially Deep Convolutional Neural Networks (CNN).

Batch active learning is highly suitable for deep networks which are trained on minibatches of data at each iterations. Indeed training with minibatches help the training of deep networks, and we empirically noticed that the size of the minibatch is a major hyperparameter. Thus it makes sense to query a batch of unlabelled data whose size would be proportionnal to the size of a minibatch. In our work, batches have the same size as the minibatches but they could be decorrelated by considering importance sampling techniques.

Our model focuses on log loss which is involved in the training process of most neural networks. To achieve the required scalability of active learning for deep architectures, we step away from traditional active learning methods and focus our attention on a more general setting: Maximum Likelihood Estimation (MLE) and Bayesian inference. Provided certain assumptions, our active selection relies on a criterion which is based on Fisher information and is obtained from the minimization of a stochastic method for variational inference. Our active selection relies on the Fisher matrices on the unlabeled data and on the data selected by the active learning step. An approximation of Fisher information, based on a diagonal Kronecker-block decomposition makes our criterion computationally affordable for an active greedy selection scheme.

Variational methods have been previously explored like in Graves (2011) as a tractable approximation to Bayesian inference for Artificial Neural Networks (ANNs). One advantage of such a representation is that it leads to a two-term active learning criterion: one related to the prediction accuracy on the observed data and the second term expressing the model complexity. Such a two-fold criterion is, to the best of our knowledge, the first of the kind scalable for active learning. Such an expression may help both to analyze and to optimize ANNs, not only in an active learning framework but also for curriculum learning.

We dedicate section Related works to the presentation of active learning literature. Section Covering presents the theoretical aspects of our active learning framework, while section Active learning as a greedy selection scheme details our greedy algorithm. We then assess our active learning method through experiments on MNIST and USPS benchmarks. We discuss possible improvements of our approach and connections with previous MLE-based active learning methods before concluding.

2. RELATED WORKS:

Active learning is a framework to automatize the selection of instances to be labeled in a learning process. Active learning offers a variety of strategies where the learner actively selects which samples seem "optimal" to annotate, so as to reduce the size of the labeled training set required to achieve equivalent performance.

We consider the context of pool-based active learning where the learner selects its queries among a given unlabeled data set. For other variants (query synthesis, (stream-based) selective sampling) we refer the reader to Settles (2012).

2.1. POOL-BASED ACTIVE LEARNING:

When it comes to pool-based active learning, the most intuitive approaches focus on minimizing some error on the target classifier. Uncertainty sampling minimizes the training error by querying unlabeled data on which the current classifier (i.e. from previous training iteration) is assigning labels with the weakest confidence. This method, uncertainty sampling, while being the least computational consuming among all active learning techniques has the main drawback of ignoring much of the output distribution classes, and is prone to querying outliers. Thanks to its low cost and easy setup, uncertainty has been adapted to deep architectures for sentiment classification Zhou et al. (2010) . However, deep architectures are subject to adversarial examples, a type of noise we suspect uncertainty selection to be highly sensitive to Szegedy et al. (2014) ; Goodfellow et al. (2015) . Other strategies (expected error reduction, expected output variance reduction) directly minimize the error on a validation set after querying a new unlabeled sample. However they are computationally expensive especially when considering neural networks.

Traditional active learning techniques handle selection of one sample at-a-time. One of the main drawbacks of the aforementioned active learning techniques is that is does not pay attention to the information held in unlabeled data besides considering it as potential queries. Hence once the strategy for selecting samples to be labeled and added to the training set is defined, the question on the impact of the possible correlation between successive selected samples remains.

To that end, one recent class of methods deals with the selection of a batch of samples during the active process, batch mode active learning. Batch mode active learning selects a set of most informative unlabeled sample instead of a unique sample. Such a strategy is highly suitable when retraining the model is not immediate and require to restart the training from scratch at each iteration as it is the case for neural networks. A simple strategy (whose has also been used for previous deep active learning strategy Zhou et al. (2010) ) is to select a batch of top scoring instances. However that strategy fails to consider the correlation among pairs of samples. The redundancy between the so-selected samples may therefore hinder the learning process.

In the context of a deep learning scenario, if several elements related to the same direction of the gradient are in the same minibatch, the gradient descent in the next learning step may lead at once too close to a local minimum, diverting the process away from the global minimum.

While one sample at-a-time can prevent from being misled that way, it gets prohibitive when considering big data because the number of iterations is equal to the number of learning samples, unlike the batch-based strategies.

Recently some solutions have been proposed for choosing an appropriate subset of samples so as to minimize any significant loss in performance. Those methods consider the minimization of the Kullback-Leibler (KL) divergence between the resampled distribution of any possible subset selected for the active process and the whole unlabeled data set distribution. A lower bound of the negative of this KL divergence is then defined and rewritten as a submodular function. The minimization of the initial KL divergence becomes then a problem of submodular maximization Hoi et al. (2006) . In Wei et al. (2015) , Wei et al have designed several submodular functions so as to answer at best the need of specific classifiers (Naive Bayes Classifiers, Logistic Regression Classifier, Nearest Neighbor Classifier). However, their approach is hardly scalable to handle all the information from non-shallow classifiers such as deep networks.

Another solution to minimize the correlation among a set of queries is to perform bayesian inference on the weights of a neural network. In a bayesian context, a neural network is considered as a parametric model which assigns a conditional probability on the observed labeled data A given a set of weights w. The weights follow some prior distribution P (α) depending on the parameter α and the posterior distribution of the weights P (w | A, α) is deduced. The goal is thus to maximize the posterior probability of the weights on the observed data A. Indeed bayesian inference expresses the uncertainty of the weights which consequently leads to a relevant exploration of the underlying distribution of the input data X. When it comes to active learning, the learner needs not only to estimate the posterior given the observed data A but also to consider the impact of new data on that posterior Golovin et al. (2010) . In our context, bayesian inference is intractable, partially due to the high number of weights involved in a deep network. To solve this issue, Graves (2011) introduced a variational approximation to perform bayesian inference on neural networks. Specifically he approximates the posterior distribution P (w | A, α) with a tractable distribution of his choice Q(w | β) depending on a new parameter β. The quality of the approximation Q(w | β) compared to the true posterior P (w | A, α) is measured by the variational free energy F with respect to the parameters α and β. F has no upper bound but gets closer to zero as both distributions become more and more similar.

EQUATION

Finally in Graves (2011) , F is then expressed as a minimum description loss function on α and β:

F(α, β) = E w∼Q(β) (L(A; w)) + KL(Q(w) || P(α)) (2)

where KL is the Kullback Leibler divergence between Q(β) and P(α).

Under certain assumptions on the family distribution for the posterior and prior of the weights ( diagonal gaussian ...), Graves proposed a backpropagation compatible algorithm to train an ensemble of networks, whose weights are sampled from a shared probability distribution.

The primary purpose of the variational free energy is to propose a new training objectif for neural network by learning α and β. In an active learning context, the main drawback of variational free energy based method is that it requires to update by backpropagation the parameters for each unlabeled data submitted as a query. However we know from statistical assumption on the maximum likelihood the posterior and prior distribution of trained weights given the current labeled training set: if and only if we consider trained networks, we know how to build α and β in a unique iteration without backpropagation. This knowledge helps us to extend Graves first objectives to the use of variational free energy to measure how new observations affect the posterior.

3. COVERING:



3.1. VARIATIONAL INFERENCE ON NEURAL NETWORK WITH GAUSSIAN POSTERIOR WEIGHT:



DISTRIBUTION:

As done for the majority of neural networks, we measure the error of the weights w by the negative log likelihood on an observed set of annotated data A:

EQUATION

We consider the Maximum Likelihood Estimator (MLE) W as the value which makes the data observed A as likely as possible for a fixed architecture. Note than even for a fixed A in the case of neural network, W may not be unique.

EQUATION

When assuming that an arbitrary parameter W * is governing the data generation process, we know that the expected negative log likelihood is lower bounded by the expected negative log likelihood of the true parameter W * governing the data generation process. What it means is that no distribution describes the data as well as the true distribution that generated it. It turns out that, under certain assumptions, we can prove using the central limit theorem that the MLE is asymptotically normal with a mean equal to the true parameter value and variance equal to the inverse of the expected Fisher information evaluated at the true parameter.

If we denote by X the underlying data distribution, W X the MLE and W * X the true parameter, we know that W X is a sample from a multivariate gaussian distribution parametrized by W * X . Note that in this context we assume the Fisher matrices are invertible.

EQUATION

However the expected Fisher information on the underlying distribution is intractable. Eventually, using the law of large numbers, we know that the observed Fisher information converges to the expected Fisher information as the sample size increases. Another theoretical limitation is that the true parameter is unknown but we can approximate its observed Fisher information with the observed Fisher information at the MLE because of the consistency of the MLE. For a sake of simplicity we keep the same notation for observed and expected Fisher matrix.

Let denote by Y the random variable after resampling the underlying distribution X using an active learning strategy. W * X , W * Y are the true parameters with respect to their respective data distributions and their respective MLE variables W X , W Y , then the following relations hold:

EQUATION

We thus notice than in an active learning context, the learner is trained on data uniformly sampled from Y, while the optimal solution would be when training on data uniformly sampled from X.

The asymptotic behaviour provides us with a prior distribution of the weights based on the data distribution X. In our context of active learning, we approximate the posterior distribution with the MLE distribution induced by the resampling Y. Hence we define a prior and posterior distribution which did not require to be learnt by backpropagation directly but depend on the two data distribution X and Y.

EQUATION

Our active learning scheme relies on the selection of input data whose induced MLE distribution Q(β Y ) is minimizing the variational free energy.

EQUATION

It consists in the minimization of the sum of two terms which we denote respectively by the training factor E W∼Q(β) (L(A; W)) and the generalization factor KL(Q(β) || P(α)). It is possible to analyze both terms independently and explain their role into the minimization:

• Training factor: Ideally, the Cramer Rao bound implies that the minimum on the training factor is reached when Q(β) matches the asymptotically most efficient estimator of the optimal parameter on the error loss on the observed data. Hence the training factor corresponds to the minimization of the error on the observed data A.

• Generalization factor: Empirical results Choromanska et al. (2015) tend to show that the variance of the accuracy diminishes as the depth of the classifier increases. So our ultimate goal would be to converge to any set of parameters of the MLE distribution as their effectiveness is similar. The goal of the generalization factor is to converge to the asymptotic distribution on the whole input distribution X and to minimize the error of prediction of new input data.

If the expression of the variational free energy provides us a theoretical context to work on, the usage of Fisher matrices of deep networks renders it computationally unaffordable. Indeed the Fisher matrix is a quadratic matrix in terms of the number of parameters of the deep networks. Because of the huge number of parameters involved, such matrices takes a lot of memory and processing them costs a lot of ressources, especially if the operations may be repeated often in the framework (as it would be the case for every possible query processed by an active learning scheme.)

The next section 3.2 explains the different approximation proposed to deduce a more friendly user criterion.

3.2. APPROXIMATIONS:



3.2.1. KRONECKER FACTORED APPROXIMATION OF THE FISHER INFORMATION OF A CNN:

Recently an approximation of the Fisher information for deep architectures has been proposed first for fully connected layer in Martens & Grosse (2015) , and then for convolutional layer as well in Grosse & Martens (2016) . The block kronecker decomposition content (ψ, τ ) is explained in Martens & Grosse (2015) ; Grosse & Martens (2016) Based on their decomposition definition, we define the evaluation of blocks of the Fisher information at a certain point x i (ψ xi,l , τ xi,l ) and an empirical estimation of the Fisher matrix on a set of data A.

A sum up of their decomposition is presented in Eq. ( 9) while the exact content of the kronecker blocks ψ and τ is left as undescribed in this paper for the sake of concision.

I A (W) = diag([ψ A,l (W) ⊗ τ A,l (W)] L l=1 ) ψ A,l (W) = 1 | A | xi∈A ψ xi,l (W) τ A,l (W) = 1 | A | xi∈A τ xi,l (W) (9)

The strength of this decomposition lies in the properties of block diagonal combined with those of the kronecker product. ψ and τ are respectively related to the covariance matrix of the activation and the covariance of the derivative given the input of a layer. Recent deep architectures tend to prevail the depth over the width (i.e. the number of input and output neurons) so this expression becomes really suitable and tractable.

3.3. APPROXIMATION OF THE TRAINING FACTOR:

Despite the block kronecker product approximation of the Fisher matrix, sampling on Q(β) requires to compute the inverse. Because the kronecker blocks may still have an important number of parameters involved (especially the first fully connected layer suceeding to a convolutional layer), the inverse of the blocks may be still too computationally expensive. To approximate the training factor, we opt for a second order approximation of the log likelihood for parameters W close to the mean parameter

EQUATION

Our first approximation consists in assuming that the MLE parameter ŵ of the currently trained network is a good approximator of W * Y . Because the network has converged on the current set of observed data A the first derivative of the log likelihood is also set to zero. Hence Eq. ( 10) thus becomes:

EQUATION

To compute the expectation over the range of weights sampled from Q(β) we need to upperbound the expectation of the dot product of W given the Fisher matrix. Because we assume our Fisher matrices invertible, and because a covariance matrix is at least semi-definite, our Fisher matrices are positive definite matrix. Hence every eigenvalue is positive and the trace of the Fisher matrix is greater than its maximum eigenvalue. From basic properties of the variance covariance matrix, if we denote by N the number of parameters in the network we obtain the following upperbound for the training factor:

EQUATION

When it comes to the trace of the inverse, we approximate it by the closest lower bound with the inverse of the trace like in Wei et al. (2015) .

E W∼Q(β) (L(A; W)) ∝ ∼ L(A; ŵ) + N √ π N 2 T r(I Y ( ŵ)I A ( ŵ)I Y ( ŵ) T ) (13)

3.4. APPROXIMATION OF THE GENERALIZATION FACTOR:

Our generalization factor corresponds to the KL divergence between the approximation of our posterior Q(β) and the prior P(α). Because both distributions are multivariate gaussians, we have a direct formulation of the KL which is always definite since the Fisher matrices are invertible:

EQUATION

Our first approximation consists in assuming that the MLE parameter ŵ of the currently trained network is a good approximator of both optimal parameters W * X , W * Y like in Zhang & Oles (2000) . We also upper bound the determinant with a function of the trace and the number N of parameters. When it comes to the trace of the inverse, we approximate it again by the closest lower bound with the inverse of the trace.

EQUATION

3.5. APPROXIMATION OF THE VARIATIONAL FREE ENERGY:

In the previous subsections, we proposed independent approximations of both our sub-criteria: the training factor and the generalization factor. However the scale of our approximations may not be balanced so we sum up our criterion with an hyperparameter factor γ which counterparts the difference of scale between the factors:

EQUATION

We approximate the expected Fisher matrices on the underlying distribution Y and X by the observed Fisher matrices on a set of data sampled from those distributions. This approximation is relevant due to the consistenty of the MLE.

As we are in a pool-based selection case, we dispose at first of two sets of data: A and U which denote respectively the annotated observed data and unlabeled data. Note that the derivatives in the Fisher matrix computation implies to know the label of the samples. Thus at each active learning step, an unknown label is approximated by its prediction from the current trained network. We denote by S the subset of data to be queried by an oracle. The size of S is fixed with | S |= K. S is the subset sampled from Y while U is sampled from X. Finally an approximation of F will be:

EQUATION

Now we express the trace based on the approximation of the Fisher matrix: we consider that every Fisher matrix for CNN is a L diagonal block matrix, with L the number of layers of the CNN. Every block is made of a kronecker product of two terms ψ and τ . We rely on the properties involved by the choice of this specific matrix topology to obtain a more computationally compliant approximation of F in Eq. ( 18):

EQUATION

4. ACTIVE LEARNING AS A GREEDY SELECTION SCHEME ON THE VARIATIATIONAL FREE ENERGY:

The selected subset S selected at one step of active learning is only involved through the kronecker product of the Fisher matrix I S ( ŵ). We express our approximation of the free energy by a criterion

Under review as a conference paper at ICLR 2016 on the subset S in Eq. ( 19):

EQUATION

Finally we estimate our subset S by a greedy procedure: to be more robust to outliers and for reasons of computational efficiency, we select first a pool of samples D ⊂ U which we will use as the set of possible queries. We recursively build S ⊂ D by picking the next sample x i ∈ D which minimizes C(S ∪ {x i }; A, U) among all remaining samples in D. When it comes to the training factor coefficient, we notice that it is a quadratic term in I S ( ŵ) which increases the complexity in a greedy selection scheme. Our choice is to estimate the trace in the following way:

T r(ψ S∪{x},l ( ŵ)ψ A,l ( ŵ)ψ S∪{x},l ( ŵ) T ) ≈ T r(ψ S,l ( ŵ)ψ A,l ( ŵ)ψ S,l ( ŵ) T )+T r(ψ {x},l ( ŵ)ψ A,l ( ŵ)ψ {x},l ( ŵ) T )

Pseudo-code and illustration of the algorithm are provided in table 1 in appendix.

5. EXPERIMENTS:

We demonstrate the validity of our approach on two datasets: MNIST (28-by-28 pictures, 50.000 training samples, 10.0000 validation samples and 10.000 test samples) and USPS (16-by-16 pictures, 4185 training samples, 464 validation samples and 4649 testing samples) both gray scaled digits image datasets. We describe the CNN configuration and the hyperparameters settings in table 2 in appendix. Note that we do not optimize the hyperparameters specifically for the size of the current annotated training set A. We picked those two similar datasets to judge of the robustness of our method against different size of unlabeled datasets, as expected our method is efficient on both small and large databases.

5.1. TEST ERROR:

We run 10 runs of experiments and average the error on the test set of the best validation error before a pass of active learning. We start from an annotated training set of the size of one minibatch selected randomly. We stop both set of experiments after 30% of the training set has been selected (15.000 image for MNIST, 1255 for USPS). We compare the lowest test error achieved so far by our MLE based method against naive baselines: uncertainty sampling, curriculum sampling and a random selection of a minibatch of examples. We measure both uncertainty and curriculum scores based on the log likelihood of a sample using as label its prediction on the full network. While uncertainty selects samples with the highest log likelihood, our version of curriculum does the exact contrary. We select randomly the set of possible queries D among the unlabeled training data. Its size is set to 30 times the minibatch size. We present the results in two phases for the sake of clarity in figure 1 for MNIST and figure 2 for USPS: the first rounds of active learning when the annotated training set is almost empty, and the second round which is more stable in the evolution of the error. In both phases and for both databases we observe a clear difference between the test error achieved by our MLE method with the test error obtained by selecting randomly the data to be queried. Moreover the error achieved by our method on 30 % is close (even equivalent in the case of USPS), to the error achieved using the standard full training sets defined for both datasets (this error rate is defined as yellow line groundtruth on the figures). The experiments made appear that curriculum learning is not a good active learning strategy for both tested datasets. As for the uncertainty selection, it works really well on MNIST while it fails on USPS. While MNIST is a pretty clean database, USPS contains more outliers and noisy samples rendering it more difficult in terms of accuracy even though both databases are designed to assess digit classification. As other works we mentioned in the related work section, we are led to explain uncertainty selection to select useless samples with the amount of outliers and noisy samples in USPS. 

5.2. TIME COMPLEXITY:

To validate our method in terms of scalability and time complexity, we measured in seconds, the current processor time for one pass of active learning. We repeated this evaluation for different size of query (8 to 128 unlabeled samples added to the currrent training set). For this experiments we used a laptop with a Titan-X (GTX 980 M) with 8 GB RAM GPU memory. Metrics were reported in figure 3 . Our criterions takes few seconds to select a batch of query of hundreds of unlabeled data. Moreover the evolution of the time given the size of the query is less than linear. 

6. DISCUSSION:

We believe our method is a proof of concept for the use of variational inference for active learning on deep neural networks. However our approximations are subject to improvements which may lead to faster convergence and lower generalization error.

The first point to raise is that our approximation of the posterior is an asymptotic distribution which may be unstable on a small subset of observed data, as it is the case for active learning. Such a distribution may be regularized by taking the probability provided by the central limit theorem about how well our data fits to the asymptotic gaussian distribution. When it comes to the KFAC approximation, it suffers from the same issue and could be regularized when evaluating on small subset. A refinement of the approximations, especially for the generalization factor, following the approaches of submodular functions may be investigated.

Finally, an interesting observation is that our formulation of the variational free energy finds similarities with other MLE based active learning criteria previously proposed in the litterature. Indeed, in Zhang & Oles (2000) the authors study active learning by looking among the possible resampling of the input distribution. They formulate their criterion as the minimization of the trace of the inverse Fisher of the resampled distribution multiplied by the Fisher matrix on the input distribution: min S T r(I -1 S ( ŵ)I U ( ŵ))

7. CONCLUSION:

In a nutshell, we proposed a scalable batch active learning framework for deep networks relying on a variational approximation to perform bayesian inference. We deduced a formulation of the posterior and prior distributions of the weights using statistical knowledge on the Maximum Likelihood Estimator. Those assumptions combined with an existing approximation of the Fisher information for neural network, lead us to a backpropagation free active criterion. Eventually we used our own approximations to obtain a greedy active selection scheme.

Our criterion is the first of the kind to scale batch active learning to deep networks, especially Convolutional Neural Networks. On different databases, it achieves better test accuracy than random sampling, and is scalable with increasing size of queries. It achieves near optimal error on the test set using a limited percentage (30%) of the annotated training set on larger and more reduced dataset.

Our works demonstrated the validity of batch mode active learning for deep networks and the promise of the KFAC approximations for deep Fisher matrices for the active learning community. Such a solution is also interesting as a new technique for curriculum learning approach. [200, 200, 50, 10] [300, 50, 10] # batch size 64 8

Table 2 : Set of hyperparameters used respectively on the CNN for MNIST and SVHN 

:

Other settings exist but are not considered here for the sake of clarity, we refer the reader toSettles (2012)

Figures and Tables:

Figure1: Error on the test set splits in two figures : the first rounds of active learning and the second round which is more stable in the evolution of the error (MNIST)

Figure 3: Average processing time for one pass of our active learning strategy vs the size of the selected samples for annotation (USPS).

Greedy selection of the final query S Require :A set of initial annotated training examples Require :U set of initial unlabeled training examples Require :D set of possible queries, D ⊂ U Require :N number of parameters Require :L number of layers Require :K number of samples to query Require :γ scaling hyperparameter 1S 0 = {}; D 0 = D 2 for k [| 1, L |] do 3# init the coefficients of each layer with 0 value: Compute the inverse Fisher information on the whole input distribution10 for l in [| 1, L |] do 11 a l = ψ l (A) -1 ; b l = τ l (A) -1 12 c l = ψ l (U) -1 ; d l = τ l (U) -1 a 13 end 14 # coefficient for the greedy selection for x∈ D k do 15 for l in [| 1, L |] do 16 V l,0 (x) = T r(ψ x,l a l ψ T x,l ) 17 V l,1 (x) = T r(τ x,l b l τ T x,l ) 18 V l,2 (x) = T r(ψ x,l c l ) 19 V l,3 (x) = T r(τ x,l d l ) best sample in D k based on the criterion C: x k = argmax x γ 1 Z0(x) + N (ln(Z 1 (x)) + N Z1(x) ) 26 S k+1 ← S k ∪ {x k } 27 D k+1 ← D k \ {x k } 28 end 29 S ← S Ka If blocks are too big, an approximation of the inverse by a Woodburry-Nystrom is processedTalwalkar (2010)

Figure 4: Error on the test set when we do not sample first a larger subset where to pick the queries (D ≡ U, USPS)

Pseudo code for the greedy selection given the variatiational free energy For reasons of computational efficiency; we first sample a larger subset D ⊂ U from which we select the queries. To assert that this subset is not detrimental for uncertainty selection, we present a one shot experiments on USPS with D ≡ U in fig ??.

Table content: <table><tr><td>hyper parameters</td><td>MNIST</td><td>USPS</td></tr><tr><td># filters</td><td>[20, 20]</td><td>[20, 20]</td></tr><tr><td>filter size</td><td>[(3,3), (3,3)]</td><td>[(3,3), (3,3)]</td></tr><tr><td>pooling size (no stride)</td><td>[(2,2), (2,2)]</td><td>[None, (2,2)]</td></tr><tr><td>activation</td><td>Rectifier</td><td>Rectifier</td></tr><tr><td># neurons in full layers</td><td/><td/></tr></table>

Peer review (iclr2017):
Quality: The paper initiates a framework to incorporate active learning into the deep learning framework, mainly addressing challenges such as scalability that accompanies the training of a deep neural network. However, I think the paper is not well polished; there are quite a lot of grammatical and typing errors. Clarity: The paper needs major improvements in terms of clarity. The motivations in the introduction, i.e., why it is difficult to do active learning in deep architectures, could be better explained, and tied to the explanation in Section 3 of the paper. For example, the authors motivated the need of (mini)batch label queries, but never mention it again in Section 3, when they describe their main methodology. The related work section, although appearing systematic and thorough, is a little detached from the main body of the paper (related work section should not be a survey of the literature, but help readers locate your work in the relevant literature, and highlight the pros and cons. In this perspective, maybe the authors could shorten some explanations over the related work that are not directly related, while spending more time on discussing/comparing with works that are most related to your current work, e.g., that of Graves *11. Originality & Significance: The authors proposed an active learning training framework. The idea is to treat the network parameter optimization problem as a Bayesian inference problem (which is proposed previously by Graves) and formulate the active learning problem as that of sampling the most informative data, where the informativeness is defined by the variational free energy, which depends on the Fisher information. To reconcile the computational burden of computing the inverse of Fisher Information matrix, the authors proposed techniques to approximate it (which seems to be novel) I think that this paper initiates an interesting direction: one that adapts deep learning to label-expensive problems, via active learning. But the paper needs to be improved in terms of presentation.