Title: Distributed Transfer Learning for Deep Convolutional Neural Networks by Basic Probability Assignment
Authors: Arash Shahriari
Venue: ICLR 2017
Keywords: Deep learning
Transfer Learning
Supervised Learning
Optimization

Abstract: Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.


Review discussion:
1. p. 3, 2.1, paragraph 2, In which cases could\mathcal{C} and \mathcal{L} be different?
2. p. 4, 2.2, paragraph 3, It's not clear what the authors mean by "we start by learning a classifier \phi". What features are used to train this classifier? Which subset of data is \phi trained on?
3. Do I understand correctly that BPA is computed on the _validation_ set of the target domain? I'm not sure why validation is always coupled with a training set in the paper: "train/validation" (p4, 2.1, last paragraph). If BPA is calculated over the training set, it's not going to be very indicative of the performance of the classifier.
4. p. 4, 2.2, Could the authors please elaborate on how they build and train weak classifiers for some pretrained base NN? A diagram depicting the proposed approach would be highly desirable.
We would like to highly appreciate our reviewer's efforts and provide some detailed explanations in regards to his/her questions as follows.
1. p. 3, 2.1, paragraph 2, In which cases could\mathcal{C} and \mathcal{L} be different?
Combination of different classes under unit category is a common practice in visual classification.
For example; buildings, trees and street lights can be merged as a new class called vertical objects.  The benefit lies in the fact that preliminary layers of deep convolutional architectures make better contribution to detect first and second order features that are highly of specific directions (vertical vs horizontal) rather than distinguishing details (tree vs street lamp). This leads to a powerful hierarchical feature learning in the case that |C|<<|L|.
In contrast, some classes can be divided to various sub-categories although they all get the same initial labels and hence this holds |C|>>|L| for the sake of deeper layers.  In this work, we do not merge or divide the original setup of the datasets under study although it seems that our BPA-based approach is also able to boost the trained classifiers for each of the merge/divide scenarios.
2. p. 4, 2.2, paragraph 3, It's not clear what the authors mean by "we start by learning a classifier \phi". What features are used to train this classifier? Which subset of data is \phi trained on?
By learning, we mean a revised transfer learning practice of target's fine-tuning using the pre-trained weights and biases on the original dataset. The difference with the standard transfer learning methods is fine-tuning with respect to each individual convolutional filters. Suppose that we aim at transferring MNIST (original) to CIFAR (target). We start by training of the last three fully-connected layers (Softmax) of MNIST for the features calculated by presenting CIFAR train set to the convolutional layers of MNIST. We repeat this procedure for each of the 20 convolutional filters separately and compute the BPA for each of them on train/validation sets.
Finally, we combine unary potentials of all the trained classifiers by employing BPA weights to come up with a unit set of class probabilities (potentials).
3. Do I understand correctly that BPA is computed on the _validation_ set of the target domain? I'm not sure why validation is always coupled with a training set in the paper: "train/validation" (p4, 2.1, last paragraph). If BPA is calculated over the training set, it's not going to be very indicative of the performance of the classifier.
This exactly matches our observations that computing of BPA on validation set gives better generalisation and consequently, higher performance on testing set.
But our experiments also show that fine-tuning on training set only and then, calculating BPA on training plus validation sets generates competitive results especially in cases that the number of classes in original/target datasets are largely different. We will discuss this in more details in our final manuscript.
4. p. 4, 2.2, Could the authors please elaborate on how they build and train weak classifiers for some pretrained base NN? A diagram depicting the proposed approach would be highly desirable.
As described above, we present the target's samples to the convolutional layers of original's pre-trained network, feed the features to the fully-connected layers (Softmax) and calculate BPA for the target's validation set after classified by theses fine-tuned layers. We try this pipeline for each of the convolutional filters and merge the distributed outcomes by the BPA coefficients. We will add a diagram to our next revision.
Why should training set be used to compute BPA and not just validation set?
How are the training data for the weak classifiers constructed?
We would like to thank our reviewer's time and provide our reply as follows.
Why should training set be used to compute BPA and not just validation set?
Our observations show that fine-tuning on training set and calculating BPA on validation, result in better generalisation of the transferred model on testing set.
On the other hand, computing of BPA on training plus validation sets gives higher performance in case of hugely different number of classes in original/target datasets. Since we employ BPA to address the class-imbalance problem, we reckon that it better captures the distribution of data by adjoining both train/validation sets especially when we intend to transfer few classes of original dataset to the larger number of classes in the target.
How are the training data for the weak classifiers constructed?
Suppose that we aim at transferring original pre-trained network to the target dataset.
First, we present the target's training set to the convolutional layers of original deep architecture. The calculated features are fed into the fully-connected layers (Softmax) to fine-tune. It is worth mentioning that these convolutional features belong to each individual filters separately. We move forward by computing BPA on target's validation set classified by the recently fine-tuned layers.
Finally, we boost the single-filter (weak) classifiers using BPA weights to figure out the probability of each class. We will provide a proper visualisation in our final revision.
This paper proposed to use the BPA criterion for classifier ensembles.
My major concern with the paper is that it attempts to mix quite a few concepts together, and as a result, some of the simple notions becomes a bit hard to understand.
For example:
(1) "Distributed" in this paper basically means classifier ensembles, and has nothing to do with the distributed training or distributed computation mechanism.
Granted, one can train these individual classifiers in a distributed fashion but this is not the point of the paper.
(2) The paper uses "Transfer learning" in its narrow sense: it basically means fine-tuning the last layer of a pre-trained classifier.
Aside from the concept mixture of the paper, other comments I have about the paper are:
(1) I am not sure how BPA address class inbalance better than simple re-weighting. Essentially, the BPA criteria is putting equal weights on different classes, regardless of the number of training data points each class has. This is a very easy thing to address in conventional training: adding a class-specific weight term to each data point with the value being the inverse of the number of data points will do.
(2) Algorithm 2 is not presented correctly as it implies that test data is used during training, which is not correct: only training and validation dataset should be used. I find the paper's use of "train/validation" and "test" quite confusing: why "train/validation" is always presented together? How to properly distinguish between them?
(3) If I understand correctly, the paper is proposing to compute the BPA in a batch fashion, i.e. BPA can only be computed when running the model over the full train/validation dataset. This contradicts with the stochastic gradient descent that are usually used in deep net training - how does BPA deal with that? I believe that an experimental report on the computation cost and timing is missing.
In general, I find the paper not presented in its clearest form and a number of key definitions ambiguous.
We thanks the reviewer's detailed comments and try to clarify some critical points as follows.
1.  "Distributed" in this paper basically means classifier ensembles, and has nothing to do with the distributed training or distributed computation mechanism.
By "distributed transfer learning", we mean parallel learning of single-filter neural architectures initialized by original-domain weights but optimized by gradient descent through backpropagation. Actually, BPA is the glue that joins these local optimizers with a common gradient that makes this works as a global optimizer in standard convolutonal networks. To our knowledge, ensembling of several week classifiers is the process of selecting only those that improve the predictive power of the model but here, we do not weight single-filter classifiers but update them again for the target-domain.
2. The paper uses "Transfer learning" in its narrow sense: it basically means fine-tuning the last layer of a pre-trained classifier.
The general practice of fine-tuning is the start point of our method not the method itself. Since we update the convolutonal layers of distributed single-filter networks by the gradient coming after the BPA block, our approach is a learning procedure initiated by the fine-tuning. We will visualize this backpropagation in the figure to prevent this misunderstanding.
3. Essentially, the BPA criteria is putting equal weights on different classes, regardless of the number of training data points each class has.
Actually BPA is calculated on a confusion matrix which is well-connected to the overall number of samples and correct/miss-classified data for each of the classes separately. Re-weighting by the inverse number of class data seems a solution but at the same time it is highly restricted and cannot expose the overall trend towards major/minor classes. Since we propagate error after BPA, fixing of weights may address the complexity issue but does not provide any useful information about the direction of gradients for each of the single-filter optimizers.
4. Algorithm 2 is not presented correctly as it implies that test data is used during training, which is not correct: only training and validation dataset should be used.
Since algorithm 2 presents the learning process, by test sample we mean either a sample from validation set only or the combination of training/validation sets. That's why we did not use the term "test set" but "test sample". We agree that this may be misleading and will fix it in our next revision.
5. why "train/validation" is always presented together? How to properly distinguish between them?
Our observation shows that learning by training set and calculating BPA on validation set, result in better generalization on testing set.
On the other hand, on large variety of class numbers between original-target domains, joint of training and validation sets gives better BPA evaluation. We used "training/validation" term to present that both of scenarios are beneficial to improve the overall performance of our proposed transfer learning and discussed it in a paragraph before conclusion session in our last revision.
6. BPA can only be computed when running the model over the full train/validation dataset. This contradicts with the stochastic gradient descent.
Wherever a confusion matrix is available, BPA can be deployed. The final BPA weights after transfer learning are not calculated but applied at the test time. Since SGD only applies to the training procedure and computing of BPA is not deployed for the testing set, backpropagation of BPA with SGD does not make any contradiction.
7. I believe that an experimental report on the computation cost and timing is missing.
We agree but since there is no counterpart distributed algorithm, we cannot provide fare comparison to the conventional transfer learning practices.
Update: I thank the author for his comments! At this point, the paper is still not suitable for publication, so I'm leaving the rating untouched.
This paper proposes a transfer learning method addressing optimization complexity and class imbalance.
My main concerns are the following:
1. The paper is quite hard to read due to typos, unusual phrasing and loose use of terminology like “distributed”, “transfer learning” (meaning “fine-tuning”), “softmax” (meaning “fully-connected”), “deep learning” (meaning “base neural network”),  etc. I’m still not sure I got all the details of the actual algorithm right.
2. The captions to the figures and tables are not very informative – one has to jump back and forth through the paper to understand what the numbers/images mean.
3. From what I understand, the authors use “conventional transfer learning” to refer to fine-tuning of the fully-connected layers only (I’m judging by Figure 1). In this case, it’s essential to compare the proposed method with regimes when some of the convolutional layers are also updated. This comparison is not present in the paper.
Comments on the pre-review questions:
1. Question 1: If the paper only considers the case |C|==|L|, it would be better to reduce the notation clutter.
2. Question 2: It is still not clear what the authors mean by distributed transfer learning. Figure 1 is supposed to highlight the difference from the conventional approach (fine-tuning of the fully-connected layers; by the way, I don’t think, Softmax is a conventional term for fully-connected layers). From the diagram, it follows that the base CNN has the same number of convolutional filters at every layer and, in order to obtain a distributed ensemble, we need to connect (for some reason) filters with the same indices. This does not make a lot of sense to me but I’m probably misinterpreting the figure. Could the authors revise the diagram to make it clearer?
Overall, I think the paper needs significant refinement in order improve the clarity of presentation and thus cannot be accepted as it is now.
We appreciate the reviewer's new points and comments on pre-review questions.
1. The paper is quite hard to read due to typos, unusual phrasing and loose use of terminology like “distributed”, “transfer learning” (meaning “fine-tuning”), “softmax” (meaning “fully-connected”), “deep learning” (meaning “base neural network”),  etc.
We reckon that our terminology is rather restrict not loose. We talked about "distributed" because of several single-filter networks learned separately for domain transfer. We used "transfer learning" because conventional fine-tuning is the initialization point of our method not the method itself. We deliberately put "Softmax" to show that our approach can be employed with other classifiers like SVM as top layer.
Finally, we say "deep learning" because all the single-filter networks follow a conventional deep architectures (convolutional, ReLU, pooling layers) and are trained by backpropagation.
2. The captions to the figures and tables are not very informative.
Thanks and we try to improve their clarity in our next revision.
3. It’s essential to compare the proposed method with regimes when some of the convolutional layers are also updated.
We agree and try to include this in our coming revisions.
4.  If the paper only considers the case |C|==|L|, it would be better to reduce the notation clutter.
Thanks for the comment and we will apply it.
5.  It is still not clear what the authors mean by distributed transfer learning.
The reviewer's impression from the figure is right but the missing point here is backpropagation. Actually we need to visualize the backpropagation of gradients after BPA's block to all the single-filter networks. The contribution of our distributed architecture is the local optimization of a set of single-filter networks by means of a global gradient calculated over BPA's output. We try to add this concept to the figure in our next revision.
This work proposes to use basic probability assignment to improve deep transfer learning. A particular re-weighting scheme inspired by Dempster-Shaffer and exploiting the confusion matrix of the source task is introduced. The authors also suggest learning the convolutional filters separately to break non-convexity.
The main problem with this paper is the writing. There are many typos, and the presentation is not clear.
For example, the way the training set for weak classifiers are constructed remains unclear to me despite the author's previous answer. I do not buy the explanation about the use of both training and validation sets to compute BPA.
Also, I am not convinced non-convexity is a problem here and the author does not provide an ablation study to validate the necessity of separately learning the filters. One last question is CIFAR has three channels and MNIST only one: How it this handled when pairing the datasets in the second set of experiments?
Overall, I believe the proposed idea of reweighing is interesting, but the work can be globally improved/clarified.
I suggest a reject.
We would like to appreciate reviewer's comments.
1. The main problem with this paper is the writing. There are many typos, and the presentation is not clear.
Although it is a hard mission to describe our entire contribution in the limited number of pages, we will put all our efforts to enhance the presentation of our next revision and correct the typos.
2. The way the training set for weak classifiers are constructed remains unclear to me despite the author's previous answer.
Let's provide a practical example for further clarification. Suppose that we aim at transferring MNIST (original) pre-trained network to CIFAR (target). To initialize, we present CIFAR images to the MNIST-network but filter by filter for 20 convolutonal filters.
Then, the outputs are passed to BPA and the gradient of errors calculated. This gradient backpropagates to all 20 single-filter networks to update their weights. After some iterations, the MNIST-network is expected to be transferred to a CIFAR-network through our joint BPA-gradient scheme running on 20 fine-tuned networks.
3. I do not buy the explanation about the use of both training and validation sets to compute BPA.
We agree that this observation should receive a better justification in the experiment section and will update it in our revision.
4. I am not convinced non-convexity is a problem here and the author does not provide an ablation study to validate the necessity of separately learning the filters.
The benefit lies in both higher precision and lower number of parameters. In the above case (MNIST-CIFAR) the global optimization problem breaks to 20 local optimizations (single-filter networks) which are connected through BPA to form its dual (distributed vs all-out fine-tuning).
5. CIFAR has three channels and MNIST only one: How it this handled when pairing the datasets in the second set of experiments?
Thanks for pointing out, we repeated MNIST gray channel three times to make and RGB-like color representation for CIFAR-network. For CIFAR, we convert RGB channels to gray-scale for presenting to MNIST-network. We will clarify this further in our next revision.

Meta-review:
role: PC
data: comment: All three reviewers appeared to have substantial difficulties understanding the proposed approach due to unclear presentation. This makes it hard for the reviewers to evaluate the originality and potential merits of the proposed approach, and to assess the quality of the empirical evaluation. I encourage the authors to improve the presentation of the study.

Decision: Reject