Title: DISTRIBUTED TRANSFER LEARNING FOR DEEP CONVOLUTIONAL NEURAL NETWORKS BY BASIC PROBABILITY ASSIGNMENT

Abstract: Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.

1. INTRODUCTION:

In supervised learning, many classification algorithms assume the same distribution for training and testing data. Consequently, change of distribution requires rebuilding of the statistical models which is not always practical because of the hardship of recollecting of training data or heavy learning process. One of the solutions is transfer learning that transfers the classification knowledge into a new domain Pan & Yang (2010) . This aims at learning of highly-generalized models with different probability distributions across domains to learn novel domains without labeled data Wang & Schneider (2014) Zhang et al. (2013) . Here, the main challenge is to reduce the shifts in data distribution between domains by algorithms that minimize the discriminant of the domains. It is worth mentioning that this could not get rid of domain-specific variations Long et al. (2016) .

Transfer learning for deep neural networks has been proved highly beneficial to boost their overall performance. Deep learning practices usually require huge amount of labeled data to learn powerful models. The transfer learning enables adaptation to a different source with small training samples. On the other hand, deep neural networks practically learn intermediate features. They could provide better transfer among domains because some of them generalize well among various domains of knowledge Glorot et al. (2011) . These transferable features generally underlies several probability distributions Oquab et al. (2014) which reduce the cross-domain discrepancy Yosinski et al. (2014) .

The common observation among several deep architectures is that features learned in bottom layers are not that specific, but transiting towards top layers makes them tailored to a dataset or task. A recent study Yosinski et al. (2014) of the generality or specificity of deep layers for the sake of transfer learning reveals two difficulties which may affect the transfer of deep features. First, top layers get quite specialized to their original tasks and second, some optimization difficulties rise due to the splitting of the network between co-adapted layers. In spite of these negative effects, it Algorithm 1 Basic Probability Assignment (BPA)

Input: train/validation set X Output: basic probability assignment BPA(φ)

1: compute R(φ) and P(φ) (Eqs.1-2) 2: calculate recall and precision assignments(Eq.3) 3: apply Dempster rule for accumulation (Eq.4)

is shown that transferred features not only perform better than random ones but also provide better initialization. This gives a boost to the generalization of deep neural networks as well.

In this paper, we propose a framework for distributed transfer learning in deep convolutional networks. This tries to alleviate the burden of splitting networks in the middle of fragile co-adapted layers. The intuition is that above difficulty relates to the complexity of deep architectures and also, class-imbalance in the transferred domain.

On the matter of network complexity, we argue that the splitting of layers leads to a hard optimization problem because of high complexity in the interconnections between neurons of co-adapted layers. It seems that transfer learning is not able to thoroughly reconstruct the original powerful wiring for the transferred domain. This is due to the size of network and large number of interconnections across neurons. To address this issue, we fine-tune the convolutional filters separately and hence, reduce the complexity of the non-convex optimization.

On the other hand, it seems that the class-imbalance problem rises form different distribution of data in original and transferred domains. This issue can be handled by cost-sensitive imbalanced classifications methods. By class-imbalance in transferred domain, we mean variable coverage of common classes in this domain and the ones from the original domain. It is probable that both original and transferred datasets have uniform distributions of data among their classes, but some classes in one domain may be fully or partly covered by the other domain. This results in imbalance class distribution in the transfer learning.

The determination of a probabilistic distribution from the confusion matrix is highly effective to produce a probability assignment which contributes to class-imbalance problems. This basic probability assignment can be either constructed from recognition, substitution and rejection rates Xu et al. (1992) or both precision and recall rates of each class Deng et al. (2016) . The key point is harvesting of maximum possible prior knowledge provided by the confusion matrix to overcome the imbalance classification challenge.

Since the power of deep convolutional models come from mutual optimization of all parameters, we join the above distributed fine-tuned filters by a boosting scheme based on basic probability assignment. Our experiments confirm the functionality of our distributed strategy for deep transfer learning. The rest of paper is organized as follows. We present the formulation of our method in Section 2, report our experiments in Section 3 and conclude in Section 4.

2. FORMULATION:

In general, a confusion matrix represents the class-based predictions against actual labels in form of a square matrix. Inspired by Dempster-Shafer theory, construction of basic probability assignment (BPA) Sentz & Ferson (2002) gives a vector which is independent of the number of class samples and sums up to one for each individual label. This basic probability assignment provides the ability to reflect the difference contributions of a classifier to each individual classes or combine the outcomes of multiple week classifiers.

2.1. BASIC PROBABILITY ASSIGNMENT:

A raw two-dimensional confusion matrix indexed by predicted classes and actual labels provides some common measures of classification performance. They are accuracy (the proportion of the total number of predictions that were correct), precision (a measure of the accuracy provided that a specific class has been predicted), recall (a measure of the ability of a prediction model to select instances of a certain class from a dataset) and F-score (the harmonic mean of precision and recall) Sammut & Webb (2011) .

Suppose a set of train/validation samples

X = {X 1 , . . . , X |X | } from C = {C 1 , . . . , C |C| } different classes are assigned to a label set L = {L 1 , . . . , L |L| } by a classifier (φ) such that |C| = |L|. If each element (n ij ) of the confusion matrix C(φ)

is considered as the number of samples belonging to class C i which assigned to label L j , then we can define recall (r ij ) and precision (p ij ) ratios as follows Deng et al. ( 2016)

r ij = n ij |C| j=1 n ij p ij = n ij |L| i=1 n ij (1)

It can be seen that the recall ratio is summed over the actual labels (rows) whilst the precision ratio is accumulated by the predicted classes (columns) of the confusion matrix C(φ). Now, we are able to define recall and precision matrices as

EQUATION

The basic probability assignments of these matrices contain recall and precision probability elements for each individual class C i such that

EQUATION

These elements are synthesized to form the final probability assignments representing the recognition ability of classifier φ to each of the classes of set C

m i = mr i ⊕ mp i = mr i × mp i 1 - |C| i=1 mr i × mp i (4)

Here, operator ⊕ is an orthogonal sum which is applied by Dempster rule of combination Sentz & Ferson (2002) . The overall contribution of the classifier φ cab be presented as a probability assignment vector

EQUATION

It is worth mentioning that BPA(φ) should be computed by the train/validation set because we assume that the test set does not include actual labels. Besides, combination of different classes under vertical or horizontal categories is a common practice in visual classification. The benefit lies in the fact that bottom layers of deep convolutional architectures make better contribution to detect first and second order features that are usually of specific directions (vertical vs horizontal) rather than detailed distinguished patterns of the objects. This leads to a powerful hierarchical feature learning in the case that |C| |L|. In contrast, some classes can be divided to various sub-categories although they all get the same initial labels and hence this holds |C| |L| to take the advantage of top layers. In the above formulation, we do not merge or divide the original setup of the datasets under study (|C| = |L|) although it seems that our BPA-based approach is also able to boost the trained classifiers for each of the merge/divide scenarios. 

2.2. DISTRIBUTED TRANSFER LEARNING:

A general practice in transfer learning includes training of an original deep neural network on a dataset and then, fine-tuning of learned features for another dataset on a new target network. Bengio et al. (2012) . The generality of selected features for both original and target domains is critical to the success of the transfer learning. For implementation, we train the original network and copy its bottom layers to form the target network. The top layers of the target network are initialized randomly and trained on the target dataset. We are able to employ backpropagation from top to bottom layers and fine-tune their parameters for the target task or freeze the copied originals and only update top target layers. This can be decided by size of the target dataset and number of parameters in the original layers. Fine-tuning of large networks for small dataset leads to overfitting but for small network or large dataset, performance will be improved Sermanet et al. (2013) .

Based on our formulation for basic probability assignment (BPA) on Section 2.1, we are able to follow the above transfer learning procedure by learning of a classifier φ (SVM or Softmax) and computing BPA(φ) using Algorithm 1. Here, the learning means fine-tuning of target domain using the rained weights and biases of the original network. To implement this, we train the original fullyconnected layers by the features calculated by presenting target's train set to convolutional layers of the same original network. We deploy this procedure for each of the available convolutional filters separately and compute the BPA of each individual single-filter network for train/validation sets. Then, we combine unary potentials of all the fine-tuned classifiers by employing BPA weights to come up with a unit set of class probabilities. Figure 1 provides an overview of conventional and distributed transfer learning processes.

Suppose that C i is the predicted class for a test sample T provided by classifier φ. To revise the classification outcome by the BPA calculation, we multiply the test sample's unary potentials U(T ) = {u 1 , . . . , u |C| } (probabilities of belonging to each class) by an assignment vector M(φ) = {1 -m 1 , . . . , 1 -m |C| } (contributions of the classifier φ to each class) and pick the maximum index as the revised predicted label

C(T ) = I arg max {u 1 × (1 -m 1 ), . . . , u |C| × (1 -m |C| )} (6)

This implies that if classifier φ performs well on class C i (high m i ), it is highly probable that C(T ) leans towards C i . At the same time, other minority classes like C j (low m j ) have a chance to win if their unary potentials would be high enough (u j > u i ). In contrast, if φ does poor classification on class C i (low m i ), the possibility of updating C(T ) to another class (C j ) with even worse unary potential (u j < u i ) would be higher. Therefore, BPA shows quite successful in handling imbalance data distribution among classes.

Algorithm 2 Distributed Transfer Learning Input: train/validation set X , test sample T , set of week classifiers F Output: predicted class C F (T )

for i = 1 to |C| do for j = 1 to |F| do 1: compute m ij ∈ BPA(F) (Alg.1

) 2: calculate unary potential u ij ∈ U F (T) end for end for 3: predict boosted output C F (T ) (Eq.8) 4: employ error backpropagation for fine-tuning As described in Section 1, employing probability assignment addresses the class-imbalance problem but does not reduce the complexity of optimization because of the fact that both forward learning and error backpropagation are applied to all the model parameters. To break this non-convex optimization, we introduce our distributed transfer learning strategy. For implementation, we replace the mutual learning of all the parameters with learning of each individual convolutional filter in a separate classifier fed by the bottom original layer. It means that we train a set of week single-filter classifiers F = {φ 1 , . . . , φ |F | } which |F| equals the number of convolutional filters in the deep neural architecture.we follow the recipe of single classifier in Equation 5 but extend it to redefine

EQUATION

such that m ij is the probability assignment of class C i to week single-filter classifier φ j . To come up with class of the test sample T , we update the Equation 6 as follows

EQUATION

Here, u ij is the unary potential of class C i determined by the week single-filter classifier φ j . Building on the above formulations, we are able to distribute the transfer learning among convolutional filters and join them later to implement a better fine-tuning for the target deep convolutional network according to the Algorithm 2.

3. EXPERIMENTS:

We We consider two different scenarios to evaluate the performance of our distributed transfer learning algorithm. In the first experiment, we try to observe the performance of fine-tuning for pairs Before moving forward to discuss the experiments, we report the baseline train-test errors for the datasets in Table 1 . These results are produced by the deep learning library provided by the Oxford Visual Geometry Group Vedaldi & Fulkerson (2008) . In this experiment, we target two pairs of datasets (original-target domains) which contain similar data and perform number/object recognition tasks. We report the results for both conventional and our distributed transfer learning methods. By conventional Bengio et al. (2012) , we mean training the original dataset and fine-tuning of the target one. With distributed, we aim at training the original dataset but employing the basic probability assignment for the transfer learning.

It can be seen that the results for the conventional transfer learning follows our argument on size of network and number of model parameters Sermanet et al. (2013) . Compared to CIFAR-100 0.05 -18.24 -On the hand, transferring of SVHN from MNIST does not overfit when our distributed transfer learning is employed. In both settings of original-target domains, our distributed strategy outperforms the conventional transfer learning approach.

The experiment on CIFAR pair exposes more interesting results due to the fact that both datasets have the same number of samples but completely different distributions among the classes. In practice, CIFAR-100 includes all the classes of CIFAR-10 but CIFAR-10 does not have any clue of the several classes of CIFAR-100. The conventional experiments show that CIFAR-10 transfers well on CIFAR-100 but it cannot perform transferring although the target network does not overfit.

All in all, the performance of our distributed transfer learning (bold values) is better than the conventional scheme and also, outperforms the baseline deep learning practices.

3.2. EXPERIMENT 2:

In Table 3 , we reports the results for both conventional and distributed transfer learnings on the second scenario. Here, we pair datasets such that the similarity of their data distributions and number of classes get minimized and they are originally trained for different tasks. It is obvious that our distributed transfer learning outperforms all the conventional results.

For the first setup, CIFAR-10 does a better transfer learning than MNSIT although the number of classes are the same. It seems that CIFAR-10 provides better generalization due to higher diversity among its classes. Here, our distributed algorithm performs better than the conventional process and,

:



Figures and Tables:

Figure 1: Conventional and Distributed Transfer Learning. The blue blocks (Conv) represent convolutional layers in the original domain, the red blocks (Softmax) show fine-tuned layers for the target domain and the green block corresponds to the basic probability assignment (BPA) respectively.

Figure 2: Examples of MNIST, CIFAR and SVHN Datasets

conduct our experiments on MNIST, CIFAR and Street View House Numbers (SVHN) datasets. The MNIST dataset LeCun et al. (1998) contains 60, 000 training examples and 10, 000 test samples normalized to 20 × 20, centered by center of mass in 28 × 28 and sheared by horizontally shifting such that the principal axis is vertical. The foreground pixels were set to one and the background to zero. The CIFAR dataset Krizhevsky & Hinton (2009) includes two subsets. CIFAR-10 consists of 10 classes of objects with 6, 000 images per class. The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. It was divided to 5, 000 randomly selected images per class as training set and the rest, as testing samples. The second subset is called CIFAR-100 having 600 images in each of 100 classes. These classes also come in 20 super-classes of five class each. The SVHN dataset Netzer et al. (2011) was extracted from a large number of Google Street View images by automated algorithms and the Amazon Mechanical Turk (AMT) framework. It consists of over 600, 000 labeled characters in full numbers and MNIST-like cropped digits in 32×32. Three subsets are available containing 73, 257 digits for training, 26, 032 for testing and 531, 131 extra samples.

Table content: <table/>

Baseline Performances of Deep LearningTable2shows the performance of conventional and distributed transfer learnings for the first scenario. The first values before dash correspond to the training errors (left) and the second ones present the testing errors (right).

Table content: <table><tr><td/><td>Baseline</td><td/></tr><tr><td/><td colspan="2">Train Error (%) Test Error (%)</td></tr><tr><td>MNIST</td><td>0.04</td><td>0.55</td></tr><tr><td>SVHN</td><td>0.13</td><td>3.81</td></tr><tr><td>CIFAR-10</td><td>0.01</td><td>19.40</td></tr><tr><td>CIFAR-100</td><td>0.17</td><td>50.90</td></tr><tr><td>3.1 EXPERIMENT 1</td><td/><td/></tr></table>

MNIST does a poor job on transferring of SVHN due to the overfitting of SVHN over MNIST network. In contrast, SVHN perform quite well on transferring MNIST.

Table content: <table/>

Performance of Conventional and Distributed Transfer Learning for Experiment 1

Table content: <table><tr><td/><td/><td colspan="2">Target</td></tr><tr><td/><td>Conventional</td><td>MNIST</td><td>SVHN</td></tr><tr><td>Original</td><td>MNIST SVHN</td><td>-0.35 -1.04</td><td>0.01 -29.57 -</td></tr><tr><td/><td/><td colspan="2">Target</td></tr><tr><td/><td>Distributed</td><td>MNIST</td><td>SVHN</td></tr><tr><td>Original</td><td>MNIST SVHN</td><td>-0.16 -0.46</td><td>0.24 -5.18 -</td></tr><tr><td/><td/><td colspan="2">Target</td></tr><tr><td/><td>Conventional</td><td>CIFAR-10</td><td>CIFAR-100</td></tr><tr><td>Original</td><td colspan="2">CIFAR-10 CIFAR-100 0.11 -24.08 -</td><td>0.53 -68.44 -</td></tr><tr><td/><td/><td colspan="2">Target</td></tr><tr><td/><td>Distributed</td><td>CIFAR-10</td><td>CIFAR-100</td></tr><tr><td>Original</td><td>CIFAR-10</td><td>-</td><td>0.29 -54.32</td></tr></table>

Peer review (iclr2017):
Update: I thank the author for his comments! At this point, the paper is still not suitable for publication, so I*m leaving the rating untouched. This paper proposes a transfer learning method addressing optimization complexity and class imbalance. My main concerns are the following: 1. The paper is quite hard to read due to typos, unusual phrasing and loose use of terminology like “distributed”, “transfer learning” (meaning “fine-tuning”), “softmax” (meaning “fully-connected”), “deep learning” (meaning “base neural network”), etc. I’m still not sure I got all the details of the actual algorithm right. 2. The captions to the figures and tables are not very informative – one has to jump back and forth through the paper to understand what the numbers/images mean. 3. From what I understand, the authors use “conventional transfer learning” to refer to fine-tuning of the fully-connected layers only (I’m judging by Figure 1). In this case, it’s essential to compare the proposed method with regimes when some of the convolutional layers are also updated. This comparison is not present in the paper. Comments on the pre-review questions: 1. Question 1: If the paper only considers the case |C|==|L|, it would be better to reduce the notation clutter. 2. Question 2: It is still not clear what the authors mean by distributed transfer learning. Figure 1 is supposed to highlight the difference from the conventional approach (fine-tuning of the fully-connected layers; by the way, I don’t think, Softmax is a conventional term for fully-connected layers). From the diagram, it follows that the base CNN has the same number of convolutional filters at every layer and, in order to obtain a distributed ensemble, we need to connect (for some reason) filters with the same indices. This does not make a lot of sense to me but I’m probably misinterpreting the figure. Could the authors revise the diagram to make it clearer? Overall, I think the paper needs significant refinement in order improve the clarity of presentation and thus cannot be accepted as it is now.