# Improvement of Heterogeneous Transfer Learning Efficiency by Using Hebbian Learning Principle
## 4 Experiments
### Experiment Study  
#### 4.2.1 Experiment A: CIFAR-10 to CIFAR-100 Transfer  
In the first experiment, we use all 10 classes of the CIFAR-10 for source domain training to get the parameters \(W^{5}\). For the target domain, we made 10 different subsets of CIFAR-100 categories by grouping 2 similar superclass categories together like 'aquatic mammals' and 'fish', and performed fine-tuning for those 10 subsets and compared the accuracy. Tables 2 and 3 describe the source and target datasets for experiment A. First, we trained the CNN model on the CIFAR-10 dataset for the classification task on ten different data classes. Under our experimental setup, we set the learning rate of {0.0001, 0.00001, 0.000001}, which, after reaching a particular epoch level, decreases by unit place. Training batch size used as one image per training cycle. We use cross-entropy loss. Second, we use the pre-trained model in the heterogeneous source domain to initialize the model for transfer learning on the target domain. The last layer of CNN is set to random values. Our experimental setup for fine-tuning the connection weights for transfer learning. We used a standard fine-tuning algorithm to do transfer learning on ten different category classes subsets, from CIFAR-100 datasets (as described in Table 3). Cross-entropy loss is used as in training source dataset. Next, we repeat the experiment with the same pre-trained model using our proposed transfer learning algorithm, Hebbian transfer learning. The experimental setup for fine-tuning the connection weights for transfer learning is same as source domain training. It works in phases, where a lifetime is mapped to one epoch or one cycle of the fine-tuning process [64]. Each lifetime contains n-number of episodes; n is a meta-parameter for the algorithm. At each time step, a transfer learning process uses a batch size of 1, which is for one iteration of a forward pass for the CNN and the Hebbian trace is updated. At the end of each episode, we calculate loss and compute the eta (\(n\)) and alpha (\(\alpha\)) by using backpropagation and update \(w_{i,j}\) and \(a_{i,j}\) by using backpropagation. After every episode, Hebbian trace is re-initialized to all zero. \[BCE=-\frac{1}{N}\sum_{i=0}^{N}y_{i}.log\Big{(}\dot{y_{i}}\Big{)}+(1-y_{i}\ ).log\Big{(}1-\dot{y_{i}}\Big{)} \tag{4}\]  
HTL propagate back the gradient at the end of each episode as mentioned above. For keeping the configuration comparable, the validation loss function is used in HTL transfer learning setup, cross-entropy. To calculate the gradient, the loss is calculated using Equation (4). The error is then back \begin{table}
\begin{tabular}{c l l} \hline \hline
**Dataset** & & **Classes** \\ \hline D\({}_{1}\) & vehicles 1: & bicycle, bus, motorcycle, pickup truck, train \\ D\({}_{2}\) & vehicles 2: & lawn-mower, rocket, streetcar, tank, tractor \\ D\({}_{3}\) & large carnivores: & bear, leopard, lion, tiger, wolf \\ D\({}_{4}\) & large carnivores and herbivores: & camel, cattle, chimpanzee, elephant, kangaroo \\ D\({}_{5}\) & household furniture: & bed, chair, couch, table, wardrobe \\ D\({}_{6}\) & household electrical devices: & clock, computer keyboard, lamp, telephone, television \\ D\({}_{7}\) & people: & baby, boy, girl, man, woman \\ D\({}_{8}\) & reptiles: & crocodile, dinosaur, lizard, snake, turtle \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Datasets from CIFAR-100 used for source and target in experiment B.  
propagated. For keeping the configuration comparable, i.e., the validation loss function used in HTL is kept the same as the one used in standard transfer learning setup, Cross-entropy. We record validation loss, top-1, and top-5 accuracy results for each lifetime. We must note here that, for each epoch in the standard transfer learning algorithm, SGD is performed, which is for each training image, because the batch size is 1. However, in Hebbian transfer learning, the algorithm only back- propagates at the end of each episode, after every n-time steps. HTL is performing n-times fewer gradient updates.