Progressive Learning and Network Growing in TensorFlow | by Viviane | Towards Data Science

In many real world applications new training data becomes available after a network has already been trained. Especially with big neural networks, it would be very tedious to retrain the complete model every time new information becomes available. It would be much easier to simply add new nodes to the network for each new class or other information that was introduced and keep all the other previously trained weights. In this post I will give a brief overview on how to do this in TensorFlow and how the network performance is affected by it.
I will demonstrate the procedure and results on the CIFAR-10 data set but the Git repository linked at the end of this post also provides code and results for MNIST and CIFAR-100 and can easily be adapted to any other data set of this kind. The network I use has a very simple structure of two convolutional layers followed by one fully connected layer with 512 neurons and one read out layer with as many neurons as classes in the data set.
In the example of a progressively learning network here, training starts with six of the ten classes in CIFAR-10. After each epoch, one new class is introduced until, after five epochs, all ten classes are in the data set. In order for the network to train on a newly added class, it needs to have a new output node for this class. This means that the last layer of the network as well as the weights connected to it grow in size with every new class that is added. To implement this in TensorFlow, I reinitialize the complete last layer with a new layer of the new size.
# Define initializer (old weights + random samples from them for the new node(s) )initializer = tf.constant_initializer(np.append(weights,np.random.choice(weights.flatten(),(512,1)),axis=1)) # Initialize new last layer of size (512 x numClasses)            tf.variable_scope(tf.get_variable_scope(),reuse=tf.AUTO_REUSE):                outLog,Wname =      newLastInit(hidden_layer,numOutP,initializer)                acc,CE = evaluate(outLog,desired)                accuracy = tf.reduce_mean(tf.cast(acc, tf.float32))                cross_entropy = tf.reduce_mean(CE)                                   learning_rate = 1e-4                optimizer = tf.train.AdamOptimizer(learning_rate,name='op_0')# get name of new layerweightName = Wname# define gradientsgradient = tf.gradients(cross_entropy, weightName)[0]# define training goaltraining_step = optimizer.minimize(cross_entropy)# get all uninitialized variablesuninitialized_vars = getUnititialized(tf.all_variables())# initialize all uninitialized variablesinit_new_vars_op = tf.initialize_variables(uninitialized_vars)# Start training for the next epochsession.run(init_new_vars_op)
The weights connected to the new last layer are best initialized with the already trained weights from the previous epoch. The weights connected to the newly added node can be randomly sampled from the old weights or a random distribution. This weight initialization leads to significantly better results than initializing all weights with random numbers or a small constant since it preserves all information learned in the previous epoch(s).
For all three weight initializations you can see a drop of performance whenever a new class is introduced. However, the drop for the first two initializations (constant and random initialization) is much bigger than when initializing with the old weights since the old weight initialization preserves already learned information and only the new class needs to be trained from the beginning on. When comparing the network performance of the old classes to the performance of the new class, one can see how the performance drop is mainly caused by the latter. The performance on the old classes is relatively untouched by a new class introduction.
Looking at these graphs it seems as if adding a new class and growing the network has little to no effect on the already learned information on the old classes when preserving the old weights. But how does it effect the overall test accuracy (black line)?
To compare network growing to a conventional training with all classes in the data set from the beginning onward, I train the network under each condition a hundred times and then compare the distribution of the resulting 100 test accuracies of each condition. Additionally, I add a condition called No Growing in which the last layer stays static (always containing 10 nodes) and only the data set grows over time. This corresponds to initializing a bigger output layer from the beginning on to leave some room for new classes without having to change the network structure during training but limits the maximal number of classes that can be added.
The normal network training (M=0.50, SD=0.01) is significantly better than both of the continuously learning conditions (growing (M=0.48, SD=0.01) and no growing(M=0.49, SD=0.01)), p<0.001. However, the normal training had a strong advantage since it could train on each example for each class five times. The two continuously learning networks had much less time to train on the classes that were introduced later on. In the most extreme case, the images of the class introduced last were only seen once by the networks as opposed to five times by the normally trained network.
When training for longer after the last class has been introduced one can see how the two continuously learning networks catch up to the normal network after 30 epochs of post training, from then on no significant difference can be found anymore, F(2,297), p=0.06.
These results show that a network can be trained with iterative class introduction and can still reach a comparable performance to a network trained with all classes from the beginning on. The continuously learning networks take a bit more time to reach a performance comparable to the performance of a normally trained network but this effect can be explained by the lower exposure of the network to the classes which were added later on. In a real world application this can be a simple solution for adding new information to an already trained network without having to start training from scratch again.