# Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning
## 1 Introduction  
Recently, computer vision technologies, including image recognition and object detection, have developed rapidly in the field of deep learning [1, 2]. Despite these remarkable achievements, one of the significant challenges in neural network-based computer vision algorithms is learning new tasks continually, like the cognitive process of human learning [3, 4, 5]. Because in practical situations, new datasets for new tasks are obtained in an incremental manner, meaning that the network only accesses the dataset for new tasks [6]. Three conditions are needed for the successful incremental learning algorithm:  
1. The subsequent data from new tasks should be trainable and be accommodated incrementally without forgetting any knowledge in old tasks, i.e., it should not suffer from catastrophic forgetting [7]. 2. The overhead of incremental training should be minimal. 3. The previously seen data of old task should not be accessible when it is training incrementally. For incremental learning, most of previous works have focused on utilizing knowledge acquired from previously learned tasks and transferring them to a new task, avoiding \"catastrophic forgetting\" [8, 9, 10]. [PERSON] and [PERSON] [11] proposed 'Learning withoutForgetting (LWF)' to exploit knowledge distillation [12] as a solution to catastrophic forgetting. Using only datasets for the new tasks, they encourage reducing the variance between the output probabilities of old classes in the original network and that in the updated new network. In addition, [PERSON] et al. [13] proposed incremental learning for sementic segmentation, while previous works focused on only object detection or classification problems. This methodology introduced four types of knowledge distillation strategies such as knowledge distilled on the output layer, on the intermediate feature spaces, on dilation layers, and on the intermediate layers of the decoding. They also tried to freeze only the first couple of convolutional layers allowing to preserve the lower level descriptions. However, this method only freezes the first couple of convolutional layers, rather than considering how much layer can be shared to preserve the feature extraction capabilities unchanged from previously learned task. [PERSON] et al. [12] introduced a methodology to select how much convolutional layers can be shared for incremental learning by considering where the forgetting issue comes from. There are two steps of the methodology, which are 'Catastrophic Forgetting Dissector (CFD)' and 'Critical Freezing' approaches. The first step is to explain how forgetting happens, based on the vision network models to figure out the most flexible convolutional blocks in a network, and the second step is to freeze weights of the earlier blocks than the most flexible convolutional layers. However, although the relationship between the accuracy and the ratio of freezing weights is a trade-off to be balanced, this methodology is limited to find the most flexible convolutional blocks adaptively with given requirements. [PERSON] et al. [14] focused on hardware and energy requirements for model update [15; 16; 17], then proposed a _'clone-and-branch'_ technique leveraging general knowledge from previous tasks to learn subsequent new tasks by sharing some initial convolutional layers of the base network as fixed extractors and fine-tuning in the new branch. This is because a deep convolutional neural network (DCNN) denotes the neural network that consists of multiple convolutional layers to extract hierarchical visual features [9; 18]. Thus, when the image is fed into a DCNN, the network transforms the image hierarchically layer by layer. Observing features learned by a convolutional neural network, the earlier convolutional layers extract basic features like color and texture, while the later layers extract more detailed and sophisticated features like certain parts of objects [19; 20; 21].