Patent Publication Number: US-11645509-B2

Title: Continual neural network learning via explicit structure learning

Description:
PRIORITY APPLICATION DATA 
     This application claims priority to U.S. Provisional Patent Application No. 62/737,636 filed on Sep. 27, 2018, which is incorporated by reference in its entirety. 
    
    
     TECHNICAL FIELD 
     The disclosure generally relates to training a neural network and more specifically to improving the neural network to process multiple tasks without catastrophic forgetting. 
     BACKGROUND 
     Conventional deep learning neural networks are designed to process a single task and are trained on a large dataset that corresponds to the task. In this case, all training samples may be drawn from the same domain. However, many real-world applications, such as applications in robotics, machine vision, etc., may have multiple tasks and have training samples from different domains. In this case, multiple tasks may be performed using multiple neural networks, where each network is designated to process a particular task. However, storing multiple neural networks in resource-constrained platforms, such as mobile device platforms, may be impractical. Further, if multiple neural networks are implemented in hardware, mobile device platforms may become heavy and/or bulky with the additional hardware. This is particularly true when a number of tasks increase over time. 
     Accordingly, there is a need to train a neural network to process multiple tasks. However, training a neural network on a new task by directly modifying the weights or parameters of the neural nodes, may lead to deterioration in the neural network performance when the neural network processes previous tasks. This is because the neural network may forget how to process previous tasks, also known as “catastrophic forgetting.” 
     Accordingly, there is a need for developing a training model for a neural network that trains the neural network to process multiple tasks while reducing “catastrophic forgetting.” 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a simplified diagram of a computing device, according to some embodiments. 
         FIG.  2    is a block diagram of an architectural search, according to some embodiments. 
         FIG.  3    is a block diagram for training a neural network on an individual task in a sequence of tasks, according to some embodiments. 
         FIG.  4    is a simplified diagram of a method for training a neural network, according to some embodiments. 
     
    
    
     In the figures, elements having the same designations have the same or similar functions. 
     DETAILED DESCRIPTION 
     Common approaches to overcoming catastrophic forgetting in a neural network are elastic weight consolidation (“EWC”) and learning without forgetting (“LwF”). The EWC approach is discussed in “Overcoming Catastrophic Forgetting in Neural Networks” and the LwF approach is discussed in “Solving Catastrophic Forgetting in Class Incremental Learning with Support Data,” both of which are incorporated herein by reference. These approaches focus on alleviating catastrophic forgetting by applying constraints on updating the neural network weights. However, while applying constrains may reduce catastrophic forgetting, catastrophic forgetting still exists when using the EWC and LwF approaches. Another common approach to catastrophic forgetting includes memory-based methods that store some key data points for every task and jointly train with the current task data. Yet another approach, such as “piggyback” or “residual adapter” approach, includes learning multiple visual domains and avoiding the catastrophic forgetting in these domains by adding a small portion of parameters while keeping the original weights fixed. These approaches, however, rely on a strong base neural network and knowledge that can only be transferred from one task to another. 
     The embodiments below describe a training module that uses a novel approach for training a neural network to perform multiple tasks. The training module employs an architectural search for each of the sequential tasks which identifies an optimal neural network structure for the current task in the sequential tasks. In identifying an optimal neural network structure, the training module may determine whether the neural network may share each layer&#39;s parameters (also referred to as weights) with the current task, adapt weights to the existing parameters, spawn new parameters for the current task, etc. Once the training module determines the parameters in the neural network architecture for the current ask, the training module may then train the parameters using the current task. 
       FIG.  1    is a simplified diagram of a computing device  100  according to some embodiments. As shown in  FIG.  1   , computing device  100  includes a processor  110  coupled to memory  120 . Operation of computing device  100  is controlled by processor  110 . And although computing device  100  is shown with only one processor  110 , it is understood that processor  110  may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), and/or the like in computing device  100 . Computing device  100  may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine. 
     Memory  120  may be used to store software executed by computing device  100  and/or one or more data structures used during operation of computing device  100 . Memory  120  may include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read. 
     Processor  110  and/or memory  120  may be arranged in any suitable physical arrangement. In some embodiments, processor  110  and/or memory  120  may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor  110  and/or memory  120  may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor  110  and/or memory  120  may be located in one or more data centers and/or cloud computing facilities. In some examples, memory  120  may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor  110 ) may cause the one or more processors to perform the training methods described in further detail herein. 
     As illustrated in  FIG.  1   , memory  120  may include a neural network  130 . Neural network  130  may be implemented using hardware, software, and/or a combination of hardware and software. Neural network  130  typically includes one or more interconnected layers with each layer having one or more nodes. The nodes at each layer typically include one or more parameters (also referred to as weights), that may manipulate the data received at the node before passing the data to one or more nodes at the next layer in the neural network. 
     In some embodiments, neural network  130  may generate a result when data from a data set is passed through neural network  130 . For example, neural network  130  may recognize an object in an image from the data submitted as input to neural network  130 . In another example, neural network  130  may determine a word or words based on data that includes a sequence of sounds submitted as input. 
     Prior to neural network  130  generating an expected result, neural network  130  may be trained. During training, one or more known datasets are passed through the layers of neural network  130  to generate an output. The weights of individual nodes in neural network  130  may be modified until the output of neural network  130  converges toward the expected output. 
     In some embodiments, memory  120  may include a training module  150 . Unlike a conventional training module, training module  150  may train neural network  130  to process multiple tasks, such as tasks  140 , while minimizing catastrophic forgetting. In some embodiments, training module  150  may learn each task in tasks  140  sequentially and using a two-step process on each task. To learn each task, training module may include a structure optimizer  160  and parameter optimizer  170 . Structure optimizer  160  may perform an architecture search step of the two-step process and may identify an optimal choice for a parameter or weight in each layer of neural network  130 . The parameter or weight in the optimal choice minimizes the loss function, described below. An example choice may be to share or reuse a parameter in neural network  130 . The existing parameter may have been determined while neural network  130  has been trained on one or more previous tasks in tasks  140 . Another choice may be to skip a layer in neural network  130 . Yet another choice may be to add a small amount to the existing parameter or determine a new parameter from scratch to process the task. In some embodiments, the loss function may be a multi-objective loss function that includes the normal learning loss (e.g. cross-entropy loss for classification) and budget control loss (penalize the choices that add a larger parameter size), etc. Once the architecture search step completes for each task, the two-step process proceeds to a parameter optimization step where parameter optimizer  170  may retrain neural network  130  on the task. 
       FIG.  2    is a block diagram of a neural network  200  undergoing an architectural search, according to some embodiments. Neural network  200 , which may be consistent with neural network  130  of  FIG.  1   , may process two tasks, task A (T A ) and task B (T B ) as tasks  140 . In the embodiment in  FIG.  2   , training module  150  may be training neural network  200  to process task B, and as part of the training add an adaptation to a parameter that adds a small amount of data, reuse a parameter or generate a new parameter, as will be discussed below. 
     For example, neural network  200  in  FIG.  2    includes five layers, where the first layer includes parameter S 1 , the second layer includes parameter S 2 , the third layer includes parameter S 3 , the fourth layer includes parameter S 4  and S′ 4  and the fifth layer includes task specific layers, such as layer T A  for task A and layer T B  for task B. 
     In the architectural search illustrated in  FIG.  2   , training module  150  may have already trained neural network  200  on task A and is training neural network  200  on task B. As illustrated in  FIG.  2   , training module  150  may determine that parameters S 1  and S 2  that were generated while training task A may also be re-used for tasks B. As also illustrated in  FIG.  2   , training module  150  may determine that parameter S 3  may include an adaptation  202  that is associated with task B, and add a new parameter S′ 4  to layer four that may be used to process task B. As also illustrated in  FIG.  2   , layer T A  is a layer in neural network  200  that is specific to processing task A, and layer T B  is a layer in neural network  200  that is specific to processing task B. 
     Going back to  FIG.  1   , in some embodiments, neural network  130  may be trained using multiple tasks  140 . Tasks  140  may be modeled as a test sequence designated as T=(T 1 , T 2 , . . . , T N ), where N is a number of tasks in tasks  140 . Each task T i  in T may be composed of dataset 
                 D   i     =     {       (       x   1     (   i   )       ,     y   1     (   i   )         )     ,     (       x   2     (   i   )       ,     y   2     (   i   )         )     ,   …   ⁢           ,     (       x     N     T   i         (   i   )       ,     y     N     T   i         (   i   )         )       }       ,         
where each dataset D i  may be of size N T     i   . In some embodiments, training module  150  may train neural network  130  to observe tasks from 1 to N sequentially. This means that after neural network  130  has finished learning or training on task T i , neural network  130  may no longer access task T i . Accordingly, the data from dataset D i  that training module  150  used to train neural network  130  on task T i  may not be available when learning tasks T i+1  to T N . Further, even though training module  140  may train tasks  140  sequentially, the actual order of tasks  140  in the sequence may be random.
 
     In some embodiments, while training neural network  130 , training module  150  may minimize the loss function in a continuous learning setting. Example loss function L for neural network  130  may be: 
                     L   ⁡     (   θ   )       =       ∑     i   =   1     N     ⁢       L   i     ⁡     (   θ   )                 (     Equation   ⁢           ⁢   1     )                   L   i     ⁡     (   θ   )       =       1     N     T   i         ⁢       ∑     n   =   1       N     T   i         ⁢       l   i     ⁡     (         f   θ     ⁡     (     x   n     (   i   )       )       ,     y   n     (   i   )         )                   (     Equation   ⁢           ⁢   2     )               
where f θ  is the model and l i  is the loss function for task T i  that neural network  130  is trained to process. In some embodiments, because training module  150  may not have access to an entire dataset at the same time, loss function L(θ) of Equation 1 may be difficult to minimize. However, if training module  150  ignores the data access issue while training neural network  130  on each task T i , the training may lead to catastrophic forgetting. In some embodiments, the loss function L(θ) of Equation 1 may also fail to account for structural changes made to neural network  130 .
 
     In some embodiments, while learning a structure of neural network  130  for new tasks that are highly dissimilar from the current and previously seen tasks in tasks  140 , optimizing the neural network parameters conventionally may cause forgetting. This is because neural network  130  may function differently between new tasks and previously seen tasks in tasks  140 , and it is unlikely that optimal parameters generated for a new task may also be a good solution for the previous tasks. In some embodiments where the new and previous tasks in tasks  140  are similar, it also may not be ideal to share the complete structure of neural network  130  to process the new and previous tasks. This is because there might be fine grained details among similar tasks in tasks  140  that make a part of neural network  130  focus on extracting different types of representations. To alleviate the deficiencies above, training module  150  may include a s i (θ) that indicates the structure for task T i  in the loss function L(θ). In this case, the loss function for individual task T i  may be: 
                       L   i     ⁡     (   θ   )       =       1     N     T   i         ⁢       ∑     n   =   1       N     T   i         ⁢       l   i     ⁡     (         f       s   i     ⁢   θ       ⁡     (     x   n     (   i   )       )       ,     y   n     (   i   )         )                   (     Equation   ⁢           ⁢   3     )               
In this way, the structure s i (θ) of each individual task T i  may explicitly be taken into consideration when training module  150  may train neural network  130  using tasks  140 .
 
     In some embodiments, when optimizing the updated loss function L i (θ) in Equation 3, training module  150  may determine the optimal parameter based on the structure s i . In some embodiments, training module  150  may interpret the loss as selecting a task specific network from a super network, such as neural network  130 , that has parameter θ using s i . In this case, there is a constraint on a total size of neural network  130 . In other embodiments, for each task T i  in tasks  140  that training module  150  may train neural network  130  with parameter s i (θ). In this case, the total size of neural network  130  may increase as a number of tasks  140  increases because the total size of neural network  130  would be the total size of neural network models for tasks  140 . 
     In some embodiments, the loss function may also be defined as follows: 
     
       
         
           
             
               
                 
                   
                     
                       L 
                       i 
                     
                     ⁡ 
                     
                       ( 
                       θ 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         1 
                         
                           N 
                           
                             T 
                             i 
                           
                         
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             n 
                             = 
                             1 
                           
                           
                             N 
                             
                               T 
                               i 
                             
                           
                         
                         ⁢ 
                         
                           
                             l 
                             i 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   f 
                                   
                                     
                                       s 
                                       i 
                                     
                                     ⁢ 
                                     θ 
                                   
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     x 
                                     n 
                                     
                                       ( 
                                       i 
                                       ) 
                                     
                                   
                                   ) 
                                 
                               
                               , 
                               
                                 y 
                                 n 
                                 
                                   ( 
                                   i 
                                   ) 
                                 
                               
                             
                             ) 
                           
                         
                       
                     
                     + 
                     
                       
                         β 
                         i 
                       
                       ⁢ 
                       
                         
                           R 
                           i 
                           s 
                         
                         ⁡ 
                         
                           ( 
                           
                             s 
                             i 
                           
                           ) 
                         
                       
                     
                     + 
                     
                       
                         λ 
                         i 
                       
                       ⁢ 
                       
                         
                           R 
                           i 
                           p 
                         
                         ⁡ 
                         
                           ( 
                           θ 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     4 
                   
                   ) 
                 
               
             
           
         
       
     
     where β i &gt;0, λ i ≥0, R i   s  and R i   p  indicate regularizers for structure and parameter, respectively. Regularizers may be terms that are added to the loss function that cause optimal parameters for task T i  to remain close, that is within configurable range to optimal parameters of previous tasks. Regularizers are further discussed in “Overcoming catastrophic forgetting in neural networks,” which is incorporated herein by reference. In this way, training module  140  may set an upper bound to the total number of parameters and may avoid a scenario where the size of neural network  130  increases proportionally with the increase in the number of tasks  140 . 
     In some embodiments, training module  150  may optimize the loss function L i (θ) in Equation 4 using a two-step process. The first step in a two-step process may include structure optimization, and the second step may include parameter learning. As will be discussed below, structure optimizer  160  may perform an architectural search on neural network  130  for each task T i  in tasks  140  that identifies optimal parameters for each task T i  and parameter optimizer  170  may train the optimal parameters in neural network  130  to process task T i . 
     As discussed above, structure optimizer  160  may perform an architecture search of neural network  130  that optimizes the structure of neural network  130 . During training, neural network  130  may be referred to as super network S. Additionally, neural network  130  may be trained to include L shareable layers that may be shared by tasks  140  and one task-specific layer that is specific for each task T i  in tasks  140 . In some embodiments, the task specific layer may be the last layer in neural network  130 . 
     In some embodiments, structure optimizer  160  may maintain super network S so that the new task-specific layers and new shareable layers may be stored in super network S once structure optimizer  160  completes optimizing task T i . 
     In some embodiments, the parameters or weights in super network S may be initialized with predetermined values or may be initialized with random values. 
     In some embodiments, structure optimizer  160  may search for a task specific neural network for each individual task T i . To search for the task specific neural network, structure optimizer  160  may make a copy of super network S. Structure optimizer  160  may then generate a task specific neural network from a copy of the super network S for task T i  by identifying an optimal choice for each of the shareable layers in the copy of the super network given new task dataset D i  and the weights of the parameters in shareable layers stored in the super network S. In some embodiments, structure optimizer  160  may select among the “reuse”, “adaptation,” and “new” choices to optimize the task specific neural network. The “reuse” choice may occur when structure optimizer  160  may reuse the same parameters in the task specific neural network for the new task as for a previous task. In other words, structure optimizer  160  may use parameters in the task specific neural network that were included in the super network S from previous tasks. The “adaptation” choice may occur when structure optimizer  160  adds a small parameter overhead that trains an additive function to the original layer output. In other words, the “adaptation” choice may add an additive function to one of the parameters that were previously stored in super network S. The “new” choice may cause structure optimizer  160  to spawn one or more new parameters that do not exist in super network S. In some embodiments, structure optimizer  160  may spawn the one or more new parameters in a current layer with the same size as the parameters that already exist in current layer. 
     In some embodiments, the size of the l th  layer in super network S may be denoted as |S l |. Accordingly, the total number of choices in the l th  layer C l  is 2|S l |+1, because structure optimizer  160  may include |S l | “reuse” choices, |S l | “adaptation” choices and generate one “new” choice. Thus, the total search space may be Π l   L  C l . 
     In some embodiments, to prevent the search space from growing exponentially with respect to a number of tasks  140 , structure optimizer  160  may limit the total number of possible choices and make a priority queue for managing different choices. 
     In some embodiments, structure optimizer  160  may make the search space continuous. To make the search space continuous, structure optimizer  160  may relax the categorical choice of the l th  layer as a softmax over all possible C l  choices according to Equation 5: 
                     x     l   +   1       =       ∑     c   =   1       C   l       ⁢         exp   ⁡     (     α   c   l     )           ∑       c   ′     =   1       C   l       ⁢     exp   ⁡     (     α     c   ′     l     )           ⁢       g   c   l     ⁡     (     x   l     )                   (     Equation   ⁢           ⁢   5     )               
where the vector α l  of dimension C l  may be the architecture weights that may be used for mixing the choices for each shareable layer, and g c   l  may be an operator for the choice c at layer l which may be expressed according to Equation 6 as:
 
                       g   c   l     ⁡     (     x   l     )       =     {           S   c   l             if   ⁢           ⁢   c     ≤          S   l                          S   c   l     ⁡     (     x   l     )       +       γ     c   -          S   l            l     ⁡     (     x   l     )                 if   ⁢           ⁢          S   l            &lt;   c   &lt;     2   ⁢          S   l                          o   l     ⁡     (     x   l     )               if   ⁢           ⁢   c     =       2   ⁢          S   l            +   1                       (     Equation   ⁢           ⁢   6     )               
where γ is the adapter operator and o is the new operator trained from scratch. In some embodiments, after the above relaxation, the task of discrete search becomes optimizing a set of continuous weights α={α l }. In some embodiments, after structure optimizer  160  may complete the architectural search, structure optimizer  160  may determine an optimal structure by taking the index with the largest weight α c   l  for each layer l, i.e. c l =arg max α l .
 
     In some embodiments, structure optimizer  160  may use the validation loss L val  to update the architecture weights α, and optimize the operator weights by the training loss L train . In some embodiments, structure optimizer  160  may update the architecture weights and the operator weights alternatively during the architectural search process. In some embodiments, structure optimizer  160  any use a first order approximation or a second order approximation process to update the architecture weights and/or the operator weights. 
     The example below describes how structure optimizer  160  may use the “reuse”, “adaptation” and “new” choices. Suppose the task specific neural network associated with an individual task T i  is a convolutional neural network where all layers have a 3×3 kernel size. When structure optimizer  160  selects a “reuse” choice, the existing weights in the shareable layers are reused and are kept fixed during the training process. When structure optimizer  160  selects an “adaptation” choice, structure optimizer  160  may, for example, add a 1×1 convolutional layer in parallel with the original 3×3 convolutional layer. In this case, during training (described below), parameter optimizer  170  may fix the parameters of the 3×3 convolutional kernel, but the parameters of the 1×1 convolutional layer may be modified. In this case, the cost of adding an additional parameter may be 1/9 of the original parameter size. When structure optimizer  160  selects a “new” choice, structure optimizer  160  may add a new parameter that may be initialized and trained from scratch. In some embodiments, structure optimizer  160  may use the loss function L val  to implement the regularizer for the structure R i   s (s i ). In some embodiments, the value of the loss function may be set as being proportional to the product of the additional parameter size z c   l  and its corresponding weight α c   l . In some embodiments, structure optimizer  160  may optimize the architectural weights α in terms of accuracy and parameter efficiency at the same or approximately the same time. 
     Once structure optimizer  160  obtains optimal choices for each layer of the task specific neural network, parameter optimizer  170  may retrain the optimal architecture on task T i . In some embodiments, parameter optimizer  170  may retrain the “reuse” parameters either by fixing the parameter and keeping the parameter unchanged or tuning the parameter using regularization, such as l 2  regularization or more sophisticated regularization such as elastic weight consolidation. In some embodiments, when structure optimizer  160  selects to “reuse” a parameter at layer l, the l th  layer may learn a very similar representation during the current task T i , as it learned during one of the previous tasks. This may be an indication of a semantic similarity learned at layer l between the current and previous tasks. 
     In some embodiments, parameter optimizer  170  may retrain the “reuse” parameter by tuning the l th  layer with some regularization. Tuning the parameters may be accomplished by slightly modifying the parameters within a preconfigured range. Tuning the parameters may also benefit the previous tasks or at least reduce catastrophic forgetting due to the semantic relationships. 
     In some embodiments, after parameter optimizer  170  retrains the optimal architecture on a current task T i , parameter optimizer  170  may merge the newly created and tuned parameters in the layers, task specific adapters and classifier(s) in the task specific neural network with the maintained super network S. In this way, neural network S may be used for model inference and be basis for further architecture search on subsequent tasks. Subsequently, training module  150  may repeat the process on the next sequential task in tasks  140  until training module  150  trains neural network  130  on all tasks in tasks  140 . 
       FIG.  3    is a block diagram  300  of training a neural network on individual task T k , according to some embodiments.  FIG.  3    illustrates a current state of super network  302  that includes layers and parameters determined using one or more previous tasks. Super network  302  may be neural network  130 , discussed above. As illustrated in  FIG.  3   , super network  302  may have one or more shareable layers, that include parameter S 1  in the first layer, parameters S 2  and S′ 2  in the second layer, parameter S 3  in the third layer, and parameters S 4 , S′ 4 , and S″ 4  in the fourth layer. Additionally, super network  302  may have a task specific layer T. Task specific layer T may have layers that include parameters that correspond to the one or more previous tasks. 
       FIG.  3    also illustrates neural network  304 . In some embodiments, structure optimizer  160  may generate neural network  304  by obtaining a copy of super network  302  and training the copy of super network  302  on task T k . As part of the training, structure optimizer  160  may perform an architectural search on neural network  304  for task T k . As discussed above, structure optimizer  160  may obtain choices that indicate whether to reuse parameters, add an adaptation to the parameter, or add a new parameter to each layer of neural network  304 . Additionally, structure optimizer  160  may also add a task specific layer  310  to neural network  304  for task T k . 
     As illustrated in  FIG.  3   , structure optimizer  160  may add a “reuse” choice, an “adaptation” choice, and/or a “new” choice to neural network  304  using “reuse,” “adaptation,” and/or “new” operators. For example, as illustrated in  FIG.  3   , structure optimizer  160  may add an adaptation to the parameter S 1  and generate a new parameter S′ 1  in the first layer. In another embodiment, structure optimizer  160  may add an adaptation to the parameter S 2 , add an adaptation for the parameter S′ 2  and generate a new parameter S″ 2  in the second layer. In another example, structure optimizer  160  may add an adaptation to the parameter S 3  and generate a new parameter S′ 3  in the third layer. In another example, structure optimizer  160  may add an adaptation to the parameter S 4 , add an adaptation to the parameter S′ 4 , add an adaptation to the parameter S″ 4  and generate a new parameter S′″ 4  in the fourth layer. 
     In some embodiments, when structure optimizer  160  may add a “reuse” choice, an “adaptation” choice, and a “new” choice to each layer, structure optimizer  160  may add a total of 2|S l |+1 choices. In further embodiments, structure optimizer  160  may obtain a set of continuous architecture weights α for each layer, shown as parameter α 1  for the first layer, parameter α 2  for the second layer, parameter α 3  for the third layer, and parameter α 4  for the fourth layer S 4 . 
     Once structure optimizer  160  adds parameters for each layer of neural network  304  using “reuse,” “adaptation,” and “new” operators, structure optimizer  160  may determine the optimal parameters for neural network  304  for task T k . To determine the optimal parameters, structure optimizer  160  may train neural network  304  using task T k  to determine which parameters generate the largest architectural weight in each layer. For example, structure optimizer  160  may select an architecture for task T k  that is illustrated as neural network  306  in  FIG.  3    and includes parameter S 1  in the first layer, parameter S 2  with an adaptation in the second layer, the new parameter S 3  in the third layer and a reused parameter S″ 4  in the fourth layer. 
     In some embodiments, once structure optimizer  160  determines neural network  306 , parameter optimizer  170  may optimize the parameters in neural network  306  by retraining neural network  306  using the task T k . 
     Once parameter optimizer  170  completes optimizing the parameters in neural network  306 , parameter optimizer  170  may update super network  302  with the architecture and parameters of neural network  306 , as shown by neural network  308 . After parameter optimizer  170  updates super network  302  with the architecture and parameters in neural network  306 , neural network  308  become the super network for task T k+1  (not shown). 
       FIG.  4    is a simplified diagram of a method  400  for training a neural network, according to some embodiments. One or more of the processes  402 - 418  of method  400  may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes  402 - 418 . 
     At process  402 , a sequence of tasks is received. For example, training module  150  may receive tasks  140  from memory  120 . As discussed above, training module  150  may train neural network  130  using tasks  140 . As further discussed above, tasks  140  may be sequential tasks. 
     At process  404 , a determination whether the last task in tasks  140  has been reached. For example, training module  150  may determine whether the last tasks in tasks  140  has been reached and used to train neural network  130 . If the last task has been reached, the super network S becomes the trained neural network  130 , and method  400  ends. If there are more tasks in tasks  140  that training module  150  may use to train neural network  130 , method  400  proceeds to process  406 . 
     At process  406 , a current task is retrieved. For example, training module  150  may retrieve a current task in tasks  140 . The current task may be a task that training module  150  may use to train neural network  130  in the sequence of tasks  140 . 
     At process  408 , a copy of a super network S is retrieved. As discussed above, a super network S may store the architecture that includes shareable layers, parameters included in the shareable layers, and task specific layers of previously trained tasks in tasks  140 . 
     At process  410 , a task specific neural network is determined. For example, structure optimizer  160  may determine an architecture for the task specific neural network for the current task by taking a copy of the super network retrieved in process  408  and adding the parameters from the “reuse,” “adaptation,” and “new” choices to the copy of the super network S at each layer. 
     At process  412 , optimal parameters in the task specific neural network are determined. For example, structure optimizer  160  may determine which parameters in the task specific neural network maximize architectural weights for each layer. The parameter that maximizes the architectural weight at that layer is the optimal parameter for that layer. 
     At process  414 , the optimal parameters in the task specific neural network are retrained. For example, parameter optimizer  170  optimizes the optimal parameters in the task specific neural network by retraining the parameters using the current task. As parameter optimizer  170  retrains the optimal parameters, parameter optimizer  170  may also tune the parameters that structure optimizer  160  identifies as the “reuse” parameters. 
     At process  416 , the optimized parameters are merged into the super network. For example, parameter optimizer  170  may merge the adaptations to the existing parameters, the new parameters, and the tuned parameters into the super network S. 
     At process  418 , the current task becomes the next task. For example, training module  150  may increment the current task, such that processes  404 - 416  may be repeated on the task following the current task in tasks  140 . 
     In some embodiments, once training module  150  trains neural network  130 , neural network  130  may be used to process multiple tasks that neural network  130  was trained to process without or with reduced catastrophic forgetting. 
     In some embodiments, neural network  130  trained using training module  150  may process multiple tasks without catastrophic forgetting and thus is an improvement over the conventional neural networks trained using other models. Table 1, below, demonstrates a table that includes ten known image classification tasks, such as tasks ImageNet (“ImNet”), C100, SVHN, UCF, OGlt, GTSR, DPed, VGG-Flower (“Flwr”), Aircraft (“Airc.”), and DTD. The images in the ten tasks may be resized so that the lower edge may be 72 pixels. Further the ten tasks are across multiple domains and the dataset seizes are highly imbalanced. Table 1 also illustrates various neural network models, such as an Individual model, a Classifier model, an Adapter model, and the training module  150  that train the ten tasks. In the Individual model, each of the ImNet, C100, SVHN, UCF, OGlt, GTSR, DPed, Flwr, Airc., and DTD tasks is trained individually. Further, the weights in the Individual model are initialized randomly. In the Classifier model, the classifier in the last layer of the neural network may be tuned, while the former twenty file layers are transferred from the ImageNet pretrained model and are kept fixed during training. In this case, each task may add a task specific classifier and batch normalization layers, thus keeping the size of the Classifier model small. In the Adapter model a 1×1 convolution layer is added next to each 3×3 convolution layer and the outputs of the 1×1 convolution layer and the 3×3 convolution layer may be passed to the next layer of the neural network. Because of the lightweight convolution layer, each task may add approximately 1/9 to the size of the whole model. 
     Neural network  130  that training module  150  may train using the ten tasks may be a 26-layer ResNet network that is described in “Deep Residual Learning for Image Recognition” which is incorporated herein by reference. In this case, neural network  130  may include three residual blocks with each output being 64, 128, and 256 channels. Each residual block may also contain four residual units, each of which consists of two convolutional layers with 3×3 kernels and a skip connection. At the end of each residual block, the feature resolution may be halved by average pooling. Further, when an adaptation choice is included in neural network  130 , the adaptation may be a 1×1 convolution layer with channels equal to the current layer output channels that is added for the corresponding layer. The convolved results may be added back to the convolution output. During training, parameter optimizer  170  may also fix the parameters while the 1×1 layer is adjusted. When training module  150  trains neural network  130  on the ten tasks, training module  150  may train neural network  130  in a random sequence, except for the ImageNet being the first task. This is to make a fair comparison with the Individual, Classifier, and Adapter models that adapt the ImageNet pretrained model to the other nine tasks. However, in other embodiments, training module  150  may train neural network  130  using all ten tasks in a random sequence. As training module  150  trains neural network  130 , the super network S is maintained as discussed above and newly created parameters and task-specific layers are stored in the super network S. Further, in neural network  130 , the batch normalization layers may be treated as task specific layers, which means that each task has its own batch normalization layer(s). Further, in Table 1, training module  150  may fix the parameters during the retraining phase when a “reuse” choice is selected during the architectural search phase. 
     
       
         
           
               
               
               
               
               
               
               
               
               
               
               
               
               
             
               
                 TABLE 1 
               
               
                   
               
               
                 Model 
                 ImNet 
                 C100 
                 SVHN 
                 UCF 
                 OGII 
                 GTSR 
                 DPed 
                 Flwr 
                 Airc. 
                 DTD 
                 avg. 
                 #params 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
               
               
               
               
               
               
               
               
               
            
               
                 Individual 
                 69.84 
                 73.96 
                 95.22 
                 69.94 
                 86.05 
                 99.97 
                 99.86 
                 41.86 
                 50.41 
                 29.88 
                 71.70 
                 58.96 
               
               
                 Classifier 
                 69.84 
                 77.07 
                 93.12 
                 62.37 
                 79.93 
                 99.68 
                 98.92 
                 65.88 
                 36.41 
                 48.20 
                 73.14 
                 6.68 
               
               
                 Adapter 
                 69.84 
                 79.82 
                 94.21 
                 70.72 
                 85.10 
                 99.89 
                 99.58 
                 60.29 
                 50.11 
                 50.60 
                 76.02 
                 12.50 
               
               
                 Search (Ours) 
                 69.84 
                 79.59 
                 95.28 
                 72.03 
                 86.60 
                 99.72 
                 99.52 
                 71.27 
                 53.01 
                 49.89 
                 77.68 
                 14.46 
               
               
                   
               
            
           
         
       
     
     As illustrated in Table 1, neural network  130  trained as discussed above and designated as “Search (Ours)” generates best results in five out of nine tasks. Table 1 further illustrates that neural network  130  is effective for tasks that have a small data size, such as VGG-Flowers and Aircraft tasks, where neural network  130  outperforms other models by a large margin and has the largest average for correctly processing the ten tasks. As also illustrated above, the size of neural network  130  is relatively small, and is comparable to the Adaptor model. 
     Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.