METHOD AND DEVICE WITH CONTINUAL LEARNING

A method and device for performing continual learning are provided. The method of performing continual learning of tasks in a set of tasks includes learning a first model based on training data corresponding to a current task in the set of tasks, learning a second model based on information on the current task and information on a previous learning task in the set of tasks, and resetting the first model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119 (a) of Korean Patent Application No. 10-2023-0062655, filed on May 15, 2023 and 10-2023-0121054 filed on Sep. 12, 2023, in the Korean Intellectual Property Office, the entire disclosures of which are incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to a method and device with continual learning, and more particularly, to a method of resetting and distillation for continual reinforcement learning (CRL).

2. Description of Related Art

Continual learning technologies involve learning from a data stream with the goal of remembering previously learned knowledge and succeeding at a current task. Existing continual learning technologies often include task continual learning (TCL), in which data arrives sequentially in a task group. To mitigate the risk of catastrophic forgetting (CF), knowledge may be refined by adapting a model trained with new data and generalizing the new data by overwriting a weight of the trained model. Particularly, regularization-based, memory replay-based, and dynamic model-based continual learning methods may be considered as a strategy to mitigate the CF problem.

SUMMARY

In one general aspect, a method of performing continual learning of a plurality of tasks includes learning a first model based on training data corresponding to a current task in a set of tasks, learning a second model based on information on the current task and information on a previous learning task in the set of tasks, and resetting the first model.

The learning of the first model may include learning the first model based on a reinforcement learning algorithm.

The learning of the second model may include performing knowledge distillation from the first model to the second model and performing behavioral cloning (BC) of the second model based on the information on the previous learning task.

The method may further include storing the information on the current task in a first buffer and maintaining a second buffer including the information on the previous learning task.

The learning of the second model may include receiving the information on the current task from the first buffer and receiving the information on the previous learning task from the second buffer.

The method may further include, when the learning of the second model is completed, updating the second buffer based on the first buffer and resetting the first buffer.

The updating of the second buffer may include storing, in the second buffer, a portion of the information on the current task stored in the first buffer.

The learning of the second model may include determining a first loss function based on the information on the current task, determining a second loss function based on the information on the previous learning task, and learning the second model based on the first loss function and the second loss function.

In another general aspect, an inference method includes receiving input data and outputting a task, the task corresponding to the input data among tasks in a set of tasks, by inputting the input data to a continual learning model, wherein the continual learning model is trained based on a reinforcement learning model that is distinct from the continual learning model, and wherein the reinforcement learning model is reset each time learning of a task in the set of tasks is completed.

In another general aspect, an electronic device includes a memory configured to store at least one instruction and a processor configured to, by executing the instruction stored in the memory, learn a first model based on training data corresponding to a current task in a set of tasks, learn a second model based on information on the current task and information on a previous learning task in the set of tasks, and reset the first model.

The processor may be configured to learn the first model based on a reinforcement learning algorithm.

The processor may be configured to perform knowledge distillation from the first model to the second model and perform behavioral cloning (BC) of the second model based on the information on the previous learning task.

The processor may be configured to store the information on the current task in a first buffer and maintain a second buffer including the information on the previous learning task.

The processor may be configured to receive the information on the current task from the first buffer and receive the information on the previous learning task from the second buffer.

The processor may be configured to, when the learning of the second model is completed, update the second buffer based on the first buffer and reset the first buffer.

The processor may be configured to store, in the second buffer, a portion of the information on the current task stored in the first buffer.

The processor may be configured to determine a first loss function based on the information on the current task, determine a second loss function based on the information on the previous learning task, and perform the learning the second model based on the first loss function and the second loss function.

In another general aspect, an electronic device includes a memory configured to store at least one instruction and a processor configured to, by executing the instruction stored in the memory, receive input data and output a task, the task corresponding to the input data among tasks in a set of tasks, by inputting the input data to a continual learning model, wherein the continual learning model is trained based on a reinforcement learning model that is distinct from the continual learning model, and the reinforcement learning model is reset each time learning of a task in the set of tasks is completed.

DETAILED DESCRIPTION

Throughout the specification, when a component or element is described as being “connected to,” “coupled to,” or “joined to” another component or element, it may be directly “connected to,” “coupled to,” or “joined to” the other component or element, or there may reasonably be one or more other components or elements intervening therebetween. When a component or element is described as being “directly connected to,” “directly coupled to,” or “directly joined to” another component or element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.

FIG.1Aillustrates an example of continual reinforcement learning (CRL) according to one or more embodiments.

Reinforcement learning is an area of machine learning and may include, as an example, a method in which an agent defined within an environment recognizes a current state and, according to the current state, selects, among selectable actions, an action or action sequence that maximizes a reward. Reinforcement learning differs from general supervised learning in that a training set of input-output pairs is not necessarily presented and correction does not necessarily explicitly occur for an incorrect behavior. Instead, the focus of reinforcement learning is on-line performance, which may be improved by balancing exploration and exploitation.

Recently, following the impressive success of reinforcement learning in various application fields, numerous studies have been conducted to improve the learning efficiency of a reinforcement learning algorithm, and CRL is one of the results of the studies. CRL aims to have an agent continually learn to improve a decision-making policy for multiple tasks.

CRL is a type of continual learning, which refers to a method in which a deep learning model continually learns based on new data. A general deep learning model learns from a large dataset and learns a generalized pattern based on the dataset. However, in a real environment, new data is continually generated, and the new data may significantly differ from existing data. Continual learning may continually learn new data and gradually expand the learned knowledge to solve the issues of existing deep learning (e.g., inability to adapt to new data). This is because it may be more efficient and economical to only add new data to an already learned model rather than retraining a model anew every time new data comes out.

Referring toFIG.1A, CRL may be effective when learning tasks that arrive sequentially to an actor are similar to each other, such as in robot behavior learning. For example, a button pressing task (hereinafter, task1)110, a door opening task (hereinafter, task2)120, and a drawer closing task (hereinafter, task3)130may be learned all at once through CRL. However, CRL has an issue in that catastrophic forgetting (CF) of previously learned knowledge may occur. For example, in task1, a model may be trained so that a robot may perform a button pressing action. Subsequently, the trained model may be trained again so that the robot may perform a door opening action. Next, the trained model may be trained again so that the robot may perform the drawer closing action. The biggest problem of continually training new tasks on a previously trained model in this way is that the previously learned content may be forgotten (lost from the state of the trained model). After training the drawer closing task, the accuracy of the button pressing task that was learned first may decrease significantly. In other words, as the model is gradually trained, the previously learned content may be gradually forgotten.

CRL may have an issue of negative transfer as well as CF. Negative transfer in CRL refers to a phenomenon in which learning of a new task fails, even when fine-tuning is performed, due to intrusive information that is learned in a previous task (previous learning interferes with new learning). Negative transfer does not occur in general continual supervised learning and may only occur in CRL. Because, unlike supervised learning, which may always use a true label for a new task, an agent in reinforcement learning sometimes may not correctly modify a previously learned policy for a new task due to the lack of true labels and weak reward signals.

FIG.1Billustrates an example of negative transfer in CRL, according to one or more embodiments.

Referring toFIG.1B, a first diagram150illustrates the learning success rate (vertical axis) of a sweep task for three million steps (horizontal axis) using the Soft Actor-Critic (SAC) algorithm and the Proximal Policy Optimization (PPO) algorithm, which are examples of reinforcement learning algorithms. However, the reinforcement learning algorithm is not limited to the SAC and PPO algorithms. Referring to the first diagram150, both the algorithms quickly achieve a success rate of 1, and from this it is apparent that the sweep task is an easy task to learn from the beginning.

A second diagram160illustrates the learning success rate when a three million-step door locking task is first learned and a three million-step sweep task is subsequently learned using the SAC algorithm and the PPO algorithm. Referring to the second diagram160, it is apparent that the success rate of the sweep task converges to 0, which indicates that negative transfer occurs in the CRL.

As described in detail below, in performing CRL, a model may be reset each time a task is learned in order to prevent negative transfer. However, since CF may occur when a model is intentionally reset, knowledge distillation (KD) may be performed on a continual learning model in the method of performing CRL.

FIG.2Aillustrates an example of a deep learning operation method using an artificial neural network (NN), according to one or more embodiments.

An artificial intelligence (Al) algorithm including deep learning may input data to an NN, train the NN with output data through operations such as convolution, and extract features using the trained NN. The NN may be a computational element with a network architecture. In the NN, nodes are connected to each other and collectively operate to process the input data. Various types of neural networks include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), and a restricted Boltzmann machine (RBM) model. However, examples are not limited thereto. In a feed-forward neural network, nodes have links to other nodes. The links may expand in one direction, for example, a forward direction, through a neural network. A NN model may have an input layer, hidden layers, and an output layer. Each layer may be made of nodes. There may be connections or links between the nodes of a layer and the nodes of a following layer. The NN model may map inputs at the input layer based on weights of the nodes, among other things.

FIG.2Aillustrates a structure of an NN (e.g., a CNN) for receiving input data and outputting output data. The NN may be a deep neural network including at least two layers.

FIG.2Billustrates an example of a CRL system, according to one or more embodiments.

Referring toFIG.2B, the CRL system may include a training device200and an inference device250. The training device200may correspond to a computing device having various processing functions such as generating a neural network, training (or learning) a neural network, or retraining a neural network. For example, the training device200may be implemented as various types of devices such as a PC, a server device, a mobile device, and the like.

The training device200may generate a trained neural network210by repetitively training (or learning) a given initial neural network. The generating of the trained neural network210may involve determining neural network parameters. I neural network parameters may include various types of data, for example, input/output activations, weights, and biases, any of which may be changed by training of the trained neural network210. When the neural network210is repeatedly trained, the parameters of the neural network210may be tuned to calculate a more accurate output for a given input.

The training device200may transmit the at least one trained neural network210to the inference device250. The inference device250may be, for example, a mobile device or an embedded device. The inference device250may be dedicated hardware for driving a neural network and may be an electronic device including at least one of a processor, memory, an input/output (I/O) interface, a display, a communication interface, or a sensor. For example, the sensor may include one or more cameras or other imaging sensors to capture images of scenes. To summarize, training of the neural network210and use of the neural network (performing inferences) may be performed on respective different computing devices.

The inference device250may be any digital device that includes a memory element and a microprocessor and has an operational capability, such as a tablet PC, a smartphone, a PC (e.g., a notebook computer), an Al speaker, a smart TV, a mobile phone, a navigation, a web pad, a personal digital assistant (PDA), a workstation, and the like.

The inference device250may drive (execute) the at least one trained neural network210without a change thereto or may drive a neural network260obtained by processing (for example, quantizing) the at least one trained neural network210. The inference device250for driving the neural network210/260may be implemented in a separate device independent of the training device200. However, examples are not limited thereto. The inference device250may also be implemented in the same device as the training device200.

FIGS.3A to3Billustrate a CRL method, according to one or more embodiments.

Referring toFIG.3A, a training device (e.g., the training device200ofFIG.2B) may perform continual learning of multiple tasks. The training device may learn a NN model, and the NN model may include a first model310and a second model320.

The first model310may be a reinforcement learning-based neural network. For example, the first model310may be learned according to an actor-critic architecture. In this case, the first model310may include an actor network that learns a policy and a critic network that learns a value function. The critic may evaluate a policy (useable to perform actions) by estimating the values of respective state-action pairs in the policy, while the actor may improve the policy by maximizing an expected reward.

The second model320is a neural network that receives knowledge distillation (KD) from the first model310. The first model310may be referred to as a teacher model and the second model320may be referred to as a student model in that the first model310performs knowledge distillation on the second model320. Alternatively, the first model310may be referred to as an online model in that the first model310learns a new task in an online way by interacting with an environment, and the second model320may be referred to as an offline model in that the second model320replicates a behavior of the online model in an offline way without interacting with the environment. Alternatively, the first model310may be referred to as a single task model and the second model320may be referred to as a continual model. The training device may perform CRL without (or with minimalized) CF and negative transfer using the first model310and the second model320.

More specifically, the first model310may learn a current task T according to a reinforcement learning algorithm and store information on the current task T (e.g., state of the first model310) in a replay buffer DT. Subsequently, the first model310may distill knowledge about the current task T to the second model320using state information stored in the replay buffer DT. Hereinafter, the replay buffer DT may be referred to as a first buffer.

The second model320(θT), to prevent CF in a distillation process thereof, may use an expert buffer MT. The expert buffer MT may include information on a previous task (previous relative to a current task). After the distillation process, the first model310may be reset to learn a next task from the beginning. The above-described learning algorithm may be referred to as a Reset and Distill (R&D) algorithm and as illustrated inFIG.3B.

When the actor θTof the second model320replicates a behavior of an actor of the first model310in the current task T, the loss function of Equation1below may be used to compute lost.

In Equation 1,andrespectively mini-batches sampled from DTand MT, respectively. The term θonlinedenotes an actor network parameter of the first model310, s, and aTdenote the set of all possible states and actions for the taskT, π(St, at) denotes a reward function that generates a scalar value at each transition, KL denotes Kullback-Leibler (KL) divergence, and k denotes the previous task.

FIG.4illustrates an example of a method of performing continual learning, according to one or more embodiments. The description provided with reference toFIGS.1A to3Bmay also apply toFIG.4.

For ease of description, operations410to430are described as being performed using the training device200shown inFIG.2B. However, operations410to430may be performed by another suitable electronic device in any suitable system.

Referring toFIG.4, in operation410, a training device may learn a first model based on training data corresponding to a current task. The training data may be specific to training/learning for the current task. The training device may learn the first model based on a reinforcement learning algorithm as applied to the current training data.

In operation420, the training device may learn a second model based on information on the current task and information on a previous learning task. The training device may perform knowledge distillation from the first model to the second model and may perform behavioral cloning (BC) of the second model based on the information on the previous learning task.

The training device may store the information on the current task in a first buffer and maintain a second buffer including information on the previous learning task. The training device may receive the information on the current task from the first buffer and receive the information on the previous learning task from the second buffer.

When learning of the second model is completed, the training device may update the second buffer based on the first buffer and then reset the first buffer. The training device may store, in the second buffer, a portion of the information on the current task stored in the first buffer.

The training device may determine a first loss function based on the information on the current task, determine a second loss function based on the information on the previous learning task, and learn the second model based on the first loss function and the second loss function.

In operation430, the training device may reset the first model. Resetting may involve, for example, an operation of randomly initializing a parameter of the model.

FIG.5illustrates an example of an inference method, according to one or more embodiments.

Referring toFIG.5, an inference device (e.g., the inference device250ofFIG.2B) may output a task corresponding to input data using a learned NN model.

More specifically, the inference device may receive input data (e.g., a door locking command) and input the input data to a second model (e.g., the second model320ofFIG.3A) of a learned NN model. The second model320may output a task corresponding to the input data among a plurality of tasks. Since the second model320is trained based on the R&D algorithm, negative transfer and CF phenomena may not occur (or may be minimalized), and accordingly, an accurate task suitable for the input data may be output.

FIG.6illustrates an example of an inference method, according to one or more embodiments. The description provided with reference toFIG.5also generally apples toFIG.6.

For ease of description, it is described that operations610and620are performed using the inference device250shown inFIG.2B. However, operations610and620may be performed by another suitable electronic device in any suitable system.

Referring toFIG.6, in operation610, an inference device may receive input data.

In operation620, the inference device may input the input data to a continual learning model and output a task corresponding to the input data among a plurality of tasks. More specifically, the inference device may input the input data to a second model (e.g., the second model320ofFIG.3A) of a learned NN model.

The continual learning model may be trained based on a reinforcement learning model that is distinct from the continual learning model, and the reinforcement learning model may be reset each time learning of each of the plurality of tasks is completed.

FIGS.7A and7Beach illustrate an example effect of a CRL method described herein, according to one or more embodiments.

Referring toFIG.7A, graphs710to740illustrate the average success rate of various methods for four types of task sequences learned using the SAC algorithm (solid line) and the PPO algorithm (dashed line). Other methods are represented by lines that are below the dashed and dotted lines of the SAC and PPO algorithms (with the R&D algorithm). Note that the “EWC” inFIG.7Astands for “elastic weight consolidation”.

Referring to the graphs710to740, it may be seen that in all four types of task sequences, the performance of the R&D algorithm is significantly higher than that of other methods. Furthermore, the average success rate of the R&D algorithm approaches closely to “1,” which indicates that the R&D algorithm may be successfully overcoming both CF and negative transfer.

Referring toFIG.7B, a graph750illustrates the results of measuring negative transfer of three different methods in four types of sequences. Referring to the graph750, it may be seen that the R&D algorithm has a much lower degree of negative transfer than other methods.

FIG.8illustrates an example of a configuration of a training device800, according to one or more embodiments.

Referring toFIG.8, the training device800may include a processor801, a memory803, and a communication module805. In practice, the processor801may be multiple processors.

The processor801may perform at least one of the operations described above with reference toFIGS.1A to4. The processor801may learn a first model based on training data corresponding to a current task, learn a second model based on information on the current task and information on the previous learning task, and reset the first model.

The memory803may be a volatile memory or a non-volatile memory, and the memory803may store data needed to perform CRL. The memory803may include a first buffer and a second buffer.

The communication module805may provide a function for the training device800to communicate with another electronic device or another server through a network. In other words, the training device800may be connected to an external device through the communication module805and exchange data with the external device.

The training device800may further include components not shown in drawings. For example, the training device800may further include an I/O interface including an input device and an output device as the means of interfacing with the communication module805. In addition, for example, the training device800may further include other components such as a transceiver, various sensors, a database, and the like.

FIG.9illustrates an example of a configuration of an inference device900. according to one or more embodiments.

Referring toFIG.9, the inference device900may include a processor901, a memory903, and a communication module905.

The processor901may perform at least one of the operations described above with reference toFIGS.1A to4. The processor901may receive input data, input the input data to a continual learning model, and output a task corresponding to the input data among a plurality of tasks.

The communication module905may provide a function for the inference device900to communicate with another electronic device or another server through a network. In other words, the inference device900may be connected to an external device through the communication module905and exchange data with the external device.

The inference device900may further include components not shown in drawings. For example, the inference device900may further include an I/O interface including an input device and an output device as the means of interfacing with the communication module905. In addition, for example, the inference device900may further include other components such as a transceiver, various sensors, a database, and the like.

The R&D algorithm is described in part with mathematical notation. However, the mathematical notation is a convenient shorthand (or “language”) for describing the operations of physical computing devices. With the description herein of the R&D algorithm (including mathematical notation), one may readily use tools (e.g., software and/or circuit engineering tools) to implement the R&D algorithm, and it is those physical device implementations of the R&D algorithm to which this disclosure is directed, whether in the form of specially constructed integrated circuits, processor(s) in combination with memory storing instructions that implement the R&D algorithm, or combinations thereof. Moreover, such physical devices configured to implement the R&D algorithm can be used to better control the actions thereof (or of another device) in order to perform physical tasks, for example, such as moving a robot, controlling movement of a robotic arm, and so forth. Such robotic control is just one example of an application of the R&D algorithm.