Energy-Based Models for Continual Learning
Authors: Shuang Li
Yilun Du
Gido Martijn van de Ven
Antonio Torralba
Igor Mordatch
Venue:  (2021)
Keywords: Continual learning
Energy-based model

Abstract: We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems. Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support a dynamically-growing number of tasks and classes and less interference with old tasks. We show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks. We also find that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks. These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime.

Under review as a conference paper at ICLR 2021
ENERGY-BASED MODELS FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We motivate Energy-Base Based Models (EBMs) as a promising model class for con- tinual learning problems. Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support dynamically-growing number of tasks and classes and less interference with old tasks. We show that EBMs are adaptable to more general continual learning setting where the data distribution changes without the notion of explic- itly delineated tasks. We also find that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks. These observations point towards EBMs as class of models naturally inclined towards the continual learning regime.
INTRODUCTION
Humans are able to rapidly learn new skills and continuously integrate them with prior knowledge. The field of Continual Learning (CL) seeks to build artificial agents with the same capabilities (Parisi et al., 2019). In recent years, CI. has seen increased attention, particularly in the context of classification problems. A crucial characteristic
continual learning is the ability to learn new data without forgetting prior data. Models must also be able to incrementally learn new skills, without necessarily having notion of an explicit task identity. However, standard neural networks (He al., 2016; Simonyan Zisserman, 2014 Szegedy et al., 2015) experience the catastrophic forgetting problem and perform poorly in this setting. Different approaches have been proposed to mitigate catastrophic forgetting, but many rely on the usage of external memory (Lopez-Paz & Ranzato 2017: Li & Hoiem, 2017). additional models (Shin et al. (017), or auxiliary objectives and regularization (Kirkpatrick et al., 2017 Schwwrr al., 2018 Zenke et al. 2017 Maltoni & Lomonaco 2019), which can constrain the wide applicability of these methods.
In this work, we focus on classification tasks. These tasks are usually tackled by utilizing normalized probability distribution (i.e., softmax output layer) and trained with cross-entropy objective. In this paper, we argue that by viewing classification from the lens of training an un- on-normalized probability distribution, we significantly improve continual learning performance in classification problems. In particular, we interpret classification as learning an Energy- Based Model (EBM) across seperate classes (Grathwohl input data and its ground truth label is decreased while the energy of the input and another selected class is increased. This offers freedom to choose what classes to update in the CL process. By contrast, the cross entropy objective reduces the likelihood of all negative classes when given new input, creating updates that lead forgetting.
al. 2019). Training becomes wake-sleep process, where the energy of an
The energy function, which maps a data and class pair to scalar energy, also provides way for the model to select and filter portions of the input that are relevant towards the classification on hand. We show that this enables EBMs training updates for new data to interfere less with previous data. In particular, our formulation of the energy function allows us to compute the energy of a data by learning conditional gain based on the input label which serves an attention filter to select the most relevant information. In the event of a new class, a new conditional gain can be learned.
These unique benefits are applicable across range of continual learning tasks. Most existing works KKikkptriik et al., 2017 Zhao et al.) 2020) toward continual learning typically learn a sequence of distinct tasks with clear task boundaries (Boundary-Aware). Many of these methods depend on knowing the task boundaries that can provide proper moments to consolidate knowledge. However, this scenario is not very common in the real world, and more natural scenario is the Boundary-

---

Under review as a conference paper at ICLR 2021
Agnostic setting (Zeno al. 2018 Rajasegaran et al. 2020), in which data gradually changes with- out a clear notion of task boundaries. This setting has also been used as a standard evaluation in the continual reinforeement learning (Al-Shedivat et al. 2017 Nagabandi et al. 2018). Many common CL methods are not applicable to the Boundary-Agmont
scenario as the task boundaries are un-
known undefined. In contrast, EBMs are readily applied to this setting without any modification and are able to support both Boundary-Award and Boundary-Agnosee settings.
There are four primary contributions of our work. First, we introduce energy-based models for clas- sification CL problems in both boundary-awan and koundary-agnostie regimes. Secondly, we use the standard contrastive divergence training procedure and show that significantly reduces catas- trophic forgetting. Thirdly, we propose to learn new conditional gains during the training process which makes EBMs parameter updates cause less interference with old data. Lastly, we show that in practice EBMs bring significant improvement on four standard CL benchmarks, split MNIST, permuted MNIST, CIFAR-10, and CIFAR-100. These observations towards EBMs as class of models naturally inclined towards the CL regime.
2
RELATED WORK
2.1 CONTINUAL LEARNING SETTINGS
Boundary-awarr versus boundary-agnostic. In most existing continual learning studies, models are trained in a "boundary- aaware" setting. in which a sequence of distinct tasks with clear task boundaries is given (e.g., Kirkpatrick et al.) 2017 Zenke al. 2017: Shin et al.) 2017). Typically there are no overlapping classes between any two tasks; for for example task 1 has data with ground truth class labels "1,2" and task has data with ground truth class labels "3,4". In this setting, models first trained on the entire first task and then move to the second one. Moreover, models are typically told when there is a transition from one task the next. However, could be argued that it is more realistic for tasks to change gradually and for models to not be explicitly informed about the task boundaries. Such boundary-agnostic setting has been explored in (Zeno et al. 2018 Rajasegaran et al. 2020; Aljundi et al., (019). In this setting, models learn in a streaming fashion and the data distributions gradually change over time. For example, the percentage of "Is" presented to the model might gradually decrease while the percentage of "2s" increases. Importantly, most existing continual learning approaches are not applicable to the boundary-agnostic setting as they require the task boundaries to decide when to consolidate the knowledge (enoo et al.) 2018)) In this paper, will show that our proposed approach can also be applied to the soundary-agnossee setting.
Task-ineremental versus class-incremenie learning. Another important distinction in continual learning and class-incremental learning (van de Ven & Tolias 2019 Prabhu et al. 2020). In task-incremental learning. also referred to the multi-head setting Farquhar& & Gal 1018), models have to predict the label of an input data by choosing only from the labels in the task where the data come from. On the other hand, in class-incremental learning, also referred to as the single-head setting, models have to chose between the classes from all tasks so far when asked to predict the label of an input data. ctussinncremenal learning substantially more challenging than task-incremental learning, as it requires models to select the correct labels from the mixture of new and old classes. So far, only methods that store data or use replay have been shown to perform well in the class-incremental learning scenario (Rebuffi et al.) 2017 Rajasegaran et al.) 2019). In this paper, we try to tackle dass-inccremnntl learning without storing data and replay.
learning is between
2.2 CONTINUAL LEARNING APPROACIIES
In recent years, numerous methods have been proposed for CL. Here we broadly partition these methods into three categories: task-specific, regularization, and replay-based approaches.
Task-specific methods. One way to reduce interference between tasks is by using different parts of a neural network for different problems. For a fixed-size network, such specialization could be achieved by learning a separate mask for each task (Fernando et al. 2017 [Serra et al. 2018), by priori defining a different, random mask for every task to be learned (Masse et al. 2018) or by using a different set of parameters for each task (Zeng et al., 2019 Hu et al. 2019). Other methods let a neural network grow recruit new resources when it encounters new tasks, examples of which are progressive neural networks (Rusu et al. 2016) and dynamically expandable networks (Yoon et al. 2017). Although these task-specific approaches are generally successful in reducing catastrophic forgetting, an important disadvantage is that they require knowledge of the task identity at both training and test time. These methods are therefore not suitable for class-incremental learning.
2

---

Under review as a conference paper at ICLR 2021
Regularization-basse methods. Regularization is used in continual learning to encourage stability of those aspects of the network that are important for previously learned tasks. A popular strat- egy is to add regularization loss to penalise changes to model parameters that are important for previous tasks. EWC (Kirkpatrick et al. 2017 and online EWC (Schwarz et al. 2018) evaluate the importance of each parameter using the diagonal elements in the fisher information matrices, while SI (Zenke et al.) (017) tracks the past and current parameters and estimates their importance online. An alternative strategy is regularize the network the functional level. Learning without Forgetting (Li & Hoiem 2017) uses knowledge distillation to encourage stability of the network's s learned input-output mapping. However, these regularization- based approaches gradually reduce the model's capacity for learning new tasks and they have been shown to consistently fail in the class-incremental learning (Farquhar & Gal 2018 van de Ven Tolias 2019
Replay methods. To preserve knowledge, replay methods periodically rehearse previously acquired information during training (Robins, 1995). One way to do this, referred to as exact or experience replay, is to store data from previous tasks and revisit them when training on new task. Although this might seem straightforward, critical non-trivial questions are how to select the data to be stored as well as exactly how to use them (Rebuffi et al. 2017 Lopez-Paz & Ranzato 2017] Rajasegaran jet al. 2019; Hou et al. 2019; Wu et al. 2019:] Mundt et al. 2020). An alternative to storing data is generate the data be replayed. In such generative replay (Shin et al. (017), a generative model is sequentially trained to generate input samples representative of those from previously seen tasks. While both types of replay can prevent catastrophic forgetting in both task- and class-incremental learning settings, an important disadvantage is that these methods are computationally relatively expensive as each replay event requires at least one forward and one backward pass through the model. In addition, storing data might not always be possible while incrementally training gener- ative model is a challenging problem in itself (Lesort et al. 2019 van de Ven et al. (0220).
In contrast, our EBMs reduce catastrophic forgetting without requiring knowledge of task- iies-iidntity, without gradually restricting the model's learning capabilities and without using stored data.
BACKGROUND
In this section, will introduce traditional continual learning methods, which are typically modi- fied based on the feed-forward classifier. We will show the limitation of using such structures and how EBMs can be applied these problems in the next section.
3.1 CONTINUAL LEARNING WITH FERD-FORWARD CLASSIFIER
Define T as classification task with inputs x and associated discrete classification labels y E Y, where Y = {I,...,.V }} contains N classes. Traditional classifier approaches to continual learning typically predict the probabilities of all the N classes using a feed-forward network
exp(if(x)lx)
po(y|x) [xx)( Y:EY
(1)
The input is a data x R.D and the output is probabilities of all classes po R.V. f(x) RD. + R is the feed-forward function that maps data into a N N-dimensional vector. The final layer in such classifier networks is predefined to predict up to N different classes.
Existing continual learning methods (Kirkpatrick et al. 2017 Masse et al. 2018) tackle the continual learning problem typically using a feed-forward classifier. However, such classifier architectures do not perform well in the Class-IL setting, where models have to predict the label of given data from all classes (Tao et al., 2020a; He et al., 2018). The standard feed-forward classifier computes the Softmax and cross-entropy loss over all seen classes. When training on new task, they improve the likelihood of ground truth classes but suppress the likelihood of old classes since they are all negative classes for data in the current task. This operation introduces competitive, winner-take-all dynamics that impact the strength of old classes and thus make the classifier forget past tasks.
3.2 ENERGY-BASED MODELS
Energy based models (LeCun et al. 2006) are class of maximum likelihood models that define the likelihood of data point x E R using the Boltzmann distribution
(x(x)= = exp( (ap(-Eo(x) Z(x) f. exp( Ex(
(2)
Z(x)
3

---

Under review as a conference paper at ICLR 2021
where Ex(x) : R R. known the energy function, maps each input data to scalar, and ((x)
is the partition function over all data points.
EBMs are powerful generative models that have been applied to different tasks, such structured
prediction [Belanger & McCallum 2016 Gygl et al. 2017] Rooshenas et al. 2019; Tu & Gim-
pell 2 machine translation (Tu et al. (0200) and image generation (Xie et al. 2016 2018
Grathwohl et al. 2019). (Belanger & McCallum 2016) introduce energy networks for structured prediction. They propose energy networks can be naturally posed as structured prediction, since the labels exhibit rich interaction structure. [Tu & Gimpel (2019) propose a structured energy func- tion for sequence labeling tasks. [Rooshenas al. (2019) introduce a search- guided approach to obtain training pairs using combination of the energy function and reward function that allows the lighll-supervisse training of structured prediction energy networks. TTu al. (2020) treat the non- autoregressive translation system as an inference network that finds the translation by minimizing the energy of a pretrained autoregressive model. Xie et al. (2016) is the first paper to use ConvNet- parameterized EBMs with Langevin dynamics for image generation. They show that generative random field model can be derived from the commonly used discriminative ConvNet. (Xie et al. 2018) cooperatively train the descriptor and generator using MCMC sampling.
4 CONTINUAL LEARNING WITH ENERGY- ENERGY-BASED MODELS
In this section, we describe how EBMs can be used for classification and we discuss how they can overcome the above mentioned problems in the continual learning setting. Note our analysis is based on the Class-II. setting which is one of the most natural and also the hardest settings used in CL (van de Ven & Tolias, 2019:
He et al. 2018 Tao al. 20205).
4.1 ENERGY-BASED MODELS FOR CLASSIFICATION
Different from existing works using EBM for structured prediction (Belanger & McCallum 2016 Gygli et al. 2017 Rooshenas et al. 2019), such multi-label classification tasks, CL does not access previous data while training on new data and thus the biggest challenge is to prevent catas- trophic forgetting. The main focus of this paper is to mitigate the catastrophic forgetting in CL without using replay buffers to store past data or models.
To make the EBM suitable for classification, we the Boltzmann distribution to define the likeli- hood of y conditioned on x:
Po (y|x) = exp( exp(-E((x,y)) Z(ylx) = Σ Exp((-E(xx,y.)) x
(3)
Y.
where Ex(x, y): (RD,N) N) + R is the energy function that maps x and y to a scalar energy value, and Z(y x) the partition function that ensures Ey Po (y|x) 1 for any given x.
To minimize the negative log likelihood of the ground truth label y inspired by the contrastive divergence proposed by (Hinton 2002), we minimize the energy of x at the ground truth label y+ while increasing the energy of x at other labels. We use the following loss and its corresponding gradient (see Appendix B for the derivation):
LMu Ex (x, + log Σ YEE
exp(-Eo(xx
(4)
0LML 00
ExxT (iEo(x,y*) By:~(Nx++) (oEofxe 00
(x,
(5)
00
whereby N is the set of 'negative classes'. Following common practice in contrastive divergence, we sample the set of negative classes N from the set of class labels in the current batch Yp, using only a single negative class per training sample. We note however that it is possible to use different strategies for choosing the negative classes, and in the experiments reported in the top half of Table we explore alternative strategies. If there are batches that contain only a single class label, sampling negative class from the current batch does not work, but one could instead select whichever other class was seen most recently as a negative class, or one could sample from all other classes seen so far. Another solution might be to always use fake label representing "not exist" as negative sample.
As opposed to the classifier architectures that output a fixed N N-dimensional probability vector of all classes, this formulation thus provides freedom in the choice of which class labels to train
4

---

Under review as a conference paper at ICLR 2021
This is important because important issue with standard classifier models is that they suppress the strength of old classes and thus make the classifier forget past tasks as mentioned in Section 3.1 In contrast, EBMs define the probability without normalizing over all classes but instead sample limited number of negative classes. This operation cause less interference with old classes without using any extra information which contributes to why EBMs suffer less from catastrophic forgetting.
Another advantage of our EBMs is that the choice of model archi- tectures also becomes more flexible. The traditional classification models can only feed in x as input and the classification results only depend on the neural network trained on the current data x. Thus the models usually fail on the old data after updating network pa- rameters trained on the new data. different ways to combine x and y in the energy function Egg Fx(x.y) y) with the only requirement that Eg(x. y): N) - R. EBMs, we can treat y an attention filter or gate to select the most relevant information between x and y as described in Section 4.2
Weight layer/ random projection
Weight layer
(x)
La())
Normalization
contrast, EBMs have many
m(x,y)
ReLU
Weight layer
Energy: Eg(x,))
Since the negative sample can be sampled from the current batch, EBMs do not require knowledge of the task on hand, allowing ap- plications across different continual learning scenarios, including those when the underlying task is unclear. More broadly speaking, EBMs can go out of the scope of classification problems, and sim- ilar ideas can be applied to regression (Gustafsson et al. (2020), generation (Xie et al. 2016 Du al. 2020), and reinforcement learning tasks (Parshakova al. 2019), although we do not explore them in this work.
Model architecture used EBM. EBM takes a data x and a class y as input and out- puts their energy value.
Figure 1:
4.2 ENERGY NETWORK.
To compute the energy of any data x and class label y pair, we use y to influence a conditional gain on x. which serves as an attention filter (Xu al. 2015)) to select the most relevant information between x and y. In Figure [ we first send x into a small network to generate the feature fx). The label y is mapped into same dimension feature space g(y) using FC layer or random projection. We use the Softmax function over each channel of g(y) and combine it with x:
exp(g.(y))
mx = fc(x) Σj (x))
where fe and ge are the th channel of fxx and g(y) respectively. The output is finally sent to a fully connected layer to generate the energy value Ex(x, y). For more details on model architectures, see Appendix
Our EBMs allow any number of classes in new batches by simply training or defining new con- ditional gain g(y) for the new classes and generating its energy value with data point x. This formulation gives us freedom learn new classes without pre-defining their number in advance. We note that also in standard classifier new classes can be dynamically added, but there new class heads need to be added the softmax output layer.
Class inference. During inference, the model must predict class label from all classes seen so far. let X2 be one data point from a batch B₃ with an associated discrete label y E Yg, where Yn contains classes in Bg. There are Y = k= Ys different classes in total after seeing all the batches. The MAP estimate
y = arg min Esg. (x), y yEL +UUY..
y
where Es,, (Xt, y) is the energy function with parameters 6x resulting from training on the batches (B..... ]. J. The energy function can compute an energy for any discrete class input, including unseen classes. This avoids needing to predefine the number of classes in advance, as is necessary with traditional CL models.
EXPERIMENTS
In this section, we want to investigate several questions, including how does the proposed EBMs per- form on different CL settings? Can we qualitatively understand the differences between EBMs and baselines? Is the proposed EBM training objective better than other objectives? Can we apply the EBM training objective to other baseline methods? And finally what is the the best architecture for
5

---

Under review as a conference paper at ICLR 2021
Table 1: Evaluation of the Class-IL performance the Boundary- Aware setting compared to different baselines. The test accuracy on four datasets are reported. Each experiment is performed 20 times with different random seeds, the results are reported as the mean + SEM over these runs. Note both EBMs and baselines do not use extra memory to store data or models for fair comparison.
After task
After 10
CLS
Method
splitMNIST
permMNIST
Cifar10
Cifar100
EBM
CLS
19.91 = 0.01
16.95 = 0.31
19.06 + 0.01
8.18 0.10
EWC
20.03 = 0.09
25.14 = 0.71
18.99 1 0.01
8.20 0.09
Online
19.89 = 0.02
33,20 = 0.74
19.07 + 0.01
8.38 0.15
Dan
19.98 0.03
29.35 0.90
19.14
9
Figure 2: Energy landmaps of CLS and EBMs after training on task T' and Tio on permuted MNIST. The darker the diagonal is, the hetter the model is in preventing forgetting previous tasks.
LWF
20.53 = 0.11
43.88 = 0.88
19.20 + 0.30
10.71 = 0.11
MAS
19.50 0,30
20.25 + 1.54
8.44 0.27
BGD
19.64 = 0.03
84.78 = 1.30
EBM
53.12 0.04
87.78
0.01
40.19 + 0.01 30.28
0.28
label conditioning that causes the minimum interference with old data? To answer these questions, we first introduce experiments the Boundary-Aware setting which typically learn sequence of distinct tasks with clear task boundaries Section [.T] We then show that EBMs can naturally applied to the Boundary-Agnosse setting where the data gradually changing in steaming fashion without the notion of task boundaries Section 5.2
5.1 EXPERIMENTS ON BOUNDARY-WWARE SETTING
5.1.1 DATASETS AND EVALUTION PROTOCOLS
Datasets. We evaluate the proposed EBMs the split MNIST (Zenke et al. 2017), permuted MNIST (Kirkpatrick et al. alla[2211111) 2017). CIFAR-I0 (Krizhessky et al. 2009). and CIFAR- 100 (rizhevsky al. 2009) datasets. We follow existing approaches and separate the datasets into several splits. The split MNIST dataset is obtained by splitting the original MNIST dataset (LeCun et al., 1998) into 5 tasks with each task has 2 classes. has 60,000 training images and 10,000 test images. The permuted MNIST has 10 tasks, each task with 10 classes. For each task, the original images pixels are randomly permuted to generate 32 x 32 images. We separate CIFAR-10 into tasks, each task with 2 classes. There are 50,000 training images and 10,000 test images. Similarly, CIFAR- 100 is split into ten tasks with each task has 10 classes. Evaluation protocols.
Learning (Task-IL), Domain iinaiinincrrmental Learning (Domain-IL), and Class-incremental Learning (Class-IL) are three common evaluation metrics used by existing works (van de Ven & Tolias 2019 Prabhu et al. 2020). Most approaches perform fairly good on the first two simpler settings, but fail on the Class- IL setting, which is considered as the most natural and also the hardest setting for continual learning (Tao et al.] 2020a) He et al.) 2018] Tao et al. (0200b). Class-II. predicts the label of given data from all seen classes. In this paper, we consider Class-Il. our evaluation metric.
5.1.2 COMPARISONS WITH EXISTING METHODS
Due to the difficulty on the class incremental setting, most approaches rely an external quota of memory. Such replay based methods (Shin al., 2017 Rebutli et al. 2017) use extra memory, larger or extra models, and are computationally relatively expensive. In this paper, we focus on continual learning without using replay and without storing data.
We compare the proposed method with available bascline models that do not use replay or stored data, including a standard classification model (CLSS, EWC (Kirkpatrick et al. 2017), Online EWC (Schwarz et al.) 2018). SI (Zenke et al. 2017), LwF (Li & Hoiem] 2017). MAS (Aljundi et al.) 2019), and BGD (Zeno et al.) 2018). The continual learning results on four datasets are shown in Table All the baselines and EBMs are based on the same model architecture. For split MNIST and permuted MNIST, we use several fully-connected layers as in (van de Ven & Tolias 2019). For CIFAR-10 and CIFAR-100.
the convolutional network (see Appendix C). For all the baselines and EBMs on CIFAR-100, 100, we pre-train the convolutional layers on CIFAR- 10 and only fine-tune the fully-connected layers.

---

Under review as a conference paper at ICLR 2021
Similar training regimes are used for the EBMs and baselines. On the split MNIST, permuted MNIST, and CIFAR-10 10 datasets, we train for 2000 iterations per task. On the CIFAR- 100 dataset train for 5000 iterations per task. For all experiments,
use the adam optimizer with learning rate 1e_1. Each experiment in Table 1 is run 20 times with different random seeds, with results reported the mean + SEM. EBMs have a significant improvement all the datasets, showing that EBMs forget less when updating models for new tasks.
5.1.3 QUALITATIVE ANALYSIS
Energy landscape. To better understand why EBMs suffer less from catastrophically forgetting, we qualitatively compare the change in energy landscapes of EBMs and baseline methods as the learning progresses. We show the energy landscapes after training on task 9 and task 10 on the permuted MNIST dataset in Figure 2 For the classifier, we show its negative probabilities on all classes. Each datapoint has 100 energy values (EBM) or probabilities (CLS) corresponding to the 100 labels in the dataset and the values are normalized over 100 classes for each datapoint. The dark diagonal elements indicate the model predictions are correct for both old and new data. After training on task Tg. CLS assigns high probabilities to classes from task T9 (80-90) for almost all the data from task Ti to Tg. However, the highest probabilities shift to classes from task T1o (90-100) after learning task Tio. This means classifier tends to assign high probabilities to new classes for both old and new data, indicating forgetting. EBM on the other hand, tends to have low energies across the diagonal, which means that after training on new tasks, EBM still assigns low energies to true labels of previous data. This indicates that EBM is better at learning new tasks without catastrophically forgetting of old tasks.
Predicted class distribution. In Figure 3. we plot the proportional distribution of all seen classes so far on the splitMNIST dataset. Taking the second figure in the first row as an example, it shows the distribution of predicted labels after training on the second task which means there are four seen classes {1, 2, 3, 4). For each image from the first and second task, we compute its top-1 class predic- tion and accumulate the prediction on each class to obtain the proportional distribution graph over all seen classes. Since the number of images in each class almost the same, the ground truth proportional distribution should be uniform over all the classes. CLS can predict the first two classes uniformly after training on task (first column). However, after learning new tasks, classes from the new task have larger proportions than old classes which means the model tends to predict new classes and forget the old classes. The last figure in row one gives very high proportions for classes from the last task but very low proportions for previously classes. CLS almost forgets all previous tasks and only predict classes from the last task even the input images are from old tasks. EBM covers all seen classes, indicating memorizing the old task and preventing forgetting.
CLS
Figure 3: Predicted label distribution af- ter learning each task on the split MNIST dataset. CLS only predicts classes from the current task while EBMs can predict classes for all seen classes.
We provide more analysis about EBMs and baselines from different perspectives in the Appendix, including the confusion matrix in Appendix A.2 the model capacity in Appendix A.3 and parameter importance in Appendix A.4
5.1.4 IS THE STRONG PERFORMANCE OF EBMS DUE TO THE ENERGY TRAINING OBJECTIVE OR DUE TO THE LABEL CONDITIONING?
Effect of energy training objective. We conduct an experiment on the 100 dataset to inves- tigate how different training objectives influence the CL results. In Equation ares sampled from YB. We test three different sampling strategies in Table
the negative labels y:
The first one uses all
seen classes so far as negative labels which is similar to the traditional feed classifiers (All Neg Seen (V4)). The second one takes all the classes in the current batch as negative labels (All Neg Batch (V4)). The last one randomly selects one class from the current batch as the negative as we reported in Section[411 truth class in all these three strategies. We found using only one negative sample generates the best result and using negatives sampled from classes in the current batch is better than from all seen classes. This conclusion is consistent with our analysis in Section 4.1 Since the loss in Equation 4 aims at improving the energy of negative samples while decreasing the energy of positive ones, sampling negatives from the current batch has less interference with previous classes than sampling
Neg Batch (V4)). Note that the negative labels do not include the ground
7

---

Under review as a conference paper at ICLR 2021
from all seen classes. Using a single negative causes the minimum suppression on negative samples and thus has the best result. Overall, our results indicate that surprisingly, and counterintuitively, directly optimizing the cross-entropy loss used by existing approaches may not be the best way for continual learning.
Effect of label conditioning. Next, test whether the label conditioning in our EBMs is important for their performance. We test this by modifying the training objective of a standard classifier to that of our proposed energy objective, which is the same as running our EBMs without the label conditioning. The results of this comparison are listed in Table 3 The new training objective does not suppress the probability of old classes when improving the probability of new classes and thus achieves better results. However, EBMs still outperform the baselines, implying that the label conditioning architecture also contributes to why EBMs suffer less from catastrophic forgetting. We also show the testing accuracy curve in Appendix A.
To summarize, we showed that the strong performance of our EBMs due to both the energy training objective and the label conditioning.
5.1.5 COMPARISON OF DIFFERENT ENERGY NETWORK ARCHITECTURES
EBMs allow flexibility in integrating data information and label information in the energy function. To investigate where and how to combine the information from and y. we conduct a series of experiments on the CIFAR-10 dataset. Table 2 shows four model architectures (V1-V4) that combine x and in the early, middle, and late stages, respectively (see Appendix [ for the architecture details). We find combining x and y in the late stage (V4) performs the best. We note that instead of learning feature embed- ding of label y. we can use fixed projection matrix which randomly sampled from the uniform distribution (4(0, 1). Even using this fixed random projection can already gen- crates better results than most basclines in Table [ Note further that the number of trainable parameters in the "Fix" setting is much lower than that of the baselines. Using a learned feature embedding of y can further improve the performance. We may also apply different normalization methods over the feature channel y. We find that Softmax (End Softmax (V4)) is better than the L2 normalization (End Norm2 (V4)) and no normalization (End (V4)).
Table 2: Different EBM training objec-
tives and label conditioning architectures.
Dataset
Cifar100
All Neg Seen (V4)
8.07 + 0.10
All Neg Batch (V4)
29.03 + 0.53
1 Neg Batch (V4)
30.28 + 0.28
Dataset
Cifar10
Beginning (V1)
13.69 + 0.02
Middle(V2) V2)
20.16 + 0.02
Middle(V3)
18.36 1 0,02
End Fix (V4)
34.30 + 0.03
End Fix Norm2 (V4)
33,91 + 0.02
End Fix Softmax (V4)
35.97 + 0.01
End (V4)
38.13 + 0.01
End Norm2 (V4)
37.23 + 0.02
End Softmax (V4)
40.19 + 0.01
5.1.6 INTERFERENCE WITH PAST DATA
Here we test the effects of label conditioning and the energy training objective in a different way, by evaluating their ability to prevent interference with past data when learning classification on new classes. Formally, let x, and x; be two data points and 0; be the model parameters after training on
x, The model parameters change () after learning new data x,. We test the difference between the losses LML. (x; 0,) and LML (x; 0, + A0(x))) Ideally, we expect LMi(x; 0, + (0(x)) < EML(X, 8,) for x; * Xj. which means the update on new data has no influence or positive
influence on the old data. The first-order expansion gives LML(x; 0 + A0(x,)) = LML X;;
0,) ) + Va, CML( x; 0,, 1x(x)) To make our desired equality hold, the gradient at x; should be Vo, CMu. |0;) A)(x)) < 0. s = Vo, Lm
( x; 0, )z (xxxx) can be used to measure the influence of updating models on new data on the performance of old data. The smaller the value is, the less negative influence old data, and thus the better the model is in preventing forgetting.
We compare EBMs with the standard classifier (CLS) and CLS using our EBM training objective (CLS*) (see Appendix A.5) for the implementation details). For results split MNIST, the average value of á over all tasks of CLS, CLS* and EBMs are ØeLs = 0.168, ØCLS* = 0.004, and ΦLEM = -3. -3.743e-5 respectively. For CIFAR-10, the results are ØcLs 1.104, OCLS= 0.043, and δEBM 4.414e -5 respectively. We found EBMs have smaller & than CLS and CLS* which is consistent attention filter that could select the
with analysis that the conditional gain gy)) serves
8

---

Under review as a conference paper at ICLR 2021
Table 3: Results of baselines using our
Table 4: Evaluation of the Class-IL performance on the Boundary-Agnostic setting compared to different baselines. The test accuracy four datasets
training objective and their original one.
split MNIST
Ori
Ours
reported. Each experiment performed times with different random seeds, the results are re- ported as the mean + SEM over these runs. Note both EBMs and baselines do not use extra memory to store data or models for fäir
CLS
19.90 0.02
44.98 0.05
BWC
20.01 + 0.06
51.68 + 0.04
Online
19.96 0.07
50.99 = 0.03
SI
19.99 0.06
49,44 0.03
comparison.
EBM
53.12 0.04
Method
splitMNIST
permMNIST
Cifar10
Cifar100
CIEAR-10
Ori
Ours
CLS
24.03 + 0.01
21.42 0.01
23.30 + 0.01
9.85 + 0.02
CLS
19.06 0.01
19.22 = 0.02
Online
39.62 4: 0.04
41.37 0.04
22.53 4: 0.01
9.57 = 0.02
BWC
18.99
36.51 0.03
SI
28.79 0.01
35.71 0.01
26,26 0.01
1042 0.01
Online
19.07 0.01
36.16 = 0.02
BGD
$1.65
26.15 0.02
17.03
1: 0.01
8.50 = 0.02
SI
0.00
35.12 = 0.02
BBM
40.19 0.01
EBM
81.78 +: 0.06
92.35 0.01
49.47 + 0.03
34.39 + 0.24
relevant information between x and y and make parameter updates cause less interference with old data, which contributes why EBMs suffer less from catastrophic forgetting.
5.2 EXPERIMENTS ON BOUNDARY-AGNOSSS SETTING
When applying continual learning in real life, boundaries are not usually well defined between different tasks. Most existing continual learning methods are often tailored the problem setup where a fixed boundary between tasks is defined, and must be informed when the tasks are switched during training. We show that EBMs are able to flexibly perform continual learning across different problem setups, and perform well on the Boundary-Agnose setting as well.
5.2.1 DATASETS AND EVALUTION PROTOCOLS
We evaluate our proposed continual learning method the Boundary-Agnostic setting on four datasets, including split MNIST, permuted MNIST, CIFAR-10 10, and CIFAR-100. We use the code of "continuous task-agnostic learning" proposed by (Zeno et al. 2018) to generate the training/testing data. The transition between tasks goes slowly over time using sampling function based on the probability of each task. There mixture of data from two different tasks during the transition and the proportion of new data gradually increases, Similar to the Boundary-Aware setting, We use Class-IL as our evaluation metric for the Boundary-Agnosit continual learning.
5.2.2 COMPARISON WITH EXISTING METHODS
We focus on the investigation on the performance of different model architectures with similar model size and memory footprint. We do not directly compare with replay-based methods as they use extra memory, larger or extra models. Since there is no knowledge on the number of tasks, previous methods of continual learning that rely on task boundaries are generally inapplicable. One trivial adaptation is to take the core action after every batch step instead of every task. However, doing such adaptation is impractical for most algorithms, such as EWC, because of the large computational complexity. We successfully get the result of CLS, Online EWC, SI, and BGD baselines. Overall classification results on four datasets are shown Table
All the baselines and EBM are based on the same model architecture in the Boundary-Aware setting. Each experiment was performed times with different random seeds, the results are re- ported the mean + SEM over these runs. We observe that EBMs have significant improvement on all the datasets. EBMs can be naturally applied to both the Boundary-Aware and Boundary- Agnostic settings without any modification, experiments demonstrate EBMs have good generalization ability for different continual learning problems. EBMs can naturally handle different number of new classes, new tasks, data streams with and without clear task boundaries.
EBMs have a very flexible training objective. The
CONCLUSION
In this paper, we found that energy-based models are a promising class of models towards va- riety of different continual learning settings. We demonstrated that EBMs exhibit many desirable characteristics to prevent catastrophic forgetting in CL, and experimentally showed that EBMs obtain the state-of-the-arr performance on the Class- and Boundary-Agnostt settings on multiple benchmarks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, llya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments, arXiv preprint arXiv:1710.0364 2017.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019.
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning. pp. 983-992, 2016.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models. arXiv preprint arXiv:2004.060300 2020.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv:1805.09733 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
Will Grathwohl, Kuan-Chieh Wang, förn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: 02.03663, 2019.
Fredrik K Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B Schön. Energy- based models for deep probabilistic regression. In Proceedings the European Conference on Computer Vision (ECCV), 2020.
Michael Gygli, Mohammad Norouzi, and Anelia Angelova. Deep value networks learn to evaluate and iteratively refine structured outputs. arXiv preprint a7X3v:/703.04530 2017.
Chen He, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exemplar-supportel generative repro- duction for class incremental learning. BMVC, pp. 98, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp- 770-778, 2016.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2019..
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In International Conference Learning Representations, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(1))3521-3522, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers 2009.
of features from tiny images.
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits, 1998. URLhtpp:///e http://yann lecun. com/exdb/nmist, 10:34, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. 2006.
10

---

Under review as a conference paper at ICLR 2021
Timothée Lesort, Hugo Caselles- Michael Sarcia-Ortizz Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. In 2019 International Joint Con- ference Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Iransactions on pattern analysis and machine intelligence, 40(12):2935-2247) 2017.
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single- inallleinncementllttasse scenarios.
Neural Networks, 116:56-73, 2019.
Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, (1(44))E10467-1007) 2018.
Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. arXiv preprint arXiv:2009.01797, 2020.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint arXiv: 3147, 2018.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional reinforcement learn- ing for energy-based sequential models. arXiv preprint arXiv:19/2.085/1, 2019.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. 2020.
Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. In Advances in Neural Information Processing Systems, pp. 22669-12679, 2019.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 13588-13597 2020.
Sylvestre-Alvise Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):
123-146, 1995.
Anirmohammad Rooshenas, Dongxu Zhang, Gopal Sharma, and Andrew McCallum. Search- guided, supervy- training of structured prediction energy networks. In Advances in Neural Information Processing Systems, pp. 13522-13532, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 162116666667777 2016.
Jonathan Schwarz, Jelena Luketina, Wojciech M Czamnekki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell, Progress & compress: A scalable frame- work for continual learning. arXiv preprint 1rxiv:1805.06370 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. arXiv preprint arXiv:1801.011 2018.
11

---

Under review as a conference paper at ICLR 2021
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. pereboygg ilasm-nncremental learning. 2020a.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot caass-incremenall learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12183-12192, 2020b.
Lifu Tu and Kevin Gimpel. Benchmarking approximate inference methods for neural structured prediction. arXiv preprini
138, 2019.
Lifu Tu, Richard Yuanzhe Pang. Sam Wiseman, and Kevin Gimpel. Engine: Energy-based based inference networks for aon-autoregressivv machine translation. arXiv preprint arXiv:2005.00850 2020.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.077 07734, 2019.
Gido M van de Ven, Hava Siegelmann, and Andreas S Tollias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11: 11:4069, 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644, 2016.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. (0448-2057. 2015.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 01547, 2017.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent pro- cessing in neural networks. Nature Machine Intelligence, 1(8):364-37,, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Chen Zeno Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint 1003::1803.1112.. 2018.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 13208-13217, 2020
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL ANALYSES
Extending the results presented in Section 5.1 here we further compare EBMs with the baseline models by providing additional quantitative analyses of their performance. We show testing accu- racy curves in Section A.1] confusion matrices between the ground truth labels and model predic- tions in Section A.2 model capacity comparisons in Section A.3 and parameter importance mea- surement in Section A.4 In Section A.5 we provide more details of the label conditioning analysis
mentioned in Section [51.5] Finally, in Section A.6 we perform the split CIFAR-100 protocol with several different number of tasks to test the generality of the proposed EBMs.
A.1 TESTING ACCURACY CURVE
In Section 5.1.4 we show that the proposed EBM training objective is also applicable baseline approaches and improves their performance significantly on the Class-IL setting. Figure 4 shows the testing accuracy of each task as the training progresses. We compare the standard classifier (CLS), classifier using our training objective (CLS*), and our EBMs. The left figures show results on the split MNIST dataset while the right figures show results on the permuted MNIST dataset. We observe that the accuracy of old tasks in CLS drop sharply when learning new tasks, while the EBM training objective can mitigate the forgetting problem. The curve on EBMs drops even slower than CLS and CLS*, implying the proposed energy objective can mitigate the catastrophic forgetting problem.
CLS
CLS
EBM
CL.S
CLS*
EBM
Figure 4: Class-IL lesting accuracy the standard classifier (CLS), classifier using our training objective (CLS*), and EBMs on each task on the split MNIST dataset (left) and permuted MNIST dataset (right).
A.2 CLASS CONFUSION MATRIX AT THE END OF LEARNING
In Section 5.1.3 we show the qualitative analysis of EBMs and baselines. Except the energy land- scape and predicted class distribution, we also generate the confusion matrices of EBMs and baseline approaches. Confusion matrix shows the relationship between ground truth labels and predicted la- bels. In Figure 5 draw the confusion matrices after training all the tasks on the split MNIST dataset and permuted MNIST dataset. The classifier tends to predict the classes from the last task (class 8, 9 for split MNIST and 90-100 for permuted MNIST). EBMs have high values along the diagonal that means the predicted results matching the ground truth labels for all the sequentially learned tasks. This demonstrates that EBMs are better at learning new tasks without catastrophically forgetting of old tasks.
TEA looe 9.2
Predicted label
Predicted labe
Plnditee biel
FrodTin bam
CLS (split MNIST)
EBM (split MNIST) CLS (permuted MNIST) EBM (permuted MNIST)
Figure 5: Confusion matrices between ground truth labels and predicted labels at the end of learning on split MNIST (left) and permuted MNIST (right). The lighter the diagonal is, the more accurate the predictions are.
13

---

Under review as a conference paper at ICLR 2021
A.3 MODEL CAPACITY
Another hypothesized reason for why EBMs suffer less from catastrophic forgetting than standard classifiers is potentially their larger effective capacity. To analyze effective capacity of our models, we test the model capacity of the standard classifier and EBMs on both the generated images and natural images.
Model capacity on generated images. We generate a large randomized dataset of 32 x 32 images with each pixel value uniformly sampled from to Each image is then assigned a random class label between O and 10. We measure the model capacity by evaluating to what extent the model can fit a such dataset. For both the standard classifier and the EBM, we evaluate three different sizes of models (small, medium, and large). For fair comparison, we control the EBM and classifier have similar number of parameters. The Small EBM and CLS have 2, 348, 545 and 2, 349, 032 parameters respectively. The medium models have 5, 221, 377 (EBM) and 5, 2212 352 (CLS) parameters while the large models have 33, 468, 417 (EBM) and 33, 465, 320 (CLS) parameters. We use the model architectures in Table 5a and Table 5b for EBMs and classifiers.
The resulting training accuracies are shown in Figure 6 with the number of data ranges from one to five millions. Given any number of datapoints, EBM obtains higher accuracy than the classifier, demonstrating that indeed EBM has larger capacity to memorize data given a similar number of parameters. The gap between EBM and CLS increases when the models become larger. The larger capacity of EBM potentially enables it to memorize more data and mitigate the forgetting problem.
Model capacity on natural images. We also compare classifiers and EBMs natural images from CIFAR-10. 10. Each image is assigned random class label between 0 and 10. We use the same network architecture in Table Saland Table 5b] but with hidden unit size of h 256. Since there are only 50, 000 images on CIFAR-10 10, we use small classifier and EBM and train them on the full dataset. After training 100000 iterations, the EBM obtains top-1 prediction accuracy of 82.81, while the classifier is 42.19 19. We obtain the same conclusion that EBM has larger capacity to memorize data given a similar number of parameters.
Table 5: The model architectures used for the model capacity analysis. h are 512, 1024, and 4096 for the small, medium and large network, respectively.
(a) The architecture of the EBM.
(b) The architecture of the standard classifier..
x = FC(784, h) (x)
FC(784, h) (X)
X = ReLU (x)
x ReLU (x)
y = Embedding (y)
x FC(h, h) (x)
X y
x ReLU (x)
ReLU (x)
out = FC(h, 10) (x)
out FChh, (x)
A.4 PARAMETER IMPORTANCE
To further understand why EBMs suffer less from catastrophic forgetting, we design an experiment to test the importance of model parameters on past data. Inspired by the elastic weight consolidation (EWC) (Kirkpatrick et al.) 2017). we estimate the importance of parameters for each tasks using the diagonal elements of the Fisher information matrix (FIM) F. Let 0, be the model parameters after training on task T. Given one of previous tasks Ty, < i, we evaluate how important each parameter is for tasks Tj. The 4tth diagonal of I' is defined as the gradient on the EBM loss
2
FiL = Ex-T, VoLa (x, + log Σ exxp(-Eo,
(8)
where x is sampled from tasks T3 and E(x, yt) is the energy value of the imput data x and ground truth label y y' Yj are classes randomly selected from the current batch. Here single
14

---

Under review as a conference paper at ICLR 2021
1.0
0.9
0.8
Accuracy
0.7
0.6
Small CLS
Medium CLS
0.5
Large CLS
0.4
Small EBM
Medium EBM
0.3
Large EBM
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Training datapoints
166
Figure 6: Model capacity of classifier and EBM using different model sizes.
negative class. The above equation assigns high values to parameters crucial to task T3 as their gradients with respect to the loss are larger. Since the diagonal elements of the fisher information matrix measure the importance of each parameter to a given task, the density of diagonal elements represents the proportion of important parameters over all parameters. More density means more parameters are important for the given task and less parameters can be recruited for new tasks. Ideally, we expect these values to be sparse.
In Figure 7 show the diagonal elements of the standard classifier (CLS), classifier using training objective (CLS*), and our EBMs on the split MNIST dataset. For CLS and CLS* we follow (Kirkpatrick
al., 2017) to compute their fisher information matrices. For comparisons across multiple models, we normalize the FIM diagonal elements of each method to be between O and 1 and report the normalized results Figure 7 For example, "Fisher on data 1" shows the diagonal elements of the Fisher information matrix obtained by Equation [ using the model parameters 0g (after training on task TT3) and data x, y yy from task Ti. The distribution of EBMs is sparser than CLS and CLS" indicating that EBMs have fewer important parameters for previous data. Updating parameters for the new task will have less negative impact on old tasks. In addition, more parameters can be used for learning new tasks if the distribution is sparse. This may provide another explanation for why EBMs can mitigate catastrophic forgetting.
A.5 MORE DETAILS THE INTERFERENCE WITH PAST DATA ANALYSIS
In Section 5.1.5] we test the interference models trained on new data with past data. We compare the proposed HBMs with standard classifier (CLS) and CLS using our EBMs training objective (CLS*). For CLS and CLS** we use the cross-entropy loss to get their o, i.e., the cross-entropy losses on the old data before and after training on new data.
We obtain 0, by training on 200 randomly selected data from taskl and compute its gradient af- ter training. Then we perform single step optimization from task2 and average their parameters to get 0z- We compute & using the inner product of A = 02 - 0; and the gradient Ve, CML(xx 0 J. For comparisons among multiple approaches, we normalize the á of each method by dividing its maximum element value of 0; and scale é based on the length of f, as CLS and EBMs have different number of parameters.
200 data that randomly selected
A.6 SPI.IT CIFAR-100 WITH DIFFERENT NUMBERS TASKS
To test the generality of our proposed EBMs, in Table we repeat the boundary-aware experiments on spit CIFAR-100 (see Table | in the main text) for different number of classes per task. In Table the CIFAR-100 100 dataset was split up into 10 tasks, resulting in 10 classes per task. Here we addition- ally perform the split CIFAR- 100 protocol with 5 different tasks (i.e., 20 classes per task), with 20 different tasks (i.c.. 5 classes per task) and with 50 different tasks (i.e., classes per task). For all settings, we find that our EBM substantially outperforms the baselines.
15

---

Under review as a conference paper at ICLR 2021
Fisher on data
4212
ESM
1.00
1334
1302
Fisher data
Fisher data 2
0.31
CLS*
CL5* ESM
3:30
EBM
0.54
0.344
1.30
0.02
1.00
0.00
Fisher 4 on data
Fisher 4 on data 2
Fisher 4 on data 3
43.5
C1.
CLS*
5.29
EBM
0.224
6.24
0.50
8.02
FRM
8.52
0.00
0.00
02
15
25
35
20
30 16
Fisher on data 1
Fisher 5 on data 2
Fisher 5 on data 3
Fisher 5 on data 4
8.315
0.03
0.000
8.00
0.00
0480
2.244
0.323
0.032
07
2.5
30
3.5
6.5 3.5
20
2.5
6.
85
30
3.5
20
255
30
3.5
10
26 38
12
Figure 7: Parameter importance on different tasks. The x-axis represents each different parameter and y-axis is the FIM value in Equation 8 The sparser the parameters which may contribute to why EBMs has less influence on previous tasks after training on new data. "Fisher on data 1" means the diagonal elements of the fisher information matrix obtained by Equation g using the model parameters 05 (after training on task T5)) and data from task Ti-
the fewer important parameters for previous data,
Table 6: Comparison of EBM with baselines different variants of the split CIFAR- 100 protocol. The CIFAR-100 dataset split up into tasks (= 20 classes per task), 10 tasks (= classes per task), 20 tasks (= classes per task) or 50 tasks (= 2 classes per task). Shown is the Class-IL performance on the Boundary-Ann setting after all tasks have been learned. Each experiment is performed 10 times with different random seeds, the results reported as the mean + SEM over these runs.
CIFAR-100 split up into:
Method
5 tasks
10 tasks 20 tasks
50 tasks
CLS
14.74 + 0.20
8.18 0.10
4.46 =: 0.03
1.91 + 0.02
EWC
14.78 1 0.21
8.20 0.09
4.46
1.91 : 0.02
14.07 - 0.24
9.24
4.37 0.04
1.88 = 0.03
LwF
25.75
+ 0.14
10.71 0.11
12.18 1118:000 0.16
7.68 = 0.16
EBM
34.88 1 0.14
30.28 + 0.28
25.04 1 0.33
13.60 1 0.50
16

---

Under review as a conference paper at ICLR 2021
B DERIVATION OF LOSS GRADIENT
The derivation of the loss gradient Equation
a log Eyle(Nsst) exp( Ex V:))
@Cml
dEa(x,yt)
Ex-T
00
00
00
0 [ exp( (x,
(Ft(x,y+)
1 Exte(NYyt) exp(-Ee(x,,))
ExxT
yE((x,y*)
1 ) Eap(-Eo(x,,y)
lexp(- ([6g(xxx))) y.))
-
00
00
y.E(N.y
0Eo(x,y*)
1
0Eo yi)
=xxrT
Sap(-Eofo (xp(-Eo(x,
00
00
y.E(N.tt)
ExxT
0Ea(x,
Σ
Evie(N exp(- Exp(-Eo(x,) y:))
1
(E.
00
00
ME(WY++)
exp(-Eo(x,y)) is constant and can be moved inside the summation
y
0Eo(x,y+)
(xx(-Eo(xxy,))
(x,
ExxT
Σ
06
yEE(W EylE(N exp( - Fo(x,
00
(Eo(x,y+)
(Eo(x,
Exx-T
Σ
po(y:/x)
00
00
@oo yx
FEs(x y:)
Ex-T
Ey--(N
00
00
C
MODEL ARCHITECTURES
Here provide details of the model architectures used the different datasets.
Images from the split MNIST and permuted MNIST datasets are grey-scale images. The base- line models for these datasets, similar as in (van de Ven & Tolias 2019), consist of several fully- connected layers. For the EBMs we use similar number of parameters. The model architectures of EBMs on the split MNIST dataset and permuted MNIST dataset are the same, but have different in- put and output dimensions and hidden sizes. The model architectures of EBMs and baseline models on the split MNIST dataset shown in Table T Table 7b respecctvvel.. The model architectures of EBMs and baseline models on the permuted MNIST dataset are shown in Table Sa and Table 8b]
Images from the CIFAR-10 and CIFAR-100 datasets are RGB images. For CIFAR-10, small convolutional network for both the baseline models and the EBMs. The model architectures of EBMs the CIFAR-10 dataset are shown in Table 9a| Table 9b Table Table 9d and Table Pe
We investigate different architectures to search for the effective label conditioning on EBMs training as described in Section 5.1.5 The baseline models on CIFAR-I0 are given Table 9f
The model architectures used on the CIFAR-100 dataset are detailed in Table [0
17

---

Under review as a conference paper at ICLR 2021
Table 7: The model architectures used on split MNIST. h 400.
(a) The architecture of the EBMs.
(b) The architecture of the baseline models.
x FC(784, h) (x)
FC(784, h) (x)
X ReLU (x)
x ReLU (x)
y= Embedding (y)
x FC(h, h) (x)
" Norm2 (y)
x ReLU (x)
x ReLU (x)
out FC(h, 10) (x)
out = FC(h, 1) (x)
Table 8: The model architectures used on permuted MNIST. h 1000.
(a) The architecture of the EBMs.
(b) The architecture of the baseline models.
x FC(1024, h) (x)
FC(1024, h) (x)
X ReLU (x)
= ReLU (x)
y Embedding (y)
X PC(h, h) (x)
Norm2 (y)
ReLU (x)
x ReLU (x)
out = FC(h, 100) (x)
out = FC(h, 1) (x)
18

---

Under review as a conference paper at ICLR 2021
Table 9: The model architectures used on the CIFAR-10 datasct.
(a) EBM: Beginning (V1)
(b) EBM: Middle (V2)
(c) EBM: Middle (V3)
Input: x,
Input: x, y
Input: y
Embedding(N, 3) (y)
x Conv2d(3x3, 3,3 (x)
x 3onv2d(3x3.3.32) (x)
= Softmax(dim=-1) (y)
x ReLU (x)
X ReLU (x)
y = y y.shape[-1]
x Conv2d(3x3, 32, 32) (x)
X = Conv2d(3x3, 32, 32) (x)
= y
X ReLU (x)
X ReLU (x)
33n32d(3x3,3332) (X)
y Embeddiig((N, 32) (y)
Maxpool(2, (x)
ReI.U (x)
Softmax(dim=-11 1) (y)
Conv2d(3x3, 32, 64) (x)
Conv2d(3x3, 32, 32) (x)
= y.shape[-1]
X ReLU (x)
X = ReLU (x)
X = x y
X Conv2d(3x3, 64, 64) (x)
Maxpool 2) (x)
X Maxpool(2, (x)
X ReLU (x)
Conv2d(3x3, 32, 64) (x)
X Conv2d(3x3, 32, 64) (x)
y Embedding(N, 64) (y)
ReLU (x)
ReLU (x)
y Softmax(dim=-1) (y)
Conv2d(3x3, 64, 64) (x)
Conv2d(3x3, 64, 64) (x)
y * y.shape[-1]
= ReLU (x)
ReLU (x)
X x= x y
X (2 (x)
X Maxpool(2. 2) (X)
X Maxpool(2, (x)
FC(2304, 1024) (x)
X FC(2304, 1024) (x)
x FC(2304, 1024) (x)
ReLU(x)
RcLU(x)
x ReLUU(x
out FC(1244 1) (x)
out FC(104,, (x)
FC(11244 (x)
(d) EBM: End Fix Softmax (V4)
(e) EBM: End Softmax (V4)
(f) Baseline models
Input: y
Input: x,
Input: x
x Conv2d(33,,33 3, 32) (x)
x Conv2d(3x3, 3, 32) (x)
x = 3, (x).
ReLU (x)
X ReLU (x)
x ReLU (x)
Conv2d(3x3, 32, 32) (x)
Conv2d(3x3, 32, 32)
X = Conv2d(3x3, 32, 32) (x)
ReI.U (x)
ReLU (x)
X ReLU (x)
= (2x) (x)
(2x (x)
Maxpool (x)
X = Conv2d(3x3, 32, 64) (x)
X Conv2d(3x3, 32, 64) (x)
Conv2d(3x3, 32, 64) (x)
= ReLU (x)
X ReLU (x)
X ReLU (x)
Conv2d(3x3, 64, 64) (x)
Conv2d(3x3, 64, 64) (x)
X Conv2d(3x3, 64, 64) (x)
ReLU (x)
ReLU (x)
X ReLU (x)
Maxpool(2, 2) (x)
Maxpool(2, 2) (x)
X Maxpool 2) (x)
FC(2304, 1024) (x)
FC(2304, 1024) (x)
X FC(2304, 1024) (x)
= Random Projection (y)
Embedding(N, 1024) (y)
x ReLU (x)
= Softmax(dim=-1) (y)
y Softma(ddi==111 1) (y)
X FC(1024, 1024) (x)
y = y y.shape[-1]
y = y y.shapee[11]
X ReLU (x)
y
x
out FC(1024, 10) (x)
out = FC(1024, 1) (x)
FC(1024, 1) (x)
19

---

Under review as a conference paper at ICLR 2021
Table 10: The model architectures used the CIFAR- 100 dataset. Following van de Ven et al. 2020 for all models the convolutational layers were pre-trained CIFAR-JO. The 'BinaryMask -operattin fully gates a randomly selected subset X% nodes, with different subset for each y. Hyperparameter X was set using a gridsearch.
The architecture of the EBMs.
(b) The architecture of the bascline models.
Input: x,
Conv2d(3x3, 3, 16) (x)
Input: x
BatchNorm (x)
3, 16) (x)
ReLU (x)
x BatchNorm (x)
X Conv2d(3x3,, 16, 32) (x)
X ReLU (x)
X BatchNorm (x)
X Conv2d(3x3, 16. 32) (x)
X ReLU (x)
x BatchNorm (x)
Conv2d(3x3, 32, 64) (x)
ReLU (x)
BatchNorm (x)
33 32, 64) (X)
ReLU (x)
BatehNorm (x)
X Conv2d(3x3, 64, 128) (x)
x ReLU (x)
X BatchNorm (x)
x Conv2d(3x3, 64, 128) (x)
X ReLU (x)
x BatchNorm (x)
X Conx2d(3x3, 128, 256) (x)
X ReLU (x)
FC(1024, 2000) (x)
Conv2d(3x3, 128, 256) (x)
ReI.U (x)
FC(1024, 2000) (x)
X BinaryMask (x. y)
ReLU (x)
X
FC(2000, 2000) (x)
x FC(2000, 2000) (x)
X ReLU (x)
x ReLU (x)
X BinaryMask (x, y)
out FC(2000, 100) (x)
FC(2000, 1) (x)
20


Reviews:
[{"review_id": "qZ74kfSktaL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work shows that energy-based models (EBMs)  are a promising model class for continual learning problems. According to the experiments, EBMs outperform the baseline methods by several continual learning benchmarks.\n \n+ves: \n1. It is interesting to see that energy-based models are introduced for the classification continual learning problems. In the paper, the authors show that EBMs achieve a significant improvement on four standard CL benchmarks. It is really surprising that the improvements are so large. \n \n2. Some analysis, for example,  “energy landscape” and “Interference with past data.” seems interesting. It shows benefit of energy-based model for continual learning. Both of them indicate that y EBMs suffer less from catastrophic forgetting.\n \n \nConcerns\n1. It seems like the gradient computation of equation (5) is not corresponding to equation (4). \n\n2. It is nice to see the improvement of EBMS. In this work, a difference architecture is usually in EBMS. The architecture is different from the baseline. It is nice that EMBS has a more flexible architecture to score the input x and output y. However, the improvement of your work is from architecture or from the training. It is better to do a clear claim. If the benefit is from the architecture, do you treat it as a black box? Or it is from the training objective?  It is possible to show some learned structure in your formulation. The architecture of EBM seems large. \n\n\nQuestions during the rebuttal period: \nPlease address and clarify the cons above:\n \n1. The experiment setting detail is unclear. The training details of the proposed method and baseline are unclear.  And the baseline detail is a little confusing to me. In section 5.1.2, it says “ All the baselines and EBMs are based on the same model architecture”. It seems the architecture of EBMs is different.\n\n2. Some related work on energy-based model: multiple label classification[1] , sequence labeling [2] and machine translation [3]\n[1] Structured Prediction Energy Networks, ICML 2016\n[2] Benchmarking Approximate Inference Methods for Neural Structured Prediction. NAACL 2019\n[3] ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation\nACL 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g3L9ZPZ84I1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the usage of EBMs in continual learning for classification. Although the application of EBMs in continual learning is novel, the general idea is a special case of the usage of EBMs for structured prediction, which has been widely studied. For instance, multi-class classification can be considered as a special version of multi-label classification, which has been studied in Belanger and McCallum (2016) and a set of follow-up works. The main difference here is that multi-class classification is a simpler problem, and all possible classes can be enumerated in O(N), but in multi-label classification, more complicated inference such as gradient-descent based approaches must be used.\nThe contrastive training can be seen as a special case of margin-based training (Belanger and McCallum, 2016; Rooshenas et al. 2019), where the margin is infinity.\nI believe the works in using EBMs for structured prediction must be cited here as they are closely related.\n\nThe authors also explored the effect of ML training on interference with past data and showed that using single sample ML approximation can significantly alleviate the catastrophic forgetting problem.  \nI believe that this is an interesting observation. \n\nTypo: \"current bath\" in Section 5.1.4\n\nBelanger and McCallum (2016), Structured Prediction Energy Networks.\nGygli et al. (2017), Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs.\nRooshenas et al. (2019), Search-Guided, Lightly-supervised Training of Structured Prediction Energy Networks", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzeltLz-bzH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The writing of the paper needs improvements. I am still unsure how the paper learns each new task. You talked about batches but never talked about how each new task is learned specifically. It creates doubts in my mind. E.g., does each batch contain some examples from old tasks? Is this training like for multi-task learning?\n\nYou wrote “In the continual learning setting, we assume classes in Y_B are uniformly distributed in every new batch.” Is Y_B fixed for each task or each batch? \n\nYou need a negative class in each batch. This means that your algorithm cannot work in the scenario where a task has one class only. Most of existing techniques can handle this case although they use 2 or more classes in a task in their experiments. I think this is a serious limitation. It is only suitable for Task-IL, which is an easier problem to solve. \n\nI think the claim that existing techniques need to fix the number of classes beforehand is not correct. I know hat most of them fix the number in their code or experiments, but I don’t see why they cannot use a large number or dynamically add new class heads when needed, say, using cross-entropy as the loss function. \n\nThe experiments are not well described, and baselines are old. Many newer baselines should be included, e.g., \n\nLearning a Unified Classifier Incrementally via Rebalancing. CVPR 2019. \nOvercoming catastrophic forgetting for continual learning via model adaptation. ICLR, 2019. \nRandom path selection for continual learning. NeurIPS 2019\nContinuous learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019\nLarge scale incremental learning. CVPR 2019\n\nFor each dataset, you used one setting for tasks only, e.g., CIFAR100, 10 tasks. More than one setting should be tried to show the generality of the approach. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3dStbOjNRc2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==========================\n\nbefore revision\n\n==========================\n\nReview: Motivated by the effectiveness and naturalness of energy-based models, this paper proposes to use energy-based learning framework for continual learning. Empirical studies are performed to validate the proposed strategy on several continual learning benchmarks.\n\nStrength: \n+ This paper applies EBMs to the task of continual learning, which is interesting and relevant to ICLR conference. \n+ The paper is well written and easy to follow. \n+ The paper is technically sound, since the formulation of the EBMs are well derived by other prior works.  \n\n\nConcerns: \n+ The contribution of the paper is insufficient for publication. The energy-based learning framework for discriminative purpose has been developed for a long time, even though recently researchers in the field machine learning are enthusiastic about developing energy-based models for data generation. \n+ The underlying theory of the proposed method is developed by other papers, the only contribution of this paper is to apply the EBM to the continual learning, which is quite straightforward. \n+ Missing key reference about EBM for discriminative learning. The core of this paper is mainly based on the finding of the transition between discriminative EBM and generative EBM, which is originally presented in reference [a]. The current paper misses to discuss and cite this paper. \n+ missing relevant reference about generative EBMs in related work. Even though this paper is not directly related to EBMs for data generation, but it DID discuss the development of it in its paper. The current related work about EBMs for generative purpose is incomplete in the sense that it skipped some pioneering works and important application with EBMs. For examples,  [1] is the first paper to use ConvNet-parameterized EBMs with Langevin for image generation.  Training EBMs with assisting networks can be found in [2] and [3].  Also, writing a section of comprehensive related works about energy-based learning is not necessary but encouraging.  \n\nreferences\n+ [1] A Theory of Generative ConvNet (ICML 2016)\n+ [2] Cooperative learning of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI 2018).\n+ [3] Divergence triangle for joint training of generator model, energy-based model, and inference model. (CVPR 2019)   \n\n===================================\n\nAfter a revision\n\n===================================\nThank you for your efforts to revise the paper. The revised parts about related work look good to me. I agree on that citing all those EBM application papers is not necessary. But doing so can provide a comprehensive and complete development of  the DeepNet-EBM. Again, this is not required and it will not affect the rating.   \n\nI also acknowledge the existing contributions in the current paper and admit that such a direction is promising, but I still feel that the current paper doesn't fully explore this area with more solid experiments. Thus, the whole contribution is quite marginal. By taking into account all these concerns, I will change my rating from 4 to 5.    \n  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There were opinions on both sides of this paper from the reviewers.  Reviewers were excited by the novel application of energy-based models (EBMs) to continual learning and the resulting performance gains, but were concerned by the more direct application of EBMs (which has been explored in other work, and here adapted to the continual learning setting, so its contribution is marginal) and with the depth of the evaluation, which they thought could be pushed farther. Overall, the reviewers agreed that this paper could benefit from another round of revisions to strengthen its contribution, incorporating many of the excellent points made by the authors in their responses.
Decision: Reject