Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use a pretrained model for its multi-scale feature map for better performance in continual learning along with cross-task structure-wise distillation and a new normalization layer. The proposed method is compared to recent work including DualNet (NeurIPS 2021), CTN (ICLR 2020), MIR (NeurIPS 2019) and etc on SVHN, CIFAR100, miniImageNet and CORe50 dataset. The proposed method improves the performance over the compared methods.
strength_and_weaknesses: **Strength**
- S1: Pretraining the model for continual learning brings a noticeable gain in performance.

**Weakness**
- W1: Pretraining using the online stream and complicated normalization scheme incurs a long pipeline for learning. One of the most important feature in continual learning is the short training time (i.e., online learning). This proposal sacrifice the efficiency for the continual learning.
- W2: Most of the gain comes from using multi-scale feature extracted from the pretrained model, which is an existing idea (not applied to CL context yet). Gains by other proposals are marginal (at the expense of larger model than prior arts (Table. 8).
- W3: The `pretraining' is not well suited to continual learning setup as it learns the streamed data by storing them in advance (this part is not very clear in description) for representation learning. As the continual learning setup encourages to learn the data with limited access over the tasks, the notion of pretraining may not be widely applicable in practice.
clarity,_quality,_novelty_and_reproducibility: The paper does not clearly describe the pretraining procedure; how do they train the pretraining model - what data, how may epochs, what is the pretext task and etc.
summary_of_the_review: Given that the paper uses seemingly expensive pretraining schemes and the gain is marginal except the gain by the existing idea of using multi-scale feature of pretrained model, the method is not very interesting to be reported in the community. Moreover the description (esp., the pretraining procedure) is not clear enough to judge the value of the work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper argues that a good continual learner should strike a different stability - plasticity tradeoff in online and offline CL, the former being more stable and the latter more plastic. The authors propose a new method MuFAN, which comprises 3 new components : the use of multi-scale features of pretrained models, an added cross-task distillation loss, and a new normalization technique. The authors evaluate their method on standard online CL benchmarks, and provide some ablation studies to highlight the effectiveness of the components. 
strength_and_weaknesses: Strengths : 
The paper positions itself well in the current literature. 
The paper is clear in detailing has many ablations to evaluate the utility of the proposed components

Weaknesses : 
1. *On the stability / plasticity conclusions drawn from Figure 1*. 
The authors argue because online methods have a lower accuracy after a training task (LA), then they are less plastic. I strongly believe that this is property is not intrinsic to online learning. Rather, this property is a direct result of **the number of training iterations spent on each task.** An easy way to increase the number of training iterations in the online setting is to simply do multiple gradient optimization steps (each with different rehearsal batches) as in ([1], [2]). It would be good for the authors to redo figure 1, where the number of iterations on a datapoint is equal to the number of epochs in the offline setting (so that online and offline methods use the same compute). 
2. *On the use of pretrained models in online CL* . 
The authors propose a novel (to the best of my knowledge) yet quite convoluted way to leverage pretrained models. Yet, highly simple and straightforward baselines (such as a nearest-class-mean classifier in the feature space of the pretrained model, or downright finetuning of such model) are omitted. Given that there is ample evidence [3,4,5] that these methods are surprisingly effective, it is crucial that one compares to such naive approaches : if one cannot beat them, then why bother with the complexity of a new method ?
Here is a small code snippet [6] that evaluates one such simple approach using the setting described on the Split CIFAR100 setting. **Without any training, one can boost the accuracy from 75% to 85% using solely the pretrained model!** 
3. *On the added complexity of the method*
From the paper, it is unclear how many additional forward passes are required at each step. Looking at 1 and 2, it seems that the full buffer is forwarded through the model at each step ? If this is the case, it is important to note that this comes at a very high additional cost in compute, and should be properly discussed. Also, what is the cost of the additional forward passes through the old model for distillation ? Given that data augmentations are used, I am assuming hidden activations cannot be cached and have to be recomputed ? 
On a similar note, it would be much more interesting to replace table 8 with a FLOP per iteration count for several methods. 
Similarly, what is the additional compute cost of using a top-down module ? my understanding is that upsampling the representation before feeding it to the classifier will have a significant increase in compute cost. It would be nice if the authors could analyse this. 
4. *On the task-free version of MuFAN* 
This version uses a hyperparameter $\mathcal{S}$, which essentially determines the size of each task (in number of unique labels). In a "real" online setting, the learner does not know how long the learning stream is, nor does it know how many unique labels it will see. Therefore, how would one set $\mathcal{S}$ a priori ? Moreover, in this section you are comparing to baselines which **do not** use a pretrained model; I don't see how this is a fair comparison



[1] Drinking from a Firehose: Continual Learning with Web-scale Natural Language https://arxiv.org/abs/2007.09335

[2] Online Continual Learning with Maximally Interfered Retrieval https://arxiv.org/abs/1908.04742

[3] REMIND Your Neural Network to Prevent Catastrophic Forgetting https://arxiv.org/abs/1910.02509

[4] A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning https://arxiv.org/abs/2210.04428

[5] Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis https://arxiv.org/abs/1909.01520
[6]
```
import os
import PIL
import torch
from torchvision import datasets, transforms

if not os.path.exists('efficientnet_lite0.pth'):
    os.system('wget https://github.com/RangiLyu/EfficientNet-Lite/releases/download/v1.0/efficientnet_lite0.pth')

if not os.path.exists('eff_net_code'):
    os.system('git clone git@github.com:RangiLyu/EfficientNet-Lite.git eff_net_code')

# load torch model
from  eff_net_code.efficientnet_lite import build_efficientnet_lite
model = build_efficientnet_lite('efficientnet_lite0', 1000)
model.load_state_dict(torch.load('efficientnet_lite0.pth'))
model = model.cuda()
print(f'total of {sum(x.numel() for x in model.parameters()) / 1e6 :.2f} M params.')

# remove linear head
model.dropout = model.fc = torch.nn.Identity()
model.avgpool = torch.nn.Identity()
DIM = model(torch.cuda.FloatTensor(2, 3, 224, 224)).size(-1)

# default transforms for the model in question
tfs = transforms.Compose([
    transforms.Resize(224 + 32, interpolation=PIL.Image.BICUBIC),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.498, 0.498, 0.498], [0.502, 0.502, 0.502])
    ])

ds_train = datasets.CIFAR100('./', train=True, download=True, transform=tfs)
ds_test  = datasets.CIFAR100('./', train=False, download=True, transform=tfs)
BS, n_cls, n_way = 256, 100, 5 # use n_way == 100 for the single-head task-free result


@torch.no_grad()
def get_prototypes(model, dataset, n_classes):
    protos = torch.zeros(n_classes, DIM).cuda() 
    counts = torch.zeros(n_classes, dtype=torch.int64).cuda()

    loader = torch.utils.data.DataLoader(dataset, batch_size=BS, num_workers=8, shuffle=True)
    offset = torch.arange(DIM).cuda()

    print(f'processing : {(str(type(dataset)))}')
    for i, (x,y) in enumerate(loader):
        print(f'{i}/{len(loader)}', end='\r')
        x, y = x.cuda(), y.cuda()
        feats = model(x)
        
        # accumulate
        counts.scatter_add_(0, y, torch.ones_like(y))
        idx = offset.view(1, -1) + y.view(-1, 1) * DIM
        protos.view(-1).scatter_add_(0, idx.view(-1), feats.view(-1)).view_as(protos)

    return protos / counts.unsqueeze(-1)


@torch.no_grad()
def evaluate(model, dataset, protos, n_way):
    loader = torch.utils.data.DataLoader(dataset, batch_size=BS, num_workers=8, shuffle=True)
    n_cls = protos.size(0)

    n_ok, n_tot = 0, 0

    for i, (x,y) in enumerate(loader):
        print(f'{i}/{len(loader)}', end='\r')
        x, y = x.cuda(), y.cuda()
        feats = model(x)

        dist = (feats.unsqueeze(1) - protos.unsqueeze(0)).pow(2).mean(-1)
        pred = dist.argmin(1)
        acc  = (pred == y).float().mean()

        # since in CL we typically partition and eval on a subset of classes, let's mimick this
        _, preds_in_order = dist.sort(1, descending=False)
        pos_of_correct_answer = torch.where(preds_in_order == y.unsqueeze(-1))[1]

        n_extra = n_way - 1

        # if we pick any `n_extra` classes at random, what is p(correct) ?
        p_correct = dist.new_ones(size=y.size())
        for it in range(n_extra):
            n_worse_left = ((n_cls - 1) - (pos_of_correct_answer) - it).clamp_(min=0)
            n_better = pos_of_correct_answer # fixed
            p_correct = p_correct * (n_worse_left / (n_worse_left + n_better))

        n_ok  += p_correct.sum().item()
        n_tot += x.size(0)

    print(f'\nTest Acc : {n_ok / n_tot  * 100 :.2f}')

protos = get_prototypes(model, ds_train, n_cls)
evaluate(model, ds_test, protos, n_way)
```
clarity,_quality,_novelty_and_reproducibility: The paper is very dense. The authors do a reasonable job explaining most of the contributions, however some parts are hard to read : for example, please consider using a different notation than $\mathcal{X}^{1,2}_{i,j}$ : you could just list the subsets in task $i$ and task $j$ one after the other. 


summary_of_the_review: Overall, given that the main contribution relies on proposing a new way to leverage pretrained models, but that key baselines are missing from this analysis, my opinion is that the paper should make these changes and be resubmitted. A proper evaluation of the computation complexity of the method is missing as well. While these two key issues are not properly addressed, my rating will stay the same. 

Post rebuttal : The authors have addressed some of my concerns, so I am raising my score accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors address the problem of stability and plasticity in continual learning in a fully online manner. They do that by introducing the extraction of multi-scale feature maps from shallow and deep layers of a pre-trained model, a structure-wise distillation loss across multiple tasks, and a parallel normalization module for stability and plasticity. The proposed approach outperforms most of the current SOTA methods in regard to accuracy (ACC), stability (FM), and plasticity (LA). In addition, the authors have performed extensive experiments in their ablation study foolproofing the significance of their approach.
strength_and_weaknesses: Strengths
1. The paper essentially describes a novel way to address the problem of catastrophic forgetting.
   a. First, similar to the architectures used in the domain of semantic segmentation, extracting the features from shallow to deep layers becomes essential especially when extracting deep high-level features and shallow low-level features. In this task, the extraction of multi-scale feature maps helps the model learn its intrinsic features when implying the addition of the top-down module.  
   b. Although the significance of sequential hierarchical training in the domain of continual learning has not been explored widely, the authors have used a similar concept to force the model to learn the relationship between two consecutive tasks by maintaining an angular distance. Based on their ablation study, this has shown some improvements in stability, but not so much in accuracy and plasticity. 
   c. The proposed parallel stability-plasticity normalization module tackles the variability issues of the incoming data and problems of normalization in the inference step. For example, the inconsistency in the data during training and inference can cause a significant drop in the performance of the previous task, leading to catastrophic forgetting. Here, this method finds a balance between stability and plasticity. 
2. The authors have performed comprehensive experiments against other SOTA methods. Their extensive ablation study further solidifies their claims. 
3. The authors have made use of all the most common benchmark datasets that are widely used in this area, furthering the compatibility of their approach in regard to its reproducibility. 
4. The paper is easy to follow. 

Weaknesses
1. Why are the ablation studies in Tables 3 and 4 performed on ER-Ring? Also, it seems to be the case with table 6 as well with DER++. The authors’ proposed approach is MuFAN. (It is either not clearly stated in the paper or there can be a  typo)
2. In table 9, it can be observed that by using all three proposed components and data augmentation, only the best overall accuracy can be achieved. However, as stated by the authors, for tackling the stability-plasticity dilemma,  row-8 has shown significant improvement in handling stability whereas row-7 handles plasticity quite well. Therefore, what is affecting these components when used altogether?
3. The authors have mentioned that the architecture is similar to that of U-Net and discussed the improvements over utilizing standard and bottom-up extraction methods; however, as mentioned in this work [1], it would be interesting to see the compatibility of the top-down (without multi-scale feature extraction) approach with auto-encoders in their ablation study.

[1] A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder Based Lifelong Learning,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 2017, pp. 1329–1337. doi: 10.1109/ICCV.2017.148. 

clarity,_quality,_novelty_and_reproducibility: Quality: The authors have shown significant improvement in online continual learning especially when there is a task-free setting in addition to outperforming most SOTA methods in a standard setting.

Novelty: Although more recent approaches like [1, 2] have used knowledge distillation between models as a similarity function (some settings of these approaches differ), using a structural distillation function for hierarchical training is a somewhat mildly explored concept, the authors have used this effectively to maintain the similarity of the tasks embedding space with the model as it is learning incrementally in an online fashion. In addition to that, the encoder-based top-down approach in this paper is somewhat explored in this work [3]; however, the authors have shown that multi-scale feature extraction, similar to U-net, can improve the accuracy significantly.

Reproducibility: based on the datasets the authors have used, the detailed methodology, and the clarity of the work, I believe this work is reproducible. 

[1] M. H. Phan, T.-A. Ta, S. L. Phung, L. Tran-Thanh, and A. Bouzerdoum, “Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 16845–16854. doi: 10.1109/CVPR52688.2022.01636.

[2] M. Kang, J. Park, and B. Han, “Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 16050–16059. doi: 10.1109/CVPR52688.2022.01560.

[3] A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder Based Lifelong Learning,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Oct. 2017, pp. 1329–1337. doi: 10.1109/ICCV.2017.148. 

summary_of_the_review: The authors have shown improvements over SOTA methods in the online continual learning paradigm. Their claims are supported with empirical evidence. In addition, each of the components in their work are contributing to the approach. I believe this would be a good contribution. Therefore, I am leaning towards accepting this paper; however, the authors will need to clarify the issues mentioned in the weakness section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method MuFAN that strikes a better stability-plasticity tradeoff compared to existing methods in an online (single-pass) continual learning setting. The main idea of the method is to use a pre-trained model as an encoder, aggregate the feature maps from different layers, and then pass the aggregated/ fused feature map to a classifier (typically a ResNet model). During training the pre-trained encoder is kept fixed and the classifier is updated. The authors propose a structured knowledge-distillation loss that maintains the angular distance between the successive tasks. Additionally, the authors propose a new normalization technique that splits the channels at a given layer and apply a combination of BatchNorm, InstaceNorm and GroupNorm on different splits of the feature map. The experiments are reported on several supervised continual learning benchmarks. 

strength_and_weaknesses: **Strengths**

1. The paper studies an important problem of online continual learning. 
2. The use of a pre-trained model as an encoder and fusing features of the encoder from different scales is a nice idea which is not explored in the context of continual learning extensively. 

**Weaknesses**

1. **Overall the method seems very ad-hoc**: While the use of a pre-trained model as an encoder that generates a fused feature map for the classifier is a nice idea, the other two contributions of the paper seem very ad-hoc. The structured-distillation loss seems like a cumbersome way of enforcing knowledge-distillation, and it is not clear if the so-called point-wise knowledge-distillation on more points would achieve similar or better results than structured distillation. The overall training objective (Eq. 4) is the combination of everything, experience-replay, point-wise knowledge distillation, and structured distillation and it is not clear if everything is needed in that objective. Second, the SP-normalization seems even more ad-hoc where half of the features are passed through BatchNorm layers, and the other half are sent to a combination of GroupNorm and InstanceNorm, without any criterion. If BatchNorm is bad for continual learning then there would be excessive forgetting through the half that was sent to the batchnorm. As for Group or InstanceNorm, if the features drift from one task to the next, how well these normalization techniques would be able to cope with forgetting. Overall, it seems that in designing SP-normalization the whole kitchen sync is thrown at the features. 

2. **Writing is not clear**: The writing of the paper is not clear and makes the paper a difficult read. The tuple notation for denoting data samples is incredibly cumbersome and non-intuitive. Sec 3.2 is not clear at all, terms are used without any definitions. For example, the functional form of the so-called structure-wise potential $\psi$ is not provided until the end of the section making it difficult to understand what is happening in Eq. 2. Similarly, throughout Sec 3, intuitions are lacking and it seems that everything is combined in an ad-hoc way. What the authors refer to as knowledge-distillation based on ER, is simple experience replay, there is no knowledge-distillation there. 


3. **Experiments**:

* Do the baselines, GEM, ER, MER etc also use the pre-trained model? This is not clear from the experiments. 

* Do all the baselines use the same codebase?

* Could the authors compare the compute and memory cost of their method vs the baselines?

clarity,_quality,_novelty_and_reproducibility: The paper writing needs improvement. Please refer to the strengths and weaknesses section above for details.
summary_of_the_review: Please refer to the strengths and weaknesses section above for details.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper describes an approach to online class-incremental learning based on exploiting pre-trained models. The authors argue that a different stability-plasticity tradeoff is needed for online continual learning (i.e. that *stability* is far more important) and propose the MuFAN model which incorporates a new, structure-based distillation to mitigate forgetting and a stability-plasticity normalization layer that tries to simultaneously maintain both stability and plasticity. Experimental results are given on standard online CIL benchmarks and a comparison is made with recent work from the state-of-the-art.

# Strengths

+ **Novel Application of Pre-trained Models**: This paper proposes to fuse features from a pre-trained encoder during online continual learning. While using a pre-trained model seems like an obvious approach to generically improve performance, the authors perform a methodical and well-motivated analysis of its incorporation into an online continual learning pipeline -- and the pre-trained encoder is *not* applied in an obvious way.

+ **Comprehensive Experimental Evaluation**: The reviewers a nearly unanimous in their praise of the experimental evaluation in the submitted manuscript, from the comparison with the state-of-the-art to the comprehensive ablation studies performed.

+ **Novelty**: The paper makes contributions in several, orthogonal directions. Apart from exploiting a pre-trained encoder, the authors also propose a new distillation technique based on the structure of fused, multi-scale features, as well as a new normalization layer specifically designed to maintain stability and plasticity.

# Weaknesses

+ **Clarity and Reproducibility**: Reviewers had a somewhat mixed opinion regarding the clarity of the paper. Some found the paper to be "dense" and "a hard read". In rebuttal the authors worked to improve several aspects regarding clarity.

+ **Computational Burden**: Exploiting pre-training goes somewhat contrary to the philosophy of continual learning -- as pointed out by reviewers -- and entails additional computational costs as well. During the discussion phase the authors provided additional analysis showing that their proposed online CIL pipeline (post pre-training) does incur additional computational costs compared to DualNet (about 10% more), but also results in nearly 10% points in accuracy gain.

# Summary

During the discussion phase there was a lively back-and-forth between the authors and reviewers (and between reviewers and AC). The authors vigorously engaged with reviewers and multiple levels, and provided new experimental results comparing with very recent works (published after the ICLR cutoff). Given the novel contributions of the work (which must be emphasized does *not* simply reduce to "using a pre-trained backbone) in multiple dimensions, the paper merits a place at ICLR as it takes a thoughtful and methodical look at *how* pretrained encoders can and should be exploited for online CIL.

justification_for_why_not_higher_score: While the novelty of the contributions are not in question, there are some methodological issues raised by reviewers that bring into question the long-term impact of the work. However, since it is a first-of-its-kind work on exploiting pre-trained encoders for online CIL, such methodological issues can and should be discussed at a forum like ICLR -- as a poster.
justification_for_why_not_lower_score: Again, the novelty of the multiple contributions and the thoroughness of the experimental evaluation earn this work a place at ICLR. Although reviewer opinion was somewhat split, there were no vigorous detractors of the work as submitted.
Decision: Accept: poster