Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper deals with the training of GAN in a continual learning setting, i.e. different tasks are sequentially coming. The authors designed learnable task-specific adapters (transformations) for avoid the catastrophic forgetting problem. And task-similarity based initialization is adopted to improve the performance further. Experiment results show that the proposed method achieves better or comparable performance compared to previous state-of-the-arts, but with remarkably fewer parameters. 
main_review: Pros:
1) The paper is well-written and easy to follow.
2) The motivation is clear and reasonable. Ablations verify the effect of each proposed module. 
3) The proposed method achieves a good trade-off between introducing additional parameters and model performance. 

Cons or questions:
1) How does the model performance change with different amount of parameters introduced by the adapters? For example, the effect of choices of channels and groups in the adapters. 

2) The global parameters are fixed across tasks. And the authors adopted the parameters learned from the first task as the global parameters. So I just wonder if the performance of continual learning will be largely affected by the characters of the first task, e.g. the similarities between the first task and the following ones. An ablation study and more discussions relating to this may be valuable. 
limitations_and_societal_impact: The authors adequately addressed the potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an approach for continual learning in GANs by learning new task-specific modules over a base network. This work explores the contributions of different continual learning approaches for generative models and proposes an expansion-based model which learns new task-specific modules for each new task. These modules augment the features of the base task layer. To prevent instability issues, the model also creates additional residual pathways. To choose the parameters for a new task, a task similarity-based initialization is proposed which initializes the new task network parameters with the learnt parameters of the previous task which is most similar to the current task. Experiments are performed on conditional and unconditional image tasks, and ablations are performed on the effectiveness of the proposed modules.

main_review: [comment]: <> (Originality)

The paper takes inspiration from expansion-based CL models which are dynamically expanding with the number of tasks, and do not suffer from catastrophic forgetting. The closest related work the paper mentions is GAN-memory [1], which also uses task-specific parameters to adapt the base parameters for the given task. Unlike the GAN-memory method which uses a StyleGAN-like framework to modify the weight parameters, the proposed method learns additional lightweight modules (namely, $3\times 3$ and $1 \times 1$ group convolutions) which are applied on top of the base modules to adapt features to the task. The FIM-based task similarity is also seemingly novel. 

Here are some of my comments and concerns:

[comment]: <> (Quality)

- One of the potential problems with the proposed method may be w.r.t. domain shift between the base task and the new tasks. Suppose the base task is a classification over medical images. If all the subsequent tasks are for natural images, then the initial filters chosen by the base network will pick out features that are for medical images (in the initial layers) and are very likely non-informative w.r.t natural images. The task-specific layers will try to build upon this set of uninformative features which is expected to have suboptimal performance. GAN-memory does not have this problem since it adapts all base layers by augmenting them (instead of building on top of base features), and the augmentation will be stronger for a completely different task. The paper can investigate the nature of the task-specific features learned from completely orthogonal tasks.
- Moreover, the next base layers are trained on the feature outputs of the previous base layers. Modifying the feature inputs with a task-specific layer will yield a task-specific feature that may have a different feature distribution. For example, $f_{\theta_{l+1}}(.)$ is trained with features $F_l^t$ as inputs (on the base task). Modifying  $F_l^t $ in the previous layer by task-specific networks to obtain $\tilde{F_l^t} = g_\phi(F_l^t)$ as input to the layer $f_{\theta_{l+1}}(.)$ is unintuitive because $f_{\theta_{l+1}}(.)$ expects to extract new features from feature vectors $F_l^t$ and not  $\tilde{F_l^t}$. This may be the reason why the proposed method doesn't work right out of the box and a residual path is added from the previous layer as a "correction term". In Line 176, the paper mentions that the method doesn't work right out of the box because the depth effectively doubles. However, the architecture they adapted from [2] has only a few resnet blocks. Resnets are known to work for large depths [3], therefore the paper's argument is unconvincing. To support this argument (L176), experiments can be run on a smaller network to see if it has a significant stability improvement over the originally chosen architecture.
- What is the effect of choosing a different base task $\mathcal{T}_0$ (like Brain MRI)? How does the performance of GAN-Memory and CAM-GAN change with the choice of the base task? An ablation on this experiment may be very helpful, not just for comparison of the paper, but in general for continual learning approaches.
- In Line 329, "The results are shown ... is significant" - this argument is not a convincing enough argument. However, the FID only decreases from 1-2 points. Bad images are known to have a high FID (>50) and real images are generally in the range of 10-30 [4]. 
- In Figure 6, results without TSL (50%) can also be added for completeness. 

[comment]: <> (Clarity)

- The details of task similarity-based transfer (Section 2.3) is not very well-explained. Specifically, for a given task id $t$, we have a set of learned adapter weights $\phi_1 .... \phi_{t-1}$. How are they used to calculate task similarity with previous tasks. I assume a task representation for all previous tasks needs to be saved as well. Are task embeddings computed for all previous task-weight combinations or only a few? At task $t$, what is the task vector that is compared to all previous representations? These details are missing from the paper. The paper can be more mathematically rigorous in adding these details.
- In Line266, does the paper mean "6-class generation problem"? Since the section is on conditional generation.
- Figure 2 is way too cluttered and the figures are too small. Similarly, Figure 3 is redundant. Since GAN-memory and CAM-GAN are expansion-based approaches with fixed base weights and fixed task weights, the loss plots are going to be horizontal lines. Only MerGAN shows degradation, being a non-expansion-based method. Either more baselines can be added, or just a table-based comparison may also suffice instead of having Figure 3.
- why are FLOPS not mentioned for GAN-memory in Table 1? I believe it should be much lower than the FLOPS for CAM-GAN since only the weights are updated (and conv weights are very small in size compared to the intermediate feature layers), and the effective depth is half of CAM-GAN. This number should also be included for a fair comparison (tradeoff b/w FLOPS and performance).
- In Figure 4 (Right), bar plots can be used instead of line plots. 


[comment]: <> (Significance)

The paper demonstrates a new continual learning approach for GAN-based models. It does outperform closely related work, though not by a significant margin. Using a task-similarity based on Fisher Information scores to select the closest task is an interesting idea that can be adapted by other works in continual learning, domain adaptation, and transfer learning. 

Ref:

[1] https://arxiv.org/pdf/2006.07543.pdf

[2] https://arxiv.org/pdf/1801.04406.pdf

[3] https://arxiv.org/pdf/2103.07579.pdf

[4] https://arxiv.org/pdf/1706.08500.pdf
limitations_and_societal_impact: The authors have touched upon the limitation of the method when a bad base task is chosen to train their base model. However, there are no experiments to demonstrate that (although the paper does have the data). The negative societal impact of this work is not very apparent, although generative models can have harmful societal impacts due to the generation of convincing fake data.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a lightweight adapter architecture, CAM-GAN, for generative adversarial networks (GAN) and its training method. This adapter enables continual adaptation to new tasks through feature transformation. The proposed CAM-GAN shows the state-of-the-art performance in continual adaptation for various tasks while minimizing the increase in adapter parameters and computation.
main_review: The strength of the proposed method is that it greatly reduces the size and computation of the adapter for continual adaptation. However, there are no big differences from the existing methods regarding adaptation through expansion or adding a separate module for each task. And using 1x1 convolution or group convolution to reduce adapter parameters and computation amount is commonly used in efficient CNN. So its contribution is minimal.
Furthermore, compared with GAN Memory, GAN Memory also includes a parameter compression method, but the comparison is missing.

The paper has several critical typographical errors. In residual learning, element-wise addition of the output of the $(l-1)^{th}$ layer is a shortcut connection, not a residual connection. So, the explanation in Session 2.2 is a bit confusing. And session 5.2, an experiment related to Residual Bias, is not cleared. The "w/o RB" needs to clarify whether it is not using an additional shortcut connection or is using a shortcut connection without $f^r_{\phi^t_{lr}}$. And in order to check the effectiveness of residual bias, it is necessary to compare the case where only shortcut connection is used.

[minor] line 134: $f_g$ > $f^g$
limitations_and_societal_impact: This work has no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed to learn a set of global and task-specific parameters to get task-specific feature maps for different tasks. And the proposed method leverages the Fisher information matrix to measure task similarities for parameter initialization for expansion-based continual learning. The proposed method is demonstrated on unconditional generation tasks and label-conditioned generation tasks.
main_review: - Pros
    - The paper is well written and easy to read.
    - The tasks are constructed from different datasets, instead of different categories from same dataset, which is more challenging.
    - The idea of considering task similarities has some merits in it.
    - The proposed approach works well for unconditional generation tasks.


- Cons
    - The contribution is marginal. The idea of considering task similarities share some similarities with [R1], but with key difference. However, the idea of global/task-specific parameters is very similar to [22, 15, R2]. [22] uses task-specific parameters on top of the feature maps to obtain new feature maps for different tasks, while [15] uses the frozen parameters from the initial task then reconstruct task-specific filters from them. The word "global" is bit inappropriate, since same as [15] the "global parameters" are parameters trained on the initial tasks and are frozen for later tasks, and unlike [12, R2] in which parameters are adapted for all later tasks using knowledge distillation. Can authors please explain in details how the proposed approach is **different and better** than [22, 15, R2]?\
    \
    [R1] Expert Gate: Lifelong Learning with a Network of Experts, cvpr'17 \
    [R2] hyper-lifelong gan: scalable lifelong learning for image conditioned generation, cvpr'21\
    $~$
    - Can author please report the average FID over all tasks after the final task is learned for both unconditional and conditional generations?\
    $~$
    - Can proposed approach be applied to image-to-image translation tasks? Would the gradients be similar for tasks like edges2shoes and edges2handbags? What do authors imagine/think the TSL perform will be on these types of tasks?\
    $~$
    - For TSL, what is the performance of random initialization? Given a new task, corresponding task-specific parameters are additional parameters which do not exist in previous models. For a baseline approach, maybe it is more natural to initialize them randomly instead from the most recent task.
limitations_and_societal_impact: Please refer to the comments in Cons in the main review sections.
review_title: null

Meta-review: The paper addresses continual learning for GANs with two major novel techniques: 1) learnable task-specific adapters to combat catastrophic forgetting, 2) a task-similarity measure based on Fisher information matrix accelerated by computational approximations. Although the method appears a bit ad hoc, the experimental results show effective performance, and the rebuttal has addressed the bulk of the concerns (some of which are quite insightful).  I suggest that the authors well incorporate the rebuttal to the paper.  Overall it is a neat and practical approach that is a good addition to the proceedings.
Decision: Accept (Poster)