Lifelong Learning in StyleGAN through Latent Subspaces
Authors: Adarsh Kappiyath
ANMOL GARG
Ramya Hebbalaguppe
Prathosh AP
Venue: Accepted by TMLR ()

Abstract: StyleGAN is one of the most versatile generative models that have emerged in recent times. However, when it is trained continually on a stream of data (potentially previously unseen distributions), it tends to forget the distribution it has learned, as is the case with any other generative model, due to catastrophic forgetting. Recent studies have shown that the latent space of StyleGAN is very versatile, as data from a variety of distributions can be inverted onto it. In this paper, we propose StyleCL, a method that leverages this property to enable lifelong learning in StyleGAN without forgetting. Specifically, given a StyleGAN trained on a certain task (dataset), we propose to learn a latent subspace characterized by a set of dictionary vectors in its latent space, one for each novel, unseen task (or dataset). We also learn a relatively small set of parameters (feature adaptors) in the weight space to complement the dictionary learning in the latent space. Furthermore, we introduce a method that utilizes the similarity between tasks to effectively reuse the feature adaptor parameters from the previous tasks, aiding in the learning process for the current task at hand. Our approach guarantees that the parameters from previous tasks are reused only if they contribute to a beneficial forward transfer of knowledge. Remarkably, StyleCL avoids catastrophic forgetting because the set of dictionary and the feature adaptor parameters are unique for each task. We demonstrate that our method, StyleCL, achieves better generation quality on multiple datasets with significantly fewer additional parameters per task compared to previous methods. This is a consequence of learning task-specific dictionaries in the latent space, which has a much lower dimensionality compared to the weight space. Code for this work is available at \href{https://github.com/kadarsh22/StyleCL}{link}

Published in Transactions on Machine Learning Research (12/2024)
Lifelong Learning in StyleGAN through Latent Subspaces
Adarsh Kappiyath
kadarsh220@mmaiilccomm
TCS Reseorch Delhi, India
Anmol Garg
agazmol19080@mai..
Indian Institute Science (ISSe), Bengatura, India TCS Research, Delha, India
Ramya Hebbalaguppe
wwm.g.hhchbllnggppe@ittt.ccmm
TCS Research Delli, India
Dr. Prathosh A. P
prathosh www.ooshhiiiiscc.ooo
Indian Institute of Science (IISe), Bengaluru, India
Reviewed on OpenReview: https: // openreview. net/ ht///forur forum? id= I4IAWVOZrM
Abstract
StyleGAN is one of the most versatile generative models that have emerged in recent times. However, when it is trained continually stream of data (potentially previously unseen distributions), it tends to forget the distribution it has learned, as is the case with any other generative model, due to catastrophic forgetting. Recent studies have shown that the latent space of StyleGAN is very versatile, as data from a variely of distributions can be inverted onto it. In this paper, we propose StyleCL, a method that leverages this property to enable lifelong learning StyleGAN without forgetting. Specifically, given a. StyleGAN trained on a certain task (dataset), we propose learn latent subspace characterized by set of dictionary vectors in its latent space, one for each novel, unseen task (or dataset). We also learn relatively small set of parameters (feature adaptors) in the weight space to complement the dictionary learning in the latent space. Furthermore, we introduce method that utilizes the similarity between tasks to effectively reuse the feature adaptor parameters from the previous tasks, aiding in the learning process for the current task at hand. Our approach guarantees that the parameters from previous tasks are reused only they contribute a beneficial forward transfer of knowledge. Remarkably, StyleCL avoids catastrophic forgetting because the set of dictionary and the feature adaptor parameters unique for each task. We demonstrate that method, StyleCL, achieves better generation quality multiple datasels with siguificantly fewer additional parameters per task compared to previous methods. This is a consequence of learning task-specific dictionaries in the latent space, which has a much lower dimensionality compared to the weight space. Code for this work is available at linkk
1 Introduction
Continual learning (CL) is a fundamental machine learning paradigm that focuses on the model's ability to learn and adapt to new tasks or evolving data streams over time while ensuring that previously acquired knowledge remains intact. Extensive research has explored continual learning within the context of discriminative models
De Lange al. (2022). but relatively less attention has been devoted the application of this paradigm in the realm of generative models. However, recent. progress in the field of generative modelling has brought.

---

Published in Transactions on Machine Learning Research (12/2024)
them to the forefront of application domains. Specifically, models such as Generative Adversarial Networks (GANs) Goodfellow et al. (2014) and denoising diffusion models Ho et. al. (2020) have found their utility in a wide variety of tasks such as semantic editing Ling et al. (2021), image in-painting, Yu et al. (2018) etc. Thus,
it is imperative to consider the problem of continual learning in the context of gencrative models Lesort et al. (2019).
It essential to note that Diffusion models offer remarkable generation quality enhancements, albeit with trade-off of increased inference time. On the other hand, GANs excel in efficiency, requiring only single forward pass for generation. Furthermore, the introduction of GigaGAN, Kang et al. (2023) and StyleGAN-T Sauer et al. (2023) have illustrated their ability to provide competitive generation quality while maintaining faster inference speeds in text-conditioned image generation tasks. Given these advantages, we turn our attention to continual learning in GANs for unconditional generation. Also, recently many recent stato-of-thh-art GANs for various tasks, as seen in works like Kang et al. (2023); Sauer et al. (2023) Fu et al. 2022)) employ StyleGAN- based architectures. This inspires our investigation into StyleGAN-basel architectures for continual learning.
We hypothesize, that StyleGAN is suited for generative continual learning because of its versatility, in that, large variety of datasets can be inverted onto its extended latent space (W+) as observed in Abdal et al. (2019). Motivated by these observations, we investigate whether the latent space of StyleGAN can be exploited to generate data from a stream of datasets without forgetting. Towards that end, we propose a method titled StyleCL to learn a per-task, style-wise dictionary vectors that define a subspace in the latent space of StyleGAN. In addition to latent dictionary learning, we also learn a set of parameters in the weight space, accommodate a richer knowledge in tandem with the learned latent subspace.
Knowledge transfer, cornerstone of continual learning, assumes a central role in StyleCL. StyleCL utilizes the latent space to identify the most similar task unlike GAN Memory Cong et al. (020) and CAM-GAN Varshney et al. (2021) where the most similar task is characterized using the most recent task or the task with high Fisher information respectively. We also determine the nature of forward knowledge transfer (positive or negative) by measuring the cosine similarity of dictionary vectors to its projection onto the latent subspace of the most similar task which is then used to prevent negative forward transfer.
The following is a summary of our contributions:
Latent subspace learning for StyleGAN:
We propose a latent subspace learning approach that enables learning without forgetting for StyleGAN.
Improved generation quality: By harnessing the versatility of the latent space StyleGAN, our method outperforms contemporary approaches like CAM-GAN and GAN Memory in terms of generation quality, all while employing fewer parameters (28.95% reduction) and FLOPs (11.6% reduction).
Prevention of negative forward transfer: We further propose a simple way to identify the most similar previous task and also characterize the nature of forward transfer between any two tasks to prevent negative forward transfer. To the best of our knowledge, StyleCL stands out as the sole method capable of averting negative forward transfer.
2 Related Work
Generative Continual Learning: Continual Learning methods are broadly categorized into three categories: replay-based, regularization-bent based and parameter isolation- based methods. These categorizations are defined for discriminative continual setting but they can be applied to generative continual learning well.
Chenshen et al. (2018) introduces MerGAN, a replay-based GAN that combines generated samples from previous tasks with new task data to form an extended training dataset. They also introduce a replay- aligument loss to ensure consistent generation for previous tasks as the number of tasks increases. Zhai. et al., presents Lifelong GAN for continual image-conditioned image generation, employing knowledge distillation and auxiliary data generation by creating patch montages from training batches to mitigate catastrophic
2

---

Published in Transactions on Machine Learning Research (12/2024)
forgetting. However, replay-based approaches face scalability issues due to cumulative inaccuracies when single generator is incrementally updated.
Parameter isolation techniques like PiggybackGAN Zhai et al. (2020) freeze old task parameters and
introduce smaller new parameters for learning without forgetting. GAN memory Cong et al. (2020) employs
normalization parameters to adapt the generator's 's weights incoming data. streams. CAM-GAN Varshney
iet al. (202I) introduces adaptation modules via group-wise convolutions at the output of each convolution
layer in the base network. Similar to CAM-GAN Verma et al. (2021), proposes efficient feature transformation
enabling continual learning with minimal parameter expansion for both generative and discriminative tasks, Recently, Seo et al. (2023) proposed LFS-GAN for continual learning in few-shot setting. This is achieved by learning an efficient task-specific modulator on top of & pre-trained StyleGAN and utilizing these task-specific modulators for generating from each of the tasks during inference. Contrary to all these methods, StyleCL takes a different approach by learning latent subspace alongside shared weight space parameters, facilitating continual lcarning rather than solcly relying on efficient feature transformations.
Even though few regularisation- based approaches like Liang al. (0118) and Seff et al. (0117) use regulari- sations to enable continual learning, their generation quality still degrades over time and thus parameter isolation methods appear to be better choice and have been receiving more attention.
Knowledge transfer in continual learning: Knowledge transfer is a crucial aspect of continual learning, predicated on the notion that similar tasks inherently possess shared knowledge that can be effectively transferred between them. However, previous approaches, like MerGAN Chenshen et al. (2018), Lifelong GAN Zhai et al. (2019), and Piggyback GAN ZZhai et al. (2020), often lack explicit, mechanisms to facilitate this positive knowledge transfer. While GAN Memory Cong ct al. (2020) demonstrates promisc in enabling knowledge transfer, it relies on the assumption that the most recent Lask is invariably the most similar, a notion that does not consistently hold. In contrast, CAM-GAN Varshney et al. (2021) quantifies task similarity by approximating the Fischer information matrix (FIM) and posits that initializing the current task with parameters from the most similar task would consistently yields positive forward transfer which may not always hold true. StyleCT. distinguishes itself by characterizing both the most similar task and the nature of forward transfer using the latent space, thus effectively capturing the state of the generator while identifying the most similar task and clucidating the nature of the forward knowledge transfer.
3 Proposed Method: StyleCL
3.1 Problem Setting
We introduce model for learning from sequence of datasets, where each datasct is identified by unique task ID and is presented sequentially for training. Let {x*) denote these datasets, with X comprising N instances {x{) each sampled from the task-specific distribution pt. Our challenge is to obtain Generative Adversarial Network (GAN) that generates samples from the current task distribution without forgetting the distributions from previous tasks.
Our method starts by training GAN as in (arrra et al., 2020a) on the first (or base) dataset (task), denoted by G². The parameters of 9 are denoted by Ø² and are shared by all the subsequent tasks. For each subsequent dataset ('(t > 1), we learn latent subspace characterized by U and b' on the latent space of G¹ as detailed in Sec. 3.2 Next, determine the most similar previous task to the current task, designated as task & by leveraging b'_ We further claborate this process in Sec. 3.3.1 Upon identifying the task k, we acquire ils corresponding generator, denoted by g*. We then evaluate w whether the parameters from task k are advantageous for learning the present task as elaborated in Sec. 333.2] When the parameters from task are beneficial, we Iearn subspace U and fcature adaptors &² within the latent and weight space of g*, otherwise, we shift this training to the latent and weight spaces of the base generator G', as described in Sec. 3.4 Fig. presents overview of our method, which name 'StyleCT.
3

---

Published in Transactions on Machine Learning Research (12/2024)
Most Similar Task
X!
X
Time
Fload
gh
D
ge
generator
Stylhbook
Discriminaor
a)
10-1
Const 4x
Ni0.1)
Disioney Vedios
S
Adaptor 4
wi
pm
S
fi-1
SY
Stt M
U
dg
b)
g
Figure 1: Overview of StyleCL: Fig. (a) illustrates the generative continual learning problem, where a continuous stream of data arrives, and the objective is to generate samples from both current and past data without forgetting. At time 6, the framework begins by learning the parameters U and b' the latent space of the generator G² using the dataset X", as shown in Fig. (b). Task similarity then evaluated identify the most similar previous task k, along with its corresponding generator g* and the degree of forward transfer is computed. As demonstrated in Fig. (c), the feature adaptor block oi along with the parameters of S* is used to obtain the activation map fhe If negative forward transfer occurs, k is set to 1, meaning only the style blocks from G (S), are used to generate ftn
3.2 Latent Dictionary Learning
We employ StyleGAN2 Karras et al. 2020aa architecture for the generators 9 that contains M style blocks and for simplicity discussion, we assume each of these style blocks comprises just layer. The first stage of our method is learn set of dictionaries on the extended latent space (W+ space) of the StyleGAN. Given dataset X dictionary Uf (u'mi...., .......,,
.., u'nk u E Rd. containing K vectors are learned for each of the 1, 2, 2., M style blocks of the generator along with bias vector b Note that the elements of Ut define latent subspace within the (W+) space of StyleGAN while the learned bias vector by serves to translate this subspace. Consequently, by characterizes the position of the latent subspace with respect to the origin.
During the training, the parameters of Ut m are initialized randomly. First, batch vectors stochastically sampled from each dictionary U as follows:
w =ml u 2m² u ... ... Zmn u K + b
(1)
4

---

Published in Transactions on Machine Learning Research (12/2024)
where Zm [2m1, ................................ 2mk] ~ I0. Further, each w corresponding to every style block is concatenated to form w as:
w' [w] win. wh w' W
(2)
Finally, w passed as the input to the corresponding generator to generate images from pe (x'x..
3.3 Forward Transfer
3.3.1
Choosing the most similar previous task
This section outlines method for identifying the task & that is most similar to the current task t. Determining the closest prior lask enables the transfer of knowledge from that lask to the current task, concept parallel to forward transfer in continual learning literature Chen & Liu 2018). In order to find the task that is most similar to an incoming task, it is necessary to charactcrize both the previous as well as the current task in the latent space. We characterize the current task by learning U and be using the base generator G is important mention that Um and bm are already learned for the previous tasks from to 1
For given task we define the most similar task & as the task whose latent subspace is closest to the subspace of task t. This assumption is based on the observation that the latent vectors of similar tasks tend to be close in proximity, while those of dissimilar tasks are farther apart, as illustrated in Fig. 5 The bias vector b'n represents the extent to which the latent subspace U is shifted, and we can use the Euclidean distance between bias vectors to quantify the similarity between subspaces across tasks. To motivate the comparison of bias vectors as a measure of subspace similarity, consider a simplified scenario where the dictionary consists of a single vector along with a bias term. In this case, the latent vectors lic within an alfine subspace, represented as b+ owi,, where is the bias and in is the dictionary vector. Similarly, we could define latent space for another dataset 8' + d'u' To measure this similarity, begin by computing the Euclidean distance between the bias vectors b and R. This distance provides mcasure of how closcly the subspaces are aligned in the feature space. Furthermore, to assess the aligument between the subspaces more comprehensively, we define a similarity metric sim(t, k) that captures the orientation and overlap of the subspaces defined by the corresponding dictionaries.
The most similar task is then identified as the task whose bias vector b" is closest to the bias vector bs of task t. Formally, this is expressed as:
k argmin (1..(2....-1)
|lb - b
(3)
Intuitively, equation equation B quantifies the discrepancy in how the subspaces of task / are translated relative to the subspaces of previous tasks. This choice is grounded in the assumption that subspaces of similar tasks should be closer in the latent space, evidenced by the empirical results shown in Fig. 5
3.3.2 Preventing negative forward transfer
Choosing the most similar task facilitates selecting a task with similar set of features as that of the current. task, although this may result in negative forward transfer. This situation arises because two subspaces U and U m might be orthogonal to each other despite being translated by the same bias vector. To alleviate this problem, we estimate the nature of forward transfer (positive or negative) by computing the cosine similarity of dictionary vectors of the current task to projection onto the latent subspace of the most similar task k. To compute the task similarity, both set of latent vectors U and V should be in the same latent space. We learn U in the latent space of G to facilitate this before computing the tisk-similarity. Let Vk (v va mj..... ..., v " v E Rd correspond to the orthonormal vectors obtained using the Gram-Schmidt orthogonalization procedure on the dictionary vectors U* The projection of u'm onto the latent subspace characterized by the orthonormal vectors V for style block m is defined
5

---

Published in Transactions on Machine Learning Research (12/2024)
Σ
v
(4)
Subsequently, the nature of forward transfer is estimated as follows: :
M
K
sim(tt, k:)
cos (u's Poni)
M
m==
(5)
M
K
u
yii
tk mi
M
P 2/
The value of sim(t,k) k) lies in the range [-1, 1], indicating the degree of forward transfer. Specifically, sim(t, k) > 0 significs potential positive forward transfer. We do not rouse the paramcters from the most similar task when sim(t, k) I 0 to prevent negative forward transfer. Intuitively, sim(t, k) quantifies the similarity or alignment between the latent subspaces of tasks. sim(t, k) S 0 denotes orthogonality or higher degree of misalignment between the latent subspaces of the current task t and the most similar task k, where feature reuse could be detrimental to the performance. Conversely, higher values of sim(t, k) suggest strong aligmment of these subspaces, which may lead to more effective positive knowledge transfer.
3.4 Feature Adaptors in the Weight Space
We observed empirically that learning U m alone may not be fully sufficient to model Px(x)). Therefore, we also introduce additional fcature adaptor blocks on the weight space of the generator to obtain g'. Since the latent subspace would have already captured some characteristies of the datasets, the nunber of fealure adaplor parameters to be learned would be lesser (Tab. 2). Theoretically, the necessity of feature adaptors arises from the limitation of dictionary learning alone generating all possible task subspaces. Specifically, dictionary learning may not be sufficient to capture the full range of task-specific features, the base generator may lack the required features that can be modulated to represent the target. features of semantically distinct tasks. By introducing feature adaptors, we extend the capacity of the base generator to learn additional features that are not present cannot be obtained through modulation of the dictionary features alone.
Let 5t denote the m style block within the generator g' for task t. We present an approach to compute the activation map f which essentially corresponds to the output of the style block S Following Sec. 3.3 we obtain the most similar task & and assess the potential negative forward transfer when considering the utilization of parameters from task k learn the parameters of the current task. Let @ represent the whether the
generator corresponding to task k. The computation of activation map f varies depending task & leads to positive forward transfer or not. Let consider the scenario in which task k results in a negative forward transfer. In such instances, we refrain from using the parameters of task k, and the activation map ft corresponding to task is obtained using only styleblocks of g as in equation G Alternatively, in case of negative transfer, one could resort to identifying the next most similar task, estimate its nature of forward transfer and use its parameters for facilitating forward transfer subject computationa and time constraints.
Sth = (Jth + ain x oh (Jm-1)
(6)
Here, 6n corresponds to the trainable feature adaptor which is 1 x 1 Convolution layer. Both oi and scaling cocfficient af are learnable and jointly learned with the latent dictionary.
When task results in a positive forward transfer, we utilize the parameters of task k to determine the activation map ft. through the following process: :
f Stt (f--)) + an (Sm-))

---

Published in Transactions on Machine Learning Research (12/2024)
where
Sth (fm²) -mm=
k +a Om (m
activations from similar tasks of &t (if any)
In this scenario, the feature adaptor block & is designed acquire additional information that is not present in G By adopting this approach obtain feature activation maps, ensure that the relevant. parameters reused for the current task, while any irrelevant parameters discarded. Additionally, given the recursive nature of the computation, it ensures that the parameters of the tasks similar to task & are also employed in obtaining the feature activation maps for the current task t.
We follow the training paradigm in Karras et al. (2020a), which includes adversarial loss L1. We also utilizo the Perceptual Path Regularizer (PPL) Karras et al. (2020b) and R1 regularization Mescheder as in Karras al. (2020b) ensure smoothness and facilitate better convergence.
2018
(. 9) E (x)r (x) llog (x)(
(8)
log D (g(w"))]
We provide comprchensive overview of the StyleCL training process in Algorithm 1
Algorithm StyleCL: Training Procedure
Input: { Stream of T datasets.
Output: Learned parameters: b generator g and adaptors {T')
1: Train AlyleGAN2 with dataset X2 to obtain ga
for t= do
Initialize discriminator v., Ut, b' Optimize U and b' using (D g¹) in equation
5: Determine most similar task kt using equation to get G
6: for each training iteration do
7: Sample latent vector w' using equation and equation
8: Optimize U and b' using C (D g%) as in equation end for
10: Calculate task similarity sim(t, k) using equation
11: Initialize feature adaptor parameters de and az.
12:
if sim(t, k) 0 then
13:
Obtain f of go using equation
14:
Initialise U b² using weights from step 8
15:
else
16:
Obtain f G using equation 6
17: Initialise Us, be using weights from step
18: end if
19:
Optimize U bt, and of using CI(D,, G¹) as in equation
20:
end for
4 Experiments and Results
4.1 Baselines and Metrics
We employ StyleGAN2 Karras al. (2020a) as the base architecture for all experiments. StyleCL comparod GAN Memory, CAM-GAN with task similarity learning, and MerGAN. We implemented all the basclines with the StyleGAN2 backbone to ensure fair comparison. Fvaluation metries include the Fréchet Inception

---

Published in Transactions on Machine Learning Research (12/2024)
Task
Task
Task 3
Task 4
Task 7
Ours
CAMGAN
GAN Memory
MeRGAN
Real Data
xl
X
Figure 2: Qualitative Evaluation: Generated Images by StyleCL compared against various basclines across six distinet lasks, including the initial base lask. Each row showcases results from different generative method, while each column displays the progression of generated samples, illustrating the model's capacity to retain and refine image quality throughout the sequence of tasks. This visual representation highlights the competence of StyleCL in maintaining ligh-iielity generation across diverse datasets.
Distance (FID) Hensel et al. (0017). Density, and Coverage Nacem et al. (2020). which are pivotal for assessing the quality and diversity of generated images. Additionally, we examine the computational and memory overhead using FLOPs and parameter count Dehghani et al. (2022), essential for evaluating the scalability of contimual learning approaches. As pointed out in Dehghani et al. 20022). data from major cloud service providers indicate that 85-90% of ML workload is in inference processing, considering this we measure FLOPs during inference stage using fveore package
For comprehensive definitions of these metrics, please refer to Appendix B.2.
4.2 Results for perceptually distant tasks
Adhering to protocols established in CAM-GAN and GAN-Memory, our initial model training employs the CelebA-HQ Karras et al. (2018) dataset. This is followed by sequential training six datasets that are significantly varied in visual content, thus "perceptually distinct': Oxford 102 Flowers Nilsback & Zisserman
(2008) LSUN Church Yu et al. (2155). LSUN Cats Yu et al. (2015), Brain MRI Cheng et al. (2016). Chest X-Ray Kermany et al. (208)), and Anime Faces] Qualitative comparisons of generated samples (Fig. 2
demonstrate that StyleCI consistently produces superior image quality compared to the basclines. We provide
additional qualitative results Appendix B.2 of supplementary materials.
Quantitative results, detailed in Tab. D confirm that StyleCL surpasses the baseline methods on most fronts according to FID, Density, and Coverage metrics. Additionally, Tab. illustrates StyleCL's reduction in parameters and the marginal increase in FLOPs, training and inference time for each of the methods. We provide an expanded version of scalability metrics in Appendix B.2 of supplementary materials, Despite lacking specific adaptation modules present in methods like CAM-GAN, StyleCL efficiently modulates the latent space, achieving high-quality generation with fewer parameters. While MerGAN maintains static

---

Published in Transactions on Machine Learning Research (12/2024)
parameters and FLOP counts, it sacrifices generation quality for previous tasks issue StyleCL circumvents with its no-forgetting approach.
Methods
MerGAN
GAN Memory
CAM-GAN
StyleCL
Metrics
FID J
D
Cov
FID +
D
Cov t
FID
f
Cov
FID
f
Cov
Dataset
Flowers
45.14
0.6
0.49
23.97
0.73
23.38
0.89
18.48
0.67
0.77
LSUN Church
31.41
0.56
0.18
37.9
0.30
0.11
24.25
0.20
0.17
17.36
0.59
0.41
LSUN Cat
53.52
1.10
0.20
53.22
0.86
0.32
52.59
0.62
0.22
34.43
1.15
0.41
Brain MRT
78.80
0.16
0.29
45.78
0.32
0.55
31.26
0.18
0.77
29.42
0.38
0.82
Chest X-Ray
38.51
0.13
0.11
58.82
0.23
0.3
24.81
0.36
0.73
25.83
0.55
0.75
Anime
39.83
0.35
0.09
16.20
0.63
0.38
21.52
0.50
0.27
12.38
0.62
0.39
Table 1: Quantitative performance metrics of StyleCL compared with McrGAN, GAN Memory, and CAM- GAN across six datasets. Fréchet Inception Distance (FID), Density (D), and Coverage (Cov) are used evaluation metrics, with the ideal direction of metric improvement indicated in parentheses. The table clearly demonstrates the superior performance of StyleCL in terms of lower FID scores and higher Density and Coverage across most datasets, signifying better image quality and greater diversity in generated samples.
Algorithm
Params Increase per Task
FLOPs Increase
Training Time (s)
Inference Time (ms)
(%)
(per iteration)
+
GAN Memory
4.21M
15.7
230
7.32
CAM-GAN
1.52M
23.32
180
4.74
StyleCL
1.08M
4.1
110
3.92
Table 2: Comparison of our approach (StyleCL) against the baselines GAN Memory and CAM-GAN regarding parameter increase per task, percentage increase in FT.OPs, training and inference time.
(a) Real data samples (b) Real data samples (c) Generated samples (d) Generated samples (e) Generated samples from 5
from X
from 2
a
k
Figure 3: Qualitative illustration of forward transfer in StyleCL: Fig. 3a and Fig. 3c corresponds to
real and generated samples from current task X StyleCL employs feature adaptors from previous tasks (samples of which are shown in Fig. 3bb to generate shared features across tasks (Fig. 3db.
Meanwhile, it
utilizes newly added feature adaptors exclusively for the unique features of the current tasks (Fig. 3el.
4.3 Results perceptually similar tasks
In order to evaluate the forward transfer capability of StyleCL, consider six varieties of butterflies from
ImageNet to create sequence of perceptually similar generation tasks, X' to 26, We consider 2 scenarios (a) StyleCL that enables forward transfer by considering the generator of the most similar previous task, and (b) StyleCL with parameter sharing only with the base task g² (without forward transfer). Tab. summarizes the results for both scenarios. Additional comparisons for more diverse datasets are given in section B.4 of the supplementary material. We observe improved performance on most datasets for scenario (a) compared scenario (b), confirming the benefit of forward transfer, inherent in our method. Also, the amount of knowledge that could be reused varies (positive negative forward transfer) which leads to varying degrees

---

Published in Transactions on Machine Learning Research (12/2024)
of improvement. As observed from Tab. in case of 2 sim(t, < O indicates potential negative forward transfer and hence when the model is forced to reuse the most similar task, it results in performance drop. This empirically validates our characterization of the nature of forward transfer by using sim(t, k). In such cases, we prevent nogative forward transfer by avoiding parameter reuse from the most similar task.
To qualitatively evaluate the forward transfer capability of our approach, StyleCT, we train it on dataset. X shown in Fig. 3a using the generator of the most similar previous task, X whose samples are shown in Fig. [b] The generated samples are illustrated in Fig. 3c To analyze the individual contribution of current and previous feature adaptors in StyleCL, separately disable their individual contribution by setting a and o to 0 in equation 7 The corresponding generated samples are illustrated in Fig. 3d and Fig. [c] Our results show that d% is reused to caplure shared characteristics of X and 2 such as shape and background (as seen Fig. 3dp. whereas newly introduced feature adaptors oi capture features unique to X such as the orange color of the wings (as seen in Fig. 3el. These findings confirm that StyleCL enables forward transfer by reusing knowledge from previous tasks.
X²
A
X4
26
2
StyleCT, without transfer
28.96
35.68
20.90
27.51
31.84
StyleCL with transfer
21.86
37.38
18.87
23.18
31.00
sim(t, k)
0.59
-0.02
0.21
0.35
0.37
Table 3: Comparison of StyleCL with and without forward transfer stream of perceptually similar datasets denoted as X 2 < i < 6. Results are reported in terms of FID, where lower values indicate better performance.
5 Analysis and Ablations
(a) Generated samples before training
(b) Only Latent Subspace learning
(c) Only Feature Adaptor learning
(d) Feature adaptors disabled on StyleCT.
(e) Both Dictionary and Feature adaptors.
Figure 4: Ablation Study on Feature Adaptors and Latent Dictionary. Fig. 4a shows initial generations without training. Learning only the latent subspace (Fig. 4bb or the feature adaptors (Fig, 4c captures partial data characteristics. The combined approach (Fig. 4ej yields high-quality, artifact-free samples. Disabling feature adaptors during StyleCL inference (Fig, 4d] still preserves certain data traits, such as color and background, demonstrating the effectiveness of the latent dictionary in capturing the essence of the data.
5.1 Analysis of Task Pask-Specific Latent Subspaces
The learned task-specific latent dictionary vectors, denoted as UA characterizes the task-specfic latent subspaces and is pivotal for choosing the most similar lask and averting negative forward transfer. We employ t-SNE visualizations to empirically validate the capacity of these latent subspaces to encapsulate task semantics. Utilizing equation [ and equation 2 we sample latent vectors w from the latent subspaces for a curated set tasks, aiming show that vectors from related tasks congregate, whereas those from unrelated tasks diverge.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Specifically, we select two semantically similar Butterfiy datasets (refer to Sec. 4.3] and contrasting Brain- MRT task (see Sec. 4.2 for visualization. Fig. 5 depicts the t-SNE plots, where one can observe the clustering of latent vectors within the space; vectors from related tasks (Butterfly in blue, Butterfly 2 in orange) are proximal, whilst vectors from the unrelated task (Brain MRI in green) are isolated. This spatial distribution substantiates our hypothesis that the learned latent subspaces indeed reflect task semantic similarities and supports their utility identifying similar tasks and discerning the nature forward transfer.
Dataset
Butterfly
Butterfly 2
Brain MRI
Butterfly
1
0.6
0.14
Butterfly 2
0.6
1
0.15
Brain MRI
0.14
0.15
Table Average cosine similarity dictionary vectors for similar and dissimilar tasks
Butterfly Butterily 2 Brain MRI
Figure 5: t-SNE visualization of latent vectors: Demonstrating the distinction and proximity of latent vectors within learned subspaces. Latent vectors from semantically related Butterfly datasets are depicted blue and orange, showcasing their clustering, while vectors from the dissimilar Brain-MRI task are in green, indicating significant separation. This illustrates the latent spaces' capability to differentiate and group tasks based semantic relatedness.
Furthermore, the parameter K, representing the number of dictionary vectors per style block, is critical design variable. An in-depth analysis of K's influence on the generative quality and the selection rationale behind it is presented in Section C.2 of the supplementary materials.
5.2 Ablation Studies on Latent Dictionary and Feature Adaptors
To dissect the individual contributions of our method's components, we conduct ablation studies focusing on the learned latent dictionary and feature adaptors. StyleCL is trained using each module independently and the outcomes are qualitatively assessed in Fig. 4 The isolated effects of dictionary learning (Fig. 4b)
and feature adaptor learning (Fig. 4cb yiold lower quality generations compared utilizing both modules landem (Fig. Ael.
Furthermore, explore the influence of dictionary learning StyleCL during the inference stage. By dcactivating the current feature adaptor's output and retaining the rest of the model structure, we observe that the generated samples (Fig. 4dl retain certain altributes, such as color and background, akin to the complete StyleCL-generated samples (Fig. 4ep. This phenomenon substantiates our hypothesis that StyleCL can effectively approximate the target data distribution with fewer parameters, primarily attributed to the integration of dictionary learning. qualitative comparison presented in Appendix B.3, complemented by further qualitative evaluations across additional datasets.
11

---

Published in Transactions on Machine Learning Research (12/2024)
5.3 Impact of Base task StyleCL
The generator Gt, pivotal for all subsequent lasks, is initially trained on the X dataset. To understand how different base task affect the efficacy of StyleCI, we conduct experiments with the base task being Brain MRI and ImageNett datasets. We assess the performance of StyleCL on a data sequence comprising CelebA-HO, Flowers, LSUN Church, and Chest X-Ray datasets under these conditions. The results, detailed in Tab. 5 highlight a notable improvement. performance when the base task is a heterogeneous dataset like ImageNet-1K3 as opposed to starting with a more specializod datasct like Brain MRI. This indicates that StyleCL significantly benefits from initial training on dataset with broader diversity. We believe that this could be attributed to diverse features captured by G which is subsequently useful for other tasks.
Base Task
CelebA-HQ
Flowers LSUN Church Chest X-Ray
Brain- MRR 22.82
31.98
55.45
29.93
ImageNet
15.86
14.25
11.71
23.54
Table 5: Performance of StyleCL with different base task X², measured by Fréchet Inception Distance (FID) across four datasets. Lower FID values indicate better image quality and higher similarity to the target dataset. ImageNet initialization outperforms Brain-MRI across all datasets, affirming the advantage diverse initial training.
6 Conclusions and Future Work
We introduce StyleCL, a novel generative continual learning approach utilizing the latent space of StyleGAN. This strategy employs task-specific latent subspaces and feature adaptors, circumventing the need for direct feature weight modifications and thus minimizing computational and memory demands while boosting performance. Uniquely, it also prevents negative forward transfer, previously unexplored aspect in this field. Future work aims to extend the applicability of StyleCL to various architectures and generative models and improve contimual learning through shared latent dictionaries. We will also focus on refining StyleCL for use in scenarios without clear task identification, broadening its practical real-world application. Our work opens new avenues for the efficient and effective training of gencrative models, with anticipated enhancements solidifying the role of continual learning in generalive model evolution.
Acknowledgments
Dr. Prathosh A.P would like to acknowledge the support provided by the Indian Institute of Science, for setting up the compute infrastructure. He also would like to thank TCS Research for collaborations.
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In CVPR, pp. 4432-4441, 2019.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1 207, 2018.
Jun Cheng, Wei Yang, Meiyan Huang, Wei Huang, Jun Jiang, Yujia Zhou, Ru Yang, Jie Zhao, Yanqiu Feng, Qianjin Feng, et al. Retrieval of brain tumors by adaptive spatial pooling and fisher vector representation. PloS one, 11(6):e0157112 2016.
WU Chenshen, L HERRANZ, LIU Xialei, et al. Memory replay gans: Learning to generate images from new categories without forgetting. In NeurIPS, pp. 5966-5976, 2018.
Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin. Gan memory with no forgetting. NewrIPS, 33:16481-1644 2020.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Delying forgetting in classification tasks. IEEE TPAMT, 44(7):3366 3385, 2022. doi:
Mostafa Dehghani, Yi Tay, Amurag Arnab, Lucas Beyer, and Ashish Vaswani. The efficiency misnomer. In ICLR, 2022. URL
Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen- Chen-Change Loy, Wayne Wu, and Ziwei Lin. Stylegan-human: A data-centric odyssey of human generation. arXiv preprint, arXiv:220.111823, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, volume 27, 2014. URL https:
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NewrIPS, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840 6851, 2020.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. URL
Tero Karras, Milka Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN, In Proc. CVPR, 2020b.
Danicl S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diagnoses and treatable diseases by mage-based deep learning. Cell, 172(5):1122 1131-111,,, 2018.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Jean-Frangois Goudou, and David Filliat. Generative models from the perspective of continual learning, 2019. URL Intpp:://openrerlew..ee net/ forum?id=S1ePtjjock]
Kevin J Liang, Chunyuan Li, Guoyin Wang, and Lawrence Carin. Generative adversarial network training is a continual learning problem, 2018.
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-procision semantic image editing. In NeurIPS, 2021.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML), 2018
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7176-7185, Proceedings of Machine Learning Research., 13 Jul 2020. URL https
/pprceedings. laaggss.ml... mlr. wttpp//eeee//eeeeeeeeeeee
html
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In TCML, volume abs/2330.09615, 2023. URL http://aaaiiv.orrraa///////
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative adversarial nets, 2017.
Juwon Seo, Ji-Su Kang, and Gyeong-Moon Park. Lfs-gan: Lifelong few-shot image generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2023.
Sakshi Varshney, Vinay Kumar Verma, PK Srijith, Lawrence Carin, and Piyush Rai. Cam-gan: Continual adaptation modules for generative adversarial networks. NeurlPS, 34:15175 15187, 2021.
Vinay Kumar Verma, Kevin J Liang, Nikhil Mehta, Piyush Rai, and Lawrence Carin. Efficient feature transformations for discriminative and generative continual learning. In 2021 IEEE/CVE Conference Compuler Vision and Pallern Recognition (CVPR), pp. 13860-13877 2021. doi: D01100//VVP44643. 2021.01365.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint Xrx: 03365, 2015.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. arXiv preprint 1801 07892, 2018.
Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image generation. In ICCV, pp. 2759 2768, 2019.
Mengyao Zhai, Lei Chen, Jiawei He, Megha Nawhal, Frederick Tung, and Greg Mori. Piggyback gan: Efficient lifelong learning for image conditioned generation. In ECCV, pp. 397-413. Springer, 2020.
14


Reviews:
[{"review_id": "O7Oqd4yOG6", "reviewer": "Reviewer_VJrr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Their paper is relatively clearly written, although I think there are some points that I wish to be revised. At least, I like that they included pseudocode and the overall figure to help understand readers.\n2. They emphasized their novelty throughout Section 3, which helped me easier to understand their contributions (such as finding out task similarity using cosine measure) \n\n\nWeakness:\n\n1. Though it seems like the authors tried their best to explain their models using images, I think it would be better if there's more detailed explanation for each subfigure in Figure 1. It would be a great idea to split this one figure into three, and if possible make some 'flow arrow' that directs 'this $G^t$ in Figure 1(a) (the largest concept) is constructed as Figure 1(b) ...' with a bit more detailed explanation. For example, what is that $m \\times d \\times K$ cube? What is the dimensionality of $f_m^t$ and the style block $S_m^t$? (For the beginners of this field it might be tough to understand what is going on. Maybe adding StyleGAN basics in the Appendix, for completeness?)\n\n2. What happens if the first dataset $\\mathcal{X}_1$ follows a skewed distribution? It seems like if the algorithm fails to find another similar task, then it relies on the first model for initialization, but if it has some.... poor quality, then I think it might affect the overall result. \n\n3. To be honest, now I have difficulty in understanding your Section 3.4. \n3-1) Why do we need to talk about style block $S_m^t$ and activation $f_m^t$? Why $U$ is not enough? The first paragraph of 3.4 is not enough for me to understand what is going on. Is it empirical things? \n3-2) It literally copies the parameters from $k$-th trained model in Eq. (7) without considering similarity (I think the new model should copy less if the similarity is low).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose to leverage the versatility of StyleGAN to enable long-term learning of StyleGAN without forgetting. Their method SOTA approaches while employing fewer parameters and FLOPS. Not only that, but their framework prevents negative forward transfer. According to what they said, StyleCL is the only method capable of averting negative forward transfer.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the weaknesses section above.\n\nMinor typo (maybe):\n\nSection 3.1: subsequent datasets dataset $X^t$?\nSection 3.2:  StyleGAN2 Karras et al. (2020a) --> StyleGAN2 (Karras et al., 2020a)?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "drc9DNNtTH", "reviewer": "Reviewer_XjRr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n* The lifelong learning problem and the proposed solution are practical and interesting. \n* The algorithm and mathematical formulations are clear.\n* The presented experiment results demonstrate that the proposed method is effective, especially the quantitative part.\n\n\n### Weaknesses:\n\n* As for Fig.2 for qualitative evaluation, it seems that no obvious difference presented between four methods.\n* The lifelong learning ability of the proposed method needs to be further clarified or evaluated. The proposed method targets to solve the lifelong learning problem, so that the results relative to $t$ is a very important aspect in quantitative evaluation. In the paper, Tab. 1 demonstrates that the proposed method achieves good performance throughout the process of increasing $t$. However, it is not clear how much the starting point(when t=0) contributes. These baselines perform poorly from the initial task to the final task compared to the proposed method, it is difficult to determine whether this is due to the poor performance of the generative model itself or the poor lifelong learning ability.\n\n\n### Minor problems:\n\n1).Define StyleCL before use it in abstract. \n2).  Problem setting - > 3.1 Problem Setting", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for lifelong learning of styleGAN, which can reuse the trained parameters on previous tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rGCcVd8ilP", "reviewer": "Reviewer_oQcf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\n+ Paper is well written and all components are explained in a comprehensible way\n+ Results bear out the claims and code is provided to validate. \n+ The idea of 'mixing' latent subspaces and obtaining similar task scores is intuitively appealing. \n\nWeaknesses\n-----------------\n- While I can see that the results are convincing, there are some aspects that concern me. The theoretical reasoning for comparing biases to measure similarity is not clear. Likewise, using the Gram-Schmidt procedure is sensible but looks arbitrary - here we are omitting the bias vectors as well.  \n- The impact of 'negative' subspaces could be quantified. \n- I am not very sure how the feature adaptor helps. This is again something that makes intuitive sense, but perhaps needs more insight.\n- The aspect of catastrophic forgetting is mentioned, but no evidence is given in the paper to show that the fittings address this issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of lifelong or continual learning in the context of GANs. By lifelong learning, it refers to learning over a number of datasets and tasks, which might differ in their distributional characteristics. In this connection, catastrophic forgetting is also pertinent, as the network might forget what it has learnt when new tasks or datasets arrive in the sequence. \n\nThe method consists of building upon StyleGAN2 to construct a dictionary $U_m^t$ for each task $t$, consisting of latent subspace vectors $u_{mk}$ for each of the StyleGAN codeblocks $m$. To create the latent style vector to pass into StyleGAN2, they sample from each of the latent subspace vectors, and add a bias term to it. After learning on the most recent task, it is compared with the previous tasks by using a similarity metric that checks the difference between the bias vectors. This gives us the 'most similar' generator. This most similar generator is then evaluated for fitness to see if it contributes positively or negatively in the task at hand. If it is positive, the generator in question is picked and its subspace parameters are updated. If not, they go with the 'base' generator. To further improve results, the model also has an adaptor component in weight space. \n\nEvaluations are presented over a number of datasets, ablations and baseline methods. Mainly, they show that their method shows improved results in FID, density and coverage. Ablations on model components are also shown.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can the authors expand on the reasoning for picking the 'bias' term to get similarity metric? On similar lines, some more explanation or evidence for negative subspaces and feature adaptor would help. This could be, for instance, from a carefully designed experiment that shows the effect of these fittings.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2JSMdxZzBD", "reviewer": "Reviewer_nhoR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tContinual learning for generative models is important and under-explored. The topic is potentially of interest to the TMLR community.\n\n•\tThe motivation of subspace learning based on task similarity is reasonable based on the idea of knowledge transfer in continual learning.\n\n•\tStyleCL shows good performance on a sequence of perceptually distant tasks compared to other GAN-based benchmarks based on the provided visualizations and quantitative results, which serves as convincing evidence for the applicability of the proposed method.\n\n•\tThe authors conduct a variety of ablation studies to help explain the rationale of the key components of StyleCL, which is commendable.\n\n•\tThe paper is generally easy to follow and well organized.\n\nWeaknesses:\n\n•\tThe rationale behind the recursive expression of $S_m^k (f_m^{t-1})$ in Equation 7 is not clearly explained. To be specific, how can we guarantee that activations from similar tasks of $k$ also result in a positive forward transfer?\n\n•\tLines 6-8 in Algorithm 1 are somewhat confusing. After learning $\\textbf{U}^t$ and $\\textbf{b}^t$ using $\\mathcal{L}_1 (D\\_{\\psi}, \\mathcal{G}^1)$, why cannot we determine $k$ based on the bias vector $\\textbf{b}^t$ and then immediately calculate $sim(t,k)$ based on the dictionary $\\textbf{U}^t$ and its projections? That is, why cannot we combine the steps of identifying the most similar task and preventing negative forward transfer? \n\n•\tDespite the reduction in parameters, the training process of StyleCL does not seem to be very efficient. Specifically, in Algorithm 1, $\\textbf{U}^t$ and $\\textbf{b}^t$ are optimized using the GAN loss 3 times for each task $t = 2, …, T$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes a continual learning paradigm for StyleGAN over a variety of tasks with versatile visual content, called StyleCL. Specifically, the input images are first used to fit a base generator $\\mathcal{G}^1$. The latent subspace is then initialized and learned on $\\mathcal{G}^1$. Next, for all subsequent tasks, the authors propose to identify the most similar task to the current one and then calculate the cosine similarity between the dictionary vectors and their projections onto the latent subspace to prevent negative forward transfer. Additionally, the authors introduce a learnable feature adaptor in each style block to compute the feature activations. The authors then test StyleCL on a sequence of visual tasks to showcase the adaptability of StyleCL to different visual contents across tasks. Ablation studies are also conducted to investigate the task semantics in the latent dictionary vectors, the necessity of feature adaptors, and the role of base generator $\\mathcal{G}^1$.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "•\tFor better illustration, it would be better to visualize the discriminator $D_{\\psi}$ in Figure 1.\n\n•\tMore clarifications on lines 6-8 in Algorithm 1 (see the 2nd point under weaknesses) would be very helpful.\n\n•\tTo empirically demonstrate the training efficiency of StyleCL (see the 3rd point under weaknesses), it would be helpful to compare the training and inference times with those of other GAN-based benchmarks in the experiments.\n\n•\tIn Figure 5, why do a small portion of t-SNE vectors of Brain MRI lie within the cluster of vectors of butterflies? Does this mean $\\textbf{U}^t$ still detects some similarities between these two types of visual tasks?", "broader_impact_concerns": "I do not see immediate ethical concerns about the work presented, though I think the authors should briefly discuss the potential limitations of the proposed method and the future direction of research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In a private discussion, one of the reviewers expressed the following (I'm paraphrasing):

The reviewer had difficulty understanding the paper’s approach, even after the revision, particularly the theoretical need for style blocks and activation functions. The reviewer mentioned that the paper tried to explain these aspects based on empirical observations rather than theory. They also mentioned that the similarity measure is used indirectly through the model selection, and so couldn't understand precisely how it affected the results. Consequently, they were leaning toward rejection, but said that they would be ok if this paper got accepted.
Decision: Accept (poster)