Learning Input-agnostic Manipulation Directions in StyleGAN with Text Guidance
Authors: Yoonjeon Kim
Hyunsu Kim
Junho Kim
Yunjey Choi
Eunho Yang
Venue: ICLR 2023 poster (2023)
Keywords: image manipulation
deep learning
generative adversarial network

Abstract: With the advantages of fast inference and human-friendly flexible manipulation, image-agnostic style manipulation via text guidance enables new applications that were not previously available. The state-of-the-art text-guided image-agnostic manipulation method embeds the representation of each channel of StyleGAN independently in the Contrastive Language-Image Pre-training (CLIP) space, and provides it in the form of a Dictionary to quickly find out the channel-wise manipulation direction during inference time. However, in this paper we argue that this dictionary which is constructed by controlling single channel individually is limited to accommodate the versatility of text guidance since the collective and interactive relation among multiple channels are not considered. Indeed, we show that it fails to discover a large portion of manipulation directions that can be found by existing methods, which manually manipulates latent space without texts. To alleviate this issue, we propose a novel method that learns a Dictionary, whose entry corresponds to the representation of a single channel, by taking into account the manipulation effect coming from the interaction with multiple other channels. We demonstrate that our strategy resolves the inability of previous methods in finding diverse known directions from unsupervised methods and unknown directions from random text while maintaining the real-time inference speed and disentanglement ability.

Published as a conference paper at ICLR 2023
LEARNING INPUT-AGNOSTEC MANIPULATION DIREC- TIONS IN STYLEGAN WITH TEXT GUIDANCE
Yoonjeon Kim', Hyunsu Kim², Junho Kim², Yunjey Chof,Eunho
Eunho Yang 1,3-
Korea Advanced Institute of Science and Technology (KAIST),, NAVER LAB,, AITRICS
yoonkim31 30kaist ac.kr kr hyunsul125 www.navercoop................... com
jhkim. a @navercorp. com yunjey wttpeyy.cc.@naaercrrp..oo com eunhoyGkaist. .c.
ABSTRACT
With the advantages of fast inference and human-friendly flexible manipulation, image-agnostic style manipulation via text guidance enables new applications that were not previously available. The state-of -fate-of-the-art text-guided image- aagnotic manipulation method embeds the representation of each channel of StyleGAN independently in the Contrastive Language-Iman provides it in the form of Dictionary to quickly find out the channel-wise manip- ulation direction during inference time. However, in this paper we argue that this dictionary which is constructed by controlling single channel individually is limited to accommodate the versatility of text guidance since the collective and interactive relation among multiple channels are not considered. Indeed, we show that fails to discover large portion of manipulation directions that can be found by existing methods, which manually manipulates latent space without texts. To alleviate this issue, we propose novel method Multi2One that learns a Dictionary, whose entry corresponds to the representation of a single channel, by taking into account the manipulation effect coming from the interaction with multiple other channels. We demonstrate that our strategy resolves the inability of previous methods in finding diverse known directions from unsupervised methods and unknown directions from random text while maintaining the real-time inference speed and disentanglement ability.
Pre-training (CLIP) space, and
INTRODUCTION
Wide range of generative models including adversarial networks (Goodfellow al., 2014; Karras et al., 2018; 2019; 2020b; Kim et al., 2022; Kim & Ha, 2021; Karras et al., 2021), diffusion models (Dhariwal & Nichol, 2021), and auto-regressive models (Dosovitskiy et al., 2020; Chang et al., 2022) have demonstrated notable ability to generate high-resolution image that is hardly distinguishable from real images. Among these powerful models, style-based GAN models (Karras et al., 2019; 2020b) equipped with unique latent space which enables style and content mixing of given images, manipulation of local regions (Wu et al., 2021), and interpolation between different class of images (Sauer et al., 2022).
In this paper, we focus on the image manipulation based on the pre-trained StyleGAN, considering the unique advantages mentioned above and its popularity. Based the steerability in the latent space of StyleGAN, researchers have put tremendous effort on finding a direction that causes semantically equivalent change the entire samples of image. In this work, we refer to such latent direction as global direction. Unlike local direction which is a sample-wise traversal direction found by iterative optimization using single image (Local Basis (Choi et al., 2021) and Latent Optimization of StyleCLIP (Patashnik et al., 2021)), global direction allows fast inference and is applicable to any images once found using supervised (Jahanian et al., 2019), unsupervised (Shen & Zhou, 2021; Wang & Ponce, 2021; Härkönen et al., 2020; Voynov & Babenko, 2020), or text-guided methods (Global Mapper & ClobalDirection' of StyleCLIP (Patashnik et al., 2021)).
In order to distinguish it from global direction, which means finding input agnostic directions, express the method proposed StyleCI.IP IP this way

---

Published as conference paper at ICLR 2023
Source
Manip.
Source
GD
Ours
Source
GD
Ours
GANSpace
70th Direction
Ours
Attractive Malc
Innocent Eyes
Smiling
Grinning
Evil Queen
Sexy Man
Grumpy Face
Shiny Hair
Smirking
cay
(h)
Figure 1: (a) Manipulation by the 70-th direction from GANspace generates 'a with wide smile'. GlobalDi cevirrcciion (GD), highlighted in red. fails to reproduce similar result even when provided with various text guidances. (b) Manipulation results by randomly selected text, demostrating that GD has insufficiiet manipulation ability. Same number of channels are manipulated in both methods.
Among them, the text-guided methods have a unique advantage in that they can naturally provide the flexibility of manipulation through the diversity the given driving text without human supervision to discover the direction in the latent space. However, in this paper we argue that contrary to this common belief on text-guidance, the standard method (Patashnik al., 2021) for text-based StyleGAN manipulation surprisingly fails to even find the manipulation directions that are known to be found in unsupervised approaches (Härkönen et al., 2020; Shen Zhou, 2021) (see Fig. I(a) for examples). In addition, manipulation on a large number of randomly selected texts (see Fig. 1(b) for examples). We hypothesize that the failure is due the na've approach that only considers change of image caused by a single channel in StyleSpace, neglecting diverse directions that are visible only when manipulating multiple channels
also show that this standard method does not properly perform
whole.
In order to address these issues, we propose a novel method, named Multi2One, of learning a Dictionary that manipulate multiple channels corresponding given text. However, here since there is no paired ground truth of text and manipulation direction corresponding to the text, we embed the directions found by existing unsupervised methods into the CLIP space and learn dictionary to reproduce them in the CLIP space. Note that this has more meaning than simply reproducing the known directions derived by unsupervised methods. As the dictionary learns the relationship between channels in StyleSpace and CLIP space, we can find manipulations that could not be found with unsupervised methods using diverse text inputs.
Through extensive experiments, we confirm that contrary to the state-of-the-arts method (Patashnik al., 2021) which explicitly encoded every single channel, our multi-channel based strategy not only excels in reconstruction of unsupervised directions but also in discovery of text-guided directions.
RELATED WORK
Style-based Generators Generators of style-based models (Karras et al., 2019; 2020b;a; 2021) are built upon the progressive structure (Karras et al., 2018) that generates images higher resolution in deeper blocks. The popularity of StyleGAN structure that has been employed in numerous number of researches comes from its ability to generate high-fidelity images, transfer styles to other images, and manipulate images in the latent spaces using inversion methods (Zhu et al., 2020; Roich et al., 2021; Tov et al., 2021; Collins et al., 2020). The latent spaces of StyleGAN used for manipulation intermediate space W and StyleSpace S (Wu al., 2021).
Unsupervised Global Directions Image-agnostic directions are latent vectors that create semanti- cally equivalent shift when applied to the latent space StyleGANs. In order to find such directions, SeFa (Shen & Zhou, 2021) performs PCA the first weight that comes after intermediate space W in pranted StyleGAN, deducing the principal components as the global directions. On the other hand, GANspace (Härkönen al., 2020) relies the randomly sampled latent codes in W and
2

---

Published as a conference paper at ICLR 2023
CL)P(()
$1
Style GAN
Φle
(
Little Mermaid
())
product
DiobeiDirrettion (ng.1)
9
8
GD
CLIP())
#1
Style
d1
dp
GAN
product
S
D Multi2One (400.4)
Source
Ours
Figure 2: A diagram depicting the framework of dictionary-baan -based image manipulation via text guidance. Our method, Multi20ne, is differentiated the previous methods in that learns dictionary for the text input. The proposed novel dictionary allows more flexible and expansive discovery of the manipulation direction with better result. (-)) in Dastmiiisettion is abbreviation of (-) of Eq. (I), The dictionary learning process of Multi20ne create DMuuuoue is illustrated in Sec. 4.
the eigenvectors from the latent codes proved to be global directions that share an image-agnostic modification ability.
Text-guided Image Manipulations Most of the text-guided image manipulation methods aim to find local direction (Kocasari et al., 2022; Xia al., 2021; Patashnik et al., 2021) which is an image-specific direction that applies to a single sample of image. Methods that find local directions using text guidance could be found in various models including GAN, Diffusion (Kim & Ye, 2021; Nichol et al., 2021; Avrahami et al., 2022; Liu et al., 2021), and Vision transformers (Chang et al., 2022). Two unique approaches for finding global direction using text guidance in StyleGAN are Global Mapper and GlobalDirection of StyleCLIP (Patashnik et al., 2021). Global Mapper finds an image-invariant direction for single text by optimizing fully connected layer. However, this method requires 10 hours of training time for every single text, making it less popular than Gl ClobalDirection in inference time using dictionary-baser framework that is applicable to any input image and text.
the other hand, Gl ElobalDirectton method offers a real-time manipulation
3 LIMITED COVERAGE OF STYLECLIP GLOBALDIREBS
METHOD
In this section, we briefly review the Gl Gloaall rect ii of StyleCLIP (Patashnik et al., 2021), which performs StyleGAN- based image manipulation using text guidance (Sec. 3.1). Then, we provide our key motivation that this state-of-the-ee text-guided manipulation method is surprisingly insufficient to fully utilize the manipulative power of StyleGAN (Sec. 3.2).
3.1 TEXT-GUIDED STYLEGAN IMAGE MANIPULATION
StyleSpace of StyleGAN The generators of StyleGAN family (Karras et al., 2019; 2020a;b) have number of latent spaces: Z, W. W+, and S. The original latent space is Z, which is typically the input noise
standard normal distribution. The generator transforms latent spaces W (Karras et al., 2019), W+ (Abdal et al., 2019), and S (Wu et al., 2021), sequentially. Recent study (Wu al., 2021) shows that StyleSpace S is the most disentangled such that it can change distinct visual attribute in a localized way. The number of style channels in is 6048 excluding toRGB channels for StyleGAN-ADA (resolution 1024*), and recent methods (Kocasari et al., 2022; Patashnik et al., 2021; ;; Wu et al., 2021) modify the values of the channels to edit an image. Our method also adopts StyleSpace and we use denote the total number of channels !? in S (that is, = S2, s,,]* E S and is a single parameter). For the convenience of explanation, the pre-trained generator G is re-defined with respect to StyleSpace as X Gs)) X is a generated image. The goal of StyleGAN-basee image manipulation via text is to find direction [81, ................................ ... $,.]" in StyleSpace S which generates an image Xodited = (s + 8) suitable for the provided text guidance f. Note that is the inverted style vector of image X, found via StyleGAN inversion methods such as Alaluf et al. (2021); Roich et al. (2021); 1); Tov et al. (2021). used for manipulation purpose in most cases. The main benefit of considering StyleSpace S in image manipulation that it does not change the undesired regions of the image when modifying a small number of style channels by its well disentangled property.
[0, I) into intermediate
3

---

Published as a conference paper at ICLR 2023
Table I: Measurement of CLIP similarity score cos(..) +) (1) between the manipulated image and the CLIP representation ΦCLI (-) of unsupervised direction a.
cos(ØCLIP(a)) Xunsup) cos(@cL)((t))
(10)
0.3295
0.2701
(30)
0.3558
0.3319
(50)
0.3687
0.2924
Text-guided Manipulation by StyleCLIP The GlobalDirecti (Patashnik al., 2021) is representative method of text-driven image manipulation that provides an input-agnostic direction with level of disentanglement. Intuitively, it computes the similarity between the imput text and each style channel in the CLIP space (Radford et al., 2021) to find the channels that should be modified given the text. In order to compute the similarity between the text and style channel, both should be encoded into CLIP. While the text guidance is trivially encoded via the text encoder of CLIPP as CL)P(t) E IRP, style channels in StyleSpace S need additional pre-processing for the embedding. GlobalDi rection proposes embed the manipulation effect of i-th style channel using the following mapping from the StyleSpace S to CLIP space:
(e,) == Eees (CLIP(G,(5 + e() CLIP(G.(s (se))) -eii]]
(1)
where the manipulation vector e, E Rn is zero-vector except for the i-th entry. Adding the manipulation vector e, to the original style vector indicates that only the i-th channel among channels in StyleSpace is manipulated. Note that ΦCLIP(e;) is also p-dimensional since the
CLIP encoder maps images generated by G.(.) into p-dimensional CLIP space. The above mapping in Eq. (l) enumerated across all channels in StyleSpace to create dictionary Deloba rect on == (ei), (CT,I ΦCLIP (e.)) E IRexn.
Finally, with this dictionary, the manipulation direction & R by Globa 1Di rection is given as the similarity score measured by the following equation:
D Diieetii CLIP(().
This overall manipulation procedure is visualized in Fig. 2.
In the following section (Sec. 3.2), we propose the evidence to the hypothesis that the single-channel encoding strategy with ÓCLIP (e,) to create the dictionary is the major bottleneck causing limited coverage issues that StyleCLIP suffers from.
3.2 COVERAGE ANALYSIS OF STYLECLIP GLOSALDIRECTION METHOD
In this section, we experimentally verify that GlobalDirection has limited ability to find manipulation directions, as described in Fig. I of the introduction. Toward this we show two empirical findings, which corresponds to Fig. ((a) and Fig. 1(b), respectively.
First, we show that many edits using unsupervised methods (Härkönen et al., 2020; Shen & Zhou, 2021) cannot be recovered by clobalDirect ion. For example, in Fig. 1(a). can observe that applying the 70-th GANspace direction manipulates the source image to become a man with wide smile, showing that pre-trained StyleGAN itself is capable of such manipulation. However, Globa 1Di rect i on (GD) in Fig. ((a) constantly fails to recover the same editing effect of the unsupervised direction, despite the variation on the text input such as 'smiling man', 'grinning man', 'smirking man', and 'man with coy smile'.
More quantitatively, we provide Tab. to show that GlobalDirection (Patashnik al., 2021) cannot effectively recover the directions found by unsupervised methods (Härkönen et al., 2020: Shen & Zhou, 2021). Scores in the table are the CLIP similarity between ΦCLIP (a) Eess (G. ) (G, (s - x and the modified image created by either Globalliiiiioooo bobalDirectio
(XGD: the construction of XGD is explained b or unsupervised methods (X (Xunsup). Note that ΦCLIP (ox) encodes the direction a into the CLIP embedding space as Gl oba D rection does for a single
"We performed the experiment here using the CiobalDirection method and test examples provided on the StyleCI.IP IP official site: http:///hhh...mmmmoaaassssskkkky
4

---

Published as a conference paper at ICLR 2023
Chaunel Index
Source
Manip.
Source
Manip.
(cur
Octr (o
Dereity
Dersity
4520 (single)
-7
1/2c
1/2n
(ctap(as)
4520-4426 (muitiple)
Density
Dersity
StykCLIP GD
1/2n
-1/2e
0
1/2r
XT
(50)
Cj
(a)
(b)
Figure 3: (a) Histogram of (L. + 8)) CL.P 8)) using single-channel manipulation
e, or multiple channel manipulation (50) (b) Manipulation results using channel indices 4520, 4426.
GlobalDirection (denoted as StyleCLIP GD) manipulated with text 'white hair'.
channel. From the table, we can confirm that GlobalDirection does not recover the similarity in CLIP space that the unsupervised methods have.
Here we illustrate the details on how we generate Xumsup and XGD for Tab. 1. First, the source image X x G,(ss is manipulated by each unsupervised direction E R" into X Xunsup G, (s + a Instead of naïvely using the direction a E R", we only use the channels whose magnitude the k-largest among channels in to reduce the entanglement that naturally comes with directions found by unsupervised methods (please refer to Appendix C to find the effect of using limited number of channels). We denote such sparsified directions as (k) E R and use them the experiment of Tab. 1. Then, to generate XGD = Gs 6), we compute á D DELooa ipirection CLIP(() E TR" as in Eq. (2). However, since there is no text corresponding to a and our objective in this experiment is to find direction à that may recover Xunsup by GlobalDirection we replace CLIP(() with (CLIP(( ( (ELes(CCIP (s CLIP)
a (*)]] Since ΦCLIP (ce indicates the manipulation effect of (*) encoded into CLIP space, it is used as substitute for the CLIP- encoded text guidance, CLIP((). Finally, compute the similarity score for 30 instances of source images and 1024 directions from SeFa and GANspace whose average is reported Tab. 1.
Secondly in Fig. 1(b), we provide several instances where the G1 oba 1 i rect i on of StyleCLIP fails to manipulate an image given randomly selected text. For additional examples on the failure cases of the standard method, please refer to Appendix A where provide the details on how we choose random texts and manipulated images using extensive list random texts.
Regarding the phenomena found in the above experiments, we hypothesize the crude assumption of clw.allireeeaeaaeaeaaa wwl.Dieieiioon manipulation solely based on single channel could fully represent the manipulative ability of StyleGAN leads to the limited coverage issue. To disprove this assumption of G1 oba Di rect on, in Fig. 3(a), we demonstrate that single-channel manipulation framework of G1 oba 1Di rection (using e;) leads improper construction of (CLII((e) in the dictionary Then in Fig. 3(b), we show an example where editing two channels has completely different effect from manipulating each of the channels individually, which proves the need for multi-channel manipulation.
D DelobsiDirection
In Fig. 3(a), we show that (CLI( E CLIP + e,)) CL,P es)- which forms the dictionary DelcssiDirection is not a robust representative of i-th style channel since there exists sample-wise inconsistency in the calculation of CLIP + +x)) CL,P e,)). On the (L
left side Fig. 3(a), the distribution of CLIP (G, CLIP the samples, (CL) are shown. As the samples show inconsistent angles over polar coordinate" when modified by single-channel manipulation e, we conclude that (CLIP(e,) may not be trust-worthy representative of i-th channel in StyleSpace. Thereby the basic assumption held by GlobalDirect ion which expects single channel manipulation using e; on any images to be semantically consistent in CLIP space, in fact, has numerous counter-examples showing inconsistency.
and its average over
AA more detailed explanation representing the samples as directions for an angle histogram is deferred to
Appendix due space constraints.
5

---

Published as a conference paper at ICLR 2023
CLIP
dg
de
din eyebeons Speric
(1)
di
da
da
Source
Plrt Speiie
Coniis KiD
0la
Figure 4: A concept diagram of our method in learning the dictionary D. Φ(-) is an abbreviation of $CLP S CLIP represents mapping from StyleSpace CLIP.
On the other hand, it is observable the right side of Fig. 3(a), that manipulation by unsupervised direction a (50), shows much consistent results.
Moreover, in Fig. 3(b), we show an example where single channel alone has completely different meaning from the manipulation by the combination of multiple channels. This supports our claim that the interaction between multiple channels should be considered, rather than individually encoding single channel each. In Fig. 3(b), manipulation by 4520-th 4426-th do not show "completely white hair" while the combination of the two shows white hair. Since GlobalDi Dirreciiion fails to learn that 4520-th also has role for creating white hair, it fails to successfully modify an image when text guidance is given as 'white hair'.
DICTIONARY LEARNING FOR SPARSE REPRESENTATION OF CLIP SPACE
As discussed Sec. 3.2, each style channel rather than the whole may not be meaningfully embedded in CLIP space. To address this issue, propose dictionary learning approach to find robust representation of styles that could be possibly a set of channels in StyleSpace S.
Ideally, given text input t and its CLIP encoding CLIP 12 would like construct dictionary D E R which finds a manipulation direction = D CLCI))( E R" as in Eq. (2). The direction á is expected to manipulate an image into G. &) which faithfully represents t. Let us for now assume that the 'ground truth' manipulation direction & E R" corresponding to the driving text L known. Then estimated direction
D2 CLIP(() should generate an image Xedited = Gs + D CLII((t)) close to Gs(s + 82). This leads to the objective
minimize CLIPPP(( - 8; 11
which we name ideal dictionary learning problem since the ground truth direction & for driving text is unknown. To avoid this unrealistic assumption, we substitute & with the known directions a E IR" derived from unsupervised methods (Härkönen et al., 2020; Shen & Zhou, 2021). CtIP(t) also replaced by (CLIP(() (see Eq. (1) for the definition of ΦCLIP (-)) since we would like to use a itself as a driving guidance rather than finding out corresponding text t that requires intee-intensive labeling. Then, Eq. (3) becomes
minimize Σ (TFocLiP(a) - -allz Va
(4)
We construct the set of known directions in Eq. (4) by using all 512 directions found by GANspace and directions with top 80 eigenvalues out of 512 from SeFa" (Härkönen et al., 2020: Shen & Zhou, 2021). Instead of naively using the unsupervised direction a E R" we prune the channels in directions to activate only & channels among n channels, zeroing out the other n - & channels that have small magnitudes. Such pruning is represented by (k) E R² where we use k 10, 30, 50 in this paper (the effect of k is described in Tab. in the appendix). The reason for pruning the number of activated channels is that unsupervised methods show instability in terms of maintaining
*This is based the observation that the change in image is not significant for the directions with lower eigenvalues in ScFa.

---

Published as a conference paper at ICLR 2023
Table 2: Mean Squared Error (Eq. (6); 4) measured in StyleSpace. We manipulate total of 100 channels using both GlobalDirect ion and Ours. Note that is a (200) in this metric.
Global Direction
Ours
(S Ô)
0.0315 + 0.0077
0.0057 0.0034
MSE a, (1700)
0.0267 = 0.00114
0.0005 + 0.00001
MSE a, (00))
0.006218 = 0.00658
0.00486 = 0.00363
the original image due to the entanglement (please refer to Appendix C for examples of entanglement without pruning) and we circumvent such issue by constraining the direction E R= to have limited effect on only & channels.
To help close the gap between Eq. (3) and Eq. (4), we introduce an additional loss term | (k) (CCLI((((K)) (A)) update the dictionary D IRxx,, This additional term comes from the observation in Sec. 3.2 that multi-channel manipulation should be considered. Unlike ClobalDi abbalDinn rect i on which directly encodes (e, for all n channels in StyleSpace, we aim to learn the multi-channel manipulation by (*) whose editing effect mapped from StyleSpace to CLIP space by ΦCLI (-). The main obstacle in this objective is identifying which channel is responsible for which part of the change induced by total of k channels in the given direction (1k), Here, we emphasize that augmenting the same direction a into a (10), (90), (50) can encourage the dictionary learning process identify a disentangled editing effect of each channel which will be explained further below.
Based on the concept figure in Fig. we explain how the additional loss term encourages dis- entangled learning of dictionary D. We present a simplified version where & = 1, 3, cre- ating three augmentations (21)), (2) (3) (3)) for the direction a. Suppose manipulation using (3) = [s11 83]" modifies the source image to have 1) thick eyebrows, 2) pink lipstick and 3) glowing skin. Further suppose manipulation by (2) = [s1, 0, sz]" modifies 1) thick eyebrows and 2) pink lipstick. Finally, manipulation using (11) = [s1, 0, 0j² modifies 1) thick eyebrows. If the loss term (Dx ΦCLIr (a<()) is perfectly optimized to satisfy (a = ΦCLI (a))), then sii,
CLIP(thick eyebrows & pink lipsticks) and piis + $2d2 + s3d3 s CLiP(thick eyebrows & pink lipsticks & glowing skin). Therefore, the lin- ear combination on the sparsified direction helps to specifically link 'thick eyebrows' di 'pink lipstick' to dg. and 'glowing skin' to d2. Hence our method may have greatly disentangled and diverse directions coming from the enriched coverage over manipulation in StyleGAN, thanks to the dictionary learning based on multi-channel manipulation. Moreover, even though our a comes from the unsupervised methods, our method learns the individual role of 81, 82 and 83 from the single unsupervised direction a [$1, 82, $3 L This leads to greatly diverse and disentangled results compared to unsupervised methods which will be proved in Tab.
CLII eyebrows), sid, + s3dz
Finally, combining these two ingredients together yields our dictionary learning problem by consider- ing all known directions a (k);
Σ -1 ( (112 a A DI (k) (aLIp(( 11
minimize
(5)
D
(a((k)
where X is the tunable hyper- paremmetreeter and is set as 0.01 in all our experiments below.
5 EXPERIMENTS
We evaluate the proposed method on StyleGAN2 (Karras et al., 2020a). We used StyleGAN2-ADA pretrained on multiple datasets, including FFHQ (Karras et al., 2019), LSUN (Yu et al., 2015) Car, Church, AFHQ (Choi et al., 2020) Dog, Cat. In this section, we show the widened StyleGAN manipulation coverage of our method compared to a text-guided method, GlobalDirection of StyleCLIP (Patashnik et al., 2021) mainly on FFHQ pre-trained model. For additional details in experimental design, please refer to Appendix E.
We measure the superiority of our method in terms of recovering the unsu- pervised directions a and text guidance f. The metrics are defined in StyleSpace S to evaluate over
Quantitative Results
7

---

Published as a conference paper at ICLR 2023
Table 3:
Cosine similarity ReLIP (Eq. (7); 1) measured in CLIP Space. For unsupervised method, which does not depend on text guidance, we choose the text-related unsupervised direction a that has the highest similarity value of cos cos(oc (ax), (L) given text t. We also present the similarity value on our method where ppraeeprramttrr À is O for ablation study on the second term of Eg. (5).
Global D1 rect on
Ours
Ours
Methods
Unsupervised
(A = 0)
(A 0.01)
10
0,219
0.224
0.235
0.242
Manip. Channels
30
0.221
0.228
0.240
0.249
50
0,223
0.232
0.243
0.251
100
0.225
0.237
0.248
0.254
unsupervised directions and in CLIP space for evaluation over text guidance. Multi20ne recover the unsupervised direction with maximum manipulative ability and minimum entanglement, for demonstration of which we define two metrics as follows.
StyleSpace metric for unsupervised directions (Tab. 2): We employ the following metric:
(6) where a is from a set of unsupervised directions A. Note that the ground truth direction a in this metric is a (100), where top-100 channels from unsupervised directions are modified zero entries. Therefore the reconstruction of such direction
MSE( à)) Ea~a [llá a
while other (100), given by & = D ΦCLIP (000) is expected to have 100 style channels whose values are similar to that of non- entries in a (100) while the other n - 100 styles are close O. MSE (a. â (0) denotes the difference between and á measured between the 100 non-zero style channels that represents the ability of the given dictionary D to find the precise manipulation direction. On the other hand, MSE (a, & =o) is the distance between a and & measured within - 100 zero-entries. This represents the level of disentanglement since zero-entries
100 channels
not expected be modified in &.
CLIP space metric for text guidance (Tab. 3): We measure the similarity between edited image and the text guidance in CLIP space as follows:
RCLIP (t, Xeditea) := Ex-r [cos (CLIP(t), CLIP CLIP(Xedted))]
(7)
We denote the set of text prompts that describes the pre-trained domain of StyleGAN, namely FFHQ as T whose construction is described in Appendix A.
Tab. 2 shows that Multi20ne consistently outperforms Globa IDirection in terms of MSE score between the unsupervised direction a and the estimated direction &. Especially, note that Multi20ne have lower MSE(c. a, á (===) than GiobalDirectioon proving that our method shows less entanglement.
Moreover, Tab. 3 shows that the cosine similarity between the given text guidance and the manipulated image is always the highest using our method. Most importantly, we emphasize that our method could find manipulation directions that could not have been found by unsupervised methods. We prove this claim using 57 text prompts (l ~ TT' as guidance to manipulate the images. Since we aim to find the unsupervised direction that could edit the image to become I, we select the a whose similarity cos( (a). Ct)P(t) is the largest then manipulate the image to produce X Xunsup Ga(s + ax) In Tab, 3, Multi2One scores better RELIP (t, Xedited) compared to unsupervised methods. Therefore, claim that method successfully learns to find expansive editing directions which could not be found by unsupervised methods given diverse text inputs. Furthermore, we report RELi((t, Xedited) score with ablation the second term of Eq. (5) in Tab. 3. For visual examples see Appendix G.
Qualitative Results We demonstrate the effectiveness of our method, compared to the state-of of-tt----he----- arts method, wrooblDDreectinn We conducted all experiments using the pretrained models and the precomputed set of CLIP embeddings for StyleCLII, that are readily provided for FFHQ, AFHQ Cat and AFHQ Dog. We computed the CLIP embeddings of the StyleSpace of LSUN Church and Car using the same configuration with GlobalDi rect ion. All the results are compared under same condition, where manipulation strength and disentanglement levels are set be equal. More specifically, we modify same number of channels for both methods and the magnitude of change" in StyleSpace is always fixed as 10.
'The original paper StyleCLIP refers to the magnitude of change in StyleSpace as and the disentanglement level as B which we substitute by number of channel that are changed.
8

---

Published as conference paper at ICLR 2023
SysteLi
(G.D)
Joker Smile Syvicuu (CD)
Little Mermail
Ours
Ours
StylCL
StyetT.I
(6.93
Hair
Young
Blue
Ours
Onrs
Figure 5: Manipulation results using FFH pretrained StyleGAN2-ADD via guidance method and StyleCLIP GlobalDi meriirectii ion. The samples are manipulated with increasing number of channels that are modified leading to larger changes.
The manipulated images show that text such 'Young' for FFHQ. On the other hand, successfully manipulates the images only on the basic text guidances but also on advanced semantics such as 'Little mermaid' and 'Joker smile' for FFHQ. We emphasize that despite the dictionary of our method is learned from the known directions in unsupervised approaches (Shen & Zhou, 2021; Härkönen et al., 2020), the manipulation results show that our learned dictionary could adapt to previously unseen combination of semanties such red hair, pale skin, and big eyes to represent "Little Mermaid' and unnatural smiles with red lipstick and pale face to represent 'Joker smile'.
fails to represent some of the most basic
Here we show that our method successfully discovers direction that does not exist in unsupervised direction. To more specific, we manipulate the image using an unsupervised direction a whose CLIP representation, (FCLIP(x)) is the most similar with the text "Little Mermaid' and 'Joker smile'. Then from the manipulated image, we observe that the manipulation result hardly plausible compared to the successful results in Fig. 5. Based this examples, emphasize that even though the dictionary learning process relies on the unsupervised methods, Multi20ne demonstrate wider variety of manipulation results using the versatility of text.
Source
=
k 30
k 50
100
Source
= 50
k 100
Little mermaid (similarity 0.1094)
Joker smile (similarity 0.1901)
Figure 6: Manipulation by the direction with highest cosine similarity. We find the unsupervised direction with highest similarity given text "Little mermaid" and 'Joker smile'. Then we manipulate the image using top-k channels from the found direction.
We show additional results of our method compared to Globallirecte based on AFHQ and LSUN dataset in Appendix D due to space constraints.
CONCLUSION
Text-guided image manipulation has been largely preferred over unsupervised methods in finding manipulation directions due its ability to find wide variety of directions corresponding to flexible text input. However, we have investigated that the state-of-the-zs method of text-guided image manip- ulation has limited coverage over possible editing directions using StyleGAN and this problem rises from simple assumption that single-channel manipulation could fully represent the manipulation ability of StyleGAN. To overcome this issue, we have proposed a dictionary learning framework that embeds the interactive and collective editing effect based on modifying multiple style channels into CLIP space jointly. Our method have been proven to be superior over other text-guided image manipulation methods in terms of manipulative ability and disentanglement.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our code is open sourced at here.
ACKNOWLEDGEMENTS
This work was supported by Institute of Information & communications Technology Planning Evaluation (ITP)) grant funded by the Korea gMV)) (8022019-0-000711 Artificial Intelligence Graduate School Program(KAIST). This project is also supported by KAIST-NAVER Hypercreative AT Center.
REFERENCES
Rameen Abdal, Yipeng Qin. and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
Yuval Alaluf, Patashnik, and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder via iterative refinement. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 671 1-6720, 2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.042000 2022.
Jaewoong Choi, Junho Lee, Changyeon Yoon, Jung Ho Park, Geonho Hwang, and Myungjoo Kang. Do not escape from the manifold: Discovering the local coordinates on the latent space of gans. arXiv preprini (06:::200660655)) 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8197, 2020.
Jung-Woo
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In CVPR, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurIPS, 2020.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" stccrability" of generative adversarial networks. arXiv preprint (107::1977077771 2019.
Brendan Jou, Tao Chen, Nikolaos Pappas. Miriam Redi, Merean Topkara, and Shih-Fu Chang. Visual affect around the world: A large-scale multilingual visual sentiment ontology. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 168, 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020b.
Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrIPS, 2021.
Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint I 2021.
Junho Kim, Yunjey Choi, and Youngjung Uh. Feature statistics mixing regularization for generative adversarial networks. In CVPR, 2022.
Yunji Kim and Jung-Wi Woo Ha. Contrastive fine-grained class clustering via generative adversarial networks. 2021.
Umut Kocasari, Alara Dirik, Mert Tiftikci, and Pinar Yanardag. Styleme: Multi-channel based fast text-guided image generation and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 895-904, 2022.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. ArXiv, abs/2 (/b//211,,0544,, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In ICCV, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual 2021.
models from natural language supervision. arXiv preprint
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based 2021.
editing of real images. arXiv preprini
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xI: Scaling stylegan to large diverse datasets. arXiv preprint 22t.::2202.002273: 2022.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020.
Binxu Wang and Carlos R Ponce. The geometry of deep generative image models and its applications. arXiv preprint arXiv: 1012000000000 2021.
11

---

Published as a conference paper at ICLR 2023
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, PP- 9939.9939. PMLR, 2020.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 22863-12872 2021.
Weihao Xia, Yujuu Yang, Jing- Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In CVPR, 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: 000.10006003333 2015.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka. Improved stylegan embedding: Where are the good latents? arXiv preprint arXiv:2012.09036. 2020.
12

---

Published as a conference paper at ICLR 2023
A MORE EXAMPLES ON FIG. 1(B)
In this section, we show the extensive results on the limited coverage of G1 GlobalDirection on wide variety of text guidance.
The experiment is conducted on FFHQ-preerained StyleGAN-ADA, and the text guidance are the adjective-noun pairs extracted from Visual Semantic Ontology (Jou et al., 2015) from which extract the nouns that describe human face. For example. the adjective-noun pairs contain human face descriptions such 'Handsome smile', 'Bad hair', and 'Stupi face'. The number of such adjective- noun pair amounts to 57 instances.
We manipulate the original image on the leftmost side using GlobalDirecte and Multi20ne using the same set of text description and equal configuration for both methods (manipulated channels on
5, manipulation strength 5). The result in Fig. shows that StyleCLIP GlobalDi constantly fails to represent the given text guidance faithfully, manifesting its limited capacity in exploiting the flexibility of text hence the need for Multi20ne to overcome the coverage issue.
13

---

Published as a conference paper at ICLR 2023
Source
GD
Ours
Source
GD
Ours
Crazy Eyes
Mad Eyes
Crying Eyes
Sexy Lips
Innocent Eyes
Shiny Hair
Friendly Smile
Sick Face
Hot Blonde
Smiling Eyes
Figure 7: Manipulated images using randomly selected text from Visual Semantic Ontology (VSO). We show manipulated image via G1 eba rect: on (denoted GD) and Multi20ne.
14

---

Published as a conference paper at ICLR 2023
B MORE DETAILS ON FIG. (AA)
Both Globa 1D rect i on and Multi20ne rely on multiple samples images to encode the aver- aged difference caused by manipulating sample into positive and negative direction. The difference is that unlike GlobalDirect in which relies on a single channel manipulation, Multi20ne en- code change of image caused by mmage-agnostic direction found by unsupervised methods (Härkönen et al., 2020; Shen & Zhou, 2021). Single channel manipulation is not image-agnostic since the change caused by moving single channel may not applicable to certain source images used as sample. For unconditional FFHQ pretrained StyleGAN-ADA, we have observed that a style channel which known make person's 's hair grey fails to modify half of the samples. On the other hand, the image-agnostic directions found by unsupervised methods hardly fails to manipulate any given image to become older with grey hair. As a consequence, the CLIP embeddings of difference caused by single channel manipulation may greatly vary sample by sample while image-agnostic directions show consistent direction of CLIP embeddings due to the successful manipulation on any given sample of images.
We visualize the direction of each sample when a single channel is modified, and when 100 or 200 channels from unsupervised directions modified. More specifically, for single channel manipulation of A..baaDirretiin the samples are (GG + s.)) (' 8t)) and for multiple channel manipulation, the samples are ((G.(s+ a)) - E(G. a)) where ~ S.
The visualization conducted in similar manner with the work of Wang Isola (2020). First we embed the p = 512 dimensional CLIP vectors into 2-dimension using t-SNE (Van der Maaten & Hinton, 2008). We denote each sample as (a;, b2) for i 1.2,..., 100 and the averaged direction of interest E E(G. s.)) E(G,(s s:)) in 2-dimensional space is (τ,, b). The angles
derived 0, arctan2(b, a.) for all samples which range between [-]. The histogram of 100 angles are plotted with the angle of averaged difference in IR" with the angle arctan2(b, a) is emphasized as a red line in Fig. 3(a).
15

---

Published as a conference paper at ICLR 2023
C EXAMPLES OF ENTANGLEMENT USING UNSUPERVISED METHODS
In Fig. 8, we demonstrate the effect of using limited number of channels among all non-zero channels in unsupervised direction a _
Since StyleGAN generator consists of progressiveGAN structure where the resolution of generated image progressively increases, it is possible to classify the generator structure into three blocks: coarse, medium and fine. We apply the unsupervised directions on each of the blocks. Moreover, by filtering out to only use manipulated channels, we show that the source image shows less drastic change. By manipulating all channels, which is shown at the rightmost side of the figure, the source image is drastically changed showing entanglement.
16

---

Published as a conference paper at ICLR 2023
Number of Channels Manipulated
Orig
100
300
500
700
900
1100
All
Coarse
Medium
GANspace 1st Direction
Fine
Layers
Orig
100
300
500
700
900
1100
Coarse
Medium
GANspace 10th Direction
Fine
All Layers
Figure 8: The manipulation results using two directions from GANSpace shows that the source identity is impaired when modifying all 18 layers of W. Moreover, limiting the number of manipulated channels for each coarse, medium, and fine layers leads significantly better results in of disentanglement.
17

---

Published as a conference paper at ICLR 2023
D MORE EXPERIMENTAL EXAMPLES FOR ALL DATASET
In this section, we provide more experimental results of Multi20ne with FFHQ, AFHQ, and LSUN.
Source
GD
Ours
Source
GD
Ours
Green Hair
Tanned
Pink Hair
Pale
Purple Hair
Dark Skin
White Hair
Sunlight
Red Hair
Strong Light
Figure 9: Additional manipulation results on FFHQ -reetrird StyleGAN-ADA. Equal configuration for both GlobalDirection ion (denoted GD) and Multi2One.
Fig. 10 shows the manipulation results on Multi2One and StyleCLIP Gl Global Dirretiiiiiii i on using 100, 500, 1000 channels. In addition, the manipulation results without channel pruning strategy is shown as well. Using over 500 channels is required for large structural change of the object in source image. However, StyleCLIP GlobalDi irrect i in shows signs of severe image collapse before creating an image that perfectly satisfies the given text guidance. For example, an image at the first row with 100 channels manipulated do not seem 'young' enough and the extent of manipulation is not improved with using more channels since the image collapse and entanglement becomes dominating factor. On the other hand, Multi2One shows consistently reasonable and disentangled images even when using all channels in STYLESPACE.
18

---

Published as a conference paper at ICLR 2023
Manipulated Channels
Original Image
100
500
1000
No pruning
StyleCLIP
GD
Ours
Young
StyleCLIP GD
Ours
Old
StyleCLIP GD
Ours
Joker Smile
StyleCLIP
GD
Ours
Woman with makeup
Figure 10: Manipulation results showing the disentanglement of Multi2One when using 100, 500, 1000 channels. and without pruning channels. StyleCLIP Globa DilllDireeeee rect on manifests image collapse when using over 500 channels while Multi20ne shows robustness against entanglement.
19

---

Published as a conference paper at ICLR 2023
StyleCLIP (G.D.)
Bengal Cat
Ours
StyleCLIP (G.D.)
Puss in Boots
Ours
(a)
StyleCLIP (G.D.)
Black Fur
Ours
StyleCLIP
(G.D.)
Blue Fur
Ours
(b)
Figure 11: Manipulation results using AFHQ and AFHQ dog pretrained StyleGAN2-ADA via text guidance on our method MultiZOne and StyleCLIP Gl al wewiteiiiiiiiiii The samples are manipulated with increasing number of channels that are modified leading to larger changes.
20

---

Published as a conference paper at ICLR 2023
StyleCLIP (GDA
G-Wagon
Benz
Ours
StyleCLIP (G.D.)
Yellow Cab
Ours
(a)
StyleCLIP (G.D.)
Brick Church
Ours
StyleCLIP (G.D.)
Golden Church
Ours
(b)
Figure 12: Manipulation results using LSUN car and LSUN church pretrained StyleGAN2-ADA via text guidance on our method Multi20ne and StyleCLII wecaalirreetai The samples are manipulated with increasing number of channels that modified leading larger changes.
21

---

Published as a conference paper at ICLR 2023
Source
GD
Ours
Source
GD
Ours
Black
Corgi
White
Chihuahua
Yellow
Maltese
Brown
Deberman
Orange
Gloden Retriever
Figure 13: Additional manipulation results on AFHoddog-retrraieee StyleGAN-ADA. Equal configuration for both ClobalDirect......c (denoted as GD) and Multi20ne.
22

---

Published as a conference paper at ICLR 2023
Source
GD
Ours
Source
GD
Ours
Black Cat
Sleepy Cat
Yellow Cat
Kitten
Grey Cat
Green Eyes
White Cat
In the Dark
Pink Nose
Surprised Cat
Figure 14: Additional manipulation results pretranddd StyleGAN-ADA, Equal configuration for both GlobalDi rest ion (denoted as GD) and Multi2One.
23

---

Published as a conference paper at ICLR 2023
Target Image
Source Image
GD
Ours
Source Image
GD
Ours
Figure 15: Additional manipulation results LSUNNar-pretraincc StylcGAN-ADA. Equal configuration for both G1 oba 1Di rection (denoted as GD)and Multi20ne. Used image instead of text as guidance encoded into CLIP with encoder E())
Target Image
Source Image
GD
Ours
Source Image
GD
Ours
Figure 16: Additional manipulation results LSUNNchch-pretrrinee StyleGAN-ADA, Equal configuration for both Gl r on (denoted GD) and Multi20ne. Used image instead of text guidance encoded into CLIP with encoder E(.).
24

---

Published as a conference paper at ICLR 2023
E EXPERIMENTAL DETAILS
We use unsupervised directions from SeFa and GANspace, both of which are found in intermediate space W limiting the maximum number of directions to 512, which is the dimension of the interme- diate latent space. We use 512 directions from GANSpace and 80 directions from SeFa. Since the earlier layers of StyleGAN mainly shows large structural changes while the deep layers are related to small detailed changes, we apply the 1024 vectors three groups of layers: coarse(resolution 4-~32), medium(resolution 64~~128) and fine(resolution 128~) (Karras et al., 2020a), which amounts to total of 1776 unsupervised directions.
Since the text-guided manipulation directions are found in StyleSpace S, we map the 1776 directions into StyleSpace S, where the maximum value of the style channel parameter in each direction is 15. The unsupervised directions are dense vectors, where the elements are non-zeros affecting every channel in StyleSpace. Dense manipulation vectors are prone modify multiple regions of images, causing an image to lose the original identity. Sparsifying the known manipulation vectors a into (x) where k = 10, 30, 50 boosts the disentangled property of the learned dictionary.
We train the dictionary from zero matrix with 30000 epochs. The hyper-parameter À is set as 0.01, and the learning rate is 3.0 with Adadelta (Zeiler, 2012), an optimization method with adaptive learning rate.
F COMPARISON WITH OTHER SOTA METHODS
Table 4: Evaluation on other text-guided image manipulation methods.
Metrics
Methods
Multi20ne
StyleCLIP GD
StyleCLIP LO
StyleMC
RELIP (L, X eddtted ) (1)
0.2617
0.2482
0.2698
0.2574
MSE( ② (4)
0.0054
0.0244
0.0047
0.0683
MSE a, 0 (L)
0.0012
0.0157
0.0018
0.0651
MSE a. a | (4)
0.0041
0.0086
0.0028
0,0032
Inference Time
x
100 secs
(per image-text)
In Tab. 4, we provide additional results from other image manipulation methods, StyleCLIP GD @elobalDirect.n from Patashnik et al. (2021)), StyleCLIP LO (local optimization from Patashnik et al. (2021)) and StyleMC from (Kocasari et al., 2022). Multi20ne and StyleCLIP Globa AlobalDiree roctiici in are two unique methods that provides instant manipulation on provided text and image without additional computation for optimization. Local optimization (Patashnik et al., 2021) and StyleMC (Kocasari al., 2022) are optimization based methods which require about 100 seconds of inference time using single NVIDIA 2080Ti. Morcover, such optimization based methods do not have disentanglement property unlike Multi20ne and GlobalDi Dirtt i on leading to significantly worse performance in source identity preservation.
The text-image similarity is measured using ReLIP (t, Xeditea) from Eq. (7), which measures the similarity between text t and the manipulated image. We used total of 57 text descriptions human face to evaluate the performance based on FFHQ-pretrained generator. We manipulate all 6048 channels in StyleSpace accordingly with optimization based methods which do not prune certain channels as a remedy for entanglement. Despite being an input-agnostic manipulation method, Multi2One is on par with the input-dependent StyleCLIP Local Optimization method (Patashnik et al., 1 in the CLIP similarity score, ReLIP (t, Xeditea)
The measurement on the ability to find the unsupervised directions a (100) (only the top 100 channels from unsupervised direction a are modified) is measured using mean squared error in Eq. (6). Given
'The construction of 57 random texts for manipulation on FFHQ pre-trained StyleGAN-ada is described in Appendix A
25

---

Published as a conference paper at ICLR 2023
an (CLIP(-) operation which maps the manipulation effect of a (100), is desirable to find direction that only modifies the 100 channels that were non-zero entries among n channels in a (1100) while the other n 100 channels are not modified. Therefore, MSE a. & (=)) measure the level of disentanglement which measures the distance between the n 100 channels in á and a (1100) The result in Tab. 4 shows that ours, Multi20ne, scored the lowest MSE (ox, á/=o) proving to be the most disentangled among the four text-guided manipulation methods.
26

---

Published as a conference paper at ICLR 2023
G ABLATION STUDY ON LOSS TERM
We report the similarity score as (t Xedited) (t X), to show the increase in similarity value when modified using the text guidance Note that ID loss tends increase the image is manipulated properly, therefore it is not a robust metric for evaluating the entanglement level.
Table 5: We report the effect of the hyper- papar--araetter À and the candidates for top-k for creating the sparsified unsupervised directions. ID....) indicates ArcFace Loss, which measures a change in identity, between the original image and edited image.
Similarity
ID(X, Xedit)
À
0.1
0.01
0.1
0.01
10
0.125
0.106
0.375
0.175
10, 30
0.098
0.140
0.075
0.390
Top-k
10, 30, 50
0.115
0.141
0.417
0.284
10,30, 30, 50, 100
0.120
0.121
0.225
0.007
In Fig. 17, we visualize the manipulation results show the effect of the second loss term when removed by À = () compared to when the second loss term is included by À 0.01. We observe that using the additional loss term does help finding novel directions which are not present in the unsupervised directions, leading to better manipulative ability.
Source
A == 0
= 0.01
Source
À = 0
== 0.01
Eyeglasses
Little Mermaid
Young Teen
Pink Hair
Male
Green Hair
Figure Manipulation results with ablation of second loss term using À and 0.01.
27

---

Published as a conference paper at ICLR 2023
H ABLATION STUDY ON THE EFFECT OF UNSUPERVISED METHODS
The dictionary learning process of Multi20ne employs the directions a E IR from unsupervised methods (Shen Zhou, 2021; Härkönen al., 2020). Therefore, conduct an ablation study on the effect of using unsupervised directions by comparing the two cases where directions a come from supervised method (Shen et al., 2020) and unsupervised methods (Shen & Zhou, 2021; Härkönen et al., 2020). Fig. 18 shows that using directions from supervised methods are less capable of manipulating images to have desired change. This could also be observed by quantitative measure RCLIP X edited) in Tab. 6 since the similarity score between manipulated image and the text guidance significantly lower using directions from supervised method compared to using unsupervised method. This is due to the limited number of directions that could be found by supervised method, which amounts to only 14 using InterFaceGAN (Shen et al., 2020).
Source Image
W! InterFaceGAN
W SeFa, GANspace
Source Image
Wi InterFaceGAN
SeFa, GANspace
Friendly Smile
Shiny Hair
Hot Blonde
Silly Smile
Sexy Lips
Tired Eyes
Figure 18: Ablation study the effeet of using unsupervised directions from Härkönen et al. (2020); Shen & Zhou (2021). The substitute the ground truth directions in the dictionary learning process with directions from InterFaceGAN (Shen al., 2020). We manipulate same number channels with equal manipulation strength for both.
Table 6: Evaluation other text rext-guided image manipulation methods. 100 channels manipulated with manipulation strength of Supervised method indicates using directions from Shen al. (2020) as direction. Unsupervised method indicates that dictionary learning process uses ground truth directions a from Härkönen et (2020); Shen & Zhou (2021).
Q
Supervised Method Unsupervised Method
RCLP(1, Xedited)
0.228
0.254
28

---

Published as a conference paper at ICLR 2023
I COMPUTATIONAL OVERHEAD
StyleCLIP GlobalDi rect in and Multi20ne require preprocessing time to compute ØCLIP() operation. Such operation computed using either e; over n channels (StyleCLIP) or for total of q directions. More specifically, StyleGAN-ADA generator has a fixed StyleSpace dimension consisting of n channels which varies by the resolution of output image. 1024 resolution has 6048 channels, while 512 resolution has 5952 and 256 has 5760. Therefore the total computational time is measured by the number of (CI.P( operations calculated over n. or q directions using 100 instances of source images to compute the expectation. In Tab. 7 we measure the average computation seconds per iteration using NVIDIA 2080Ti which is equal for both methods. Since one iteration means single (-LIP(.) operation, comparing the number of iterations shows that our method requires less computation time. Moreover, our method introduces an additional dictionary learning phase which only takes about 15 minutes. Therefore conclude that our method is slightly more efficient in terms of time to construct the dictionary D.
Table 7: Measured time required for forward pass with batch size 1 in NVIDIA 2080Ti-seconds. SG2 stands for StyleGAN-ADA. The first show the duration of single iteration in seconds.
Preprocessing Time
Dictionary Learning
Dataset
SG2-res1024²
SG2-res5122
SG2-res256'
All resolution
Sec/it
14.98s/it
13.1 1s/it
8. 8.19//i
GD (iters)
6048
5952
5760
Ours (iters)
5328
5328
5328
15 mins
J LIMITATIONS
The flexibility and diversity of text is not fully exerted due the limited encoding ability and the deterministic representation of CLIP (Radford et al., 2021). For example, it is possible to observe change in pose when using the eigenvectors (vectors that correspond to large eigenvalues often shows large change in image) from unsupervised methods. However, since CLIP is known to have limited ability in terms of understanding the relative position, it is impossible observe that CLIP encoder fails to
modify an image using
text guidance such 'move right' and 'turn left'. Moreover, represent some of the text instance such as 'Smiling eyes' (Fig. 7) while it is shown in Fig. I(a) that change in eyes happens using text 'Smile'. Since is proven that the editing direction of 'smiling eyes' could be discovered when provided with appropriate text, we conjecture that the CLIP encoder fails to represent some text input faithfully leading unsuccessful manipulation results.
29


Reviews:
[{"review_id": "mPq8Qgk3ok", "reviewer": "Reviewer_RdgP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for image manipulation with ‘global direction’, which is case-agnostic and applicable to all input images. The authors argue that the previous work StyleCLIP provides a dictionary to quickly find out the channel-wise manipulation while the collective and interactive relation among multiple channels is not considered. Thus the manipulation performance is limited. The authors propose a new manipulation method by considering the manipulation effect coming from the interaction with multiple other channels.\n", "summary_of_the_review": "The reviewer recommends a weak accept of the paper and is pretty confident.", "strength_and_weaknesses": "Strength:\n1. Global direction editing is charming since it can be transferred to different new images without local optimization. An effective method around text-guided editing is a promising task.\n2. The analysis is interesting. Especially the experiment is impressive, i.e., single-channel editing provides no change while joint-channel editing can change the global shape. A corresponding method is derived based on the observation, which makes it more convincing. The results look good compared with the original StyleCLIP dictionary. The joint-channel editing may provide other unsupervised editing work with several insights (the collective and interactive relation among multiple channels should be considered).\n3. The authors conduct adequate experiments on sufficient datasets to prove the manipulation performance of the proposed method surpasses StyleCLIP.\n\nWeakness:\n1. One question, beyond text-guided editing, is it possible to use such a global direction to provide a robust editing method on the large pose or expression change? Existing methods have attempted to use 3DMM representation or other unsupervised editing directions to inject into 2D StyleGAN, but the results are not that robust.\n2. Some figures are pretty messy. For instance, there are too many texts of various fonts/sizes/colors in Fig.1 and Fig.4. Please consider unifying some of them.\n3. Some terms lacked explanation when first mentioned, such as △s and e(i) in Eq(2), easily confusing the reader.\n4. The process of StyleCLIP’s obtaining a dictionary should be represented more clearly in case the reader is unfamiliar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lUnpPC_LW2H", "reviewer": "Reviewer_v5mm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on leveraging the StyleGAN model for text-guided image manipulation. Starting from latent directions discovered by the unsupervised method, the authors propose an approach, with a sparsify trick, to further disentangle the editing directions. Experiments conducted on the FFHQ, LSUN, AFHQ datasets show that the proposed method performs favorably against the state-of-the-art method.", "summary_of_the_review": "I suggest the authors to carefully revise the manuscript, especially Section 2 and 3. Although favorable quantitative and qualitative results are shown, the motivation, intuition, technique, and novelty behind this paper is not well-demonstrated.\n\n--- post rebuttal ---\nThe authors have revised the manuscript, and now Section 2 and 3 are more clear, so I'm not against to accept this work.", "strength_and_weaknesses": "*Strength*\n1. The authors promise to release the source code in the future, which can stimulate further research in this field.\n2. From the quantitative and qualitative results, the proposed method seems to have favorable editing capability compared to the state-of-the-art methods.\n\n*Weakness*\n1. The paper is not well written. Particularly, Section 3 and 4 is hard to follow. The syntaxes in the figure and the text description are not alighted. For example, $\\Delta{\\mathcal{S}}$ is used in Figure 1, but in the text descriptions it seems to be $\\Delta{s}$. There are many confusing and unclear descriptions as well. For instance, in Section 3.2, what do the authors mean by \"a large portion of the instances from unsupervised directions\"? What do the \"instance\" and \"directions\" refer to? Moreover, what does \"represent new directions to the fullest capacity\" mean? The notation $\\alpha$ is also not properly defined. What is $alpha$? What is the dimensionality? For section 4, how do the authors use the sparsified $\\alpha^{(r)}$ in Equation 4?\n2. The experiments shown in Section Table 1 are also confusing. For Table 1, how is the direction term $\\alpha$ computed for the GlobalDirection method computed?\n3. As the goal of this paper is to edit images using text, what is the importance of recovering the unsupervised directions in this paper? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_3NNZ9tEAE", "reviewer": "Reviewer_bYKo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a dictionary learning framework to improve StyleCLIP, one state-of-the-art image manipulation method via text guidance. Compared to StyleCLIP which operates on one single channel independently, this method tries to embed the multiple style channels into CLIP space jointly. Some experiments are performed to support the advantages of the proposed method.", "summary_of_the_review": "This submission details how it extends from GlobalDirection to achieve multiple channel modeling. However, this motivation seems to be somewhat trivial and just a straightforward revision, even given its superior results. The authors can also improve their paper by adding more experimental results to support their claims. But I would not like to object acceptance if other reviewers prefer to do so.", "strength_and_weaknesses": "Strengths:\n\n+ The authors design experiments to support their motivations by showing that the GlobalDirection can not provide consistent embeddings. I like these experiments, which makes the statement clear.\n+ The shown visual results look pleasing and demonstrate some improvement over GlobalDirection.\n+ The writing is clear and easy to follow.\n\nWeakness:\n\n+ The proposed approach is an incremental improvement upon StyleCLIP and GlobalDirection methods. Specifically, this submission can be regarded as a straightforward extension of GlobalDirection by extending the single-channel editing direction to multi-channel editing direction. Also, the idea of discovering the editing direction and training the dictionary also borrows the results of unsupervised latent direction discovery methods such as SeFA and GANSpace. This makes me feel like this submission might suffer from limited novelty.\n\n+ As mentioned in Section E, the proposed method uses the output of previous methods (e.g. SeFA and GANSpace). So the authors need an ablation study (only use multi-channel learning strategy) along with corresponding discussions to rule out the influence of this point and then highlight the influence of multi-channel modeling. And the fact that this method depending on SeFA and GANSpace's outputs, also might make it suffer from similar limitations to SeFA and GANSpace.\n+ How to evaluate the image synthesis results is still an open problem. Thus this submission mainly depends on the qualitative comparisons, which might be very subjective. Although this submission presents some basic quantitative comparisons, I suggest to use some widely used quantitative metrics, like Inception score or more advanced alternatives, to evaluate the generation results instead of MSE and Cosine similarity. At least, the user studies are needed for evaluations over randomly generated results.\n+ In this submission, the authors only compare with the original StyleCLIP but there are other variants. So more empirical comparisons need to be conducted with more SOTA methods.\n+ While the proposed method shows better results over chosen baselines, I am curious about the comparisons in terms of computation overhead and other additional limitations introduced by multi-channel modeling.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NndzBw9Vxl", "reviewer": "Reviewer_jy1p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve the GlobalDirection method of StyleCLIP to further consider the inter-channel correlations when finding the editing vectors in StyleSpace. A novel dictionary learning loss is proposed to learn such correlations and experiment validates the effectiveness of the method compared to the StyleCLIP. ", "summary_of_the_review": "This paper proposes to improve the GlobalDirection method of StyleCLIP to further consider the inter-channel correlations when finding the editing vector. I can generally understand the main idea of dictionary learning but cannot fully nail down the meaning and function of the second loss term. From the experiment results, it seems that the proposed dictionary learning is effective to improve the editing quality. \n", "strength_and_weaknesses": "**Strengths:**\n+ The idea to consider the inter-channel correlations during latent editing is intuitive and reasonable. The paper is well motivated. \n+ In general, the visual and editing quality of the proposed method is significantly better than the baseline method. \n\n**Weaknesses:**\n+ The idea of the second loss term in Eq. (4) is hard to understand to me. What is the intuitive motivation or physical meaning behind this loss term? Is there any visual results with and without this loss term to help readers better understand its effect?\n+ In ablation study, the effect of second loss term is better to be validated by setting lambda=0. Current, only lambda=0.1 and 0.01 is compared quantitatively. Visual results are suggested to be given for better comparison.\n+ The limitation is suggested to be discussed. For example, why in Fig. 11, the text is smiling eyes, but the eyes have nearly no changes while the mouth changes a lot.\n+ The conclusion is suggested to be given.\n\n**Some small issues:**\n+ The text in Fig. 6 is too small.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster