Zero-shot Synthesis with Group-Supervised Learning
Authors: Yunhao Ge
Sami Abu-El-Haija
Gan Xin
Laurent Itti
Venue: ICLR 2021 Poster (2021)
Keywords: Disentangled representation learning
Group-supervised learning
Zero-shot synthesis
Knowledge factorization

Abstract: Visual cognition of primates is superior to that of artificial neural networks in its ability to “envision” a visual object, even a newly-introduced one, in different attributes including pose, position, color, texture, etc.  To aid neural networks to envision objects with different attributes,  we propose a family of objective functions, expressed on groups of examples, as a novel learning framework that we term Group-Supervised Learning (GSL). GSL allows us to decompose inputs into a disentangled representation with swappable components, that can be recombined to synthesize new samples.  For instance, images of red boats & blue cars can be decomposed and recombined to synthesize novel images of red cars.   We propose an implementation based on auto-encoder, termed group-supervised zero-shot synthesis network (GZS-Net) trained with our learning framework, that can produce a high-quality red car even if no such example is witnessed during training. We test our model and learning framework on existing benchmarks, in addition to a new dataset that we open-source. We qualitatively and quantitatively demonstrate that GZS-Net trained with GSL outperforms state-of-the-art methods

Published as a conference paper at ICLR 2021
ZERO-SHOT SYNTHESIS WITH GROUP-SUPERVISED LEARNING
Yunhao Ge, Sami Abu-El-Haija, Gan Xin, Laurent Itti University of Southern California yunhaoge@use.. sami @hai ja wwwiea.aaoooooooo..oo.. org, gxin@usc.edu, itti@usc.edu
ABSTRACT
Visual cognition of primates is superior to that of artificial neural networks its ability to "envision" a visual object, even newly-introduced one, in different attributes including pose, position, color, texture, etc. To aid neural networks to envision objects with different attributes, we propose family of objective functions, expressed on groups of examples, as a novel learning framework that we term Sroup-Superviied Learning (GSL). GSL allows to decompose inputs into a disentangled representation with swappable components, that can be recombined synthesize new samples. For instance, images of red boats & blue cars be decomposed and recombined to synthesize novel images of red cars. We propose implementation based auto-encoder, termed group-supervised zero- shot synthesis network (GZS-Net) -et) trained with our learning framework, that can produce high-quality red car even no such example is witnessed during training. We test our model and learning framework on existing benchmarks, in addition to a new dataset that we open-source. We qualitatively and quantitatively demonstrate that GZS-Net trained with GSL outperforms state-of-the-art methods.
INTRODUCTION
Primates perform well at generalization tasks. If presented with single visual instance of an object, they often immediately can generalize and envision the object in different attributes, e.g., different 3D pose (Logothetis et al.) 1995). Primates can readily do so, their previous knowledge allows them to be cognizant of attributes. Machines, by contrast, are most-commonly trained on sample features (e.g., pixels), not taking into consideration attributes that gave rise to those features.
To aid machine cognition of visual object attributes, a class of algorithms focuses on learning disentangled representations (Kingma & Welling 2014, Higgins et al. 2017 Burgess al. 2018
Kim & Mnih 2018 | Chen al. 2018), which map visual samples onto a latent space that separates the information belonging to different attributes. These methods show disentanglement by interpolating
between attribute values (c.g., interpolate pose, etc). However, these methods usually process one sample at time, rather than contrasting reasoning about group of samples. We posit that semantic links across samples could lead to better learning.
We motivated by the visual generalization of primates. We seek method that synthesize realistic images for arbitrary queries (e.g., a particular car, in given pose, on a given background), which we refer to controlled synthesis. We design a method that enforces semantic consistency of attributes, facilitating controlled synthesis by leveraging semantic links between samples. Our method maps samples onto a disentangled latent representation space that (i) consists of subspaces. each encoding one attribute (c.g., identity, pose, ...). and, (ii) is such that two visual samples that share an attribute value (e.g., both have identity "car"" have identical latent values the shared attribute subspace (identity), even if other attribute values (e.g., pose) differ. To achieve this, we propose a general learning framework: Group Supervised Learning (GSL, Sec. B. which provides a leamne (e.g., neural network) with groups of semantically-related training examples, represented as multigraph. Given a query of attributes, GSL proposes groups of training examples with attribute combinations that are useful for synthesizing test example satisfying the query (Fig. [T. This endows the network with an envisioning capability. addition to applications in graphics, controlled synthesis can also augment training sets for better generalization on machine learning tasks (Sec. 6.3

---

Published as a conference paper at ICLR 2021
Decoder Enoode
g
Disentngeed Latent Peanrrss
identty
backgmund
Manily
W Image-
m
Image-2
Image-3
())
(c)
Figure 1: Zero-shot synthesis performance of our method. (a), (b), and (c) are from datasets, respectively, iLab-20M, RaFD, and Fonts. Bottom: training images (attributes are known). Top: Test image (attributes are a query). Training images go through an encoder, their latent features get combined, passed into decoder, synthesize the requested image. Sec. 4.2 shows how disentangle the latent space, with explicit latent feature swap during training.
As an instantiation of GSL, we propose an encoder-decoder network for zero-shot synthesis: Group- Supervised Zero-Shot Synthesis Network (GZS-Net, Sec. 4 While learning (Sec. 4.2 repeatedly draw group of semantically-related examples, as informed by multigraph created by GSL GZS-Net encodes group examples, to obtain latent vectors, then swaps entries for one or more attributes in the latent space across examples, through multigraph edges, then decodes into an example within the group (Sec. 4.2).
we
Our contributions are: (i) We propose Group-Supervised Learning (GSL), explain how it casts its admissible datasets into a multigraph, and show how it can be used to express learning from caaunnicall--relltee groups and synthesize samples with controllable attributes; (ii) We show one instantiation of GSL: Group-superrvsed Zero-shot Synthesis Network (GZS-Net), trained on groups of examples and reconstruction objectives; (iii) We demonstrate that GZS-Net trained with GSL outperforms state-of-the-ar alternatives for controllable image synthesis on existing datasets; (iv) We provide new dataset, Fonts]] with its generating code. It contains 1.56 million images and their attributes. Its simplicity allows rapid idea prototyping for learning disentangled representations.
2 RELATED WORK
We review research areas, that share similarities with our work, to position our contribution.
Self-Supervised Learning (e.g.. Gidaris et al. (2018)) admits dataset containing features of training samples (e.g., upright images) and maps it onto an auxiliary task (e.g., rotated images): dataset examples are drawn and random transformation (e.g., rotate 909) is applied to each. The task could be to predict the transformation (e.g.. =90)) from the transformed features (e.g., rotated image). Our approach is similar, in that it also creates auxiliary tasks, however, the tasks we create involve semantically-relatee group of examples, rather than from one example time.
Disentangled Representation Learning are methods that infer latent factors given example visible features, under generative assumption that each latent factor is responsible for generating one semantic attribute (c.g. color). Following Variational Autoencoders (VAEs, Kingma & Welling 2014), class of models (including, [Higgins al.) 2017] Chen al. 2018) achieve disentanglement implicitly, by incorporating into the objective, a distance measure e.g. KI. KT--divergence, encouraging the latent factors be statistially-indeeennentt While these methods can disentangle the factors without knowing them beforehand, unfortunately, they are unable to generate novel combinations not witnessed during training (e.g., generating images of red car, without any in training). On the other hand, our method requires knowing the semantic relationships between samples (e.g., which objects are of same identity and/or color), but can then synthesize novel combinations (e.g., by stitching latent features of "any car" plus "any red object").
Conditional synthesis methods can synthesize a sample (e-g.. image) and some use information external to the synthesized modalities, e.g., natural language sentence Zhhng et al. (2017) Hong et al.
http: http://i10 ab. usc wsw. wtt....c..bb/tttattttt///ooooa
2

---

Published as a conference paper at ICLR 2021
(0188) class Label|Miran & Osindero (2014); Tran al. (2017). Ours differ, in that our "external information" takes the form of semantic relationships between samples. There are methods based on GAN Goodfellow et al. (2014) that also utilize semantic relationships including Motion Re-targeting YYang et al. 2020), which unfortunately requires domain-specific hand-engineering (detect and track human body parts). On the other hand, design and apply our method on different tasks (including people faces, vehicles, fonts; see Fig. D.. Further, we compare against two recent GAN methods starGAN (Choi al. 2018) and ELEGANT (Xiao al. 2018), as they are state-of-the-ar GAN methods for amending visual attributes onto images. W While they are powerful in carrying local image transformations (within a small patch, e.g.. changing skin tone hair texture). However, our method better maintains global information: when rotating the main object, the scene also rotates with it, semantically coherent manner. Importantly, our learning framework allows expressing simpler model network architectures, such as feed-forward auto-encoders, trained with only reconstruction objectives, as opposed to GANs, with potential difficulties such as lack of convergence guarantees.
Zero-shot learning also consumes side-information. For instance, models of Lampert (009); Atzmon & Chechik (2018) learn from object attributes, like our method. However, (i) these models supervised accurately predict attributes, (ii) they train and infer example at time, and (iii) they are concerned with classifying unseen objects. We differ in that (i) no learning gradients (supervision signal) are derived from the attributes, (ii) these attributes are used group the examples (based on shared attribute values), and (iii) we are concerned with generation rather than classification: we want to synthesize an object in previously-unseee attribute combinations.
Graph Neural Networks (GNNs) (Scarselli et al. 2009) are a class of models described on graph structured data. This is applicable our method, as we propose to create multigraph connecting training samples. In fact, our method can be described as GNN, with message passing functions Gilmer al. 2017) that are aware of the latent space partitioning per attribute (explained in Sec. 4 Nonetheless, for self-containment, we introduce our method in the absence of the GNN framework.
3 GROUP-SUPERVISED LEARNING
3.1 DATASETS ADMISSIBLE BY GSL
Ji=1 where each example is (6) ) Each attribute value is a member of accompanied with m attributes D. {(a) a2 .... am
Formally, a dataset admissible by GSL containing n samples (2)
a countable set: a, E Aj. For instance, pertaining to visual scenes, Ai can denote foreground-cole -colors A {red, yellow,... ... J. A2 could denote background colors, A3 could correspond to foreground identity, A4 to (quantized) orientation. Such datasets have appeared literature, e.g. in Borji et al.
(0166); Matthey et al. (0117) Langner et al. (2010) Lai al. (201I.
3.2 AUXILIARY TASKS VIA MULTIGRAPHS
Given dataset of n samples and their attributes, we define a multigraph M with node set [1..n). Two nodes, i, A E [1...]] with i # are connected with edge labels M(, k) C [1..m] as:
(ii = (3 a5) = (k) [L..m]
In particular, M defines a multigraph, with [M, k)) denoting the number of edges connecting nodes i and k which is equals the number of their shared attributes. Fig. depicct (sub- )multgrraph for the Fonts dataset (Sec. 5.
Definition COVER(S, i): Given node set C [L,.| and node E [L..|Dpl] we say set covers node i if every attribute value of is in at least member of S. Formally:
COVER(S, i) + [L.m]] U (i,
kES
When COVER(S, () holds, there are two mutually-exclusiie cases: either i E S, or i & S, respec- tively shaded as green and blue in Fig. (b). The first case trivially holds even for small S, e.g. COVER(fi), holds for all i. However, we are interested in non-trine non-trivial sets where |S| > 1, as sets The second case
with SS| = 1 would cast our proposed network (Sec. 4) to standard
3

---

Published as a conference paper at ICLR 2021
S2
B
k
H
f
f
Q
K
9
S1
Vary
Vary
(1)
LITE Tam
M
M
N
N
N
M
M
Background
Figure 2: (a) Samples from our proposed Fonts dataset, shown in groups. In each group, we vary one attribute but keep others the same. (b) (Sub-)mn Juuliiuultg of our Fonts dataset. Each edge connect two examples sharing an attribute. Sets S1 and S2 cover sample i.
crucial for zero-shot synthesis. Suppose the (image) features node i (in Fig. (b)) are not given, we search for S1. under the assumption that if COVER (S1, i) holds, then S1 contains sufficient information to synthesize i's features as they are not given & S1).
Until this point, made assumptions how the pairs (S, multigraph s.t. COVER (S, i) holds. In the sequel, we train with |S| = 2 and i E S. We find that this particular specialization of GSL is easy program, and we leave-out analyzing the impact of mining different kinds of cover sets for future work.
extracted (mined) from the
4 GROUP-SUPERVISED ZERO-SHOT SYNTHESIS NETWORK (GZS-NET)
We now describe our ingredients towards our goal: synthesize volisticallyssemaane novel images.
4.1 AUTO-ENCODING ALONG RELATIONS IN M
Auto-encoders (Do E) X' X' are composed of an encoder network E X - IRd and a decoder network D Idd X. Our networks further utilize M emitted by GSL. GZS-Net consists of
an encoder
M: and a decoder
R M
(2)
M denotes the space of sample pairvise-relationships. GSL realizes such (X, M) C X M, where X contains (a batch of) training samples and M the (sub)graph of their pairwise relations. Rather than passing as-is the output of E into D, one can modify it using algorithm by chaining: Do E. For notation brevity, fold A into the encoder E, by designing swap operation, next.
4.2 DISENTANGLEMENT BY SWAP OPERATION
While training auto-encoder DEE M)), we wish to disentangle the latents output by E, to provide use for using to decode samples not given to E. (/ E) outputs (/ inputs) one or more images, onto (/ from) the image space. Both networks can access feature and relationship information.
At a high level, GZS-Ner aims to swap attributes across images by swapping corresponding entries across their latent representations. Before any training, fix partitioning of the the latent space
= E(X, M). Let rew-vector z(1) [g] 92 ,
(1) (1) 9m
the concatenation of m row vectors hyperparameters.
19 (1) E Rd 1 where d >" dj and the values of {d, )m
To simplify the notation to follow, we define an operation swap Rddd x Rdd x [1..m] - R2d Rd which accepts two latent vectors (e.g., (0)) and 2()) and an attribute (e.g., 2) and returns the input vectors except that the latent features corresponding to the attribute are swapped. E.g..
swap (2) swap (171
(1) (1)
92 .... (1) (g 992
1 1,2)
[oi 92 93
(1) (2) (1)
(1) [g]
(2) 1))
93
(2)
9 (2)
One-Overlap Attribute Swap. To encourage disentanglement in the latent representation of at- tributes, consider group S and example COVER(S, æ) holds, and for all xº E S, dº, the
4

---

Published as a conference paper at ICLR 2021
Same back No everlee
Group Imsages:
Same id.
Roconstructio::
pose
One- Overlap Attiluric Swap
Same 3d
Same back
Encoder
Feature Swap
Loss
Resenstrrctiinn
Auribuse
Cydee
Decnder
Random
Correqunding
Swap
Swap
Recorstruction
Cycle Swap
Disentansled
Image copy
Far
Racorstraaion
Loss
Franel Inase
innges,
Staned linggs.
inaige
Figure 3: Architecture of GZS-Net, consisting of an encoder E: maps sample onto latent vector, and decoder D: maps latent vector onto sample. The latent space is pre- pre-partitioned among the attribute classes shown: identity, pose, background). (a, left) considered examples: center image (x, red border) and 3 images sharing one attribute with it, as well (T, black border). (a, right) standard reconstruction loss, applied for all images.
overlap image sharing no attributes (b) One-overlap attribute swap: Two images with identical values for one attribute should be reconstructed into nearly the original images when the latent representations for that attribute are swapped ("no-op" swap; left: identity; middle: pose; right: background).
(c) Cycle swap: given any example pair, we randomly pick an attribute class j. We encode both images, swap representations of decode, re-encode, swap on again (to reverse the first swap), and decode to recover the inputs. This unsupervised cycle enforces that double-swap on j does not destroy information for other attributes.
pair (x", .)) share exactly one attribute value (M xx] []] 1). Encoding those pairs, swapping the no-op the swap did not affect
latent representation of the attribute, and decoding should then other attributes (Fig. (b)). Specifically, would like for pair of examples, (red border in Fig. 3b) and x (blue border) sharing only attribute j (e.g., identity P
with 2 = (xx and 20 E(x"), be s.t.
D (= and D (2250) se 10); with Rey 2(00 swap =))).
If, for each attribute, sufficient sample pairs share only that attribute, and Eq. 3 holds for all with zero residual loss, then disentanglement is achieved for that attribute (on the training set).
Cycle Attribute Swap. This operates all example pairs, regardless of whether they share an attribute or not. Given two examples and their corresponding latent vectors, if we swap latent information corresponding to any attribute, we should end up with a sensible decoding. However, we may not have ground- tround-truth supervision samples for swapping all attributes of all pairs. For instance, when swapping the color attribute between pair orange truck and white airplane, we would like to learn from this pair, even without any orange airplanes in the dataset. To train from any pair, we are motivated to follow a recipe similar to CycleGAN (Zhu et al., (017). As shown Fig. (.. given two examples and ã: (i) sample attribute U[I.m,] (ii) encode both examples,
E(x) and
ž E(t); (iii) swap features corresponding to attribute j with Z,o 2, swap 2, 2,j); ; (iv) decode, (==.) and 7 = Dzz); (v) on second round (hence, cycle), encode again as E(ū) and
/ E [ū);; (vi) another swap, which should reverse the first swap, 2. 2, swap(2, غ, j): (vii) finally,
one last decoding which should approximately recover the original input pair, such that:
D (2x) R and D R
If. after the encode-swap-decode attribute we sample, this implies that swapping one attribute does not destroy latent information for other attributes. As shown Sec. 3 this can be seen as data augmentation, growing the effective training set size by adding all possible new attribute combinations not already in the training set.
are able recover the input images, regardless of which
It holds that COVER((>, and COVER
5

---

Published as a conference paper at ICLR 2021
Algorithm I: Training Regime; for sampling data and calculating loss terms
Input: Dataset D and Multigraph M
Output: Tr, Tur, Tess
Sample x: E D C such that COVER(S, x) and |S| = m and Wk E S, (x, k) =
for (0)) ES do
2 - E(æ); 20) t E(xlo));
- swap (z, (%,
Ess 28
Lug + Lus + (za) - x|li + | (25 - x ||,
Swap reconstruction loss
J ~ D and j ~ U(1..mm # Sample for Cycle swap
E(x); - E()); (Ze: 2x) swap(2, 2, 2.)); - D(z.): 3 D(2.)
2 E()); + E(3;; (2 2,) t swap (2, 2,
Losr + || (2,) 2||, + +I (3a) - =1,, Cycle reconstruction loss
L. + (D(())) - x ln # Standard reconstruction loss
4.3 TRAINING AND OPTIMIZATION
Algorithm 1 lists our sampling strategy and calculates loss terms, which we combine into total loss
C(E, D; = L, + AueLs + AestLest:
(5)
where T. Lug and Teesr. respectively are the reconstruction, swap-reconstruction, and cycle construc- tion losses. Scalar coefficients Asr Aesr control the relative importance of the loss terms. The total loss L can be minimized w.r.t. parameters of encoder (E) and decoder (D) via gradient descent.
5 QUALITATIVE EXPERIMENTS
We qualitatively evaluate our method on zero-shot synthesis tasks, and on its ability to learn disentan- gled representations, existing datasets (Sec. 5.2). and on a dataset contribute (Sec. (Sec.5.11.
GZS-Net architecture. For all experiments, the encoder E is composed of two convolutional layers with stride 2, followed by 3 residual blocks, followed by convolutional layer with stride 2, followed by reshaping the response map to a vector, and finally two fully-connected layers to output 100-dim vector latent feature. The decoder 1) mirrors the encoder, and is composed of two fully-connected layers, followed by reshape into cuboid, followed by de-conv layer with stride 2, followed by 3 residual blocks, then finally two de-conv layers with stride to output synthesized image.
5.1 FONTS DATASET & ZERO-SHOT SYNTHESIS PERFORMANCE
Design Choices. Fonts is cumputer-generatie image datasets. Each image is of an alphabet letter and accompanied with its generating attributes: Letters (52 choices, of lower- and upper-case English alphabet); size (3 choices); font colors (10); background colors (10); fonts (100); giving total of 1.56 million images, each with size (128 x 128) pixels. We propose this dataset to allow fast testing and idea iteration on zero-shot synthesis and disentangled representation learning. Samples from the dataset are shown in Fig. 2 Details and source code are in the Appendix. We partition the 100-d latents equally among the 5 attributes. We trainttest split of 75:25. Baselines. We train four baselines:
The first three are a standard Autoencoder, a 9- (Higgins et al. 2017), and 3-TCVAE (Chen et al. 2018). 3-VAE and 3-TCVAE show reasonable disentanglement on the dSprites dataset (Matthey et al.) 2017). Yet, they do not make explicit the assignment between latent variables and attributes, which would have been useful for precisely controlling the attributes (e.g. color, orientation) of synthesized images. Therefore, for these methods, we designed a best-effort approach by exhaustively searching for the assignments. Once assignments are known, swapping attributes between images might become possible with these VAEs, and hopefully enabling for controllable-synn syytheess......ss...... We denote these three baselines with this Exhaustive Search, using suffix +ES. Details Exhaustive Search are in the Appendix.

---

Published as a conference paper at ICLR 2021
+85
F-CCVF -ES
ribe
ARDES
-85
rake
K
K
U
X
S
k
W
E
L
H
H
H
E
m
x
H
A
k
P
P
B
Q
W
p
a
H
W
X
Q
Q
n
g
n
J
Q
Figure 4: Zero-shot synthesis performance compare on Fonts. 7-11 and 18-22 columns are input group images and we want to combine the specific attribute of them to synthesize an new images. 1-5 and 12-16 columns are synthesized images use auto-encoder Exhaustive Swap (AE+ES), 3-VAE + Exhaustive Swap (S-VAE+ES), 3-TCVAE Exhaustive Swap (3-TCVAE+ES), auto-encoder + Directly Supervision (AE+DS) and GZS-Net respectively. 6 and 17 columns are ground truth (GT)
The fourth baseline, AE+DS, is an auto-encoder where its latent space is partitioned and each partition receives direct supervision from one attribute. Further details are in the Appendix.
As shown in Fig. our method outperforms baselines, with second-runner being AE+DS: With dis- criminative supervision, the model focus on the most discriminative information, e-g., can distinguish e.g. across size, identity, etc, but can hardly synthesize photo-realistic letters.
5.2 SERO-SHOT SYNTHESIS ON ILAB-20M AND RAFD
iLabb 20M (Borji et al. 2016): is attributed dataset containing images of toy vehicles placed on a turntable using II cameras at different viewing points. There are attribute classes: vehicle identity: 15 categories, each having 25-160 instances; pose; and backgrounds:
14 for each identity: projecting vehicles in relevant contexts. Further details are the Appendix. We partition the 100-d latent space among attributes 60 for identity, 20 for pose, and 20 for background. iLab-20M has limited attribute combinations (identity shows only in relevant background; e.g., cars on roads but not in deserts). GZS-Net can disentangle these three attributes and reconstruct novel combinations (e.g., cars on desert backgrounds) Fig. shows qualitative generation results.
We compare against (AE+DS), confirming that maintains discriminative information, and against two ttate-of-the--ee the-art GAN baselines: starGAN (Choi et al. 2018) and ELEGANT (Xiao et al. (018). GAN baselines are strong in knowing what to change but not necessarily how to change Where change is required, pixels are locally perturbed (within patch) but the perturbations often lack global correctness (on the image). See Appendix for further details and experiments on these GAN methods.
RaFD (Radboud Faces Database, Laaabase,Laagneerr Langner et al. 2010y: contains pictures of 67 models displaying emotional expressions taken by different camera angles simultaneously. There are 3 attributes: identity, camera position (pose), and expression. We partition the 100-d latent space among the attributes as 60 for identity, 20 for pose, and 20 for expression. We use a 80:20 split for train:test, and use GZS-Net to synthesize images with novel combination of attributes (Fig. 6). The synthesized images can capture the corresponding attributes well, especially for pose and identity.
6
QUANTITATIVE EXPERIMENTS
6.1 QUANTIPYING DISENTANGLEMENT THROUGH ATTRIBUTE CO- CO-PREDICTION
Can latent features of one attribute predict the attribute value? Can it also predict values for other attributes? Under perfect disentanglement, we should answer always for the first and never for the second. Our network did not receive attribute information through supervision, but rather, through swapping. We quantitatively assess disentanglement by calculating a model- based confusion matrix between attributes: We analyze models trained on the Fonts dataset. We take the Test examples from Font, and split them 80:20 for :testpp For each attribute pair j.r [1..m] [1..m]. we train a classifier (3 layer MLP) from 9j of trainpr to the attribute values of r. then obtain the accuracy of each attribute by testing with 9j of tesTDR. [
how well features of each attribute (row) can predict an attribute value (column): perfect should as close as possible to Identity matrix, with off-diagonal entries close to random (i.e., 1 |A, (). GZS-Nett outperforms other methods, except for (AE + DS) as its latent space was Directly Supervised for this particular task, though it shows inferior synthesis performance.
7

---

Published as a conference paper at ICLR 2021
ELEGANT only change bockground
StarGN only change hackground
StarGAN only change pinc
StarGAN
only
Lr Lar Lear
Ideatity
Rackground
Figure 5: Zero-shot synthesis qualitative performance on ilab-20M. Columns left of the dashed line are output by methods: the first five baselines, followed by three GZS networks. The baselines are: (1) is an auto-encoder with direct supervision (AE+DS); (2, 3, 4) are three GAN baselines changing only one attribute; (5) is starGAN changing two attributes. Then, first two columns by GZS-Net ablation experiments: trained with part of the objective function, and the third column is output by GZS-Net trained with all terms of the objective. starGAN of (Choi al. 2018) receives one input image and edit information (explanation in Appendix Section ELEGANT uses identity and background images (in Appendix Section B.3
uses all input images). Others use all three inputs.
DE
Ours
Idcutity
Pose
Expecssion
AE+DS
Ours
Mdenity
Pose
Expression
Figure 6: GZS-Net zero-shot synthesis performance RaFD. 1-2 and 6-7 columns are the synthe- sized novel images using auto-encoder + Directly Supervision (AE+DS) and GZS-Net respectively. Remaining columns are training images with their attributes provide.
6.2 DISTANCE OF SYNTHESIZED IMAGE TO GROUND TRUTH
The construction of the Fonts dataset allows programmatic calculating ground-truth images corre- sponding to synthesized images (recall, Fig. 4. We measure how well do generated images compare to the ground-trutt -truth test images. Table 2 shows image similarity metrics, averaged over the test set, comparing our method against baselines. Our method significantly outperforms baselines.
6.3 GZS-NET BOOST OBJECT RECOGNITION
We show-case that our zero-shot synthesised images by GZS-Net can augment and boost training of visual object recognition classifier (Ge al. 2020). Two different training datasets (Fig. (7)) are tailored from iLab-20M, pose and background unbalanced datasets (D (half classes with poses per object instance, other half with only poses; cut poses, some backgrounds also
8

---

Published as a conference paper at ICLR 2021
Table I: Disentangled representation analysis. Diagonals are bolded.
GZS-Net
Auto-encoder
AE
B-VAE+
ES
8-TCVAE ES
(1A))
[
S
BC
C
FC
C
S
St
BC
C
FC
Content (52)
92
_11
.13
.30
.48
.60
.71
.92
06
.99
.72
.22
.25
.35
.11
.19
1
.39
.13
.11
.01
Size
.78
1.0
.15
36
45
.77
96
.4
1.0
.19
.5
.88
.99
.11
.2
.18
.19
.11
FoniColor (10)
.70
88
1.0
.16
23
48
60
.67
95
06
19
64
1.0
66
20
02
.33
44
.11
02
.3
21
.3
.01
BackColor
.78
.64
32
29
.25
34
86
03
24
Style (100)
.70
93
12
12
43
49
.6
70
94
.6
38
29
20
20
.65
02
33
10
.11
80
02
33
10
.08
.01
Table 2: Average metrics between ground-truth test image and image synthesized by models, con- ducted over the Fonts dataset. We report MSE (smaller is better) and PSNR (larger is better).
GZS-Net
AE+DS
--TCVAE +ES
Δ-vae ES
AB +ES
Average Mean Squared Error (MSE) Average Peak Signal-to-Noise Ratio (PSNR)
0.0014
0.0254 16.44
0.2366 6.70
0,1719
0.1877
29.45
7,9441
eliminated), as well as pose and background balanced dataset (DB) (all classes with all poses per object instance).
We use GZS-Net synthesize the missing images of DUD and synthesize new (augmented) balanced dataset DB-s We alternatively use common data augmentation methods (random crop, horizontal flip, scale resize, etc) to augment the DUB dataset to the same number of images as DB-5, called DUB-a, We show object recognition performance DB with DUB shows 2 7% points improvements on classification performance, due to augmentation with synthesized images for missing poses in the training set, reaching the level of when all real poses are available (DB). Our synthesized poses outperform traditional data augmentation (DUB-a)
the test set using these four datasets respectively. Comparing
Dalase
nyU
7B
pB-s
pl
G2S-Net
Tralitional
Source
syntheszzed
sugmented
Sizee
25149
37417
37395
37305
Avcuracy
Ioe' fereeer yorkep
1aherr Ineese/ gickin Trato Used Tumk
(1)
Figure 7: (a) Dataset details for training object recognition task, where the x-axis represents different identities (1004) and the y-axis represents the backgrounds (111) and poses (6) each purple and brown pixel means our dataset covers the specific combination of attributes. (b) object recognition accuracy (%) on 37469 test examples, after training (augmented) datasets.
CONCLUSION
We propose new learning framework, Group Supervised Learning (GSL), which admits datasets of examples and their semantic relationships. It provides a learner groups of semantically-related samples, which we show is powerful for zero-shot synthesis. In particular, our Group-supervise Zero-Shot synthesis network (GZS-Net) is capable of training on groups of examples, and can learn disentangled representations by explicitly swapping latent features across training examples, along edges suggested by GSL We show that, to synthesize samples given query with custom attributes, is sufficient to find one example per requested attribute and to combine them in the latent space. We hope that researchers find our learning framework useful and extend it for their applications.
ACKNOWLEDGMENTS
This work was supported by C-BRIC (one of six centers in JUMP, Semiconductor Research Corporation (SRC) program sponsored by DARPA), the Army Research Office (W91 INF202053))) and the Intel and CISCO Corporations. The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yuval Atzmon and Gal Chechik. Probabilistic and-or attribute grouping for zero-shot learning. In Uncertainty in Artificial Intelligence, 2018.
A. Borji, S. Izadi, and L.. Itti. ilab-20m: large-scale controlled object dataset to investigate deep learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in bela-vae. arXiv preprint arXiv:1804.083 03599, 2018.
Ricky Q. Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Yunhao Ge, Jiaping Zhao, and Laurent Itti. Pose augmentation: Class-agnostic object pose trans- formation for object recognition. European Conference on Computer Vision, pp. 138-155. Springer, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In JCML, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Neural Information Processing Systems, 2014.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick. S. Mohamed, and A. Lerchner. B-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Seunghoon Hong, Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7986-7994, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 18Xiv:1802.000 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. A large-scale hierarchical multi-view rgb-d object dataset. In 2011 IEEE international conference on robotics and automation, pp. 1817-1824. IEEE, 2011.
C. H. Lampert. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Oliver I Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel HJ Wigholdus, Skyler T Hawk, and AD Van Knippenberg, Presentation and validation of the radboud faces database. Cognition and emotion, 2418):1377-1388, 2010.
Nikos K. Logothetis, Jon Pauls, and Tomaso Poggiot. Shape representation in the inferior temporal cortex of monkeys. In Current Biology, 1995.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
10

---

Published as a conference paper at ICLR 2021
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXi::1411.18%: 2014.
arXiv preprint
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE' Transactions Neural Networks, 20(1):61 (1::1))81....0111 1-00, 2009.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invee gose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 1415-1424, 2017.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-187, September 2018.
Zhuoqian Yang, Wentao Zhu, Wayne Wu, Chen Qian, Qiang Zhou, Bolei Zhou, and Chen Change Loy. Transmomo: Invariance-driven unsupervised video motion retargeting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 5306-5315, 2020.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imad translation using cycle-consistent adversarial networks. In International Conference on Computer Vision (ICCV'17), 2017.
APPENDIX
A FONTS DATASET
Vary Letter
E
f
h
A
t
D
Vary Size
G
G
G
Vary Font color
b
b
b
b
b
b
Vary Background color
g
g
g
g
g
g
Vary Fonts
M
M
M
M
M
M
Figure 8: Samples from the Fonts dataset, a new parametric dataset we created by rendering characters under distinct attributes. In each row, keep all attributes the same but vary one.
Fonts a computer-generaaed RGB image datasets. Each image, with 128 x 128 pixels, contains an alphabet letter rendered using 5 independent generating attributes: letter identity, size, font color, background color and font. Fig. shows some samples: in each row, we keep all attributes values the same but vary one attribute value. Attribute details are shown in Table 1. The dataset contains all
11

---

Published as a conference paper at ICLR 2021
possible combinations of these attributes, totaling to 1560000 images. Generating attributes for all images are contained within the dataset. Our primary motive for creating the Fonts dataset, is that allows fast testing and idea iteration, disentangled representation learning and zero-shot synthesis.
You can download the dataset and its generating code from: http://ilab. usc. eduu datasets fonts which plan to keep up-to-date with contributions from ourselves and the community.
Table 3: Attributes generating the Fonts dataset
Attribute
Number of Attribute Values
Attribute Value Details
Uppercase Letters (A-2) Lowercase Letters (a-z)
Letter
Small, Medium, Large
Size
(80, 100, 120 pixel height respectively)
Red, Orange, Yellow, Green, Cyan Blue, Purple, Pink, Chocolate, Silver
Font color
10
Red, Orange, Yellow, Green, Cyan Blue, Purple, Pink, Chocolate, Silver
Background color
10
Ubuntu system fonts
Font
100
e.g. aakar, chilanka, sarai, etc.
B-VAE IES
TC-VAE IES
AE-ES
GZS-Net
GT
Shape
Scale
Orientation
Position
Position Y
Figure 9: Zero-shot synthesis performance on dSprites. Columns 6-10 are input group images: from each, we want to extract one attribute (title of column). The goal is to combine the attributes to synthesize an new images. Columns 1-4 are synthesized images, respectively using: auto-encoder + Exhaustive Search (AE+ES), 3- VAE + Exhaustive Search (8- (A=VAE, TC-VAE + Exhaustive Search (TC- TAC-VE and GZS-Net respectively. The 5th column are ground truth (GT), which none of the methods during training synthesis
12

---

Published as a conference paper at ICLR 2021
B BASELINES
B. EXHAUSTIVE SEARCH (ES) AFTER TRAINING AUTO-ENCODER BASED METHODS
After training the baselines: standard Autoencoder, a B-VAE (Higgins et al. 2017). and TC-VAE (Chen et al. 2018). We want to search for the assignment between latent variables and attributes, as these VAES do not make explicit the assignment. This knowing the assignment should hypothetically allow to trade attributes between two images by swapping feature values belonging to the attribute we desire to swap.
To discover the assignment from latent dimension attribute, map all n training images through the encoder, giving a 100D vector per training sample E R We make an 80:20 split on the vectors, obtaining Xirainss E R0.8nx 100 and X R0.2nxx00. Then, ran- domly sample K different partitionings P of the 100D space evenly among the 5 attributes. For each partitioning p E P. we create classification tasks, one task per attribute, according to pi { ( [3pp5] E TR. 8. X [:P E (R }} } For each task j, we train a 3-layer MLP to map X to their known attribute values and measure its performance Finally, we commit to the partitioning E Р with highest average performance on the 5 attribute tasks. This p represents our best effort to determine which latent feature dimensions correspond to which attributes. For zero-shot synthesis with baselines, we swap latent dimensions indicated by partitioning p. We denote three baselines with this Exhaustive Search, using suffix +ES (Fig.
X [:. P,l-
B.2 DIRECT SUPERVISION (DS) ON AUTO-ENCODER LATENT SPACE
The last baseline (AE+DS) directly uses attribute labels supervise the latent disentangled repre- sentation of the auto-encoder by adding auxiliary classification modules. Specifically, the encoder maps an image sample z²()) to a 100-d latent vector 200) = E(x()), ), equally divided into 5 partitions
corresponding 5 attributes: (()) lo(") (i) ..... 96 1. Each attribute partition has attribute label, [v] (6) ....,, vt"), which represent the attribute value (e.g. for font color attribute, the lahel represent different colors: red, green, blue,,etc). We use 5 auxiliary classification modules predict the corresponding class label given each latent attribute partitions as input. We use Cross Entropy loss as the classification loss and the training goal is to minimize both the reconstruction loss and classification loss.
After training, we have assignment between latent variables and attributes, so we can achieve attribute swapping and controlled synthesis (Fig. 4 (AE+DS)). The inferior synthesis performance demonstrates that: The supervision (classification task) preserves discriminative information that is insufficient for photo-realistic generation. While our GZS-Net uses one attribute swap and cross swap which enforce disentangled information to be sufficient for photo-realistic synthesis.
B.3 ELEGANT XIAA ET AL.. 2018)
We utilize the author's ss open-sourced code: htttp:://gtttttii httpp//uw....er.ep.ppeeeae///e//
For ELEGANT and starGAN (Section B.4). we want to synthesis a target image has same identity as id provider image, same background as background provider image, and same pose as pose provider image. To achieve this, we want to change the background and pose attribute of id image.
Although ELEGANT is strong in making image transformations that are local to relatively-small neighborhoods, however, it does not work well for datasets, where image-wide transformations are required for mcaningful synthesis. This can be confirmed by their model design: their final output is a pixel-wise addition of a residual map, plus the input image. Further, ELEGANT treats all attribute values as binary: they represent each attribute value in a different part of the latent space, whereas method devotes part of the latent space to represents all values for attribute. For investigation, we train dozens of ELEGANT models with different hyperparameters, detailed as:
For L0M, the pose and background contain a total of 117 attribute values (6 for pose, 111 for background). As such, we tried training it all attribute values (dividing their latent space among 117 attribute values). We note that this training regime was too slow and the loss values do not seem change much during training, even with various learning rate choices (listed below).
13

---

Published as a conference paper at ICLR 2021
ELEGANT
ELEGANT chouge backgrvund (of background provider)
only change backgrouud (of idientit provider)
ELEGANT only change pose
ELEGANT
GZS with LeH.arloo
Mdnnity
Powa
Background
Figure 10: Zero-shot synthesis for ELEGANT, investigating the modification of pose and background attributes on an identity image. Details are in Section B.3
To reduce the difficulty the task for ELEGANT, we ran further experiments restricting attribute variation to only 17 attribute values (6 for pose, 11 for background) and this shows more qualitative promise than 117 attributes. This is what we report.
Fig I0 shows that ELEGANT finds more challenge in changing the pose than in changing the background. We now explain how we generated Columns and 4 of Fig 10 for modifying the background. We modify the latent features for the identity image before decoding. Since the Identity imput image and the Background input image have known but different background values, their background latent features are represented two different latent spaces. One can swap on one or on both of these latent spaces. Column and 4 of Fig swap only on one latent space. However, Fig. of the main paper, we swap on both positions. We also show swapping only the pose attribute (across 2 latent spaces) in Column I of Fig 10 and swapping both pose and background in Column 2.
To investigate if the model's performance is due poor convergence of the generator, we qualitatively assess its performance on the training set. Fig. I shows output of ELEGANT on training samples. We that the reconstruction (right) of input images (left) shows decent quality, suggesting that the generator network has converged to decently good parameters. Nonetheless, we see artefacts in its outputs when amending attributes, particularly located in pixel locations where change is required. This shows that the model setup of ELEGANT is aware that these pixel values need to be updated, but the actual change is not coherent across the image.
For the above, applied generous sweep of training hyperparameters, including:
Learning rate: author's original is 2e-4, we tried several values between le-5 and le-3, including different rates for generator and discriminator.
Objective term coefficients: There are multiple loss terms for the generator, adversarial loss and reconstruction loss. We used a grid search method by multiplying the original parameters by a number from [0.2. 0.5, 2, 5] for each of the loss terms and tried several combinations.
The update frequency of weights generator (G) and discriminator (D). Since D is easier to learn, we performing update steps on G for every update step on D. We tried A 5, 10, 15, 20, 30, 40,50 50.
14

---

Published as a conference paper at ICLR 2021
B
ELEGANT oub change background
ELEGANT only change
Figure 11: Training performance of ELEGANT. The left columns (A and B) are input image. The following 4 columns the generated synthesized images: A' and B' reconstructions of the input (acceptable quality, suggesting convergence generator), whereas and D are the result of swapping features before the generator: C C/ D) uses the latent features of A (/ B) except by swapping background with B (/ All (C, D, A', B') share the same generator.
We report ELEGANT results showing best qualitative performance.
Overall, ELEGANT does work well for holistic image manipulation (though works well for local image edits, per experiments by authors (Xiao et al. 2018)..
B.4 STARGAN (CHOI ET AL. 2018
We utilize the author's open-sourced code:
Unlike
ELEGANT (Xiao et al. 2018) and our method, starGAN only accepts one input image and an edit information: the edit information, is not extracted from another image this following their method and published code.
C ZERO-SHOT SYNTHESIS PERFORMANCE ON DSPRITES DATASET
We qualitatively evaluate our method. Group-Supervised Zero-Sho Synthesis Network (GZS-Net), against three bascline methods, on zero-shot synthesis tasks on the dSprites dataset.
C.1 DSPRITES
dSprites Matthey et al. 2017 is a dataset 2D shapes procedurally generated from 6 ground truth independent latent factors. These factors color, shape, scale, rotation, and y-positions of sprite. All possible combinations of these latents are present exactly once, generating 737280 total images. Latent factor values (Color: white; Shape: square, ellipse, heart; Scale: 6 values linearly
15

---

Published as a conference paper at ICLR 2021
spaced in [0.5, 1]; Orientation: 40 values in [0, 2 pi]: Position X: 32 values in [0. 1]; Position Y: 32 values [0, I])
C.2 EXPERIMENTS OF BASELINES AND GZS-NET
We train 10-dimensional latent space and partition the it equally among the 5 attributes: 2 for shape, 2 for scale, 2 for orientation, 2 for position X, and 2 for position Y. We use a train:test split of 75:25.
We train 3 baselines: standard Autoencoder, 3- VAV (Higgins al. 2017). and TC-VAE Chen ct al. 2018). To recover the silentto-aattrbure assignment for these baselines, we utilize the Exhaustive Search best-effort strategy, described in the main paper: the only difference is that we change the dimension of Z space from 100 to 10. Once assignments are known, we utilize these baseline VAEs by attribute swapping to do controlled synthesis. We denote these baselines using suffix +ES.
As is shown in Figure 2, GZS-Net can precisely synthesize zero-shot images with new combinations of attributes, producing images similar to the groud truth. The baselines 3-VAE and TC-VAE produce realistic images of good visual quality, however, not satisfying the requested query: therefore, they cannot do controllable synthesis even when equipped with our best-effort Exhaustive Search to discover the disentanglement. Standard auto-encoders combining latents from different examples, giving images outside the distribution training samples (e.g. showing multiple sprites per image).
synthesis meaningful images when
16


Reviews:
[{"review_id": "ued_T6d7ZTI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes Group-Supervised Learning (GSL) that can learn disentangled representations by swapping components in latent space, and enable one-shot novel view synthesis.  They created large dataset to evaluate their method, and demonstrate its effectiveness in controllable image synthesis, and disentanglement, outperforming existing baselines.\n\nPros:\n1. This paper is good-written and clear to follow. The authors can demonstrate their idea well in Section 3 and 4.\n2. Swapping attributes is an interesting and novel idea to encourage disentangled representation learning, which are easy to implement and could have wide applications in many fields.\n3. The experiments are extensive, and the results are able to support their claims.\n\nCons:\n1. In Equation (4), can you explain the defination of  separate losses, L_r, L_sr and L_csr along with the equation? They are not mentioned above, but simply in the algorithm 1. \n\n2. If the cycle loss is removed, will the performance degrade dramatically? Can you please offer ablations for each loss?", "extra_scores": {}, "extra_text": {}}, {"review_id": "tuQ6LAUysp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposed a new training framework, namely GSL, for novel content synthesis. And GSL enables learning of disentangled representations of tangible attributes and achieve novel image synthesis by recombining those swappable components under a zero-shot setting. The framework leverages the underlying semantic links across samples which could be instantiated as a multigraph. Cycle-consistent reconstruction loss as well as reconstruction loss are computed on synthetic samples from swapped latent representations.\n\nReasons for score:\n\nI vote for accepting. Overall, I think the paper is well written and the idea of GSL leveraging the underlying semantic relationships between samples to learn disentangled representations of tangible attributes is pretty novel and interesting. Experimental result on several real and synthetic datasets are also promising.\n\nConcerns:\n\n1) How are hyperparameters {d_j}^m_{j=1} selected? It would be great if the authors could elaborate more about this.\n\n2) Concerning the cycle attribute swap, how to guarantee the intermediate generated image to be reasonable instead of a trivial solution. Seems like no losses (possibly like adversarial loss) is directly added on the intermediate result.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YZ-sjmRa8nX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I. Summary of paper.\n\nThe authors propose GZS-Net, a model based on an auto-encoder that synthesizes realistic images by exploring the semantic relationships between group images. The model is trained with a family of objective functions expressed on groups of examples. \n\nIt first maps training images into disentangled latent representations, which can be decomposed into multiple components, which in the specific implementation correspond to semantic attributes. Components are then recombined to form novel images that can correspond to previously unseen combinations of attributes. \n\nThe main experimental setup considered is zero-shot image synthesis: having seen different combinations of attributes, the model relies on disentangling attribute representations to synthesize novel combinations of previously seen attributes. Good performance is achieved on the Fonts, ilab-20M and RaFD datasets. In addition, the authors perform a more detailed analysis of their proposed approach and show that it results in good downstream performance on an object recognition task.\n\nII. Strong and weak points.\n\nPositive:\n- The proposed task, zero-shot synthesis, is interesting.\n- Overall, the paper is well-written and clear\n- Good results against the selected baselines\n- The novelty of the approach itself.\n\nNegative:\n- The following justifies some of the comments below. The authors claim GAN-related training is unstable with significant mode collapse. This is simply not the case currently, and in particular in the relatively small datasets considered. A comparison with e.g. cycle-GAN is fully warranted.\n- The qualitative comparisons are limited. Yes, the model can generate good-looking images on what are mostly toy tasks. But these generations are commonly compared only to a relatively weak baseline (AE + DS) in fig 6, 5. Cycle GANs would work in this setting, and I supposed might be very competitive.\n- The paper suffers from a general lack of (serious) quantitative comparisons. The authors provide generated images but the single results table on disentanglement does not contain models that would fare well in this setting.\n\nIII. Recommendation and justification.\n\nI feel the paper is borderline, with a slight inclination to reject. On one hand, an interesting task is presented and the writing/clarity/experiments make this submission an interesting submission from a scientific standpoint. On the other hand, a lot of experimental work is needed to show this approach offers a benefit compared to serious baselines.\n\nIV. Advice for author response.\n\nI like this paper conceptually and would be prepared to raise my score. I'm interested in the following, if possible:\n- Is there a way for the authors to provide an indication of sample quality versus adversarial baselines (cycleGAN, starGAN or some related method e.g.) to see if results hold?\n- On the disentanglement side, is there a way to extend Table 1 to show whether the model performs well when compared to models that are good at disentangling, e.g. beta-VAE family.\n\nIf not feasible to include new experiments, please if possible at least discuss the issue wrt. baselines.\n\nV. Improving the paper.\n\nOn the link between zero-shot learning, disentanglement, and generative models, I would suggest including [1, 2]. To be clear, these are just suggestions, choosing to include them or not will have no impact on my rating.\n\nReferences\n\n[1] Sylvain, Tristan, Linda Petrini, and Devon Hjelm. \"Locality and Compositionality in Zero-Shot Learning.\" International Conference on Learning Representations. 2019.\n\n[2] Higgins, Irina, et al. \"SCAN: Learning Hierarchical Compositional Visual Concepts.\" International Conference on Learning Representations. 2018.\n\nEdit: I have read the author's response. They have responded to the questions I had, and they address many of the concerns I had. I am now raising \n\n\nEdit: updating score, and recommending acceptance as per my response to the author's rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4qZ0El7lm5w", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Group-Supervised learning method for zero-shot synthesis. This method can generate multi-attributes images through the GZS-Net.\nHowever, the authors may pay attention to the following questions:\n1. The authors should check for grammar mistakes, e.g.,”Fonts is a...” ->”Fonts are...”.\n2. From the description, the Group-Supervised learning method is just a special data sampling manner used in training. Besides, the total framework is commonly used in GANs for attribute control generation. \n3. The contrast methods are too simple. The reviewer thinks the authors should compare their method with some new works, e.g., “ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face”.\n4. In Section 5.3, the reasons for the results in Table 1 are not analyzed. Why is the accuracy of contrast method AE-DS higher than GZS-Net?\n5. The author should add more quantitative analysis. Such as comparing the FID/IS values of generated results using different methods.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a zero-shot generation approach by disentangling representations into swappable components (each component corresponding to an attribute) and then conditioning on any desired combination of attributes to do zero-shot synthesis of samples containing those attributes.

There were some concerns raised in the original reviews which the authors have addressed in the rebuttal and the revised submission. Post the discussion phase, all reviewers see merit in the proposed ideas and unanimously recommend acceptance. Based on my own reading of the paper and the reviews/author responses, I agree with the assessment. 
Decision: Accept (Poster)