Zero-shot Synthesis with Group-Supervised Learning

1 INTRODUCTION . Primates perform well at generalization tasks . If presented with a single visual instance of an object , they often immediately can generalize and envision the object in different attributes , e.g. , in different 3D pose ( Logothetis et al. , 1995 ) . Primates can readily do so , as their previous knowledge allows them to be cognizant of attributes . Machines , by contrast , are most-commonly trained on sample features ( e.g. , pixels ) , not taking into consideration attributes that gave rise to those features . To aid machine cognition of visual object attributes , a class of algorithms focuses on learning disentangled representations ( Kingma & Welling , 2014 ; Higgins et al. , 2017 ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ) , which map visual samples onto a latent space that separates the information belonging to different attributes . These methods show disentanglement by interpolating between attribute values ( e.g. , interpolate pose , etc ) . However , these methods usually process one sample at a time , rather than contrasting or reasoning about a group of samples . We posit that semantic links across samples could lead to better learning . We are motivated by the visual generalization of primates . We seek a method that can synthesize realistic images for arbitrary queries ( e.g. , a particular car , in a given pose , on a given background ) , which we refer to as controlled synthesis . We design a method that enforces semantic consistency of attributes , facilitating controlled synthesis by leveraging semantic links between samples . Our method maps samples onto a disentangled latent representation space that ( i ) consists of subspaces , each encoding one attribute ( e.g. , identity , pose , ... ) , and , ( ii ) is such that two visual samples that share an attribute value ( e.g. , both have identity “ car ” ) have identical latent values in the shared attribute subspace ( identity ) , even if other attribute values ( e.g. , pose ) differ . To achieve this , we propose a general learning framework : Group Supervised Learning ( GSL , Sec . 3 ) , which provides a learner ( e.g. , neural network ) with groups of semantically-related training examples , represented as multigraph . Given a query of attributes , GSL proposes groups of training examples with attribute combinations that are useful for synthesizing a test example satisfying the query ( Fig . 1 ) . This endows the network with an envisioning capability . In addition to applications in graphics , controlled synthesis can also augment training sets for better generalization on machine learning tasks ( Sec . 6.3 ) . As an instantiation of GSL , we propose an encoder-decoder network for zero-shot synthesis : GroupSupervised Zero-Shot Synthesis Network ( GZS-Net , Sec . 4 ) . While learning ( Sec . 4.2 & 4.3 ) , we repeatedly draw a group of semantically-related examples , as informed by a multigraph created by GSL . GZS-Net encodes group examples , to obtain latent vectors , then swaps entries for one or more attributes in the latent space across examples , through multigraph edges , then decodes into an example within the group ( Sec . 4.2 ) . Our contributions are : ( i ) We propose Group-Supervised Learning ( GSL ) , explain how it casts its admissible datasets into a multigraph , and show how it can be used to express learning from semantically-related groups and to synthesize samples with controllable attributes ; ( ii ) We show one instantiation of GSL : Group-supervised Zero-shot Synthesis Network ( GZS-Net ) , trained on groups of examples and reconstruction objectives ; ( iii ) We demonstrate that GZS-Net trained with GSL outperforms state-of-the-art alternatives for controllable image synthesis on existing datasets ; ( iv ) We provide a new dataset , Fonts1 , with its generating code . It contains 1.56 million images and their attributes . Its simplicity allows rapid idea prototyping for learning disentangled representations . 2 RELATED WORK . We review research areas , that share similarities with our work , to position our contribution . Self-Supervised Learning ( e.g. , Gidaris et al . ( 2018 ) ) admits a dataset containing features of training samples ( e.g. , upright images ) and maps it onto an auxiliary task ( e.g. , rotated images ) : dataset examples are drawn and a random transformation ( e.g. , rotate 90◦ ) is applied to each . The task could be to predict the transformation ( e.g. , =90◦ ) from the transformed features ( e.g. , rotated image ) . Our approach is similar , in that it also creates auxiliary tasks , however , the tasks we create involve semantically-related group of examples , rather than from one example at a time . Disentangled Representation Learning are methods that infer latent factors given example visible features , under a generative assumption that each latent factor is responsible for generating one semantic attribute ( e.g . color ) . Following Variational Autoencoders ( VAEs , Kingma & Welling , 2014 ) , a class of models ( including , Higgins et al. , 2017 ; Chen et al. , 2018 ) achieve disentanglement implicitly , by incorporating into the objective , a distance measure e.g . KL-divergence , encouraging the latent factors to be statistically-independent . While these methods can disentangle the factors without knowing them beforehand , unfortunately , they are unable to generate novel combinations not witnessed during training ( e.g. , generating images of red car , without any in training ) . On the other hand , our method requires knowing the semantic relationships between samples ( e.g. , which objects are of same identity and/or color ) , but can then synthesize novel combinations ( e.g. , by stitching latent features of “ any car ” plus “ any red object ” ) . Conditional synthesis methods can synthesize a sample ( e.g. , image ) and some use information external to the synthesized modalities , e.g. , natural language sentence Zhang et al . ( 2017 ) ; Hong et al . 1http : //ilab.usc.edu/datasets/fonts ( 2018 ) or class label Mirza & Osindero ( 2014 ) ; Tran et al . ( 2017 ) . Ours differ , in that our “ external information ” takes the form of semantic relationships between samples . There are methods based on GAN Goodfellow et al . ( 2014 ) that also utilize semantic relationships including Motion Re-targeting ( Yang et al. , 2020 ) , which unfortunately requires domain-specific hand-engineering ( detect and track human body parts ) . On the other hand , we design and apply our method on different tasks ( including people faces , vehicles , fonts ; see Fig . 1 ) . Further , we compare against two recent GAN methods starGAN ( Choi et al. , 2018 ) and ELEGANT ( Xiao et al. , 2018 ) , as they are state-of-the-art GAN methods for amending visual attributes onto images . While they are powerful in carrying local image transformations ( within a small patch , e.g. , changing skin tone or hair texture ) . However , our method better maintains global information : when rotating the main object , the scene also rotates with it , in a semantically coherent manner . Importantly , our learning framework allows expressing simpler model network architectures , such as feed-forward auto-encoders , trained with only reconstruction objectives , as opposed to GANs , with potential difficulties such as lack of convergence guarantees . Zero-shot learning also consumes side-information . For instance , models of Lampert ( 2009 ) ; Atzmon & Chechik ( 2018 ) learn from object attributes , like our method . However , ( i ) these models are supervised to accurately predict attributes , ( ii ) they train and infer one example at a time , and ( iii ) they are concerned with classifying unseen objects . We differ in that ( i ) no learning gradients ( supervision signal ) are derived from the attributes , as ( ii ) these attributes are used to group the examples ( based on shared attribute values ) , and ( iii ) we are concerned with generation rather than classification : we want to synthesize an object in previously-unseen attribute combinations . Graph Neural Networks ( GNNs ) ( Scarselli et al. , 2009 ) are a class of models described on graph structured data . This is applicable to our method , as we propose to create a multigraph connecting training samples . In fact , our method can be described as a GNN , with message passing functions ( Gilmer et al. , 2017 ) that are aware of the latent space partitioning per attribute ( explained in Sec . 4 ) . Nonetheless , for self-containment , we introduce our method in the absence of the GNN framework . 3 GROUP-SUPERVISED LEARNING . 3.1 DATASETS ADMISSIBLE BY GSL . Formally , a dataset admissible by GSL containing n samples D = { x ( i ) } ni=1 where each example is accompanied with m attributes Da = { ( a ( i ) 1 , a ( i ) 2 , . . . a ( i ) m ) } ni=1 . Each attribute value is a member of a countable set : aj ∈ Aj . For instance , pertaining to visual scenes , A1 can denote foreground-colors A1 = { red , yellow , . . . } , A2 could denote background colors , A3 could correspond to foreground identity , A4 to ( quantized ) orientation . Such datasets have appeared in literature , e.g . in Borji et al . ( 2016 ) ; Matthey et al . ( 2017 ) ; Langner et al . ( 2010 ) ; Lai et al . ( 2011 ) . 3.2 AUXILIARY TASKS VIA MULTIGRAPHS . Given a dataset of n samples and their attributes , we define a multigraph M with node set [ 1 .. n ] . Two nodes , i , k ∈ [ 1 .. n ] with i 6= k are connected with edge labels M ( i , k ) ⊆ [ 1 .. m ] as : M ( i , k ) = { j ∣∣∣ a ( i ) j = a ( k ) j ; j ∈ [ 1 .. m ] } . In particular , M defines a multigraph , with |M ( i , k ) | denoting the number of edges connecting nodes i and k , which is equals the number of their shared attributes . Fig . 2 depicts a ( sub- ) multigraph for the Fonts dataset ( Sec . 5.1 ) . Definition 1 COVER ( S , i ) : Given node set S ⊆ [ 1 .. |Dg| ] and node i ∈ [ 1 .. |Dg| ] we say set S covers node i if every attribute value of i is in at least one member of S. Formally : COVER ( S , i ) ⇐⇒ [ 1 .. m ] = ⋃ k∈S M ( i , k ) . ( 1 ) When COVER ( S , i ) holds , there are two mutually-exclusive cases : either i ∈ S , or i /∈ S , respectively shaded as green and blue in Fig . 2 ( b ) . The first case trivially holds even for small S , e.g . COVER ( { i } , i ) holds for all i . However , we are interested in non-trivial sets where |S| > 1 , as sets with |S| = 1 would cast our proposed network ( Sec . 4 ) to a standard Auto-Encoder . The second case is crucial for zero-shot synthesis . Suppose the ( image ) features of node i ( in Fig . 2 ( b ) ) are not given , we can search for S1 , under the assumption that if COVER ( S1 , i ) holds , then S1 contains sufficient information to synthesize i ’ s features as they are not given ( i /∈ S1 ) . Until this point , we made no assumptions how the pairs ( S , i ) are extracted ( mined ) from the multigraph s.t . COVER ( S , i ) holds . In the sequel , we train with |S| = 2 and i ∈ S. We find that this particular specialization of GSL is easy to program , and we leave-out analyzing the impact of mining different kinds of cover sets for future work .

Summary: The paper proposed a new training framework, namely GSL, for novel content synthesis. And GSL enables learning of disentangled representations of tangible attributes and achieve novel image synthesis by recombining those swappable components under a zero-shot setting. The framework leverages the underlying semantic links across samples which could be instantiated as a multigraph. Cycle-consistent reconstruction loss as well as reconstruction loss are computed on synthetic samples from swapped latent representations.