SPIGAN: Privileged Adversarial Learning from Simulation

1 INTRODUCTION . Learning from as little human supervision as possible is a major challenge in Machine Learning . In Computer Vision , labeling images and videos is the main bottleneck towards achieving large scale learning and generalization . Recently , training in simulation has shown continuous improvements in several tasks , such as optical flow ( Mayer et al. , 2016 ) , object detection ( Marı́n et al. , 2010 ; Vazquez et al. , 2014 ; Xu et al. , 2014 ; Sun & Saenko , 2014 ; Peng et al. , 2015 ) , tracking ( Gaidon et al. , 2016 ) , pose and viewpoint estimation ( Shotton et al. , 2011 ; Papon & Schoeler , 2015 ; Su et al. , 2015 ) , action recognition ( de Souza et al. , 2017 ) , and semantic segmentation ( Handa et al. , 2016 ; Ros et al. , 2016 ; Richter et al. , 2016 ) . However , large domain gaps between synthetic and real domains remain as the main handicap of this type of strategies . This is often addressed by manually labeling some amount of real-world target data to train the model on mixed synthetic and real-world labeled data ( supervised domain adaptation ) . In contrast , several recent unsupervised domain adaptation algorithms have leveraged the potential of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) for pixel-level adaptation in this context ( Bousmalis et al. , 2017 ; Shrivastava et al. , 2016 ) . These methods often use simulators as black-box generators of ( x , y ) input / output training samples for the desired task . Our main observation is that simulators internally know a lot more about the world and how the scene is formed , which we call Privileged Information ( PI ) . This Privileged Information includes physical properties that might be useful for learning . This additional information z is not available in the real-world and is , therefore , generally ignored during learning . In this paper , we propose a novel adversarial learning algorithm , called SPIGAN , to leverage Simulator PI for GAN-based unsupervised learning of a target task network from unpaired unlabeled real-world data . We jointly learn four different networks : ( i ) a generator G ( to adapt the pixel-level distribution of synthetic images to be more like real ones ) , ( ii ) a discriminator D ( to distinguish adapted and real images ) , ( iii ) a task network T ( to predict the desired label y from image x ) , and ( iv ) a privileged network P trained on both synthetic images x and adapted ones G ( x ) to predict their associated privileged information z . Our main contribution is a new method to leverage PI from a simulator via the privileged network P , which acts as an auxiliary task and regularizer to the task network T , the main output of our SPIGAN learning algorithm . We evaluate our approach on semantic segmentation in urban scenes , a challenging real-world task . We use the standard Cityscapes ( Cordts et al. , 2016 ) and Vistas ( Neuhold et al. , 2017 ) datasets as target real-world data ( without using any of the training labels ) and SYNTHIA ( Ros et al. , 2016 ) as simulator output . Although our method applies to any kind of PI that can be predicted via a deep network ( optical flow , instance segmentation , object detection , material properties , forces , ... ) , we consider one of the most common and simple forms of PI available in any simulator : depth from its z-buffer . We show that SPIGAN can successfully learn a semantic segmentation network T using no real-world labels , partially bridging the sim-to-real gap ( see Figure 1 ) . SPIGAN also outperforms related state-of-the-art unsupervised domain adaptation methods . The rest of the paper is organized as follows . Section 2 presents a brief review of related works . Section 3 presents our SPIGAN unsupervised domain adaptation algorithm using simulator privileged information . We report our quantitative experiments on semantic segmentation in Section 4 , and conclude in Section 5 . 2 RELATED WORK . Domain adaptation ( cf . Csurka ( 2017 ) for a recent review ) is generally approached either as domaininvariant learning ( Hoffman et al. , 2013 ; Herath et al. , 2017 ; Yan et al. , 2017 ; Ganin & Lempitsky , 2015 ) or as a statistical alignment problem ( Tzeng et al. , 2014 ; Long et al. , 2015 ) . Our work focuses on unsupervised adaptation methods in the context of deep learning . This problem consists in learning a model for a task in a target domain ( e.g. , semantic segmentation of real-world urban scenes ) by combining unlabeled data from this domain with labeled data from a related but different source domain ( e.g. , synthetic data from simulation ) . The main challenge is overcoming the domain gap , i.e . the differences between the source and target distributions , without any supervision from the target domain . The Domain Adversarial Neural Network ( DANN ) ( Tzeng et al. , 2014 ; Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ) is a popular approach that learns domain invariant features by maximizing domain confusion . This approach has been successfully adopted and extended by many other researchers , e.g. , Purushotham et al . ( 2017 ) ; Chen et al . ( 2017 ) ; Zhang et al . ( 2017 ) . Curriculum Domain Adaptation ( Zhang et al. , 2017 ) is a recent evolution for semantic segmentation that reduces the domain gap via a curriculum learning approach ( solving simple tasks first , such as global label distribution in the target domain ) . Recently , adversarial domain adaptation based on GANs ( Goodfellow et al. , 2014 ) have shown encouraging results for unsupervised domain adaptation directly at the pixel level . These techniques learn a generative model for source-to-target image translation , including from and to multiple domains ( Taigman et al. , 2016 ; Shrivastava et al. , 2016 ; Zhu et al. , 2017 ; Isola et al. , 2017 ; Kim et al. , 2017 ) . In particular , CycleGAN ( Zhu et al. , 2017 ) leverages cycle consistency using a forward GAN and a backward GAN to improve the training stability and performance of image-to-image translation . An alternative to GAN is Variational Auto-Encoders ( VAEs ) , which have also been used for image translation ( Liu et al. , 2017 ) . Several related works propose GAN-based unsupervised domain adaptation methods to address the specific domain gap between synthetic and real-world images . SimGAN ( Shrivastava et al. , 2016 ) leverages simulation for the automatic generation of large annotated datasets with the goal of refining synthetic images to make them look more realistic . Sadat Saleh et al . ( 2018 ) effectively leverages synthetic data by treating foreground and background in different manners . Similar to our approach , recent methods consider the final recognition task during the image translation process . Closely related to our work , PixelDA ( Bousmalis et al. , 2017 ) is a pixel-level domain adaptation method that jointly trains a task classifier along with a GAN using simulation as its source domain but no privileged information . These approaches focus on simple tasks and visual conditions that are easy to simulate , hence having a low domain gap to begin with . On the other hand , Hoffman et al . ( 2016b ) are the first to study semantic segmentation as the task network in adversarial training . Zhang et al . ( 2017 ) uses a curriculum learning style approach to reduce domain gap . Saito et al . ( 2017 ) conducts domain adaptation by utilizing the task-specific decision boundaries with classifiers . Sankaranarayanan et al . ( 2018 ) leverage the GAN framework by learning general representation shared between the generator and segmentation networks . Chen et al . ( 2018 ) use a target guided distillation to encourage the task network to imitate a pretrained model . Zhang et al . ( 2018 ) propose to combine appearance and representation adaptation . Tsai et al . ( 2018 ) propose an adversarial learning method to adapt in the output ( segmentation ) space . Zou et al . ( 2018 ) generates pseudo-labels based on confidence scores with balanced class distribution and propose an iterative self-training framework . Our main novelty is the use of Privileged Information from a simulator in a generic way by considering a privileged network in our architecture ( see Figure 2 ) . We show that for the challenging task of semantic segmentation of urban scenes , our approach significantly improves by augmenting the learning objective with our auxiliary privileged task , especially in the presence of a large sim-to-real domain gap , the main problem in challenging real-world conditions . Our work is inspired by Learning Using Privileged Information ( LUPI ) ( Vapnik & Vashist , 2009 ) , which is linked to distillation ( Hinton et al. , 2015 ) as shown by Lopez-Paz et al . ( 2015 ) . LUPI ’ s goal is to leverage additional data only available at training time . For unsupervised domain adaptation from a simulator , there is a lot of potentially useful information about the generation process that could inform the adaptation . However , that information is only available at training time , as we do not have access to the internals of the real-world data generator . Several works have used privileged information at training time for domain adaptation ( Chen et al. , 2014 ; Hoffman et al. , 2016a ; Li et al. , 2014 ; Sarafianos et al. , 2017 ; Garcia et al. , 2018 ) . Hoffman et al . ( 2016a ) leverage RGBD information to help adapt an object detector at the feature level , while Garcia et al . ( 2018 ) propose a similar concept of modality distillation for action recognition . Inspired by this line of work , we exploit the privileged information from simulators for sim-to-real unsupervised domain adaptation . 3 SIMULATOR PRIVILEGED INFORMATION GAN . 3.1 UNSUPERVISED LEARNING WITH A SIMULATOR . Our goal is to design a procedure to learn a model ( neural network ) that solves a perception task ( e.g. , semantic segmentation ) using raw sensory data coming from a target domain ( e.g. , videos of a car driving in urban environments ) without using any ground truth data from the target domain . We formalize this problem as unsupervised domain adaptation from a synthetic domain ( source domain ) to a real domain ( target domain ) . The source domain consists of labeled synthetic images together with Privileged Information ( PI ) , obtained from the internal data structures of a simulator . The target domain consists of unlabeled images . The simulated source domain serves as an idealized representation of the world , offering full control of the environment ( weather conditions , types of scene , sensor configurations , etc . ) with automatic generation of raw sensory data and labels for the task of interest . The main challenge we address in this work is how to overcome the gap between this synthetic source domain and the target domain to ensure generalization of the task network in the real-world without target supervision . Our main hypothesis is that the PI provided by the simulator is a rich source of information to guide and constrain the training of the target task network . The PI can be defined as any information internal to the simulator , such as depth , optical flow , or physical properties about scene components used during simulation ( e.g. , materials , forces , etc. ) . We leverage the simulator ’ s PI within a GAN framework , called SPIGAN . Our approach is described in the next section .

Summary: The paper focuses on the problem of semantic segmentation across domains. The most standard setting for this task involves real world street images as target and synthetic domains as sources with images produced by simulators of photo-realistic hurban scenes.  This work proposes to leverage further depth information which is actually produced by the simulator together with the source images but which is in general not taken into consideration.