LEARNING GENERATIVE MODELS FOR DEMIXING OF STRUCTURED SIGNALS FROM THEIR SUPERPOSITION USING GANS

1 INTRODUCTION . In this paper , we consider the classical problem of separating two structured signals observed under the following superposition model : Y = X +N , ( 1 ) where X ∈ X and N ∈ N are the constituent signals , and X , N ⊆ Rp denote the two structured sets . In general the separation problem is inherently ill-posed ; however , with enough structural assumption on X and N , it has been established that separation is possible . Depending on the application one might be interested in estimating only X , which is referred to as denoising , or in recovering both X and N which is referred to as demixing . Both demixing and denoising arise in a variety of important practical applications in the areas of signal/image processing , computer vision , and statistics Chen et al . ( 2001 ) ; Elad et al . ( 2005 ) ; Bobin et al . ( 2007 ) ; Candès et al . ( 2011 ) . Most of the existing techniques assume some prior knowledge on the structures of X andN in order to recover the desired component signal ( s ) . Prior knowledge about the structure of X and N can only be obtained if one has access to the generative mechanism of the signals or has access to clean samples from the probability distribution defined over sets X and N . In many practical settings , neither of these may be feasible . In this paper , we consider the problem of separating constituent signals from superposed observations when clean access to samples from the distribution is not available . In particular , we are given superposed observations { Yi = Xi + Ni } Ki=1 where Xi ∈ X and Yi ∈ N are i.i.d samples from their respective ( unknowns ) distributions . In this setup , we explore two questions : First , How can one learn prior knowledge about the individual components from superposition samples ? Second , Can we leverage the implicitly learned constituent distributions for tasks such as denoising and demixing ? 1.1 SETUP AND OUR TECHNIQUE . Motivated by the recent success of generative models in high dimensional statistical inference tasks such as compressed sensing in Bora et al . ( 2017 ; 2018 ) , in this paper , we focus on Generative Adversarial Network ( GAN ) based generative models to implicitly learn the distributions , i.e. , generate samples from their distributions . Most of the existing works on GANs typically assume access to clean samples from the underlying signal distribution . However , this assumption clearly breaks down in the superposition model considered in our setup , where the structured superposition makes training generative models very challenging . In this context , we investigate the first question with varying degrees of assumption about the access to clean samples from the two signal sources . We first focus on the setting when we have access to samples only from the constituent signal class N and observations , Yi ’ s . In this regard , we propose the denoising-GAN framework . However , assuming access to samples from one of the constituent signal class can be restrictive and is often not feasible in real-world applications . Hence , we further relax this assumption and consider the more challenging demixing problem , where samples from the second constituent component are not available and solve it using what we call the demixing-GAN framework . Finally , to answer the second question , we use our trained generator ( s ) from the proposed GAN frameworks for denoising and demixing tasks on unseen test samples ( i.e. , samples not used in the training process ) by discovering the best hidden representation of the constituent components from the generative models . In addition to the denoising and demixing problems , we also consider a compressive sensing setting to test the trained generator ( s ) . Below we explicitly list the contribution made in this paper : 1 . Under the assumption that one has access to the samples from one of the constituent component , we extend the canonical GAN framework and propose denoising-GAN framework . This learns the prior from the training data that is heavily corrupted by additive structured component . We demonstrate its utility in denoising task via numerical experiments . 2 . We extend the above denoising-GAN and propose demixing-GAN framework . This learns the prior for both the constituent components from their superposed observations , without access to separate samples from any of the individual components . We demonstrate its utility in demixing task via numerical experiments . The rest of this paper is organized as follows : In section 2 , we discuss relevant previous works and compare our novelty over these existing methods . In section 3 , we formally introduce our proposed approach , and in section 4 , we provide some experimental results to validate our framework . 2 APPLICATION AND PRIOR ART . To overcome the inherent ambiguity issue in problem 1 , many existing methods have assumed that the structures of sets ( i.e. , the structures can be low-rank matrices , or have sparse representation in some domain ( McCoy & Tropp , 2014 ) ) X and N are a prior known and also that the signals from X and N are “ distinguishable ” ( Elad & Aharon , 2006 ; Soltani & Hegde , 2016 ; 2017 ; Druce et al. , 2016 ; Elyaderani et al. , 2017 ; Jain et al. , 2017 ) . This assumption is a big restriction in many realworld applications . Recently , there have been some attempts to automate this hard-coding approach . Among them , structured sparsity Hegde et al . ( 2015 ) , dictionary learning Elad & Aharon ( 2006 ) , and in general manifold learning are the prominent ones . While these approaches have been successful to some extent , they still can not fully address the need for prior structure . Over the last decade , deep neural networks have been demonstrated to learn useful representations of real-world signals such as natural images , and thus have helped us understand the structure of these high dimensional signals , for e.g . using deep generative models ( Ulyanov et al. , 2017 ) . In this paper , we focus on Generative adversarial networks GANs ) Goodfellow et al . ( 2014 ) as the generative models for implicitly learning the distribution of constituent components . GANs have been established as a very successful tool for generating structured high-dimensional signals ( Berthelot et al. , 2017 ; Vondrick et al. , 2016 ) as they do not directly learn a probability distribution ; instead , they generate samples from the target distribution ( s ) ( Goodfellow , 2016 ) . Hence , they do not need to parametrize the distributions . If we assume that the structured signals are drawn from a distribution lying on a low-dimensional manifold , GANs generate points in the high-dimensional space that resemble those coming from the true underlying distribution . Since their inception by Goodfellow et al . ( 2014 ) , there has been a flurry of works on GANs ( Zhu et al. , 2017 ; Yeh et al. , 2016 ; Subakan & Smaragdis , 2018 ) . In most of the existing works on GANs with few notable exceptions Wu et al . ( 2016 ) ; Bora et al . ( 2018 ) ; Kabkab et al . ( 2018 ) ; Hand et al . ( 2018 ) ; Zhu et al . ( 2016 ) , it is implicitly assumed that one has access to clean samples of the desired signal . However , in many practical scenarios , the desired signal is often accompanied by unnecessary components . Recently , GANs have also been used for capturing of the structure of highdimensional signals specifically for solving inverse problems such as sparse recovery , compressive sensing , and phase retrieval ( Bora et al. , 2017 ; Kabkab et al. , 2018 ; Hand et al. , 2018 ) . Specifically , Bora et al . ( 2017 ) have shown that generative models provide a good prior to structured signals , for e.g. , natural images , under compressive sensing settings over sparsity-based recovery methods . They rigorously analyze the statistical properties of a generative model based on compressed sensing and provide theoretical guarantees and experimental evidence to support their claims . However , they don ’ t explicitly propose an optimization procedure to solve the recovery problem . They simply suggest using stochastic gradient-based methods in the low-dimensional latent space to recover the signal of interest . This has been addressed in Shah & Hegde ( 2018 ) , where the authors propose using a projected gradient descent algorithm for solving the recovery problem directly in the ambient space ( space of the desired signal ) . They provide theoretical guarantees for the convergence of their algorithm and also demonstrate improved empirical results over Bora et al . ( 2017 ) . While GANs have found many applications , most of them need direct access to the clean samples from the unknown distribution , which is not the case in many real applications such as medical imaging . AmbientGAN framework Bora et al . ( 2018 ) partially addresses this problem . In particular , they studied various measurement models and showed that their GAN can find samples of clean signals from corrupted observations . Although similar to our effort , there are several key differences between ours and AmbientGAN . Firstly , AmbientGAN assumes that the measurement model and parameters are known , which is a very strong and limiting assumption in real applications . One of our main contributions is addressing this limitation by studying the demixing problem . Second , for the noisy measurement settings , AmbientGAN assumes an arbitrary measurement noise and no corruption in the underlying component . We consider the corruption models in the signal domain rather than as a measurement one . This allows us to study the denoising problem from a highly structured corruption . 3 BACKGROUND AND THE PROPOSED IDEA . 3.1 BACKGROUND . Generative Adversarial Networks ( GANs ) are one of the successful generative models in practice was first introduced by Goodfellow et al . ( 2014 ) for generating samples from an unknown target distribution . As opposed to the other approaches for density estimation such as Variational AutoEncoders ( VAEs ) Kingma & Welling ( 2013 ) , which try to learn the distribution itself , GANs are designed to generate samples from the target probability density function . This is done through a zero-sum game between two players , generator , G and discriminator , D in which the generator G plays the role of producing the fake samples and discriminator D plays the role of a cop to find the fake and genuine samples . Mathematically , this is accomplished through the following min-max optimization problem : min θg max θd Ex∼Dx [ log ( Dθd ( x ) ) ] + Ez∼Dz [ log ( 1−Dθd ( Gθg ( z ) ) ) ] , ( 2 ) where θg and θd are the parameters of generator networks and discriminator network respectively , and Dx denotes the target probability distribution , and Dz represents the probability distribution of the hidden variables z ∈ Rh , which is assumed either a uniform distribution in [ −1 , 1 ] h , or standard normal . it has been shown that ifG andD have enough capacity , then solving optimization problem 2 by alternative stochastic gradient descent algorithm guarantees the distribution Dg at the output of the generator converges to Dx . Having discussed the basic setup of GANs next we discuss the proposed modifications this to basic GAN setup that allow for usage of GANs as a generative model for denoising and demixing structured signals .

Summary: This paper proposed two new GAN structures for learning a generative modeling using the superposition of two structured components. These two structures can be viewed as an extension of AmbientGAN. Experiments results on MNIST dataset are presented. Overall, the demixing-GAN structure is relatively novel. However, the potential application seems limited and the experiment result is not sufficient enough to support the idea. Detail comments are as following,