LEARNING GENERATIVE MODELS FOR DEMIXING OF STRUCTURED SIGNALS FROM THEIR SUPERPOSITION USING GANS

1 INTRODUCTION . In this paper , we consider the classical problem of separating two structured signals observed under the following superposition model : Y = X +N , ( 1 ) where X ∈ X and N ∈ N are the constituent signals , and X , N ⊆ Rp denote the two structured sets . In general the separation problem is inherently ill-posed ; however , with enough structural assumption on X and N , it has been established that separation is possible . Depending on the application one might be interested in estimating only X , which is referred to as denoising , or in recovering both X and N which is referred to as demixing . Both demixing and denoising arise in a variety of important practical applications in the areas of signal/image processing , computer vision , and statistics Chen et al . ( 2001 ) ; Elad et al . ( 2005 ) ; Bobin et al . ( 2007 ) ; Candès et al . ( 2011 ) . Most of the existing techniques assume some prior knowledge on the structures of X andN in order to recover the desired component signal ( s ) . Prior knowledge about the structure of X and N can only be obtained if one has access to the generative mechanism of the signals or has access to clean samples from the probability distribution defined over sets X and N . In many practical settings , neither of these may be feasible . In this paper , we consider the problem of separating constituent signals from superposed observations when clean access to samples from the distribution is not available . In particular , we are given superposed observations { Yi = Xi + Ni } Ki=1 where Xi ∈ X and Yi ∈ N are i.i.d samples from their respective ( unknowns ) distributions . In this setup , we explore two questions : First , How can one learn prior knowledge about the individual components from superposition samples ? Second , Can we leverage the implicitly learned constituent distributions for tasks such as denoising and demixing ? 1.1 SETUP AND OUR TECHNIQUE . Motivated by the recent success of generative models in high dimensional statistical inference tasks such as compressed sensing in Bora et al . ( 2017 ; 2018 ) , in this paper , we focus on Generative Adversarial Network ( GAN ) based generative models to implicitly learn the distributions , i.e. , generate samples from their distributions . Most of the existing works on GANs typically assume access to clean samples from the underlying signal distribution . However , this assumption clearly breaks down in the superposition model considered in our setup , where the structured superposition makes training generative models very challenging . In this context , we investigate the first question with varying degrees of assumption about the access to clean samples from the two signal sources . We first focus on the setting when we have access to samples only from the constituent signal class N and observations , Yi ’ s . In this regard , we propose the denoising-GAN framework . However , assuming access to samples from one of the constituent signal class can be restrictive and is often not feasible in real-world applications . Hence , we further relax this assumption and consider the more challenging demixing problem , where samples from the second constituent component are not available and solve it using what we call the demixing-GAN framework . Finally , to answer the second question , we use our trained generator ( s ) from the proposed GAN frameworks for denoising and demixing tasks on unseen test samples ( i.e. , samples not used in the training process ) by discovering the best hidden representation of the constituent components from the generative models . In addition to the denoising and demixing problems , we also consider a compressive sensing setting to test the trained generator ( s ) . Below we explicitly list the contribution made in this paper : 1 . Under the assumption that one has access to the samples from one of the constituent component , we extend the canonical GAN framework and propose denoising-GAN framework . This learns the prior from the training data that is heavily corrupted by additive structured component . We demonstrate its utility in denoising task via numerical experiments . 2 . We extend the above denoising-GAN and propose demixing-GAN framework . This learns the prior for both the constituent components from their superposed observations , without access to separate samples from any of the individual components . We demonstrate its utility in demixing task via numerical experiments . The rest of this paper is organized as follows : In section 2 , we discuss relevant previous works and compare our novelty over these existing methods . In section 3 , we formally introduce our proposed approach , and in section 4 , we provide some experimental results to validate our framework . 2 APPLICATION AND PRIOR ART . To overcome the inherent ambiguity issue in problem 1 , many existing methods have assumed that the structures of sets ( i.e. , the structures can be low-rank matrices , or have sparse representation in some domain ( McCoy & Tropp , 2014 ) ) X and N are a prior known and also that the signals from X and N are “ distinguishable ” ( Elad & Aharon , 2006 ; Soltani & Hegde , 2016 ; 2017 ; Druce et al. , 2016 ; Elyaderani et al. , 2017 ; Jain et al. , 2017 ) . This assumption is a big restriction in many realworld applications . Recently , there have been some attempts to automate this hard-coding approach . Among them , structured sparsity Hegde et al . ( 2015 ) , dictionary learning Elad & Aharon ( 2006 ) , and in general manifold learning are the prominent ones . While these approaches have been successful to some extent , they still can not fully address the need for prior structure . Over the last decade , deep neural networks have been demonstrated to learn useful representations of real-world signals such as natural images , and thus have helped us understand the structure of these high dimensional signals , for e.g . using deep generative models ( Ulyanov et al. , 2017 ) . In this paper , we focus on Generative adversarial networks GANs ) Goodfellow et al . ( 2014 ) as the generative models for implicitly learning the distribution of constituent components . GANs have been established as a very successful tool for generating structured high-dimensional signals ( Berthelot et al. , 2017 ; Vondrick et al. , 2016 ) as they do not directly learn a probability distribution ; instead , they generate samples from the target distribution ( s ) ( Goodfellow , 2016 ) . Hence , they do not need to parametrize the distributions . If we assume that the structured signals are drawn from a distribution lying on a low-dimensional manifold , GANs generate points in the high-dimensional space that resemble those coming from the true underlying distribution . Since their inception by Goodfellow et al . ( 2014 ) , there has been a flurry of works on GANs ( Zhu et al. , 2017 ; Yeh et al. , 2016 ; Subakan & Smaragdis , 2018 ) . In most of the existing works on GANs with few notable exceptions Wu et al . ( 2016 ) ; Bora et al . ( 2018 ) ; Kabkab et al . ( 2018 ) ; Hand et al . ( 2018 ) ; Zhu et al . ( 2016 ) , it is implicitly assumed that one has access to clean samples of the desired signal . However , in many practical scenarios , the desired signal is often accompanied by unnecessary components . Recently , GANs have also been used for capturing of the structure of highdimensional signals specifically for solving inverse problems such as sparse recovery , compressive sensing , and phase retrieval ( Bora et al. , 2017 ; Kabkab et al. , 2018 ; Hand et al. , 2018 ) . Specifically , Bora et al . ( 2017 ) have shown that generative models provide a good prior to structured signals , for e.g. , natural images , under compressive sensing settings over sparsity-based recovery methods . They rigorously analyze the statistical properties of a generative model based on compressed sensing and provide theoretical guarantees and experimental evidence to support their claims . However , they don ’ t explicitly propose an optimization procedure to solve the recovery problem . They simply suggest using stochastic gradient-based methods in the low-dimensional latent space to recover the signal of interest . This has been addressed in Shah & Hegde ( 2018 ) , where the authors propose using a projected gradient descent algorithm for solving the recovery problem directly in the ambient space ( space of the desired signal ) . They provide theoretical guarantees for the convergence of their algorithm and also demonstrate improved empirical results over Bora et al . ( 2017 ) . While GANs have found many applications , most of them need direct access to the clean samples from the unknown distribution , which is not the case in many real applications such as medical imaging . AmbientGAN framework Bora et al . ( 2018 ) partially addresses this problem . In particular , they studied various measurement models and showed that their GAN can find samples of clean signals from corrupted observations . Although similar to our effort , there are several key differences between ours and AmbientGAN . Firstly , AmbientGAN assumes that the measurement model and parameters are known , which is a very strong and limiting assumption in real applications . One of our main contributions is addressing this limitation by studying the demixing problem . Second , for the noisy measurement settings , AmbientGAN assumes an arbitrary measurement noise and no corruption in the underlying component . We consider the corruption models in the signal domain rather than as a measurement one . This allows us to study the denoising problem from a highly structured corruption . 3 BACKGROUND AND THE PROPOSED IDEA . 3.1 BACKGROUND . Generative Adversarial Networks ( GANs ) are one of the successful generative models in practice was first introduced by Goodfellow et al . ( 2014 ) for generating samples from an unknown target distribution . As opposed to the other approaches for density estimation such as Variational AutoEncoders ( VAEs ) Kingma & Welling ( 2013 ) , which try to learn the distribution itself , GANs are designed to generate samples from the target probability density function . This is done through a zero-sum game between two players , generator , G and discriminator , D in which the generator G plays the role of producing the fake samples and discriminator D plays the role of a cop to find the fake and genuine samples . Mathematically , this is accomplished through the following min-max optimization problem : min θg max θd Ex∼Dx [ log ( Dθd ( x ) ) ] + Ez∼Dz [ log ( 1−Dθd ( Gθg ( z ) ) ) ] , ( 2 ) where θg and θd are the parameters of generator networks and discriminator network respectively , and Dx denotes the target probability distribution , and Dz represents the probability distribution of the hidden variables z ∈ Rh , which is assumed either a uniform distribution in [ −1 , 1 ] h , or standard normal . it has been shown that ifG andD have enough capacity , then solving optimization problem 2 by alternative stochastic gradient descent algorithm guarantees the distribution Dg at the output of the generator converges to Dx . Having discussed the basic setup of GANs next we discuss the proposed modifications this to basic GAN setup that allow for usage of GANs as a generative model for denoising and demixing structured signals .

Summary: In this, paper a GANs-based framework for additive (image) denoising and demixing is proposed. The proposed methodology for denoising largely relies on the Ambient GAN model and hence the technical contribution of the paper in this task appears to be limited. Regarding demixing, as explained in the comments below, the proposed model appears to be superficial in the sense that neither theoretical analysis nor thorough empirical evaluation is provided. The proposed method is evaluated on both tasks (i.e., denoising and demixing) by conducting toy experiments on handwritten digits (MNIST).