Learning Discriminators as Energy Networks in Adversarial Learning

1 INTRODUCTION . This work focuses on applying adversarial learning ( Goodfellow et al. , 2014a ) to solve structured prediction tasks , e.g. , multi-label classification and image segmentation . Adversarial learning can be formalized as a minimax two-player game between structured prediction models and discriminative models . Discriminative models are learned to distinguish between the outputs predicted by the structured prediction models and the training data , while structured prediction models are learned to predict outputs to fool discriminative models . Though structured prediction models are trained by the gradients of discriminative models , existing methods rarely use discriminative models to improve structured prediction models at the inference stage . A straightforward way of utilizing discriminative models for inference is to follow the ascent gradient directions of discriminative models to refine the predicted outputs . However , due to the wellknown gradient vanishing problems , the gradients of discriminative models are almost zero for all the predicted outputs . It is difficult to resolve the gradient vanishing problems , because they are caused by the training instability of the existing adversarial learning framework . Consequently , most existing methods do not use the information from discriminative models to refine structured prediction models . Most existing adversarial learning methods take discriminative models as classifiers . If discriminative models well separate real and predicted samples , they tend to assign the same scores to all predicted samples . Differently , energy-based models ( LeCun et al. , 2006 ; Gygli et al. , 2017 ) usually predict different energy scores for different samples . Therefore , we propose to train discriminative models as energy-based models to ameliorate the gradient vanishing problems . In our framework , discriminative models are learned to assign scores to evaluate the quality of predicted outputs . Structured prediction models are learned to predict outputs that are judged to have maximum scores by discriminative models . In this way , discriminative models are trained to approximate continues value functions which evaluate the quality of predicted output . The gradients of discriminative models are not zero for predicted outputs . Thus , we can refine structured prediction models by following the ascent gradient directions of discriminative models at the inference stage . In this paper , we refer our method as learning discriminative models to refine structured prediction models ( LDRSP ) . The proposed method learns discriminative models utilizing the data generated by the structured prediction models . Gygli et al . ( 2017 ) found that the key to learning deep value networks is generating proper training data . We propose to augment the training set of discriminative models by following the data generation methods proposed in ( Gygli et al. , 2017 ) . At the training stage , we simultaneously run the inference algorithm to generate extra training samples utilizing models in previous iterations . These samples are useful since they are generated along the gradient-based inference trajectory utilized at the inference stage . We also augment the training set with adversarial samples ( Goodfellow et al. , 2014b ) . These samples are used as negative samples to train the discriminative models . To validate our method , experiments are conducted on multi-label classification , binary image segmentation , and 3-class face segmentation tasks , and experimental results indicate that our method can learn discriminative models to effectively refine structured prediction models . This work has two contributions : ( 1 ) We propose a novel adversarial learning framework for structured prediction , in which the information captured by discriminative models can be used to improve structured prediction models at the inference stage . ( 2 ) We propose to learn discriminative models to approximate continues value functions that evaluate the quality of the predicted outputs , and thus ameliorate the gradient vanishing problems . 2 RELATED WORK . Recently , adversarial learning has been well studied on producing high-resolution and high-quality images ( Denton et al. , 2015 ; Karras et al. , 2017 ) , improving the training stability and getting rid of problems like model collapse ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Mao et al. , 2017 ; Qi , 2017 ) , image to image translation ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Huang et al. , 2018 ) , semantic image segmentation ( Luc et al. , 2016 ; Hwang et al. , 2018 ) and neural dialogue generation ( Li et al. , 2017 ) . These methods learn discriminative models and generative models in an adversarial way , but discriminative models are abandoned once models are trained . Since discriminative models are learned to capture the discriminative distributions between the generated samples and the training data , we argue that they are able to be utilized to make the generated samples more real . As we know , we are the first to learn discriminative models that can be directly utilized to refine structured prediction models . There are some researches that utilize iterative inference methods . Since Denoising Autoencoders ( DAEs ) can estimate the gradients of energy distribution functions , Nguyen et al . ( 2016 ) propose to learn DAEs to capture probability distributions of realistic images . In the inference process , DAEs can be utilized to iteratively refine generated images . Romero et al . ( 2017 ) expand similar ideas to the image segmentation . Different from these methods , in which feed-forward networks are utilized to iteratively refine the generated samples , we adopt a gradient-based inference to find samples with the zero task loss . Learning discriminative models as energy-based models has been recently studied ( Zhao et al. , 2017 ; Dai et al. , 2017 ) . These works only constrain that discriminative models predict lower energy scores for the training data than the generated samples . However , our method learns discriminative models to approximate value functions that evaluate the quality of the generated samples . Moreover , we propose to generate extra training samples to learn discriminative models , so that discriminative models can better capture the probability distributions of the sample space . There is a rising interest in energy-based structured prediction ( Zheng et al. , 2015 ; Chen et al. , 2015 ; Song et al. , 2016 ) . Amos et al . ( 2016 ) proposed to add constraints to the neural network parameters such that the output of the neural network is a convex function of ( some of ) the inputs . Belanger & McCallum ( 2016 ) introduced Structured Prediction Energy Network ( SPEN ) . SPEN relies on a max-margin surrogate objective to ensure that the neural network predicts the lowest energy value for the ground-truth label . Belanger et al . ( 2017 ) improved SPEN by proposing an end-to-end version of SPEN , which directly back-propagates through a computation graph that unrolls gradient-based energy minimization . Inspired by the reinforcement learning , Gygli et al . ( 2017 ) proposed a Deep Value Network ( DVN ) that directly learns to evaluate the quality of different output configures . Compared with these methods , our method adversarially learns energy-based models and structured prediction models rather than learns energy-based models alone . 3 APPROACH . We propose a novel adversarial learning framework for structured prediction , in which discriminative models D ( x , y ; θd ) can be used to refine structured prediction models G ( x ; θg ) at the inference stage . In our method , discriminators are treated as energy-based models , which take both input objects x and possible outputs y as inputs and predict scores in the range of [ 0 , 1 ] . We assume that at the training stage , one can get access to an oracle value function v∗ ( y , y∗ ) , which evaluates the quality of y corresponding to x . Here y∗ is the ground-truth label . The discriminators are learned to mimic the behavior of the oracle value functions . Following ( Gygli et al. , 2017 ) , we utilize intersection over union ( IOU ) and F1 metrics as the oracle value functions for image segmentation and multi-label classification , respectively , which are defined on ( y , y∗ ) ∈ { 1 , 0 } M × { 0 , 1 } M , v∗IOU ( y , y ∗ ) = y ∩ y∗ y ∪ y∗ , ( 1 ) v∗F1 ( y , y ∗ ) = 2 ( y ∩ y∗ ) ( y ∩ y∗ ) + ( y ∪ y∗ ) . ( 2 ) Here y ∩ y∗ denotes the number of dimension i where both yi and y∗i are active and y ∪ y∗ denotes the number of dimensions where at least one of yi and y∗i is active . yi and y ∗ i denote the i-th variable of y and y∗ . To apply v∗ ( y , y∗ ) to the continuous output y , the notions of intersection and union are extended by using element-wise min and max operators , y ∩ y∗ = M∑ i=1 min ( yi , y ∗ i ) , ( 3 ) y ∪ y∗ = M∑ i=1 max ( yi , y ∗ i ) . ( 4 ) We propose to learn discriminators D ( x , y ; θd ) to estimate v∗ ( y , y∗ ) . The learning of discriminators can be understood from a regression setting with z = ( x , y ) as inputs and v = v∗ ( x , y ) as the target outputs . The structured prediction models are learned to predict outputs scored highly by discriminators . Discriminators and structured prediction models are respectively learned by optimizing : min θd Ex∼pX [ 1 2 ( D ( x , G ( x ; θg ) ; θd ) − v∗g ) 2 ] + Ex∼pX [ 1 2 ( D ( x , y∗ ; θd ) − 1 ) 2 ] , ( 5 ) min θg Ex∼pX [ Lg ( G ( x ; θg ) , y ∗ ) + 1 2 ( D ( x , G ( x ; θg ) ; θd ) − 1 ) 2 ] . ( 6 ) Here v∗g = v ∗ ( G ( x ; θg ) , y ∗ ) , and Lg is a surrogate loss . Equation 5 can be understood in two aspects : ( 1 ) It ’ s a modified version of least-square GAN ( Mao et al. , 2016 ) loss , and discriminators are learned to predict oracle values v∗g for the samples predicted by structured prediction models ; ( 2 ) It learns an energy-based model using training samples that consist of ground-truth samples ( x , y∗ , 1 ) and predicted samples ( x , G ( x ; θg ) , v∗g ) . The second term of Equation 6 regularizes structured prediction models such that the predicted samples tend to have higher scores . Once models are trained , discriminators can refine structured prediction models by utilizing a gradient-based inference . The outputs predicted by structured prediction models are updated following the ascent gradient directions of discriminators that lead to the high scores : y ( 0 ) = G ( x ; θg ) , ( 7 ) y ( t+1 ) = PY ( y ( t ) + η ∂ ∂y D ( x , y ( t ) ; θd ) ) . ( 8 ) Here , PY denotes an operator that projects the predicted outputs back to the feasible set of solutions . In a simple case , where Y = [ 0 , 1 ] M , the PY operator clips the predicted outputs . It is observed in experiments , i.e. , the learned discriminators tend to give small gradients during the gradient-based inference . One reason is that the predicted outputs of structured prediction models are already close to the ground-truth labels . In order to further improve the predicted outputs by using the gradientbased inference and overcome the small-gradient issue , we use the normalized gradient method , i.e. , y ( t+1 ) = PY y ( t ) + η ( ∂ ∂yD ( x , y ( t ) ; θd ) ) ∣∣∣∣∣∣ ∂∂yD ( x , y ( t ) ; θd ) ∣∣∣∣∣∣ . ( 9 ) Gygli et al . ( 2017 ) proposed to simultaneously generate training samples to learn their energy-based model , i.e. , Deep Value Network ( DVN ) in the training process . We follow their method to generate extra training samples to learn discriminators . The training samples are a set of tuples ( input x , output y , oracle value v∗ ) represented as T ≡ { ( x ( i ) , y ( i ) , v∗ ( i ) ) } Ni=1 . Here N is the size of the training set , and x ( i ) , y ( i ) and v∗ ( i ) respectively denote the i-th image , i-th output and i-th oracle value in the training set . Similar to ( Gygli et al. , 2017 ) , we utilize two different methods to generate training samples : • Inference samples . At each training iteration , structured prediction models are first used to predict samples . Then , we take these samples as initial solutions and run a gradient-based inference to find high-valued samples of discriminators . These samples are useful since they are generated along the inference trajectory at the inference stage . These samples can be generated during training by using models in a previous iteration ( Gygli et al. , 2017 ) . • Adversarial samples . Maximize the loss : 12 ( D ( x , y ; θd ) − v ∗ ) 2 with respect to y using a gradient-based optimizer ( Goodfellow et al. , 2014b ) . These samples serve as negative samples to learn discriminators . Similar to the inference samples , the adversarial samples are also generated during training . To utilize these training samples to learn discriminators , we add another loss term to the Equation 5 , and the new objective function is as follows : min θd Ex∼pX [ 1 2 ( D ( x , G ( x ; θg ) ; θd ) − v∗g ) 2 ] + Ex∼pX [ 1 2 ( D ( x , y∗ ; θd ) − 1 ) 2 ] + E ( x , y , v∗ ) ∼T [ 1 2 ( D ( x , y ; θd ) − v∗ ) 2 ] . ( 10 ) We utilize the Adam optimizer ( Kingma & Ba , 2014 ) with the momentum term β1 = 0.5 to train structure prediction models and discriminators . At each training iteration , we randomly generate inference or adversarial samples and update the parameters of the structured prediction models and the discriminators according to Equation ( 6 ) and Equation ( 10 ) . The models are trained until convergence .

Summary: This paper effectively learns a variant of a Deep Value Network (Gygli et al 2017), a model consisting of an energy network that assigns scores to input-output tuples that is trained to mimic a task-specific loss. The primary differences between the model presented in this work (titled LDRSP) and DVNs are twofold: first, the initial label prediction used at test time for inference is the output of a model rather than being initialized to all zeros. Second, a GAN-inspired loss is used to train both the scoring function and the initial prediction estimator. This new setup is compared against a variety of recent structured prediction methods on the tasks of multilabel classification, semantic segmentation, and 3-class face segmentation.