user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Structured Prediction using cGANs with Fusion Discriminator

Paper content:
We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks.
This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data.
This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree.
The proposed method is conceptually simpler than the joint convolutional neural network - conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials.
Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.
Convolutional neural networks (CNNs) have demonstrated groundbreaking results on a variety of different learning tasks.
However, on tasks where high dimensional structure in the data needs to be preserved, per-pixel regression losses typically result in unstructured outputs since they do not take into consideration non-local dependencies in the data.
Structured prediction frameworks such as graphical models and joint CNN-graphical model-based architectures e.g. CNN-CRFs have been used for imposing spatial contiguity using non-local information BID13 BID2 BID24 .
The motivation to use CNN-CRF models stems from their ability to capture some structured information from second order statistics using the pairwise part.
However, statistical interactions beyond the second-order are tedious to incorporate and render the models complicated BID0 BID12 ).
Other approaches have used task-specific perceptual losses to solve this problem BID10 .Generative
models provide another way to represent the structure and spacial contiguity in large high-dimensional datasets with complex dependencies. Implicit generative
models specify a stochastic procedure to produce outputs from a probability distribution. Such models are appealing
because they do not demand parametrization of the probability distribution they are trying to model. Recently, there has been
great interest in CNN-based implicit generative models using autoregressive BID4 and adversarial training frameworks BID16 .Generative adversarial networks
(GANs) BID6 can be seen as a two player minimax game where the first player, the generator, is tasked with transforming a random input to a specific distribution such that the second player, the discriminator, can not distinguish between the true and synthesized distributions. The most distinct feature of adversarial
networks is the discriminator that assesses the discrepancy between the current and target distributions. The discriminator acts as a progressively
precise critic of an increasingly accurate generator. Despite their structured prediction capabilities
, such a training paradigm is often unstable and can suffer from mode collapse. However, recent work on spectral normalization
(SN) and gradient penalty has significantly increased training stability BID7 . Conditional GANs (cGANs) BID19 incorporate conditional
image information in the discriminator and have been widely used for class conditioned image generation . To that effect, unlike in standard GANs, a discriminator
for cGANs discriminates between DISPLAYFORM0 Adversarial loss (a) Concatenated Image Conditioning x y Adversarial loss DISPLAYFORM1 Discriminator models for image conditioning. We propose fusing the features of the input and the ground
truth or generated image rather than concatenating.the generated distribution and the target distribution on pairs of samples y and conditional information x.For class conditioning, several unique strategies have been presented to incorporate class information in the discriminator BID23 BID22 . However, a cGAN can also be conditioned by structured data
such as an image. Such conditioning is much more useful for structured prediction
problems. Since the discriminator in image conditioned-GANs has access to
large portions of the image the adversarial loss can be interpreted as a learned loss that incorporates higher order statistics, essentially eliminating the need to manually design higher order loss functions. Consequently, this variation of cGANs has extensively been used
for image-to-image translation tasks . However, the best way of incorporating image conditional information
into a GAN is not always clear and methods of feeding generated and conditional images to the discriminator tend to use a naive concatenation approach. In this work we address this gap by proposing a discriminator architecture
specifically designed for image conditioning. Such a discriminator can contribute to the promise of generalization GANs
bring to structured prediction problems whereby a singular and simplistic setup can be used for capturing higher order non-local structural information from higher dimensional data without complicated modeling of energy functions.Contributions. We propose an approach to incorporating conditional information into a cGAN
using a fusion architecture (Fig. 1b) . In particular, we make the following key contributions:1. We propose a novel
discriminator architecture optimized for incorporating conditional
information in cGANs for structured prediction tasks. The method is designed to incorporate conditional information in feature space and thereby
allows the discriminator to enforce higher-order consistency in the model. At the same time, this method is conceptually simpler than alternative structured prediction
methods such as CNN-CRFs where higher-order potentials have to be manually incorporated in the loss function.2. We demonstrate the effectiveness of this method on a variety of structured prediction tasks
including semantic segmentation, depth estimation, and generating real images from semantic masks. Our empirical study demonstrates that using a fusion discriminator is more effective in preserving
high-order statistics and structural information in the data.2 RELATED WORK 2.1 CNN-CRF MODELS Models for structured prediction have been extensively studied in
computer vision. In the past these models often entailed the construction of hand-engineered features. In 2015, BID15
demonstrated that a fully convolutional approach to semantic segmentation could yield
state-ofthe-art results at that time with no need for hand-engineering features. BID1 showed that post-processing the results of a CNN with a conditional Markov random field led to
significant improvements. Subsequent work by many authors have refined this approach by incorporating the CRF as a layer within
a deep network and thereby enabling the parameters of both models to be learnt simultaneously BID11 . Many researchers have used this approach for other structured prediction problems, including image-to-image
translation and depth estimation BID14 .In most cases CNN-CRF models only incorporate unary and pairwise potentials. Recent work by BID0 has investigated
incorporating higher-order potentials into CNN-based models for semantic segmentation
, and has found that while it is possible to learn the parameters of these potentials, they can be tedious to incorporate and render the model quite complex. There is a need for developing methods that can incorporate higher-order statistical information with out manual modeling
of higher order potentials.
Structured prediction problems can be posed as image conditioned GAN problems.
The discriminator plays a crucial role in incorporating non-local information in adversarial training setups for structured prediction problems.
Image conditioned GANs usually feed concatenated input and output pairs to the discriminator.
In this research, we proposed a model for the discriminator of cGANs that involves fusing features from both the input and the output image in feature space.
This method provides the discriminator a hierarchy of features at different scales from the conditional data, and thereby allows the discriminator to capture higher-order statistics from the data.
We qualitatively demonstrate and empirically validate that this simple modification can significantly improve the general adversarial framework for structured prediction tasks.
The results presented in this paper strongly suggest that the mechanism of feeding paired information into the discriminator in image conditioned GAN problems is of paramount importance.
The generator G tries to minimize the loss expressed by equation 5 while the discriminator D tries to maximize it.
In addition, we impose an L1 reconstruction loss: DISPLAYFORM0 leading to the objective, DISPLAYFORM1 6.2 GENERATOR ARCHITECTURE We adapt our network architectures from those explained in .
Let CSRk denote a Convolution-Spectral Norm -ReLU layer with k filters.
Let CSRDk donate a similar layer with dropout with a rate of 0.5.
All convolutions chosen are 4 × 4 spatial filters applied with a stride 2, and in decoders they are up-sampled by 2.
All networks were trained from scratch and weights were initialized from a Gaussian distribution of mean 0 and standard deviation of 0.02.
All images were cropped and rescaled to 256 × 256, were up sampled to 268 × 286 and then randomly cropped back to 256 × 256 to incorporate random jitter in the model.

assistant: text: We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.