Probabilistic Neural Programmed Networks for Scene Generation

32nd Conference on
Neural Information Processing Systems(NeurIPS 2018)

Zhiwei Deng, Jiacheng Chen, Yifang Fu, Greg Mori
Simon Fraser UniversityOverview

PNP-Net: Probabilistic operators

Generating scenes from rich and complex semantics is an important step towards understanding thevisual world. To generate scenes from complex semantic descriptions, we develop a flexible latent
distribution built from reusable, modular components

Our Method: Probabilistic Neural Programmed Networks (PNP-Net)

Experiments

Concept mapping operator

Performance MeasureC x H x W tensor(appearance)

Note: All distributions in PNP-Net’s framework are multivariate
Gaussian, they are represented represented by mean andvariance in the whole compositional procedure

Cx1 vector(scale)concept word(cube)

Detector score for measuring semantic correctness: mAP of Faster-RCNN object detector• Objectness: whether the number of objects are consistent with semantics?
• Object type: whether the objects in generated images have correct type (shape)?

Aggregation operator
Combine• Object-attribute: whether the objects in generated images have correct visual attributes?

Describe● Programmatic /●Modular●compound red and shiny attributedistribution into red shiny attributedistribution
g-PoE: gated Product of Experts● Probabilistic●ground blue attribute to an exact objectsphere●is content aware “decorating”
distribution

Color-MNIST● Compositional
Transform● Reusable

Layout●●●Structured Semantic Concepts

Scene Latent
Distribution

instantiate an appearance distribution basedon sampling size from scale distribution
size sampler: sample bounding box sizetransform
to new spatial size

●Pixels

model interactions between
objects, arrange position forblue sphere and red cube

CLEVR-G(64x64)
Qualitative ComparisonsProblem Setting: rich semantics to complex scenes
Semantics with different forms:- sentence- syntax tree
- scene graph- ...

Complex scenes that conform to input
semantics:- deterministic images- a distribution of images constrained by the semantics

References
1. S. Reed et al. Neural programmer-interpreters. In ICLR, 2016.
2. J. Andreas et al. Deep compositional question answering with neural module networks. In CVPR, 2016.

3. E. Parisotto et al. Neuro-symbolic program synthesis.
4. T. Anh Le, et al. Inference compilation and universal probabilistic programming. In AISTATS 2017.

Illustration of Visual Concepts

PNP-Net: Overall Formulation
We integrate our probabilistic neural modules into canonical VAE framework, the reusable neural modules compose the prior ofimages constrained by given semantics.

Typical semantics contain:
Attributes, Objects, Relations
A blue metal cube is behind a cyan
rubber sphere with a red metal sphere
next to it.

Sampling

Experiments: Generalizability
Notations

CLEVR-G(64x64): Zero-shot setting

Challenges: Complex scenes contain varied visual elements, compositional visual concepts, andThe test set contains combinations of objects, attributes andrelations that are not present in training set

complicated relations between objects, which makes the generation highly challenging.
- An ideal generative model should encompass three core components:● An interpretable generation process that composes complex scenes● What are the primitive visual elements and how are they composed● The rendering of abstract concepts into their pixel-level realizations- Monolithic generative models can generate images of single objects well, but have difficulties handling more

Probabilistic andCompositional
Neural Modules

CLEVR-G(128x128): More complex scenes

complex scenes

We can train the model on simpler scenes, but test it on
more complex cases

Learning framework of PNP-Net

Contributions
We propose Probabilistic Neural Programmed Networks (PNP-Net), a programmatic generative model framework forcomplex scene generation. Specifically, we:- Introduce a set of visual elements and neural modules/programs for modifying the appearance of these visual elements- Integrate our probabilistic neural modules into the canonical VAE framework and generalize the VAE by empowering it withreusable, composable, interpretable modules- Demonstrate generalization ability for complicated scene understanding, including zero-shot learning of novel semantic
compositions

Generated
Complex
ScenesDatasets- an variant of MNIST- an image contains at most 2
digits on different locations- digit has different color, sizeand style- comparisons with monolithic
generative models- an variant of CLEVR designed forimage generation task
- up to 8 objects- different shapes, colors, materials,
relations- comparisons with monolithic
generative models- extra experiments for generalizability

More Details
Please check our project page on Github for more details about model
implementation and data.

Poster produced by Faculty & Curriculum Support (FACS), Georgetown University School of Medicine