instruction:
In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
question:
We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining. P(X)
answer:
Deep Generative Stochastic Networks Trainable by Backprop


question:
Common activation functions used in neural networks can yield to training difficulties due to the saturation behavior of the activation function, which may hide dependencies which are not visible to first order (using only gradients). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that some gradients may sometimes flow, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent to be more exploratory. By adding noise only to the problematic parts of the activation function we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by by noisy variants helps training in many contexts, yielding state-of-the-art results on several datasets, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.
answer:
Noisy Activation Functions


question:
The efficiency of algorithms using sec­ ondary structures for probabilistic inference in Bayesian networks can be improved by ex­ ploiting independence relations induced by evidence and the direction of the links in the original network. In this paper we present an algorithm that on-line exploits indepen­ dence relations induced by evidence and the direction of the links in the original network to reduce both time and space costs. In­ stead of multiplying the conditional proba­ bility distributions for the various cliques, we determine on-line which potentials to multi­ ply when a message is to be produced. The performance improvement of the algorithm is emphasized through empirical evaluations in­ volving large real world Bayesian networks, and we compare the method with the HUGIN and Shafer-Shenoy inference algorithms.
answer:

Lazy Propagation in Junction Trees