Hierarchical Variational Recurrent Autoencoder withTop-Down prediction

Kosuke Miyoshi (2,3,4), Naoya Arakawa (2,4), Hiroshi Yamakawa (1,2,4)
The University of Tokyo(1) The Whole Brain Architecture Initiative(2),Narrative Nights Inc.(3), Dwango Artificial Intelligence Laboratory(4)Abstract
A framework that deﬁnes functions and interface semantics of the cortical micro-circuit was previously proposed as The Coritical Master Algorithm
Framework (Yamakawa 2017), and a deep learning model that supports this framework is demanded.
We chose the VRNN network model (Variational Recurrent Neural Network) to add a hierarchical feature. We implemented time series prediction with topdown signal, and found the representation in the lower layers became sparsely disentangled, so that the fundamental factors in the sensor input wereextracted. We also discuss the ability of functional diﬀerentiation with HVRNN.

Cortical Master Algorithm Framework
A framework that deﬁnes the functions and interface semantics of cortical microcircuits was previously proposed as the Cortical Master Algorithm Framework (MAF).

Experiments
To examine HVRNN, we trained both VRNN and HVRNN using our own dataset
following the Moving MNIST format with unsupervised training. One sequence
consists of a 20 step image time series (64x64 pixels).

MAF requrires functional and structural prerequisites as below.

VRNN• Dimension reduction• Unsupervised learning• Time series prediction• Disentanglement• Generative model• Sparsity• Internal state• Hierarchy• Prediction with top-down signal• Attention with control signal

HVRNN

Hierarchical VRNN (HVRNN)VRNN

HVRNN (Proposed model)

The graphs on the left show show the mean of latent variable z's posterior
during 20 time steps. Figures on the right show the correlation between eachdimension of z and the factors that the input image data comprises.
VRNN (Variational Recurrent Neural Netowrk) is a deep learning model that is capableof dimension reduction, unsupervised learning, time series prediction, generative model,and internal state. VRNN has the structure in which time series prediction is processedwith latent variable z. Latent variable z's approximated posterior distribution q(zt|x t,z<t)
is predicted by the prior distribution p(zt|x<t,z<t).
By following the MAF requirements, we propose a Hierarchical VRNN model (HVRNN)
that adds hierarchy to the VRNN. In HVRNN we implemented time series prediction inlower layers with top-down signals.

Predictive Coding
Predictive Coding (Rao & Ballad, 1999)

HVRNN

In VRNN some dimensions of latent variable correspond to the movement of adigit, and in HVRNN this correspondence becomes more apparent, and they are
more sparsely disentangled in the ﬁrst layer. In HVRNN we can clearly see that in
the second layer z corresponds to the timing of the bounces of the digit at thetop and bottom walls with precise one frame delay after the bounce timing.

Functional Diﬀerentiation
VRNN has the structure like the conditional VAE whichis conditioned on RNN hidden state h. Conditional VAE
tries to ﬁnd factors of input X that are not included in thecondition h and extract them as latent z. HVRNN's latent z
is predicted by the higher layer, so we assume that theRNN state h in the lower layer is diﬀerentiated from thehigher layer.
To conﬁrm this, we suppressed the RNN state h in eachlayer of HVRNN. When h in the layer1 was suppressed, theshape of the object became corrupt while the position waskept. When there was no top-down prediction, both shapeand position became corrupt. This result implies thefunctional diﬀerentiation in the HVRNN.

Ground Truth:
The top-down prediction in HVRNN can be seen as the predictive coding (Rao 1999).
Predictive coding calculates the diﬀerence between the representation in the lower layer(r) and the top-down prediction (rtd), and sends the prediction error (rtd-r, r-rtd) to bothlower and higher layers to minimize the error.
HVRNN doesn't have these explicit prediction error signal feedback. However, throughthe back-propagation training, it contains exactly the same error feedback by minimizing
KL divergence between the posterior and the prior.

Prediction:
Suppress h of Layer1
Suppress h of Layer2
Suppress h of Layer3