Modeling correlations in spontaneous activity of visual cortex with centered Gaussian-binary deep Boltzmann machines
Authors: nan wang, Laurenz Wiskott, Dirk Jancke
ICLR 2014

Abstract: Spontaneous cortical activity -- the ongoing cortical activities in absence of sensory input -- are considered to play a vital role in many aspects of both normal brain functions and mental dysfunctions. We present a centered Gaussian-binary deep Boltzmann machine (GDBM) for modeling the activity in visual cortex and relate the random sampling in DBMs to the spontaneous cortical activity. After training on natural image patches, the proposed model is able to learn the filters similar to the receptive fields of simple cells in V1. Furthermore, we show that the samples collected from random sampling in the centered GDBMs encompass similar activity patterns as found in the spontaneous cortical activity of the visual cortex. Specifically, filters having the same orientation preference tend to be active together during random sampling. Our work demonstrates the homeostasis learned by the centered GDBM and its potential for modeling visual cortical activity. Besides, the results support the hypothesis that the homeostatic mechanism exists in the cortex.


Peer review:
[Review]:
=====Responses to the common comments from the Reviewers=====
A)  Main contribution/importance of this work
Clarified in the conclusion that the contribution of this work has two folds:

i) We applied centering trick to GDBMs and show empirically that centered GDBMs can be properly trained without layer-wise pretraining.

ii)  We mainly demonstrate that the GDBM is a meaningful model approach for basic receptive field properties and the emergence of spontaneous activity patterns in early cortical visual areas. We reproduced many aspects of the spontaneous activity with proposed model. Our results support hypotheses assuming that the brain learns to generate expectations of internal states as a generative model. The patterns of the spontaneous activity serve as such expectations. Furthermore, we gave some further insights into spontaneous activity patterns found in [1]. Please refer to E) for more details. 
  
B) The statement of V2-like features in trained GDBMs
Removed the statement suggesting that the model captures receptive field properties in V2. 

However, we want to point out that the hidden units in the second hidden layer have the tendency to have strong connections to similar features in the first layer. We demonstrate that both hidden layers of the centered GDBMs are well trained and the centered GDBMs do not suffer from the problem reported in [9] in which the higher layers were not able to learn meaningful features.

C) The sampling procedure used to generating spontaneous frames
Clarified sampling methods used in both generating orientation maps and collecting spontaneous frames.  An illustration of the sampling procedures used in this work has been added in Figure 1. We have included more details of the sampling procedure in section 3.2.1 and 3.2.2.

Clarified the reason why we do not clamp the visible units to zeroes, which is equivalent to clamp the units to the centering offsets in centered GDBMs, in the Discussion part. Briefly, there are two reasons. 

i) Spontaneous activity is the ongoing activity in the absence of intentional sensory input. Biologically, it is not necessary to cut off the bottom-up stimulus input [22] as in [6, 7], where the authors used DBMs to model the cortical activity with visual impairment or blindness. In [1], the authors reported to make the same findings during presenting a uniform gray screen to the cats as having the cats in a darkened room. 

ii) From the perspective of modeling, we want to estimate samples from the model’s prior distribution over the first-hidden-layer units P(Y) by collecting from Gibbs sampling on the P(X, Y, Z). This sampling procedure bring us the samples that can be considered as the model’s expectation of Y without any bottom-up or top-down knowledge from X and Z. 
Our results suggest that both the bottom-up and the top-down inferences are critical in generating the presented spontaneous activity patterns. By setting the visible units to zeroes, a two-layer GDBM is equivalent to a restricted Boltzmann machine and show worse match to the biological findings in [1]. This is also the prediction we have included in the revised version, i.e. the observed spontaneous is a result of the interactions between the incoming stimulation and feedback from higher areas. Please refer to E) for more details. 
  
D) The misleading usage of “homeostasis”
Removed the term “homeostasis” to avoid misunderstanding. 

We agree that the term 'homeostasis' as used could be misleading as the term is nowadays often used in structural plasticity approaches. By 'homeostasis', we mean in a very general sense that the stable status of the internal milieu in the brain as suggested in the free-energy principle. To our understanding, Friston suggests two different environments, i.e. external and internal milieu, and the biological agents are supposed to keep the homeostasis in both cases. Please refer to the following reference for more details:
Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 1-12, 2010. 
In current DBM frameworks, we lack the 'action' to interact with the external environment. Still, to learn generating a proper expectation of internal states appears a possible way to keep homeostasis of the internal milieu in the face of constantly changing sensory inputs. In contrast, [6, 7] refer 'homeostasis' as to homeostatic plasticity (see above), i.e.  neurons keep their firing rate at a certain level by adapting its bias term during inference. This can be considered as another alternative to keep homeostasis in the theory of free energy principle. Nevertheless, it is an interesting direction to point to the connection between GDBM and homeostatic plasticity in the context of the, admittedly very general, free-energy principle.

E) Comparisons between different models for modeling spontaneous activity
Added descriptions of the experiment results with different models used on trial together with the further interpretations. 

In our experiments, we compared different models (including GDBMs, GRBMs and DBNs) in modeling the spontaneous activity and found that only GDBMs can match the biological experiments faithfully.  The maximum correlation coefficient between spontaneous frames and the orientation maps was 0.50+/-0.06 in GDBMs. In comparison, other models seldom reached 0.2. Furthermore, less than 2% spontaneous frames were significantly in other models, compared with 18+/-9% in the case of GDBMs. 

A possible explanation is that GDBMs use both top-down and bottom-up interactions during inference. In comparison, GRBMs and DBN use either the bottom-up or the top-down inference, in which the samples collected from the model distribution do not show similar activity patterns as found in the spontaneous cortical activity. Besides, by clamping either the visible units or the second-layer hidden units to the centering offsets, we did not observe the reported activity patterns in the spontaneous frames from GDBMs. These results suggest that the spontaneous activity in visual cortex is a result of interaction between the incoming stimulation and feedback from higher areas.