Neuronal Synchrony in Complex-Valued Deep Networks
Authors: David Reichert, Thomas Serre
ICLR 2014

Abstract: Deep learning has recently lead to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to real cortical circuits. The challenge is to identify which neural mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile deep representations. 
We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter aspect, we demonstrate the potential of the approach in several simple experiments. Thus, synchrony could implement a flexible mechanism that fulfills multiple functional roles in deep networks.


Peer review:
[Review]:
We thank the reviewers for the fair reviews. For simplicity, below we refer to 'Anonymous 4c84', 'Anonymous ec9e', and 'Anonymous 0ae5' as reviewer 1-3, respectively.

We agree with the overall assessment of the reviewers. This is early work that intends to communicate a potentially powerful idea, backed up by simple experiments. There are several avenues for extending it towards principled theoretical frameworks and to address e.g. the need for learning, and we were hoping for feedback from the community to that end.

Perhaps the main issue, as raised by reviewer 1, is whether this work would be better suited to a short workshop paper, given its early stage. This is a fair point. At the same time, given the amount of material we had, and given the solid 'story' that we wanted to communicate, it really didn't make sense to us not to write a full-length paper. For a conference track submission, we think the strengths of the work, such as its originality (to this audience) and quality of presentation, could overcome its shortcomings (but of course that's ultimately up to the reviewers and the chair to decide). We can also address several of the concrete concerns raised by the reviewers, and will do so in the following.


****** Changes to paper ******

We are in the process of uploading an updated paper (v4, ETA Feb 18 7pm EST) to the arXiv, which takes the reviewers' comments into account. We clarified notation in the main text (now slightly breaking the 9 pages limit, which should be acceptable at this point) and wording in appendix B. We also added some of the main issues raised by the reviewers and our discussion thereof as appendix C, whenever it made sense to expand on points that were only briefly touched on in the main text.


****** Reviewer 1 & Reviewer 2 ******

'It is not clear that the segmentation is working for the bars experiment because multiple bars are colored by the same phase. What is the goal here? that each bar has a different, unique phase value? Does the underlying phase distribution effectively partition the bars? '

'there is an issue of phase resolution: it appears that four different bars are all coded the same shade of green. Is this a problem? A readout mechanism might be confused and judge all these bars to be one object, even though bars occur independently in the training data.'

This is a very important issue that we are still considering. It is perhaps an issue more generally with the underlying biological theories rather than just our specific approach. As we noted in the paper, some theories pose that a limit on how many discrete objects can be represented in an oscillation cycle, without interference, explains certain capacity limits in cognition. The references we cited (Jensen & Lisman, 2005; Fell & Axmacher, 2011) refer to working memory as an example (often 4-7 items; note the number of peaks in Figure 3c -- obviously this needs more quantitative analysis). We would posit that, more generally, analysis of visual scenes requiring the concurrent separation of multiple objects is limited accordingly (one might call this a prediction -- or a `postdiction'? -- of our model). The question is then, how does the brain cope with this limitation? As usual in the face of perceptual capacity limits, the solution likely would involve attentional mechanisms. Such mechanisms might dynamically change the grouping of sensory inputs depending on task and context, such as whether questions are asked about individual parts and fine detail, or object groups and larger patterns. In the bars example, one might perceive the bars as a single group or texture, or focus on individual bars as capacity allows, perhaps relegating the rest of the image to a general background. 

Dynamically changing phase assignments according to context, through top-down attentional input, should, in principle, be possible within the proposed framework: this is similar to grouping according to parts or wholes with top-down input, as in the experiment of Section 3.2.


****** Reviewer 1 ******

Regarding the similarity to Rao et al: as we've acknowledged in the paper, the work is similar in several points (we arrived at our framework and results independently and were not aware of Rao et al.'s work initially -- we do not think the latter is particularly well known in this community). However, we would want to counter the impression that our work does not provide additional contributions. First of all, to clarify the issue of training on multiple objects: in Rao et al.'s work, the training data consists of a small number of fixed 8x8 images (N <= 16 images *in total* for a dataset), containing simple patterns (one example has 4 small images with two faces instead). To demonstrate binding by synchrony, two of these patterns are superimposed during test time. We believe that going beyond this extremely constrained task, in particular showing that the binding can work when trained and tested on multiple objects, on multiple datasets including MNIST containing thousands of (if simple) images, is a valid contribution from our side, which is not diminished by the fact that this result relies on the capability of the DBM (indeed, showing that this works with the DBM is itself a contribution as it might tell us something interesting about the kind of representations learned by a DBM that is not usually made explicit). 

Similarly, as far as we can see, Rao et al. do not discuss the gating aspect at all (as we mentioned in our paper), nor the specific issues with excitation and inhibition (Section 2.1) that we pointed out as motivation for using both classic and synchrony terms. Lastly, the following issues are addressed in our experiments only: network behavior on more than two objects; synchronization for objects that are not contiguous in the input images, as well as part vs. whole effects (Section 3.2); decoding distributed hidden representations according to phase (Section 3.3; in particular, it seems to be the case that Rao et al.s networks had a localist (single object<->single unit) representation in the top hidden layer in the majority of cases).

'the introduction of phase is done in an ad-hoc way, without real justification from probabilistic goals [...]'

We agree that framing our approach as a proper probabilistic model would be helpful and perhaps more convincing to this audience (e.g. using an extension of the DUBM of Zemel et al., 1995, as discussed in the paper). At the same time, we think there is value to presenting the heuristic as is, based on a specific neuronal activation function, to emphasize that this idea could find application in neural networks more generally, not only those with a probabilistic interpretation/Boltzmann machines (that our approach is divorced from any one particular model is another difference when compared to Rao et al.'s work). In particular, we have performed exploratory experiments with networks trained (pretrained as real-valued nets or trained as complex-valued nets) with backprop, including (convolutional) feed-forward neural networks, autoencoders, or recurrent networks, as well as a biological model of lateral interactions in V1. We agree with the reviewer that a more rigorous mathematical and quantitative analysis is needed in any case.

'the results in the appendix appear to indicate that the approach is not working very well in general, and the best results are the ones shown in the main text.' 

We are not sure what exactly the reviewer is referring to here. If it is our statement that our approach of using pretrained real-valued networks does not always work, then yes, that is an issue that needs to be addressed. However, we should perhaps clarify that what we meant is: for some datasets and training parameters, models did not perform well; in cases where they did perform reasonably well however, that performance was relatively consistent across images, and the results show representative examples from those models. If, on the other hand, the reviewer is referring to results in the supplementary figure supposedly looking different from the figures in the main text, then no, other than perhaps with the schematic overview in Figure 6, we did not purposefully cherry-pick nicer looking results to display (most of the figures were actually cropped from the same larger figures simply for space reasons).

'How is the phase distribution segmented? Phase is a continuous variable, and the segmentation/partitioning seems to be done by hand for the examples. This needs to be addressed.'

Partitioning was done with k-means, not by hand...? Other options are possible (also depending on whether the aim is a principled machine learning framework or addressing questions about the brain with a biological model). It is true that phase is a continuous variable, however, our results indicate that there is a tendency to form discrete phase clusters, in line with biological models (e.g. the one of Miconi & VanRullen).

'Also, what about the overlaps of the bars? these areas seem to be mis- or ambiguously labeled. is this a bug or a feature?' 

This is more of a problem with the task itself being ill-defined on binary images, where an overlapping pixel cannot really be meaningfully said to belong to either object alone (as there is no occlusion as such). We plan to use (representations of) real-valued images in the future.


****** Reviewer 2 ******

'Comment: The text leading up to equation 1 is confusing regarding z. Is it an output or an input? It doesn't seem like we're dealing with a dynamical system, and the input was called x in the paragraph above. '

With x and z we just refer to, respectively, real-valued and complex-valued states in general. There are several notions of input here: the input (image) to the overall network, the units/states providing input to a specific unit, the total 'post-synaptic' input w . z, and the term that is ultimately used as input to the activation function with a real-valued domain (e.g. |w .  z|). We have attempted to clarify this in this revision of the paper by introducing some additional variables (perhaps the reviewer could check whether Section 2.1 is clearer now).

'Comment: the use of |x| in equation 2 is confusing because presumably |w . x| is a vector norm whereas in w . |x| it denotes elementwise magnitude of the complex elements of x. Right?'

Assuming the reviewer meant to write z not x: No, |w . z| is also the magnitude, and w . z happens to be a complex scalar; this is the input to a single unit, thus both w and z are vectors and this is a dot product. This should be clearer with the new notation.

'Comment: the authors mention that the two terms in f() can be weighted, but don't include those weights in Eq. 2, as I have done above (alpha, eta).'

We simply left this out for simplicity, because we do not actually explore unbalanced weightings in this paper and didn't want to introduce unnecessary notation.

'A [conventional, real-valued] DBM was trained on small pictures with horizontal and vertical bars, and then 'converted' to a complex-valued network (and was the activation function changed to the one from Eq. 2? What does that mean in terms of inference in the DBM?)'

Yes the activation function changed; we essentially use the normal DBM training as a form of pretraining for the final, complex-valued architecture. The resulting neural network is likely not exactly to be interpreted as a probabilistic model. However, if such an interpretation is desired, our understanding is that running the network could be seen as an approximation of inference in a suitably extended DUBM (by adding an off state and a classic term; refer to Zemel et al., 1995, for comparison). For our experiments, we used two procedures (with similar outcomes) in analogy to inference in a DBM: either sampling a binary output magnitude from f(), or letting f() determine the output magnitude deterministically; the output phase was always set to the phase of the total input. The first procedure is similar to inference in such an extended DUBM, but, rather than sampling from a circular normal distribution on the unit circle when the unit is on, we simply take the mode of that distribution. The second procedure should qualitatively correspond to mean-field inference in an extended DUBM (see Eqs. 9 and 10 in the DUBM paper), using a slightly different output function.

By the way: perhaps we could have framed our work in such terms to begin with, but in a way that obscures what our original line of thinking was.

'[...] 'sampling' (is this actually sampling from a probability distribution?)'

No, not exactly. We actually only used the term 'sampling' in the standard, real-valued case, other than in the caption of Figure 3. We will fix the latter.

'Figure 3 illustrates what happens after 100 iterations of sampling, what happens after more iterations? Do the co-incidentally green bars change color independently of one another?'

Phase assignments appear to be stable (see the supplementary movies), though we did not analyze this in detail. It should also be noted that the overall network is invariant to absolute phase, so only the relative phases matter.

'I would strongly suggest that the authors upload their Pylearn2 code so that others can reproduce the effects presented in this paper [...]'

We are happy to publish the code either way, but it would unfortunately take some extra work to put it into a form that is accessible to others. We will do so if the paper gets accepted as a proper conference paper.


****** Reviewer 3 ******

Unfortunately, we certainly can't lay claim to being the first to explore this idea in a computational framework (see the references cited), though we are perhaps the first to make a connection to the types of deep networks that have recently been employed in the deep learning community (DBMs in this case; also, as stated, the framework could in principle be applied to other deep nets, such as ConvNets). Apart from that, we are of course happy to agree that this a fascinating idea and that it seems worthwhile to bring it to the attention of the ICLR community.