Neuronal Synchrony in Complex-Valued Deep Networks
Authors: David Reichert, Thomas Serre
ICLR 2014

Abstract: Deep learning has recently lead to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to real cortical circuits. The challenge is to identify which neural mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile deep representations. 
We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter aspect, we demonstrate the potential of the approach in several simple experiments. Thus, synchrony could implement a flexible mechanism that fulfills multiple functional roles in deep networks.


Peer review:
[Review]:
I find this paper deeply fascinating.  It illustrates - I believe for the first time - the utility of binding by synchrony in a deep network architecture.  Although the general idea of binding by synchrony is an old one, it is mostly a vague idea that has never, to my mind, been put into a concrete computational framework.  Here, the authors propose that the phase of oscillation in neural ensembles acts as a kind of 'label' for objects being represented in a distributed fashion.  That is, no single unit represents an object, but when an object is presented to the network it activates features at each level which synchronize via two-way communication between levels.  The result is a kind of segmentation of objects within the image.  This could be useful for example in separating objects from clutter, or possibly in resolving occlusion (although that has not been demonstrated here). 

Although mostly toy examples of patterns are used, I believe that this paper taps into a powerful idea, and that it will be of high interesting to the ICLR community, and certainly to the neuroscience community.