Neuronal Synchrony in Complex-Valued Deep Networks
Authors: David Reichert, Thomas Serre
ICLR 2014

Abstract: Deep learning has recently lead to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to real cortical circuits. The challenge is to identify which neural mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile deep representations. 
We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter aspect, we demonstrate the potential of the approach in several simple experiments. Thus, synchrony could implement a flexible mechanism that fulfills multiple functional roles in deep networks.


Peer review:
[Review]:
The paper describes a method to augment pre-trained DBMs with phase variables and shows some demonstrations of binding, segmentation, and partitioning (based on latent variables).

Pros:
The paper is very well written and introduces a number of concepts clearly.
Phase is a curious neurophysiological phenomena and is deserving of modeling that addresses the representational consequences/implications.
I could see how this ad-hoc approach could be used to understand DNNs (like extended Zeiler and Fergus' visualization work).
The paper may educate the ICLR community on the binding problem and proposals from the neuroscience community that argue for phase as a solution to this problem.

Cons:
A major issue with the described work is its similarity to the work for Rao and colleagues.  The authors provide some comments about how their work is distinguished.  However these are merely rhetorical (your approach is general and theirs is not) or actually contributions that are not made by this paper but by previous work (DBMs can be trained to learn representations of multiple-simultaneously presented object/patterns).

The two major limitations of the paper in its current form:
1. the introduction of phase is done in an ad-hoc way, without real justification from probabilistic goals.  The authors appear surprised that their hack worked at all.  It seems the more rigorous approach would be to either introduce phase as a proper latent variable and train the network to optimize the distribution to match the data distribution (the usual approach to modeling), or to explain more rigorously why this ad hoc extension does not interfere with the network (however it is not even clear from the experiments that the ad-hoc model preserves the properties of the original network).  A more rigorous mathematical approach might reveal that the basins of attraction are preserved with the introduction of phase, or that the phase variables are independent of the amplitude variables (I believe they are not).
2. The results of the experiments are mostly just pictures and lack quantitative assessment or any controls.  The resulting work provides a demonstration of the phase idea for binding/grouping/segmentation, which are not new ideas (although they are probably new ideas to the ICLR community).  Furthermore, the results in the appendix appear to indicate that the approach is not working very well in general, and the best results are the ones shown in the main text.

The procedures avoid some obvious issues:
How is the phase distribution segmented?  Phase is a continuous variable, and the segmentation/partitioning seems to be done by hand for the examples.  This needs to be addressed.
It is not clear that the segmentation is working for the bars experiment because multiple bars are colored by the same phase.  What is the goal here?  that each bar has a different, unique phase value?  Does the underlying phase distribution effectively partition the bars?  Also, what about the overlaps of the bars?  these areas seem to be mis- or ambiguously labeled.  is this a bug or a feature?

Overall, I think this is an interesting direction of research and the exposition is top notch, however the underlying work falls short of some obvious extensions and methodological rigor.  I think this would make for a nice workshop paper, so that it could receive some feedback from the community and educate the community of the phase binding idea, but it lacks some ingredients for a conference paper.

Some other relevant references you might want to include:
S. Jankowski, et al. 1996. Complex-valued multistate neural associative memory.
T. Nitta. 2009. Complex-Valued Neural Networks: Utilizing High-Dimensional Parameters.
C. Cadieu & K. Koepsell 2010. Modeling Image Structure with Factorized Phase-Coupled Boltzmann Machines.